tag:blogger.com,1999:blog-90793085064594461722024-03-19T05:47:18.578-06:00SmoogeSpaceA biographical log of the various projects and plans from the mind of Stephen Smoogen.Stephen Smoogenhttp://www.blogger.com/profile/17026786034163911165noreply@blogger.comBlogger460125tag:blogger.com,1999:blog-9079308506459446172.post-10830668320856993542023-04-21T07:23:00.004-06:002023-04-21T07:23:34.324-06:00Note To Future Self: Lenovo Laptop USB-C Mini Dock Reset<p>Hello Future Self,</p><p>Past Self here leaving you a note since I forgot to do so last time.</p><h3 style="text-align: left;">The Problem </h3><p>When running Linux on a Lenovo, there are times where a firmware update will cause problems with the USB-C Mini Dock afterwards. In the previous 2 cases, the USB-C's RTL network will no longer show up as a seen device. External monitors plugged into the dock may also not function correctly, but it only happened once so I am not sure about that.</p><p>Diagnosis of the problem is that the system will complain of no internet connection, and commands will show something like the following (output altered):</p>
<pre><code>
ssmoogen@ssmoogen-rh:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s31f6: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
link/ether ab:cd:12:34:56:78 brd ff:ff:ff:ff:ff:ff
3: wlp82s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 9a:bc:de:f1:23:45 brd ff:ff:ff:ff:ff:ff permaddr aa:aa:bb:bb:cc:cc
</code></pre>
<p>The part that has confused me at least once before is that the enp0s31f6 says NO-CARRIER which made me believe that the switch had problems. Looking at the USB-C showed that the link light was on and there was traffic. This led me to try various solutions which were wrong.</p>
<h3 style="text-align: left;">Attempted Solutions</h3><p style="text-align: left;">In trying to diagnosis this in the past, I tried backing out all the firmware updates to see if they would untrigger the bricking of the connection. The <t>fwupdtool</t> worked great to do this, and I was able to back down through 8 firmwares without a hitch. However the network still said it was offline.</p><p>Next I went through older kernels and tried booting with a USB stick. All of them continued to show the e1000e as disconnected. Finally I went through the <t>journalctl</t> command to look for previous boots and what networks were shown up.</p>
<pre><code>
ssmoogen@ssmoogen-rh:~$ journalctl | grep eth0 | tail -n 100
....
Apr 20 16:54:37 ssmoogen-rh.localdomain kernel: e1000e 0000:00:1f.6 eth0: Intel(R) PRO/1000 Network Connection
Apr 20 16:54:37 ssmoogen-rh.localdomain kernel: e1000e 0000:00:1f.6 eth0: MAC: 13, PHY: 12, PBA No: FFFFFF-0FF
Apr 20 16:54:37 ssmoogen-rh.localdomain kernel: e1000e 0000:00:1f.6 enp0s31f6: renamed from eth0
Apr 20 16:54:38 ssmoogen-rh.localdomain kernel: r8152 4-2.1.2:1.0 eth0: v1.12.13
Apr 20 16:54:38 ssmoogen-rh.localdomain kernel: r8152 4-2.1.2:1.0 enp9s0u2u1u2: renamed from eth0
Apr 20 17:09:24 ssmoogen-rh.localdomain kernel: e1000e 0000:00:1f.6 eth0: (PCI Express:2.5GT/s:Width x1) aa:aa:aa:bb:bb:bb
Apr 20 17:09:24 ssmoogen-rh.localdomain kernel: e1000e 0000:00:1f.6 eth0: Intel(R) PRO/1000 Network Connection
Apr 20 17:09:24 ssmoogen-rh.localdomain kernel: e1000e 0000:00:1f.6 eth0: MAC: 13, PHY: 12, PBA No: FFFFFF-0FF
Apr 20 17:09:24 ssmoogen-rh.localdomain kernel: e1000e 0000:00:1f.6 enp0s31f6: renamed from eth0
Apr 20 17:13:56 ssmoogen-rh.localdomain kernel: e1000e 0000:00:1f.6 eth0: (PCI Express:2.5GT/s:Width x1) aa:aa:aa:bb:bb:bb
Apr 20 17:13:56 ssmoogen-rh.localdomain kernel: e1000e 0000:00:1f.6 eth0: Intel(R) PRO/1000 Network Connection
Apr 20 17:13:56 ssmoogen-rh.localdomain kernel: e1000e 0000:00:1f.6 eth0: MAC: 13, PHY: 12, PBA No: FFFFFF-0FF
Apr 20 17:13:56 ssmoogen-rh.localdomain kernel: e1000e 0000:00:1f.6 enp0s31f6: renamed from eth0
...
</code></pre>
<h3>The Solution</h3>
<p>Matching up the timestamps I see that there was an additional device see before the updates occurred. This was using the <t>r8152</t> driver but after the firmware updates it no longer showed up. This finally triggered a memory of an email I had seen where someone else had a similar problem. Going through my email archives, I found that the solution they had found was to unplug the USB-C Dock for 1 minute and then plug everything back in. Sure enough, doing this restored the RTL driver and my network was restored. The e1000 was a red herring as it is somewhere internal to the laptop and probably available through a dongle which I forgot about as there doesn't seem to be a RJ-45 jack I could find on the exterior of the laptop.</p><p>Anyway, when this happens again, please remember this letter and save yourself 2 hours of firmware resets and kernel reboots. Sometimes completely turning off the hardware (remove all power from the Dock including the laptop) and turning it back on WILL fix the problem.<br /></p>Stephen Smoogenhttp://www.blogger.com/profile/17026786034163911165noreply@blogger.com0tag:blogger.com,1999:blog-9079308506459446172.post-2124052435763037572023-04-19T13:18:00.003-06:002023-04-19T13:18:38.839-06:00~1 year to end of EPEL-7<h1>Extra Packages for Enterprise Linux 7<br /></h1>
<p>Extra Packages for Enterprise Linux (EPEL) are packages based off of various Fedora releases but built for the various distributions based off of Red Hat Enterprise Linux. In June of 2014, Red Hat Enterprise Linux 7 (EL-7) was released and over the next several months, a focus was made to make the release of EPEL possible. Much of the work was done by Kevin Fenzi and Dennis Gilmore with some additional work by anyone else who had spare time. The initial goal was to make it that core packages needed for Fedora Infrastructure to move its core servers to EL-7 were built. That had been what had been done for the initial releases of EPEL-5 and EPEL-6, and would allow for enough base 'packages' to be built for additional packages to be added by other maintainers.</p><p>In comparison to trying to get EPEL-5 working, building for EPEL-7 was fairly easy. The initial distribution came with a large set of shipped development libraries and tooling versus getting added later. Over the years, the EL-7 distribution also gained various newer gcc toolkits via software collections which also helped EPEL maintainers to keep updating software for much of the 10 year lifecycle of EL-7. However, this maintenance has been getting harder over the last 2 years, as more and more software required either newer kernels, glibc, or other libraries that aren't available for an older operating system. [This is similar to what happened with EPEL-5 and EPEL-6 where the last year or so of the repository was more and more packages being removed due to maintenance concerns.]<br /></p><p>This ties in with the general end of support for Red Hat Enterprise Linux 7 on June 30, 2024. While final plans for how EPEL-7 will be end of lifed, this is a general outline from how EPEL-5 and EPEL-6 were similarly ended.</p><ol style="text-align: left;"><li>There will be regular reminders on mailing lists that the project will no longer be supporting EL-7 after a specified date. </li><li>On that date, the following will happen: <br /></li><ol><li>The Fedora build system will not allow any more EPEL-7 builds</li><li>A final push of all updates will happen to /pub/epel/7/ <br /></li><li>The current items in /pub/epel/7/ will be archived over to /pub/archive/epel/7.final/</li><li>Symbolic links will be made to point /pub/archive/epel/7 to the 7.final</li><li>The mirrormanager program which is what yum uses to look for updates will change where it points to to /pub/archive/epel/7/</li><li>After a week to allow mirrors to catch up, /pub/epel/7/ will be removed and a line telling people where to find the archived content.</li></ol><li>Updates to lists and such explaining what happened will occur.</li></ol><h1>Why A Year Plus Notice?</h1><p>EPEL-7 is the largest release that the Fedora project supports. There are about 400,000 Fedora systems seen by countme, and somewhere between 3.4 million and 6.7 million EPEL-7 users (depending on how looks at mirrormanager statistics). Going from the long tail turn off of EPEL-5 and EPEL-6 systems over the years, many of those EPEL-7 systems will take years to move to later releases. Going from past reports, many of the system administrators are not the original admins who set up the machine, and don't even know the OS or its auxiliary repositories like EPEL are no longer updated. Putting up blog posts like this can help:</p><ul style="text-align: left;"><li>Give admins notice and a case to their management to do updates BEFORE the end of life date.</li><li>A heads up on why scripts that mirrored content from /pub/epel/7/ will no longer work.</li><li>Time to mirror the content locally for the inevitable reinstalls because management don't think an update to a newer release is needed. </li></ul><p>Whatever the case, good luck to you fellow system administrators. <br /></p>Stephen Smoogenhttp://www.blogger.com/profile/17026786034163911165noreply@blogger.com0tag:blogger.com,1999:blog-9079308506459446172.post-23562412752497462852022-04-01T09:41:00.002-06:002022-04-01T10:23:50.700-06:00Compiling openldap for CentOS 8 Stream
<header id="title-block-header">
<h1 class="title">Compiling OpenLDAP for EL8 systems</h1>
</header>
<nav id="TOC" role="doc-toc">
<ul>
<li><a href="#steps-to-compile-openldap-server-for-centos-8-stream">Steps to compile openldap-server for CentOS 8 Stream</a>
<ul>
<li><a href="#setting-up-a-build-environment">Setting up a build environment.</a></li>
</ul></li>
<li><a href="#downloading-direct-from-centos">Downloading direct from CentOS.</a></li></ul>
</nav>
<h1 id="steps-to-compile-openldap-server-for-centos-8-stream">Steps to compile openldap-server for CentOS 8 Stream</h1>
<p>The EL8 release did not ship an openldap-server like it did in previous releases. Instead only the client tools and some libraries are included for existing applications. Instead the focus from the upstream provider has been on other LDAP solutions.</p>
<p>This leaves a problem for various sites who have their data in an OpenLDAP system and do not have the time, energy, resources for moving to something else. There are several possible solutions to this:</p>
<ol start="0" type="1">
<li>Continue to use EL5/EL6 even though it is at end of open maintenance.</li>
<li>Continue to use EL7 until it is end of open maintenance around 2024-06-30.</li>
<li>Move to a different distribution which does have working openldap</li>
<li>Compile replacement tools using the Fedora src.rpm which may be closer to the ‘upstream’.</li>
<li>Compile replacement tools using the upstream source.</li>
<li>Compile using the upstream source from https://git.centos.org</li>
<li><b>[Added after initial post]</b> You can download them from https://koji.mbox.centos.org/koji/</li>
</ol>
<p>In this tutorial we will work with number 5. At the end we will cover number 6.</p>
<h2 id="setting-up-a-build-environment">Setting up a build environment.</h2>
<p>For simplicity sake, we will assume you have a working but minimally installed Fedora 35 or EL8 system (Alma, Oracle, Rocky, etc) which you can do compiles in. If we are using an EL8 system are going to need to get <code>mock</code> and <code>git</code> installed.</p>
<pre><code>$ sudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
</code></pre>
<p>For Fedora and EL8 systems the following should work the same:</p>
<pre><code>$ sudo dnf install git mock rpm-build
$ sudo usermod -a -G mock $USERNAME
$ newgrp mock</code></pre>
<p>Answer yes to the questions about adding new keys and the packages should be installed to allow for a build to occur. We now need to set up a minimal <code>.rpmmacros</code> file for the next steps:</p>
<pre><code># uncomment if you want to build in standard homedirectory
#%_topdir %(echo $HOME)/rpmbuild
# comment if want to use standard home directory
%_topdir %{getenv:PWD}
%_sourcedir %{_topdir}/SOURCES
#%_sourcedir %{_topdir}/SOURCES/%{name}-%{version}
%_specdir %{_topdir}/SPECS
%_srcrpmdir %{_topdir}/SRPMS
%_builddir %{_topdir}/BUILD
%__arch_install_post \
[ "%{buildarch}" = "noarch" ] || QA_CHECK_RPATHS=1 ; \
case "${QA_CHECK_RPATHS:-}" in [1yY]*) /usr/lib/rpm/check-rpaths ;; esac \
/usr/lib/rpm/check-buildroot
</code></pre>
<p>Once we have that in place, the following will get an openldap build going:</p>
<pre><code>$ mkdir -vp ~/EL8-sources/ ~/output-packages/
$ cd ~/EL8-sources/
$ git clone https://git.centos.org/rpms/openldap.git
$ git clone https://git.centos.org/centos-git-common.git
$ cd openldap
$ ../centos-git-common/get_sources.sh
$ rpmbuild -bs SPECS/openldap.spec</code></pre>
<p>Now depending on the host OS you are doing this on, you should see a file like <code>SRPMS/openldap-2.4.46-18.fc35.src.rpm</code> or <code>SRPMS/openldap-2.4.46-18.el8.src.rpm</code> having been created.</p>
<pre><code>$ mock -r centos-stream+epel-next-8-x86_64 --chain --localrepo \
~/output-packages/ SRPMS/openldap-2.4.46-18.fc35.src.rpm</code></pre>
<p>should then attempt to build the packages and will end up with a fully usable repo in <code>${HOMEDIR}/output-packages/results/centos-stream+epel-next-8-x86_64</code></p>
<p>If not, then there are probably some steps or problems I missed in this howto :(. At this point you can determine what to do with installing this -server package on the server needing it.</p>
<h1 id="downloading-direct-from-centos">Downloading direct from CentOS.</h1>
<p>This is the <i>‘feed the fisherman versus teaching how to fish’</i> part of the document.</p>
<p>If you are using CentOS Stream 8, you can download the build packages from the project koji. I expect similar steps can be done for other rebuilds.</p>
<ol start="0" type="1">
<li><code>dnf list openldap</code> to get which package you are looking for.</li>
<li>Open a window to https://koji.mbox.centos.org/koji/</li>
<li>Type in openldap in the <code>Search</code> box.</li>
<li>Click on the build you would have installed. For this example, we will choose https://koji.mbox.centos.org/koji/buildinfo?buildID=18688 and then scroll down to the architecture you are using.</li>
<li>Right click on the download button for openldap-servers like:https://koji.mbox.centos.org/pkgs/packages/openldap/2.4.46/18.el8/x86_64/openldap-servers-2.4.46-18.el8.x86_64.rpm</li>
<li>Install this package in the package place you want.</li>
<li>When dnf breaks because it can’t upgrade the package due to the upstream updating, go follow step 0 again.</li>
</ol>
Stephen Smoogenhttp://www.blogger.com/profile/17026786034163911165noreply@blogger.com0tag:blogger.com,1999:blog-9079308506459446172.post-39954928506236848422022-02-28T11:50:00.002-07:002022-02-28T11:50:29.930-07:00Dealing with RAID arrays<p>Dear Future Self,</p><p> We have come to another letter where we are going to better document something PastSelf thought it knew, but clearly didn't. In this case we are going to start recovering from a RAID array after a reinstall. For reasons we won't get into, PastSelf had to reinstall the home server for the 2nd time this week. [Let us just say that PastSelf is no longer allowed to use sudo without supervision and move on.] In the reinstall, we could not get the /dev/sdb and /dev/sdc RAID array to be fully recognized and realized that we had also made the original ones too small for what we needed [which is what started the whole problem when we tried to grow a partition but forgot that the external backup always becomes /dev/sda for some reason and /dev/sdb was not the RAID drive but the / drive. Live and learn, live and learn.]</p><p>Due to some bad signatures we needed to clear the drives of their current data. This was done by booting from a USB stick (which also becomes /dev/sda in this hardware.... wtf?) and clearing each drive of its signatures. </p>
<code>
<pre># wipefs -a /dev/sdb
# wipefs -a /dev/sdc
# wipefs -a /dev/sdd
# cat /proc/mdstat
Personalities :
md127 : inactive sdc1[1](S)
1464851456 blocks super 1.2
unused devices: <none>
</none></pre>
</code>
<p>The above failed because the kernel and boot had tried to make them part of a RAID array <code>/dev/md127</code> but was not able to sync them. I was also unable to </p><pre>mdadm --stop /dev/md127</pre> for some reason. At this point, PastSelf further broke his oath of <i>primum non nocere</i> by using dd on each of the disks.
<code>
<pre># dd if=/dev/zero of=/dev/sdb bs=1024 count=1000000
# dd if=/dev/zero of=/dev/sdc bs=1024 count=1000000
# dd if=/dev/zero of=/dev/sdd bs=1024 count=1000000
</pre>
</code>
A reboot and going into rescue mode still showed that some signatures were there which I realized was due these disks being formatted with GPT and being much more capable of surviving stupidity. However <code>mdadm --stop</code> now worked so I could use <code>gdisk</code> on the drives. I then reinstalled a minimal Alma8.5 onto the box and then did a manual creation of the RAID array:
<code>
<pre># gdisk /dev/sdc
GPT fdisk (gdisk) version 1.0.3
Partition table scan:
MBR: not present
BSD: not present
APM: not present
GPT: not present
Creating new GPT entries.
Command (? for help): n
Partition number (1-128, default 1): 1
First sector (34-3907029134, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-3907029134, default = 3907029134) or {+-}size{KMGTP}:
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300): fd00
Changed type of partition to 'Linux RAID'
Command (? for help): w
Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!
Do you want to proceed? (Y/N): y
</pre>
</code>
<p>At this point we were able to get the system ready for creating the RAID partition.</p>
<code>
<pre># mdadm --create --verbose /dev/md1 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1 --force
mdadm: Note: this array has metadata at the start and
may not be suitable as a boot device. If you plan to
store '/boot' on this device please ensure that
your boot-loader understands md/v1.x metadata, or use
--metadata=0.90
mdadm: /dev/sdc1 appears to be part of a raid array:
level=raid1 devices=2 ctime=Thu Dec 30 18:54:28 2021
mdadm: size set to 1953381440K
mdadm: automatically enabling write-intent bitmap on large array
Continue creating array? y
mdadm: Fail to create md1 when using /sys/module/md_mod/parameters/new_array, fallback to creation via node
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md1 started.
[root@xenadu ~]# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sdc1[1] sdb1[0]
1953381440 blocks super 1.2 [2/2] [UU]
[>....................] resync = 0.7% (15491456/1953381440) finish=158.1min speed=204199K/sec
bitmap: 15/15 pages [60KB], 65536KB chunk
unused devices: <none>
# mdadm --detail --scan
ARRAY /dev/md1 metadata=1.2 name=xenadu.int.smoogespace.com:1 UUID=c032f979:e8e4deda:a590ca5d:820a8548
# mdadm --detail --scan > /etc/mdadm.conf
# echo '/dev/md0 /srv xfs defaults 0 0' >> /etc/fstab
</pre>
</code>
<p>Now wait for the sync to be done, and then start the restore from backups... you know the ones that Past-PastSelf made just in case of this situation. Also Future-Self, could you please write up some ansible playbooks to do this from now on? Future-FutureSelf will appreciate it.</p><p>Yours Truly, PastSelf</p>Stephen Smoogenhttp://www.blogger.com/profile/17026786034163911165noreply@blogger.com0tag:blogger.com,1999:blog-9079308506459446172.post-52376606795949306582022-02-28T05:33:00.004-07:002022-02-28T05:33:51.706-07:00Getting past EL-{8,9}'s limitations with toolbx<p>Dear Future Self,</p><p>One of the biggest issues with dealing with Enterprise Linux 8 (be it Rocky to Red Hat) is the lack of additional packages which you know are in Fedora. Trying to get them into EL-8 turns into a Sisyphean task of moving the boulder of multiple python/go/ruby/etc packages into EL8 only to find that the RPM macros and other software have changed so much in 2 to 3 Fedora releases you can't. Past self spent the weekend trying to get a simple GO package backported and found that he needed to touch at least 175 src.rpms to make this 'work'. That was just too much for trying to get something else working.<br /></p><p>Thankfully, EL8 ships with a tool which will allow to get past most of these problems if you meet the following criteria:</p><ol style="text-align: left;"><li>The package must not require any kernel feature not shipped in the EL-8 kernel.</li><li>You have lots of disk space available to basically install a second OS. <br /></li><li>You can deal with some of the limitations of containers. <br /></li></ol><p>The tool which does all this is <a href="https://containertoolbx.org/">Container Toolbx</a> which uses <a href="https://podman.io/">podman</a> to create an interactive shell using the runtime space of the OS you want.<br /></p>
<code>
<pre>
$ sudo -i dnf install toolbox
Password:
$ toolbox create --distro fedora --release f35 f35
$ cat /etc/system-release
AlmaLinux release 8.5 (Arctic Sphynx)
$ ls
Ansible-smoogespace/ HUGO/ OLD/ Packages/ RPMS/ SSH-AGENT Website-smoogespace/ go/ yadm-dotfiles/
$ toolbox enter f35
$ ls
Ansible-smoogespace/ HUGO/ OLD/ Packages/ RPMS/ SSH-AGENT Website-smoogespace/ go/ yadm-dotfiles/
$ cat /etc/system-release
Fedora release 35 (Thirty Five)
$ sudo -i dnf update
< no password asked >
$ sudo -i dnf install {package I want}
$ {package_command}
</pre>
</code>
As can be seen by the example above, <code>toolbx</code> basically puts the container in the home directory in the user but using the userspace of Fedora 35. This allowed me to have some newer commands which allowed for a compiled go package which I couldn't do in EL-8 at the moment. Since go is static, I can then use this package regularly in my EL-8 environment. [I was also able to get past some similar errors in emacs where I had used some package calls from newer emacs which compile elc which works with EL-8 emacs.]
<h2>Important!</h2>
<p>This is not a cure-all. You are basically downloading basic containers and then using overlays to do updates and other magic to make this work. While it is quite likely possible one could make various daemons (say openvpn) work this way, I also expect that the network hell that comes with containers would make it fragile. However when needing <code>fedpkg</code> or some similar command it is easier to use this than try and port all the other 'packages' that it relies on if you have only a couple of hours free.</p>
<p>Anyway, this is the 2nd time I have had to re-discover this in the last 2 years so I figured I had better write a note to future me in 6 months or a year who has to do this again.</p>
<p>Yours truly, Past Self</p>Stephen Smoogenhttp://www.blogger.com/profile/17026786034163911165noreply@blogger.com0tag:blogger.com,1999:blog-9079308506459446172.post-36442063594096864662022-02-08T11:38:00.004-07:002022-02-08T14:10:15.121-07:00How to Install CentOS Stream 9 Cloud Image<p>Dear Future Self,</p><p>You have probably started to install a <a href="http://cloud.centos.org/centos/9-stream/x86_64/images/" rel="nofollow" target="_blank">CentOS Stream 9 cloud image</a>, and completely forgot all the things you learned this time around. No worries, past-self is going to write these down for your usage. </p><p>First off, download the image you want. On the day we are writing this, the latest image is <a href="http://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-GenericCloud-9-20220207.0.x86_64.qcow2">http://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-GenericCloud-9-20220207.0.x86_64.qcow2</a> but it will most likely be something much newer. They don't put a 'latest' in the directory, so open a browser, search for qcow2, and then instead of searching through 4000 entries from 2021-08-30, press the up-arrow and jump to the last entry on the web-page. </p><p>Next, we need to use virt-install to get the image imported to where virtual manager will use it. Older versions of CentOS had a default user, but CentOS Stream 9 relies on cloud-init in order set up the root user and password. This is done via the virt-install command IF you have a virt-install after version 3, so need to look at different command for EL8 and Ubuntu 18.</p>
<code>
<pre>$ sudo virt-install --name guest-cs9 --memory 2048 \
--vcpus 2 --disk ./CentOS-Stream-GenericCloud-9-20220207.0.x86_64.qcow2 \
--import --os-type Linux --os-variant centos-stream9 \
--network default --console pty,target_type=serial --graphics vnc \
--cloud-init root-password-generate=on,disable=on,ssh-key=/home/ssmoogen/.ssh/id_ecdsa.pub
</pre>
</code>
<p>You can add more cloud init options by creating data-files for meta and user-data. Go to <a href="https://cloudinit.readthedocs.io/en/latest/topics/examples.html" rel="nofollow" target="_blank">the cloud-init site</a> for that.</p><h3 style="text-align: left;">Alternative method (ok the one most likely used).</h3><p style="text-align: left;">In the case of trying to do this on EL8 or earlier Ubuntu editions, you will need to use the <a href="https://libguestfs.org/virt-customize.1.html" rel="nofollow" target="_blank">virt-customize</a> command instead. First we have to make sure it is installed.</p>
<code>
For Ubuntu:
<pre> $ sudo apt install libguestfs-tools
</pre>
For EL based distros:
<pre> $ sudo dnf install guestfs-tools
</pre>
</code>
<p>The <tt>virt-customize</tt> command is meant to alter a non-running image. If you use it on a running one, you will probably have a very dead box afterwards. <b>YOU HAVE BEEN WARNED.</b></p>
<code>
<pre>
$ virt-customize -v --uninstall cloud-init --selinux-relabel \
-a CentOS-Stream-GenericCloud-9-20220207.0.x86_64.qcow2 \
--ssh-inject root:file:/home/ssmoogen/.ssh/id_ecdsa.pub \
--root-password random |& tee CSG.out
$ grep 'virt-customize.password' CSG.out # for the random set password.
</pre>
</code>
Depending on the OS this may or may not need to be run with sudo. Check the CSG.out file for any extra errors. We sent both standard out and standard error there to make sure it all got captured. After this is done, you can import as working image with:
<code>
<pre>$ sudo virt-install --name guest-cs9 --memory 2048 \
--vcpus 2 --disk ./CentOS-Stream-GenericCloud-9-20220207.0.x86_64.qcow2 \
--import --os-type Linux --os-variant centos-stream9 \
--network default --console pty,target_type=serial --graphics vnc
</pre>
</code>
<p>And with that, past self is done recording what is needed to be done.</p>Stephen Smoogenhttp://www.blogger.com/profile/17026786034163911165noreply@blogger.com0tag:blogger.com,1999:blog-9079308506459446172.post-47954882967604955072021-09-25T11:18:00.002-06:002021-09-25T11:18:42.037-06:00Growth of Fedora Distribution over Time<h1 id="growth-of-fedora-distribution-over-time">Growth of the Fedora Distribution over time</h1>
<p>There was a conversation in IRC (libera.chat, #fedora-admin) on the amount of disk space that Fedora is using over time. It used to grow astronomically over time, but there was an idea that it might be slowing down.. and then the realization that no one had graphed it. Taking this challenge in hand I decided to look at it. Doing a complete mirror of the data would require me to have a very long time frame and 100+ TB of disk space, but luckily for me, the Fedora mirror system does a <code>du</code> every night and outputs this data to a file, https://dl.fedoraproject.org/pub/DIRECTORY_SIZES.txt</p>
<p>The file covers all the directories that the main download servers have including the archive trees which are where old releases go to live. It also puts it in a ‘human-readable’ format like</p>
<pre><code>egrep '/rawhide$|/releases/[0-9]*$|/updates/[0-9]*$|/updates/testing/[0-9]*$' DIRECTORY_SIZES.txt | egrep -v '^8.0K|^12K|^4.0K|/pub/epel|/pub/alt' > /tmp/dirs
$ grep '/7' /tmp/dirs
71G /pub/archive/fedora/linux/releases/7
55G /pub/archive/fedora/linux/updates/7
1.5G /pub/archive/fedora/linux/updates/testing/7
</code></pre>
<p>The above takes all the directories we want to worry about and avoid <code>/pub/alt</code> which is a wild west of directories and data. I also want to avoid <code>/pub/epel</code> so I don’t get a mix between EPEL-7 and Fedora Linux 7. It also allows me to save that entire long grep into a file so I don’t repeat if for every time I do the next data manipulation which is:</p>
<pre><code># Thanks to https://gist.github.com/fsteffenhagen/e09b827430956d7f1de35140111e14c4
grep '/7' /tmp/dirs | numfmt --from=iec | awk 'BEGIN{sum=0} {sum=sum+$1} END{num=split($0,a,"/"); print sum,a[num]}' | numfmt --to=iec
128G 7</code></pre>
<p>This uses a command <code>numfmt</code> that I wish I had known years before as I have ‘replicated’ it repeatedly poorly in awk and python. The first one converts it to an integer, then feeds it to awk which adds it, and then sums all that and prints the output. The conversion is lossy but ok for a quick blog post.</p>
<pre><code>$ cat foobaz.sh
#/bin/bash
for i in $( seq 7 35 ); do
grep "/${i}$" /tmp/dirs | numfmt --from=iec | awk 'BEGIN{sum=0} {sum=sum+$1} END{num=split($0,a,"/"); print sum,a[num]}' | numfmt --to=iec | awk '{print $2","$1}'
done
$ bash foobaz.sh
7,128G
8,153G
9,286G
10,207G
11,266G
12,267G
13,202G
14,229G
15,371G
16,388G
17,594G
18,669G
19,600G
20,639G
21,730G
22,804G
23,865G
24,816G
25,821G
26,971G
27,1.1T
28,1.2T
29,1.1T
30,1.3T
31,1.1T
32,1.2T
33,1.3T
34,1.3T
35,200G
</code></pre>
<p>This first run found a problem because 35 should be greater than 200G. However only <code>/pub/fedora/linux/updates/35</code> and <code>/pub/fedora/linux/updates/testing/</code> are publically readable. Getting some data from root and we correct this to 35 having 917G. Plotting this in openoffice with some magic we get:</p>
<p>This is ok for a textual map but how about a graph picture. For this we remove the conversion to human readable data (aka M,G,T) and put the data into openoffice for some simple bar graphs. And so here is our data:</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgoMD66k5I7BoRphdST3u030auMRcAy_gV8W614GzG5MVq_Q0Klge0PcCxhaavnZGv_cGhRKZq7SFd81qWPQlHJm3eOR2cDhlUFWP1vf6k5psosiCpfwABFCtWMBCx3trUEwR6EOU4xRBU/s717/ReleaseSizeChart.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="429" data-original-width="717" height="272" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgoMD66k5I7BoRphdST3u030auMRcAy_gV8W614GzG5MVq_Q0Klge0PcCxhaavnZGv_cGhRKZq7SFd81qWPQlHJm3eOR2cDhlUFWP1vf6k5psosiCpfwABFCtWMBCx3trUEwR6EOU4xRBU/w456-h272/ReleaseSizeChart.png" width="456" /></a></div><br /> <p>After this we can also look at how someone mirroring the distributions over time need more disk space:</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0UBqzU-xqvSjiLdfDUoZlLx1Da1TVW6SR_hVWnnXyEV6u7a0_Gj9GTQFzowfJ5c23oB9s7IEwwgsQwWOI_tQZ8JjLKfvmLuIQMSEMNQlJrgKhJQd3g8rhfWFR-5nOyUOu0ANA8_shWzU/s599/GrowthOverTime.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="336" data-original-width="599" height="249" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0UBqzU-xqvSjiLdfDUoZlLx1Da1TVW6SR_hVWnnXyEV6u7a0_Gj9GTQFzowfJ5c23oB9s7IEwwgsQwWOI_tQZ8JjLKfvmLuIQMSEMNQlJrgKhJQd3g8rhfWFR-5nOyUOu0ANA8_shWzU/w446-h249/GrowthOverTime.png" width="446" /></a></div><br /><p>The total growth looks to be move from exponential to linear over time. If you wanted to break out into smaller archives, you could put release 1 to 25 on one 10 TB drive, and 26 to 32 on another 10 TB drive as the releases after 26 are usually 1.4 TB in size at the end of their release cycle. <br /> </p>
Stephen Smoogenhttp://www.blogger.com/profile/17026786034163911165noreply@blogger.com1tag:blogger.com,1999:blog-9079308506459446172.post-8967797142722880902021-09-03T13:55:00.002-06:002021-09-03T13:55:58.662-06:00How to clone (a lot of) package spec files from CentOS 8 git.centos.org<p>Recently I have had to try and work out all the dependencies on a set of packages. I am writing this as a blog, as I needed to recreate work that I had done several times in the past, but past Smoogen had not fully documented. [I went looking and found I had 3 different copies of trees of packages from Fedora and CentOS but absolutely no notes and my bash_history had cycled over the 10k lines I normally keep. Past Smoogen was a BAD, BAD sysadmin.]</p>
<p>For general requires, I would do this by using <code><tt>dnf</tt><code> or </code><tt>dnf</tt></code><br /></p>
<pre><code>
$ dnf repoquery --requires bash
Last metadata expiration check: 2:44:24 ago on Fri 03 Sep 2021 10:14:22 EDT.
filesystem >= 3
libc.so.6(GLIBC_2.15)(64bit)
libdl.so.2()(64bit)
libdl.so.2(GLIBC_2.2.5)(64bit)
libtinfo.so.6()(64bit)
rtld(GNU_HASH)
</code></pre>
<p>However in this case, I needed to also work out all the buildrequires of the packages, and then the requires and buildrequires of those tools. Basically it is sort of building a buildroot for a set of leaf packages which usually means I need to get the spec files and parse them with something like spectool. </p>
<p>If I was working with Fedora, I would take the shallow git clone of the <a href="https://src.fedoraproject.org">src.fedoraproject.org</a> website which can be found at <a href="https://src.fedoraproject.org/lookaside/">https://src.fedoraproject.org/lookaside/</a>. Then I would start going down my list of 'known' software I need to work and clone out the usual 'buildroot' a <code>fedpkg mockbuild</code> of the package would give. However I am working with CentOS Stream 8 which is a slightly different repository layout, and does not have a prebuilt shallow clone.</p>
<p>Thankfully, the CentOS repository has a very useful sub-repository called <a href="https://git.centos.org/centos-git-common.git">https://git.centos.org/centos-git-common.git</a> which contains all the tools to fetch the appropriate code and tools from the CentOS src repository. The first one I need work with is <code>centos.git.repolist.py</code> to query the pagure API and get a list of packages. I then need to clean up that list a bit because it contains some forks but the following will get me a complete list of the packages I want to parse:</p>
<code>
[centos-git-common (master)]$ ./centos.git.repolist.py | grep -v '/forks/' > repolist-2021-09-03
[centos-git-common (master)]$ ./centos.git.repolist.py --namespace modules | grep -v '/forks/' >> repolist-2021-09-03
[centos-git-common (master)]$ sort -o repolist-2021-09-03 -u repolist-2021-09-03
[centos-git-common (master)]$ wc -l repolist-2021-09-03
8769 repolist-2021-09-03
</code>
<p>That is a lot more packages than CentOS ships with as there are just over 2600 src.rpm packages in vault.centos.org for AppStream, BaseOS, and PowerTools. What is going on here?</p>
<p>It turns out there are two different events happening:</p>
<ol>
<li>Buildroot packages.</li>
<li>SIG packages</li>
</ol>
<p>Buildroot packages are the ones which are not shipped with EL8 but are needed to build EL8. [EL-8 was not meant to be build complete but just the packages that would be supportable by Red Hat.]. SIG packages are various ones which CentOS SIGs are supporting for projects like virtualization, hyperscale, and automotive. I may need to trace through all of these for my own package set, so I decided to clone them all first and then try to figure out what is needed afterwords.</p>
<pre><code>
#!/bin/bash
# A silly script to clone all the repositories from git.centos.org in
# order to work out things like buildorder and other tasks.
# Set up our local repo place. I have lots of space on /srv
CLONE_DIR=/srv/gitrepos/centos/
CLONE_LIST=repolist-2021-09-03
# loop over the namespaces we want to clone. It would have been nicer
# if there was a third namespace for sigs, but I don't think
# namespaces really happened when centos-7 was setting up git.centos.org.
for namespace in rpms modules; do
mkdir -vp ${CLONE_DIR}/${namespace}
cd ${CLONE_DIR}/${namespace}
for repo in $( grep "/${namespace}/" ${CLONE_DIR}/${CLONE_LIST} ); do
repodir=$( basename ${repo} )
git clone ${repo}
if [[ -d ${repodir} ]]; then
pushd ${repodir} &> /dev/null
X=`git branch --show-current`
if [[ ${X} =~ 'c8s' ]]; then
echo "${i} ${X}"
else
git branch -a | grep c8s &> /dev/null
if [[ $? -eq 0 ]]; then
echo "${repodir} ${X} xxx"
fi
fi
popd &> /dev/null
fi
sleep 1
done
done
</code></pre>
<p>Running this takes a couple of hours, with a lot of errors about empty directories (for git repos which seem to have been created but never 'filled') and for non-existant branches (my script just looks for a c8s but some branches were called slightly differently than that.) In either case, I end up having the git repos I was trying to remember how I got earlier.</p>Stephen Smoogenhttp://www.blogger.com/profile/17026786034163911165noreply@blogger.com0tag:blogger.com,1999:blog-9079308506459446172.post-50271723378140962172021-06-21T13:41:00.000-06:002021-06-21T13:41:01.471-06:00Hello Rocky Linux 8.4 (and belated Hello to Alma Linux too)<p>So according to <a href="https://lwn.net/Articles/860420/" rel="nofollow" target="_blank">LWN.net</a>, Rocky Linux 8.4 reached <a href="https://rockylinux.org/news/rocky-linux-8-4-ga-release/" rel="nofollow" target="_blank">General Available (GA)</a>. This is great as it means that there are two* 'community rebuild' to move CentOS 8 systems to if CentOS Stream is not a good match. </p><ul style="text-align: left;"><li><a href="https://mirrors.almalinux.org/isos.html" target="_blank">AlmaLinux</a></li><li><a href="https://rockylinux.org/download/" target="_blank">Rocky Linux</a></li></ul><p>Alma is built by the same people who have built <a href="https://www.cloudlinux.com/" target="_blank">Cloud Linux</a> for years. From their downloads they focusing on x86_64 which is the most common in the cloud. </p><p>Rocky Linux was founded by a person who worked on the pre-CentOS project of ChaOS. The <a href="https://en.wikipedia.org/wiki/Rocky_Linux">wikipedia article</a> on Rocky covers in more detail than me regurgitating it. My main reason for looking at it, is that it has a ARM port which will eventually run nicely on a raspberry pi for people interested in it. </p><p>My other reason was to remind people that if they are using CentOS Linux, that the end of life for 8 is December 31, 2021. Before that time it would be good to look at switching to Alma, CentOS Stream 8, Oracle Linux, Red Hat Enterprise Linux, or Rocky. These transitions take time, and there are only ~190 days before it needs to be done. <br /></p><h4 style="text-align: left;">Caveats<br /></h4><p>[*] There is a<a href="https://springdale.math.ias.edu/" target="_blank"> third rebuild</a> which has been around for years called Springfield Linux made at Princeton University. However either due to the pandemic or other items, it does not seem to have had regular updates. <br /></p>Stephen Smoogenhttp://www.blogger.com/profile/17026786034163911165noreply@blogger.com0tag:blogger.com,1999:blog-9079308506459446172.post-5248995242882630122021-06-16T16:15:00.007-06:002021-09-07T08:43:25.668-06:00Working with Raspberry PI4 systems<p><span>While my current work is aimed at ARM-64 hardware, many of the boards are not </span><a href="https://en.wikipedia.org/wiki/Server_Base_System_Architecture" rel="noopener" target="_blank"><span>Server Ready Hardware</span></a><span>
and thus do not have things like UEFI to boot, ACPI to query the
hardware stack, or various other things which are added later as
firmware updates. They also end up having ‘developer kit boards’ of
US$6000.00+ systems which having one at home is hard to justify. {</span><i><span>Sorry kid, no college this semester… Dad bought himself a board that the customer may dump next week.</span></i><span>}</span></p><div class="markdown-body container-fluid comment-enabled" data-hard-breaks="true" id="doc"><p><span>In looking for proxy systems, my team has been focusing first on the classic ARM for small projects: </span><a href="https://www.raspberrypi.org/" rel="noopener" target="_blank"><span>The Raspberry Pi</span></a><span>.
The raspberry pi4 with 4 GB of ram works out as a proxy for all kinds
of ‘low-end’ systems where you may need to play with a small GPU and try
to make it work with a Server Ready operating system like CentOS
Stream.</span></p><h2 data-id="Getting-the-hardware" id="Getting-the-hardware"><span>Getting the hardware <br /></span></h2><p><span> </span></p><p><span>There are several places to get the hardware, and while I used to get things from </span><a href="https://www.adafruit.com/category/1000" rel="noopener" target="_blank"><span>Adafruit</span></a><span>, but they did not have an IOT kit set up for the 4 series. I ended up going with </span><a href="https://www.amazon.com/gp/product/B07V5JTMV9" rel="noopener" target="_blank"><span>CanaKit from Amazon</span></a><span>
just so I could get a bundle of parts together. Going from past
experience with MMC cards burning out after a couple of days, I bought 3
32 gig cards to put the OS on. So far the cards have lasted longer than
I expected but that just means they will burn out any day now.</span></p><p><span>When
getting a raspberry pi, I highly recommend making sure you get the
correct power supply, a USB 2 serial connector for the GPIO, and if you
are using an external drive, a seperately powered disk drive. I have
found that while the Raspberry Pi4 uses a larger power supply than the
1,2, or 3 series… attaching a non-powered USB drive can be problematic
on boot (the ssd one I had would cause me to just have the rainbow
picture of doom).</span></p><h2 data-id="Setting-up-the-hardware" id="Setting-up-the-hardware"><span>Setting up the hardware</span></h2><p><span> </span></p><p><span>For
the raspberry pi4 if you are using it to compile or build things, make
sure you have correctly sized heat dispensors for the CPU and possibly a
fan (and maybe a replacement fan for when the first one dies). Then
attach a serial cable to pins 6 (ground),8 (txd),10 (txd). Make sure you
do not attach anything to 1,2,3 as you could be looking for a new pi or
computer. The serial is useful for when you attempt to boot a new
kernel config and the HDMI connector was not functional afterwords.</span></p><p><span>On another computer attach the USB connector and you can use the </span><code>screen</code><span> or </span><code>minicom</code><span> commands to see output from the system on boot. On my test system, I was able to capture the following:</span></p><pre><code> </code></pre><pre><code>$ screen -c /dev/null -L /dev/ttyUSB0 115200
recover4.elf not found (6)
recovery.elf not found (6)
Read start4x.elf bytes 2983784 hnd 0x000013b1 hash '3f7b34a64191a848'
Read fixup4x.dat bytes 8453 hnd 0x00000d3b hash '59e66162bed1b815'
0x00c03112 0x00000000 0x000003ff
MEM GPU: 32 ARM: 992 TOTAL: 1024
Starting start4x.elf @ 0xfec00200 partition 0
MESS:00:00:05.434998:0: arasan: arasan_emmc_open
</code></pre><h2 data-id="Initial-setup" id="Initial-setup"><span>Initial setup</span></h2><p><span> </span></p><p><span>Like
any hardware setup, it is important to make sure the shipped hardware
has up to date firmware and configs. For this I took one of the MMC
cards, and burned the </span><a href="https://www.raspberrypi.org/software/operating-systems/" rel="noopener" target="_blank"><span>Raspian OS with recommended software</span></a><span>
Once this booted up, the tools did a firmware upgrade on the system
from whatever had been on the box when it was stored in a depot. This OS
is a 32 bit operating system but is maintained by the ‘manufacturer’ so
is a good test for ‘did it break in shipment’</span></p><p><span>{</span><i><span>Pro-tip:
Once you have finished updating, shutdown the system, take this card
out, and put it in a box/bag for later use. At some point things are
going to go badly in an install and you won’t know if its you, your
hardware, or something else. Having a known bootable backup that is
supposed to work is good.</span></i><span>}</span></p><p><span>After this it is time to focus on getting ‘bootable’ systems using the base OS’s we want to target:</span></p><ol><li><span>Fedora Linux</span></li><li><span>CentOS Stream</span></li></ol><h2 data-id="Fedora-34-Initial-Install" id="Fedora-34-Initial-Install"><span>Fedora 34 Initial Install</span></h2><p><span> </span></p><p><span>Fedora
34 came out as I started working on the project, so I decided to aim
for that as an initial OS. The ARM work that the Fedora team is doing is
aimed primarily at 64-bit and with Server Ready hardware. As such, they
do try to make a raspberry pi4 work, but it is listed as possibly
problematic. That said, the following worked mostly fine:</span></p><ol><li><span>Download the </span><a href="https://download.fedoraproject.org/pub/fedora/linux/releases/34/Workstation/aarch64/images/Fedora-Workstation-34-1.2.aarch64.raw.xz" rel="noopener" target="_blank"><span>raw workstation image</span></a></li><li><span>On an existing Fedora 33 system, install arm-image-installer via </span><code>sudo dnf install arm-image-installer</code></li><li><span>Insert an mmc into the computer using the appropriate adaptor, and find the disk name.</span></li><li><span>GNOME will probably be overly helpful and have mounted partitions on the card for you. You will need to unmount them. </span><code>df -a | grep mmc ; sudo umount /dev/mmcblk0p1; sudo umount /dev/...</code></li><li><span>write the image to the mmc disk with image-installer:</span></li></ol><pre><code>fedora-arm-image-installer --image=./Fedora-Server-34-1.2.aarch64.raw.xz --media=/dev/mmcblk0 --addkey=a_key.pub --resizefs --showboot --target=rpi4 --addconsole=ttyAMA0,115200
</code></pre><ol start="6"><li><span>Move the mmc card over to the powered off raspberry pi4, and prepare to boot up the hardware.</span></li><li><span>On my Fedora system I started a screen to watch the fireworks: </span><code>screen /dev/ttyUSB0 115200</code></li><li><span>Power on the raspberry pi and watch what happens. If you are
lucky then you will get the system eventually booting into Fedora 34
graphical mode. If you aren’t, then it may stay in rainbow mode (like
when I found that my SSD drive pulled too much power on boot.)</span></li><li><span>Log into the system and play around a bit. This is a good time
to do any updates and such. Getting an idea of how ‘fast’/‘slow’ the
system with defaults is good here too.</span></li></ol><h3 data-id="Get-serial-working" id="Get-serial-working"><span>Get serial working</span></h3><p><span> </span></p><p><span>At
this point I wanted to make sure I could log into Fedora from the
serial port. In order to do this, you need to edit the grub configs
which is done in </span><code>/etc/default/grub</code><span> and then
rebuilding the config. I moved the grub timeout to 10 seconds to give me
a chance to choose different options, removed the rhgb and quiet, and
added a console line.</span></p><pre><code> </code></pre><pre><code>$ sudo -i
# vi /etc/default/grub
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyAMA0,115200 "
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true
# grub2-mkconfig -o /etc/grub2-efi.cfg
</code></pre><p><span>A test reboot is good here to make sure that when you boot you get past the</span></p><pre><code>EFI stub: Booting Linux Kernel...
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
</code></pre><h2 data-id="Fedora-34-with-TianoCore" id="Fedora-34-with-TianoCore"><span>Fedora 34 with TianoCore</span></h2><p><span> </span></p><p><span>As
stated before the Raspberry Pi systems are not System Ready and do not
have a built in EFI or ACPI system for a kernel to boot from. Instead
the default kernel boots usually with </span><a href="https://source.denx.de/u-boot/u-boot" rel="noopener" target="_blank"><span>uboot</span></a><span> mapping devices via </span><a href="https://www.kernel.org/doc/html/latest/devicetree/usage-model.html" rel="noopener" target="_blank"><span>device tree</span></a><span>
in order for hardware to be discovered. I am going to say that is about
all I really know about the subject. There have been long threads in
the Fedora ARM lists over the years on going over this versus ACPI, and I
think it is better for an expert like Jon Masters or </span><a href="https://nullr0ute.com/" rel="noopener" target="_blank"><span>Peter Robinson</span></a><span> to explain why APCI is preferred in Fedora Linux and Red Hat Enterprise Linux (RHEL) versus a novice like myself.</span></p><p><span>For the raspberry pi4, the current method to implement a UEFI interface is a port of the </span><a href="https://www.tianocore.org/" rel="noopener" target="_blank"><span>Tianocore</span></a><span> by the </span><a href="https://github.com/pftf/RPi4" rel="noopener" target="_blank"><span>Pi Firmware Task Force</span></a><span>. TianoCore is an opensource implementation and extension of the Intel </span><a href="https://en.wikipedia.org/wiki/Unified_Extensible_Firmware_Interface" rel="noopener" target="_blank"><span>Extendable Firmware Interface</span></a><span> which was written to replace the 16-bit BIOS used in personal computers since the 1980’s. A further extension was with the </span><a href="http://www.linux-kvm.org/downloads/lersek/ovmf-whitepaper-c770f8c.txt" rel="noopener" target="_blank"><span>Open Virtual Machine Firmware</span></a><span> which I believe was then used by the Pi Firmware Task Force for their version.</span></p><h3 data-id="Easier-than-I-thought" id="Easier-than-I-thought"><span>Easier than I thought</span></h3><p><span> </span></p><p><span>In
reading various blogs and documentation on the status of the EFI
support, I was prepared for a system that would only work via serial
console or may not have networking or other utilities. Instead, I found
the process to be fairly ‘painless’ and I only ended up with a
non-booting system twice. The general steps were the following:</span></p><pre><code>ssmoogen@fedora ~]$ mkdir RPi4
[ssmoogen@fedora ~]$ cd RPi4/
[ssmoogen@fedora RPi4]$ wget https://github.com/pftf/RPi4/releases/download/v1.27/RPi4_UEFI_Firmware_v1.27.zip
--2021-06-16 17:36:31-- https://github.com/pftf/RPi4/releases/download/v1.27/RPi4_UEFI_Firmware_v1.27.zip
Resolving github.com (github.com)... 140.82.114.3
Connecting to github.com (github.com)|140.82.114.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
...
RPi4_UEFI_Firmware_v1.27.zip 100%[=====================================================================================================>] 2.92M 14.4MB/s in 0.2s
2021-06-16 17:36:32 (14.4 MB/s) - ‘RPi4_UEFI_Firmware_v1.27.zip’ saved [3064085/3064085]
$ wget https://github.com/pftf/RPi4/releases/download/v1.27/RPi4_UEFI_Firmware_v1.27.zip
$ unzip RPi4_UEFI_Firmware_v1.27.zip
Archive: RPi4_UEFI_Firmware_v1.27.zip
inflating: RPI_EFI.fd
inflating: bcm2711-rpi-4-b.dtb
inflating: bcm2711-rpi-400.dtb
inflating: bcm2711-rpi-cm4.dtb
inflating: config.txt
inflating: fixup4.dat
inflating: start4.elf
creating: overlays/
inflating: overlays/miniuart-bt.dtbo
inflating: Readme.md
creating: firmware/
inflating: firmware/Readme.txt
creating: firmware/brcm/
inflating: firmware/brcm/brcmfmac43455-sdio.clm_blob
inflating: firmware/brcm/brcmfmac43455-sdio.bin
inflating: firmware/brcm/brcmfmac43455-sdio.Raspberry
inflating: firmware/brcm/brcmfmac43455-sdio.txt
inflating: firmware/LICENCE.txt
</code></pre><p><span>At this point, if you haven’t already, read the
documentation. Our main tasks will be to setup the raspberry pi EFI boot
partition to have the needed data in it. I had seen that this came with
dtb files which I figured needed to be replaced. However as seen below,
all of these are in a Fedora 34 and are of a similar timeframe. The
only file we really need to work with is the </span><code>RPI_EFI.fd</code><span> and </span><code>config.txt</code><span> files.</span></p><pre><code> </code></pre><pre><code>[ssmoogen@fedora RPi4]$ sudo -i
[sudo] password for ssmoogen:
[root@fedora ~]# cd /boot/efi/
[root@fedora efi]# cp config.txt config-orig.txt # always make a backup!
[root@fedora efi]# rpm -qf /boot/efi/bcm2711-rpi-4-b.dtb
bcm2711-firmware-20210430-1.1a46874.fc34.aarch64
[root@fedora efi]# rpm -qf /boot/efi/overlays/miniuart-bt.dtbo
bcm283x-overlays-20210430-1.1a46874.fc34.aarch64
[root@fedora efi]# cp ~ssmoogen/RPi4/RPI_EFI.fd /boot/efi/
</code></pre><p><span>At this point, it is time to replace the
config.txt that came with the system with one which can be used for
booting the UEFI program. This is where I caused my system to go into
rainbow mode a couple of times. In the end, I put in the following file:</span></p><pre><code> </code></pre><pre><code>
#boot in 64-bit mode
arm_64bit=1
# boot into the RPI_EFI firmware
armstub=RPI_EFI.fd
# turn on serial and keep it on in 2ndstage
enable_uart=1
uart_2ndstage=1
bootcode_delay=1
# using this from upstream UEFI config.txt
device_tree_address=0x1f0000
device_tree_end=0x200000
# set up the miniuart and the vc4
dtoverlay=miniuart-bt,vc4-fkms-v3d
disable_commandline_tags=1
disable_overscan=1
enable_gic=1
# set up the GPU ram and HDMI
gpu_mem=128
hdmi_ignore_cec_init=1
max_framebuffers=2
start_x=1
</code></pre><p><span>A couple of the commands in it are from a side
trip on getting accelerated graphics working in CentOS Stream on Pi. I
decided to document it for later as this is getting long. At this point I
was able to get a booting system:</span></p><pre><code> </code></pre><pre><code>recover4.elf not found (6)
recovery.elf not found (6)
Read start4x.elf bytes 2983816 hnd 0x000032fe hash '210478ae179a91d0'
Read fixup4x.dat bytes 8451 hnd 0x00002c88 hash '7716af32619f0771'
0x00c03112 0x00000000 0x000003ff
MEM GPU: 256 ARM: 768 TOTAL: 1024
Starting start4x.elf @ 0xfec00200 partition 0
MESS:00:00:05.550101:0: arasan: arasan_emmc_open
MESS:00:00:05.698627:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:05.701548:0: brfs: File read: 342 bytes
MESS:00:00:05.821820:0: HDMI1:EDID error reading EDID block 0 attempt 0
MESS:00:00:05.831336:0: HDMI1:EDID error reading EDID block 0 attempt 1
MESS:00:00:05.840843:0: HDMI1:EDID error reading EDID block 0 attempt 2
MESS:00:00:05.850357:0: HDMI1:EDID error reading EDID block 0 attempt 3
MESS:00:00:05.859867:0: HDMI1:EDID error reading EDID block 0 attempt 4
MESS:00:00:05.869382:0: HDMI1:EDID error reading EDID block 0 attempt 5
MESS:00:00:05.878890:0: HDMI1:EDID error reading EDID block 0 attempt 6
MESS:00:00:05.888404:0: HDMI1:EDID error reading EDID block 0 attempt 7
MESS:00:00:05.897914:0: HDMI1:EDID error reading EDID block 0 attempt 8
MESS:00:00:05.907428:0: HDMI1:EDID error reading EDID block 0 attempt 9
MESS:00:00:05.911926:0: HDMI1:EDID giving up on reading EDID block 0
MESS:00:00:05.918043:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:06.995664:0: gpioman: gpioman_get_pin_num: pin DISPLAY_DSI_PORT not defined
MESS:00:00:07.002961:0: *** Restart logging
MESS:00:00:07.004382:0: brfs: File read: 342 bytes
MESS:00:00:07.072608:0: hdmi: HDMI1:EDID error reading EDID block 0 attempt 0
...
MESS:00:00:07.226856:0: dtb_file 'bcm2711-rpi-4-b.dtb'
MESS:00:00:07.235551:0: dtb_file 'bcm2711-rpi-4-b.dtb'
MESS:00:00:07.248011:0: brfs: File read: /mfs/sd/bcm2711-rpi-4-b.dtb
MESS:00:00:07.251301:0: Loading 'bcm2711-rpi-4-b.dtb' to 0x1f0000 size 0xc042
MESS:00:00:07.289344:0: brfs: File read: 49218 bytes
MESS:00:00:07.465024:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:07.467674:0: brfs: File read: 342 bytes
MESS:00:00:07.488759:0: brfs: File read: /mfs/sd/overlays/vc4-fkms-v3d.dtbo
MESS:00:00:07.535268:0: Loaded overlay 'vc4-fkms-v3d'
MESS:00:00:07.537248:0: dtparam: cma-256=true
MESS:00:00:07.541611:0: dtparam: miniuart-bt=true
MESS:00:00:07.552794:0: Unknown dtparam 'miniuart-bt' - ignored
MESS:00:00:07.654662:0: brfs: File read: 1446 bytes
MESS:00:00:07.658863:0: Failed to open command line file 'cmdline.txt'
MESS:00:00:08.953730:0: brfs: File read: /mfs/sd/RPI_EFI.fd
MESS:00:00:08.956193:0: Loading 'RPI_EFI.fd' to 0x0 size 0x1f0000
MESS:00:00:08.962019:0: No compatible kernel found
MESS:00:00:08.966520:0: Device tree loaded to 0x1f0000 (size 0xc622)
MESS:00:00:08.974158:0: uart: Set PL011 baud rate to 103448.300000 Hz
MESS:00:00:08.981676:0: uart: Baud rate change done...
MESS:00:00:08.983696:0:NOTICE: BL31: v2.3():v2.3
NOTICE: BL31: Built : 10:40:51, Apr 21 2020
UEFI firmware (version UEFI Firmware v1.27 built at 11:17:17 on May 25 2021)
3hESC (setup), F1 (shell), ENTER (boot)
</code></pre><p><span>On the HDMI monitor, you get a nice raspberry pi
and a timeout to choose if you want to go into setup, shell or boot. If
you hit ESC, you will get what looks like a fairly standard BIOS/EFI
screen asking if you want to change various settings. At this point it
is a good idea to make a change to allow for full memory usage (or you
will be limited to 3GB and a slower system). Following the upstream
README, </span><code>Device Manager</code><span> → </span><code>Raspberry Pi Configuration</code><span> → </span><code>Advanced Configuration</code><span>.
At this point you select ‘Limit RAM to 3GB’ and disable it. Save the
settings and escape up to the top menu. Choose the boot manager and you
should be given the choices of the following operating systems:</span></p><pre><code> </code></pre><pre><code>Fedora
UEFI Shell
SD/MMC on Arasan SDHCI
UEFI PXEv4 (MAC:??)
UEFI PXEv6 (MAC:??)
UEFI HTTPv4 (MAC:??)
UEFI HTTPv6 (MAC:??)
</code></pre><p><span> </span></p><p><span>Choose Fedora and after a short pause, you will
get the grub config file. You shouldn’t need to change any defaults, but
it is good in case you did. Once the kernel has been selected, the
system will begin booting and the scary black screen occurs. This is a
point where for some seconds nothing seems to be happening and a couple
of times I was ready to power off and go back to other configs. Then you
will see something similar to start scrolling:</span></p><pre><code class="[ hljs"> </code></pre><pre><code class="[ hljs">[<span class="hljs-meta"> 0.000000</span>] Linux version <span class="hljs-number">5.12</span><span class="hljs-number">.10</span><span class="hljs-number">-300.f</span>c34.aarch64 (mockbuild@buildvm-a64<span class="hljs-number">-03.</span>iad2.fedoraproject.org) (gcc (GCC) <span class="hljs-number">11.1</span><span class="hljs-number">.1</span> <span class="hljs-number">20210531</span> (Red Hat <span class="hljs-number">11.1</span><span class="hljs-number">.1</span><span class="hljs-number">-3</span>), GNU ld version <span class="hljs-number">2.35</span><span class="hljs-number">.1</span><span class="hljs-number">-41.f</span>c34) <span class="hljs-meta">#1 SMP T</span>
hu Jun <span class="hljs-number">10</span> <span class="hljs-number">13</span>:<span class="hljs-number">49</span>:<span class="hljs-number">00</span> UTC <span class="hljs-number">2021</span>
[<span class="hljs-meta"> 0.000000</span>] efi: EFI v2<span class="hljs-number">.70</span> <span class="hljs-keyword">by</span> https:<span class="hljs-comment">//github.com/pftf/RPi4</span>
[<span class="hljs-meta"> 0.000000</span>] efi: ACPI <span class="hljs-number">2.0</span>=<span class="hljs-number">0x30720018</span> SMBIOS=<span class="hljs-number">0x33e00000</span> SMBIOS <span class="hljs-number">3.0</span>=<span class="hljs-number">0x33de0000</span> MEMATTR=<span class="hljs-number">0x321c2418</span> RNG=<span class="hljs-number">0x33fdb798</span> MEMRESERVE=<span class="hljs-number">0x30375118</span>
[<span class="hljs-meta"> 0.000000</span>] efi: seeding entropy pool
[<span class="hljs-meta"> 0.000000</span>] ACPI: Early table checksum verification disabled
[<span class="hljs-meta"> 0.000000</span>] ACPI: RSDP <span class="hljs-number">0x0000000030720018</span> <span class="hljs-number">000024</span> (v02 RPIFDN)
...
</code></pre><p><span> </span></p><p><span>My belief is that the pause is the kernel and
initial ramdisk are getting gunzipped in memory for usage. Reading from
the MMC is slow, and uncompressing the files is slow. A future project
may be to see if there is a sizable speedup of just doing this on the
filesystem beforehand. In any case, the system will boot and be usable
as a workstation.</span></p><h2 data-id="End-of-File" id="End-of-File"><span>End of File</span></h2><p><span>This
post has gotten on in size, and there were several other side tasks I
worked on while doing it. Those will need to be seperate posts in the
near future.</span></p></div>Stephen Smoogenhttp://www.blogger.com/profile/17026786034163911165noreply@blogger.com4tag:blogger.com,1999:blog-9079308506459446172.post-78711611257862844972021-04-16T14:51:00.002-06:002021-06-07T08:16:49.048-06:00Leaving Fedora Infrastructure<p> In June 2009, I was given the opportunity to work in Fedora Infrastructure as Mike McGrath's assistant so that he could take some vacation. At the time I was living in New Mexico and had worked at the University of New Mexico for several years. I started working remote for the first time in my life, and had to learn all the nuances of IRC meetings and typing clearly and quickly. With the assistance of Seth Vidal, Luke Macken, Ricky Zhou, and many others I got quickly into 'the swing of things' with only 2 or 3 times taking all of Fedora offline because of a missed ; in a dns config file. </p><p><br /></p><p>For the last 4300+ days, I have worked with multiple great and wonderful system administrators and programmers to keep the Fedora Infrastructure running and growing so that the number of systems using 'deliverables' has grow into the millions of systems. I am highly indebted to everyone from volunteers to paid Red Hatters who has helped me grow. I want to especially thank Kevin Fenzi, Rick Elrod, and Pierre Yves-Chibon for the insights I have needed. </p><p><br /></p><p>Over the years, we have maintained a constantly spinning set of plates which allow for packagers to commit changes, build software, produce deliverables, and start all over again. We have moved our build systems physically at least 3 times, once across the North American continent. We have dealt with security breaches, mass password changes, and the undead project of replacing the 'Fedora Account System' which had been going on since before I started. [To the team which finished that monumental task in the last 3 months, we are all highly indebted. There may be pain-points but they did a herculean task.]</p><p><br /></p><p>All in all, it has been a very good decade of working on a project that many have said would be 'gone' by next release. However, it is time for me to move onto other projects, and find new challenges that excite me. Starting next week, I will be moving to a group working with a strong focus on embedded hardware. I have been interested in embedded in some form or another since the 1970's. My first computer memories were of systems my dad showed me which would have been in an A-6 plane. From there I remember my dad taking me to see a friend who repaired PDP systems for textile mills and let me work on my first Unix running on a Dec Rainbow. Whenever I came home from those visits, I would have a smile and hum of excitement which would not leave me for days. I remember having that humm when in 1992, a student teacher showed me MCC Linux running on an i386 which we had been repairing from spare parts. I could do everything and anything on that box for a fraction of the price of the big Unix boxes I had to pay account time for. And recently I went to a set of talks on embedded projects and found myself with the same hum. It was a surprise for me but I found myself more and more interested in it as the weeks have gone by.</p><p><br /></p><p>I was offered a chance to move over, and I decided to take it. I will still be in the Fedora community but will not be able to work much on Infrastructure issues. If I have tasks that you are waiting for, please let me know, and I will finish them either by myself or by doing a full handoff to someone else in Infrastructure. Thank you all for your help and patience over these last 11+ years. </p>Stephen Smoogenhttp://www.blogger.com/profile/17026786034163911165noreply@blogger.com2tag:blogger.com,1999:blog-9079308506459446172.post-68411540144957185942020-12-18T18:24:00.001-07:002020-12-18T18:24:19.819-07:00Reading: Peopleware: Productive Projects and Team by Tom DeMarco and Timothy ListerSo last month I finished a re-read of <a href="https://en.wikipedia.org/wiki/The_Mythical_Man-Month" rel="nofollow" target="_blank">"The Mythical Man-Month" by Fred Brooks</a>.. and saw in the back of the book a reference to<a href="https://en.wikipedia.org/wiki/Peopleware:_Productive_Projects_and_Teams" rel="nofollow" target="_blank"> "Peopleware:..." by Tom DeMarco and Timothy Lister.</a> I had heard of the book in the late 1990's (right before I joined Red Hat for the first time) from a fellow coder who was moving into management and had wanted to do it right. Marshall had read the book, tried to put the ideas into place and was promptly let go by the company. [Within about 6 months, we were all let go also... for a reason given in the book..] In the years since I have seen the book brought up several times, but I had never actually read it.<div><br /></div><div>First off, the book is very ranty. Very very ranty in parts about open floor plans (and the Furniture police), open organizations, and a couple of other areas of middle management. It is not that I found their comments off the mark.. but you may have to soldier onto the next chapter through a couple of them. That said there are a lot of gems in the book which 'spoke to me'.</div><h3 style="text-align: left;">Parkinson's Law Doesn't Exist</h3><div><i>Work expands to fill the time allocated to it. </i>- <a href="https://en.wikipedia.org/wiki/Parkinson%27s_law" rel="nofollow" target="_blank">C. NorthCote Parkinson</a></div><div><br /></div><div>I have had several managers quote this to teams I have been on in the last 30 years. People would bring up Parkinson's law about why we shouldn't add more time to a schedule but should cut it back. I, myself took it for certain that this was some sociological experiment which proved things about scheduling. I never realized until reading this book that a Mr. Parkinson was not a scientist, but a satirist and the things people have 'quoted' as being proven by it originally were all made up for a humorous article and books he wrote. </div><div><br /></div><div>In the extent that there may be places where people come up with busy work to meet a schedule... it usually is where people are not finding satisfaction in their current job. [chapter 5]</div><h3 style="text-align: left;">Open Floor Plans Don't Help Programmers Flow</h3><div>This book was originally written in the 1980's when open floor plans were having 6 foot high cubicles versus offices with doors. Back then there was a lot of recorded data about how various 'knowledge' workers can't get into a flow state of thought because there are too many distractions. Today's open floor plans of no cubicle walls and brutalist concrete structures in order to get some sort of energy efficiency put those to shame. [This is where the rants on furniture police start and go on for quite some time.. ] The useful information I took from this is that one should measure their uninterrupted time and not their hours. Doing this can show where productivity is actually going and work out better planning on how efficient a project can meet its goals.</div><h3 style="text-align: left;">Hiring The Right People to Teamicide</h3><div>This section of the book has a lot of useful parts, but one that stood out to me was the chapter "Happy to Be Here" with an excerpt from <a href="https://en.wikipedia.org/wiki/Robert_Townsend_(author)" rel="nofollow" target="_blank">"Up the Organization" by Robert Townsend</a>. It basically explains a method for dealing with an underperforming organization by moving it to a new location. The purported guaranteed results:</div><div><ol style="text-align: left;"><li>the best people will move to the new location.</li><li>the rest don't have to face the fact they have been fired. Instead they can say "The company moved town."</li><li>your competition will move in and hire these less desirables thinking they are getting a bargain.</li><li>the new people at your new site are infused with enthusiasm because all they know are your 'best' people. </li></ol><div>This was basically the strategy used by that company I worked for years before Red Hat. We were all told the company was moving town and we could reapply for our jobs at a different location. The problem they ran into is the one mentioned in the book.. most of the best people didn't go (they ended up being hired as contractors and paid 2.5x what they had been paid before so that deadlines could be met), and a lot of the people who did weren't the ones they wanted. Those of the best people who did go left shortly after their moves. The psychological loyalty they had with the company had been broken so any problem at the new place was enough to send them packing. </div></div><div><br /></div><div>The next sections cover more about how to make a team of people "jell" which ends up being harder to describe than the ways to make a team break down. </div><div><ol style="text-align: left;"><li> Defensive Management</li><li>Bureaucracy</li><li>Physical Seperation</li><li>Fragmentation of people's time</li><li>Quality of reduction of the product</li><li>Phony deadlines</li><li>Clique Control</li><li>"Motivational" posters and awards</li><li>Overtime causing undertime</li><li>Internal team competition</li><li>Annual salary/merit reviews</li><li>Management by Objectives</li></ol></div><div>I can see where some of the above are causes of team breakdowns but others seem to be a lot smaller concerns comparatively.</div><h3 style="text-align: left;">Making Change Possible</h3><p style="text-align: left;">The chapter opens with a quote by <a href="https://systemsguild.eu/" rel="nofollow" target="_blank">Steve McMenamin, of the Atlantic Systems Guild</a> (the organization the authors are members of).</p><p style="text-align: left;"></p><blockquote style="border: none; margin: 0 0 0 40px; padding: 0px;"><blockquote><p style="text-align: left;"><i>People hate change... and that's because people hate change... I want you to be sure that you get my point. People really hate change. They really, really do.</i> </p></blockquote></blockquote><p style="text-align: left;">People don't react to change logically and very little 'logical argument' will help them. People react to change emotionally and the most likely result is going to be failure and ridicule from ones peers and sometimes by the people who instituted the change. The book also brings up the idea of Jerry Johnson, Menninger Foundation, that there is a continuum of ways people react to the change:</p><p style="text-align: left;"></p><ol style="text-align: left;"><li>Blindingly Loyal (Ask no questions.)</li><li>Questioners</li><ol><li>skeptics (show me)</li><li>passive (what's in it for me?)</li><li>opposed due to fear of change</li><li>opposed due to fear of loss of knowledge/power</li></ol><li>Militantly Opposed (Will Undermine and Destroy).</li></ol><div>While some people think that the second 2 sets are 'non-supporters', the Blindingly loyal are also the ones to most likely jump ship at the next newest thing. The only people who will stick it through are the various questioners if they are helped through the emotional journey through the initial Chaos and failures before the change can be truly evaluated. [The rest of the chapter goes into parts of that while referencing books like <a href="https://en.wikipedia.org/wiki/The_Prince" rel="nofollow" target="_blank">"The Prince" by Machiavelli</a> and <a href="https://en.wikipedia.org/wiki/Virginia_Satir" rel="nofollow" target="_blank">Virginia Satir's model of therapy</a> change. I actually wish this chapter had been a bit more in depth but I think I will reread <a href="https://en.wikipedia.org/wiki/William_Bridges_(author)" rel="nofollow" target="_blank">Managing Transitions</a> (which was referenced in different parts of the book) instead.</div><p></p><p style="text-align: left;"> </p><p></p><div><br /></div>Stephen Smoogenhttp://www.blogger.com/profile/17026786034163911165noreply@blogger.com0tag:blogger.com,1999:blog-9079308506459446172.post-88924799127909831402020-12-14T08:41:00.003-07:002020-12-14T08:41:36.444-07:00CentOS-8 End of Life 2021-12-31<h2 style="text-align: left;">What is Happening <br /></h2><p><a href="https://blog.centos.org/2020/12/future-is-centos-stream/" rel="nofollow" target="_blank">Red Hat announced changes to CentOS Linux</a> release structure last week, and I like everyone else around is working through what that means. I have worked with and on CentOS since 2005, and saw a lot of my work in EPEL as helping people in that community able to get things done. This has been a real kick to the guts for a lot of people, and I do not have the words to express myself on this. That said, as a System Administrator, you have to take the Tango Charlie Foxtrot's as they are handed to you, plan what to do next, and execute as efficiently as possible.<br /></p><h2 style="text-align: left;">Current End of Life Dates for CentOS Releases<br /></h2><p><a href="https://en.wikipedia.org/wiki/CentOS#End-of-support_schedule" rel="nofollow" target="_blank">The following seem to be the current dates to plan against:</a></p>
<table border="1"><tbody><tr><th>CentOS Linux Release</th><th>Current End of Life</th></tr>
<tr><td>CentOS 6</td><td>2020-11-30</td></tr>
<tr><td>CentOS 7</td><td>2024-06-30</td></tr>
<tr><td>CentOS 8</td><td>2021-12-31</td></tr>
<tr><th>CentOS Stream Release</th><th>Current End of Life</th></tr>
<tr><td>CentOS Stream 8</td><td>2024-06-??</td></tr>
</tbody></table>
<h2 style="text-align: left;">What Do I Need to do Next</h2><h4 style="text-align: left;">Deal with the Grief <br /></h4><p style="text-align: left;">At this point, System Administrators need to start making a lot of assessments of what they are using CentOS for, why they are using it, and what are the alternatives to move to. This is hard to do when you are angry, pissed, depressed, betrayed, enraged, apoplectic, or the other dozen emotions that have been going through my head lately. So the first thing one has to do is try to work on those emotions as best you can. Talk them out with someone who can take one more rant from you, write them in a blog you never publish, and go for long walks away from computers for several hours. [There are a lot of other methods too and each person has their own way to deal with this.. but the point is that until you get the emotions out, you can not operate.]</p><h4 style="text-align: left;">Do an analytical assessment of why you are currently using CentOS Linux<br /></h4><p style="text-align: left;">There are a plethora of reasons why people use CentOS in their environments.. and those need to be examined to figure out what you need to do next. Sadly like most environments, I think most System Administrators rarely have time to document these and so when having to redeploy you end up with what seems like an impossible project. Like all Charlie Tango Foxtrot situations, one needs to eat the elephant one bite at a time. I find that the following can help make this easier. For each set of servers/services you answer:</p><ol style="text-align: left;"><li>Who are the consumers of this service or server? </li><li>What do they need from their services?</li><li>What is the amount of change that they can deal with on this service?</li><li>How many systems do I have in this service? How can I break them into smaller chunks?</li><li>Where are my systems? [How are <br /></li><li>What automation can I use to roll out these changes?</li><li>When are upcoming deadlines that my consumers need to deal with</li><li>When are upcoming deadlines I have to deal with?</li><li>What are my alternatives? (This is a larger question needing a breakout sheet of why this alternative, how close is it to meeting my needs, what are the costs involved, when can I switch to it, where can i run it, what do I need to change from what I do now?, etc)<br /></li><li>How long would it take to work these alternatives? <br /></li></ol><p>If you have hundreds of servers, then this still looks like a giant load of dung to crawl through. I start with my central Infrastructure server and work my way out.</p><ol style="text-align: left;"><li>Who are the consumers? [I am as use this server to run config scripts to outlying systems.]</li><li>What do they need from their services? [Good uptime and stability so that when an outage happens elsewhere I have a good homebase to work from.]</li><li>What is the amount of change they can deal with on this service? [If I use this as a learning experience, a lot. If I do not have time to learn.. then not a lot.]</li><li>How many systems do I have in this service? 1</li><li>Where are my systems? [Home base is under my desk in my office.]</li><li>What automaton can I use to roll out these changes? Ansible looks like a good choice</li><li>When are upcoming deadlines that my consumers need to deal with? [check with home calendar and mark off dates when I can actually futz with this.]</li><li>When are upcoming deadlines that I have to deal with? 2020-12-31 </li><li>What are my alternatives? [Only listing some alternatives.. rationale needs to be filled out]<br /></li><ul><li><a href="https://centos.org/distro-faq/#q7-how-do-i-migrate-my-centos-linux-8-installation-to-centos-stream" rel="nofollow" target="_blank">CentOS Stream</a> </li><li><a href="https://www.redhat.com/en/technologies/linux-platforms/enterprise-linux" rel="nofollow" target="_blank">Red Hat Enterprise Linux</a><br /></li><li><a href="https://www.oracle.com/linux/" rel="nofollow" target="_blank">Oracle Enterprise Linux</a></li><li><a href="https://springdale.math.ias.edu/" rel="nofollow" target="_blank">Springdale Linux</a></li><li><a href="https://rockylinux.org/" rel="nofollow" target="_blank">Rocky Linux (maybe 2020-1Q?)</a></li><li><a href="https://blog.cloudlinux.com/announcing-open-sourced-community-driven-rhel-fork-by-cloudlinux" rel="nofollow" target="_blank">Lenix (maybe 2020-1Q?)</a></li><li><a href="https://lwn.net/Distributions/" rel="nofollow" target="_blank">Completely different Linux distribution</a> <br /></li></ul><li>How long would it take to work these alternatives? ... unknown at this time.</li></ol><p>This list is a starting point for getting the conversation going with some questions not needed and others needing to be added. Once you have worked out the questions then it is time to work out with each of the stakeholders of your services to answer the questions for themselves and then come to a conclusion. For deadlines to work this out, I would use the rule of thumb I found when dealing with change over of Fedora services with several hundred systems and also previous Tango Foxtrot Charlie changes:</p>
<table border="1"><tbody><tr><th>Step</th><th>Soft Deadline</th><th>Hard deadline</th></tr>
<tr><td>Inventory current systems</td><td>2021-01-31</td><td>2021-02-15</td></tr>
<tr><td>Evaluate alternatives</td><td>2021-02-28</td><td>2021-03-15</td></tr>
<tr><td>Write automation rules for minimal services</td><td>2021-03-31</td><td>2021-04-15</td></tr>
<tr><td>Roll out staged minimal new infra using different choices if possible</td><td>2021-04-30</td><td>2021-05-15</td></tr>
<tr><td>Evaluate rollout and go with best choice</td><td>2021-05-15</td><td>2020-05-31</td></tr>
<tr><td>Determine convergence planning for additional services</td><td>2021-05-15</td><td>2020-05-31</td></tr>
<tr><td>Begin Rollout of New Infra</td><td>2021-06-30</td><td>2021-07-15</td></tr>
<tr><td><i><strike>A series of miracles occur </strike>sorry I mean lots of planned deliverables are met.<br /></i></td><td><br /></td><td><br /></td></tr>
<tr><td>Last systems are moved over.</td><td>2021-12-31</td><td>2020-12-31</td></tr>
<tr><td>Last updates for CentOS-8</td><td>2020-12-31</td><td>2020-12-31</td>
</tr></tbody></table>
<p><b>Note: </b>None of the above is to say 'golly gee, its no problem, you just need to throw away all your other work plans for 2021.' or any similar flipant response. It is only meant as a possible way to deal with this in a sane way. Some ways of converting/change are easier than others, but they all require some work and knowing what the tradeoffs are for each one. </p>Stephen Smoogenhttp://www.blogger.com/profile/17026786034163911165noreply@blogger.com0tag:blogger.com,1999:blog-9079308506459446172.post-23545112265942272552020-10-29T15:21:00.000-06:002020-10-29T15:21:02.384-06:00RHEL-6/CentOS-6/SciLin-6/EPEL-6 End Of Life Notice 2020-11-30<h1 style="text-align: left;">CentOS-6 End of Life Notice 2020-11-30</h1><div>This is a short reminder that Red Hat Enterprise Linux (RHEL) version 6 will enter 'Extended Lifetime Support' in about 30 days from when I am writing this. Extended Lifetime Support (ELS) is a specific contract with Red Hat for them to cover certain security fixes for some extended time to allow sites some time for last minute transitions. </div><div><br /></div><div>RHEL-6 was released in November of 2010, and was the first RHEL I got to work with/on after I returned to Red Hat in 2009. The release has seen 10 minor releases (1 less than RHEL-5), and has been in 'extended' mode since the last 6.10 release in June 2018. </div><div><br /></div><h2 style="text-align: left;">What does this mean to me?</h2><div>When RHEL-6 enters its version of ELS, then it is considered 'end of life' by its 'downstream' rebuilders CentOS and Scientific Linux (and by extension EPEL). I am not sure what Scientific Linux plans are, but for CentOS, they will follow a similar plan as they did with EL-4/EL-5 distributions end. First they will copy all the code from the master servers to the vault servers. Then they will turn off the main mirrorlist systems any references to EL-6. This will cause the 'yum' command to fail and usually causes all kinds of yelling and screaming as people realize they needed to have moved their servers to something else before December 1, 2020. </div><div><br /></div><div>For EPEL users it will be slightly different. All EPEL builds for EL-6 will be 'stopped' on December 1st. [Packages will no longer be allowed to branch to EL-6, builders will not be able to build code for EL-6. Packagers will not be able to move software from testing to prod.] At that point all RPMS on the download servers will be hardlinked to /pub/archive on the master servers. After a week, the mirrorlists will point to the archive zone, and Fedora Infrastructure will remove the code from /pub/epel/6/ trees. </div><div><br /></div><h2 style="text-align: left;">What can I do to deal with this?</h2><div>Primarily, if you are going to be affected by the end of EL-6 services, you either need to get an ELS contract, move to another OS, or move to self-support. In order to self-support, you will need to mirror the source code from your distribution provider and learn the basics of RPM building. If you are on CentOS and find your servers not able to do yum installs anymore.. you will need to mirror the EL-6 from the CentOS vault somewhere locally and use that as your new 'mirror'. Depending on time and energy, I will try to outline some of these steps in future blog posts. </div><div><br /></div><div>[references for article:<a href=" https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux#RHEL_6" target="_blank"> https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux#RHEL_6</a> ] </div>Stephen Smoogenhttp://www.blogger.com/profile/17026786034163911165noreply@blogger.com0tag:blogger.com,1999:blog-9079308506459446172.post-84869300325668365832019-10-07T08:16:00.001-06:002019-10-07T08:38:29.777-06:00Happy Halloween (Packages Not In EPEL-8 yet)It is October, and in the US it means that all the decorations for Halloween are going up. This is a time of year I love because you get to dress up in a costume and give gifts to people. In the spirit of Halloween, I am going to make various packages available in a COPR to add onto the EPEL-8 repositories.<br />
<br />
There are a lot of packages which are in EPEL-6 or EPEL-7 but are not in EPEL-8 yet. Some of these may not be possible due to missing -devel, others may just need someone interested in maintaining a branch for EPEL-8, etc etc. In order to try and get a push on this I wanted to see what packages could be built and made ready at some point. I also wanted to make it possible that if you really needed this package, that they could be available. <br />
<br />
<b>Important notes:</b><br />
<br />
<ol>
<li>These packages will not be getting updates</li>
<li> These packages will not be something you can file a bugzilla on if they don't work. </li>
<li>If they turn out to be filled with goblins, you have been warned. </li>
<li>If your system starts glowing green and moaning about Elder Gods.. I take no responsibility. </li>
</ol>
<br />
That said, this is how I am building these packages in case you want to do this also:
<br />
<code></code><br />
<pre><code>
#Look up package src git repository
$ kinit
$ fedpkg clone {src name}
$ fedpkg srpm
$ mock --chain -r epel-8-x86_64 --localrepo=/home/smooge/not-yet-in-epel8/
# See what packages were needed to build or fix any spec file changes
# if packages needed start chain of packages and fixes
$ copr-cli build not-yet-in-epel8 {src.rpm 1} {src.rpm 2}
# see what failures occurred.. see if they are fixable.
</code></pre>
<br />
Currently packages which are not fixable are anything using perl-generators. There are 2 perl-generators in RHEL/CentOS-8 with one of them being a fully built module and one being a pseudo-module. The mock configs use <tt>best=1</tt> which causes the fully built module perl to be pulled in.. which is the wrong package as it is built against <tt>perl-5.24</tt> and the main perl is <tt>perl-5.26</tt>. Other packages which will not build are ones which need items which RHEL/CentOS-8 did not ship at all (various -devel and such). <br />
<br />
In any case, the copr is at <a href="https://copr.fedorainfracloud.org/coprs/smooge/not-yet-in-epel8/">https://copr.fedorainfracloud.org/coprs/smooge/not-yet-in-epel8/</a> . I will maintain this repository until next March when it like all Halloween themed candy is not even sold at the dollar store anymore.Stephen Smoogenhttp://www.blogger.com/profile/17026786034163911165noreply@blogger.com0tag:blogger.com,1999:blog-9079308506459446172.post-14003494866093619402019-10-01T11:29:00.002-06:002019-10-01T11:29:32.403-06:00Attention: Removal of python36 from EPEL-7 on 2019-10-03With the release of RHEL-7.7, many of the packages for python36 in EPEL were replicated in the release as python3-3.6 packages. The normal pattern when this is seen is to remove the packages from EPEL so that they do not cause problems. However, this did cause problems for users of CentOS-7 who did not have access to the newer packages. Two weeks ago, CentOS-7.7.1908 was released and should have flowed out to users as needed.<br />
<br />
So it is time to remove the following src.rpm packages from EPEL:<br />
<code></code><br />
<pre><code>python36-3.6.8-1.el7.src.rpm
python3-setuptools-39.2.0-3.el7.src.rpm
</code></pre>
<pre><code>
</code></pre>
<br />
As they are duplicated by:
<code></code><br />
<pre><code>python3-3.6.8-10.el7.src.rpm
python3-setuptools-39.2.0-10.el7.src.rpm</code></pre>
<pre><code>
</code></pre>
We will be removing the python packages on 2019-10-03 so that they should disappear during the repository compose on 2019-10-04. EPEL is a rolling release locked against the latest state of the Red Hat Enterprise Linux repositories. If you are using an older snapshot of RHEL or CentOS, you should sync down versions of the repository and lock particular versions for your use.Stephen Smoogenhttp://www.blogger.com/profile/17026786034163911165noreply@blogger.com0tag:blogger.com,1999:blog-9079308506459446172.post-34321740238572745722019-09-19T07:41:00.003-06:002019-09-19T07:41:43.162-06:00Attention: Fedora Yahoo Email UsersGoing from a<a href="https://mmcgrath.livejournal.com/37248.html" target="_blank"> blast of the past</a> we are currently going through one of the Yahoo is not allowing many emails with either fedoraproject.org OR from our mail routers. It would seem that the way to get yahoo to blacklist a domain is to get subscribed to mailing lists and then report the lists as SPAM. Enough accounts (or maybe if one person does it enough times).. yahoo will helpfully blacklist the domain completely. [It then is usually a multi-month process of people explaining that no Fedora is not a spam site, hasn't been taken over by a spam site, or a bunch of other things which do happen so any mail admin is going to be wary on.]<br />
<br />
The funny thing is that their blockage doesn't work 100% so some people seem to still get emails delivered even when most of our logs show that yahoo is telling our servers various SMTP errors of GO AWAY.<br />
<br />
At this point, if you are a packager with a yahoo.com email address, you probably have not gotten an email from any lists or possibly bugzilla for a bit. Trying to email you directly from our site to tell you this isn't going to work.. so we are going back to blogs on the hopes that someone still reads them.<br />
<br />
<br />Stephen Smoogenhttp://www.blogger.com/profile/17026786034163911165noreply@blogger.com0tag:blogger.com,1999:blog-9079308506459446172.post-65726365104915584542019-09-16T11:47:00.000-06:002019-09-16T11:47:05.596-06:00EPEL Bug: Bash errors on recent EL-8 systems.Last week, I got asked about a problem with using EPEL-8 on Oracle Enterprise Linux 8 where trying to install packages failed due to bad license file. I duplicated the problem on RHEL-8 which had not happened before some recent updates.
<br />
<pre><code>
[smooge@localhost ~]$ repoquery
bash: repoquery: command not found...
Failed to search for file: Failed to download gpg key for repo 'epel': Curl error (37): Couldn't read a file:// file for file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-8.0 [Couldn't open file /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-8.0]
</code></pre>
The problem seems to be that the EPEL release package uses the string $releasever for various short-cut strings. Take for example:
<br />
<pre><code>
[epel-playground]
name=Extra Packages for Enterprise Linux $releasever - Playground - $basearch
#baseurl=https://download.fedoraproject.org/pub/epel/playground/$releasever/Everything/$basearch/os
metalink=https://mirrors.fedoraproject.org/metalink?repo=playground-epel$releasever&arch=$basearch&infra=$infra&content=$contentdir
failovermethod=priority
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-$releasever
</code></pre>
<br />
The problem is that when I wrote new versions of the EPEL-8 repo file, I replaced the old key phrase <tt>gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7</tt> with <tt>gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-$releasever</tt> . When I tested things with the dnf command it worked fine but I didn't check to see where things like bash completion would show up.<br />
<br />
Moving back to the format that EPEL-6 and EPEL-7 used fixes the problem, so I will be pushing an updated release file out this week. My apologies for people seeing the errors.Stephen Smoogenhttp://www.blogger.com/profile/17026786034163911165noreply@blogger.com0tag:blogger.com,1999:blog-9079308506459446172.post-46159838367238951982019-08-14T08:50:00.000-06:002019-08-14T13:00:28.682-06:00Announcing EPEL-8.0 Official Release<div dir="ltr" id="docs-internal-guid-a65b02ba-7fff-94c3-ac0d-e4348e18a5ca" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 26pt; vertical-align: baseline; white-space: pre-wrap;">EPEL-8.0 released</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">The EPEL Steering Committee is pleased to announce that the initial EPEL-8 is ready for release. We would like to thank everyone in the community for helping us get the initial set of builds out to mirrors and to consumers worldwide. Special thanks go to Patrick Uiterwijk, Jeroen van Meeuwen, Robert Scheck, and many others in the community who helped in the last 6 months to get this release done. </span></div>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">EPEL-8.0 has packages for the x86_64, ppc64le, aarch64, and now the s390x platforms.</span><br />
<span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="font-family: arial; font-size: xx-small;"><span style="white-space: pre-wrap;"><a href="https://download.fedoraproject.org/pub/epel/8/Everything/" target="_blank">https://download.fedoraproject.org/pub/epel/8/Everything/ </a>is the link for seeing what packages are available. <b>[Edited 2019-08-14 19:59 UTC add in link for people to follow]</b></span></span></div>
<h2 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 18pt;">
<span style="font-family: "arial"; font-size: 16pt; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">What is EPEL?</span></h2>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">EPEL stands for Extra Packages for Enterprise Linux and is a subcommunity of the Fedora and CentOS projects aimed at bringing a subset of packages out of Fedora releases ready to be used and installed on various Red Hat Enterprise Linux (RHEL). It is not a complete rebuild of Fedora or even of previous EPEL releases. EPEL is also a community and not a product. As such we need community members to help get packages into the repository more than done in Fedora.</span></div>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">If you are interested in getting a package into EPEL, contact the package maintainer through</span><a href="https://bugzilla.redhat.com/" style="text-decoration-line: none;"><span style="color: black; font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="color: #1155cc; font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">bugzilla</span></a><span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">. This way the request can be tracked, and if the primary maintainer is not interested in branching to EPEL, others can step in and do so. Optionally you can send a request to the </span><a href="mailto:epel-devel@lists.fedoraproject.org" style="text-decoration-line: none;"><span style="color: #1155cc; font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">epel-devel@lists.fedoraproject.org</span></a><span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"> mailing list. If you do so, please include why the package is needed, to help other volunteers decide whether they can support it.</span></div>
<h2 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 18pt;">
<span style="font-family: "arial"; font-size: 16pt; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">What is new?</span></h2>
<h3 dir="ltr" style="line-height: 1.38; margin-bottom: 4pt; margin-top: 16pt;">
<span style="color: #434343; font-family: "arial"; font-size: 14pt; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Playground for Rawhide like things</span></h3>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">We have added an additional set of channels for EPEL-8 called playground. It is similar to Fedora Rawhide so packagers can work on versions of software that are too fast moving or will have large API changes compared to versions in the regular channel.</span></div>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">To make this purpose transparent, when a package is built in epel8, it will normally also be built in epel8-playground. This is done via a packages.cfg file which lists the targets for fedpkg to build against. A successful package build will then go through two different paths:</span></div>
<ul style="margin-bottom: 0; margin-top: 0;">
<li dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 12pt;">
<span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">epel8 package will go into bodhi to be put into epel8-testing</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 0pt;">
<span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">epel8-playground will bypass bodhi and go directly into epel8-playground the next compose.</span></div>
</li>
</ul>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">If a packager needs to focus only on epel8 or epel8-playground they can edit packages.cfg to change the target=epel8 epel8-playground to target=epel8.</span></div>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">Packages in epel8-playground are intended to be used in the following manner:</span></div>
<ul style="margin-bottom: 0; margin-top: 0;">
<li dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 12pt;">
<span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">To test out a new version of the package that might not be stable yet.</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">To test out new packaging of the package</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">To test a major version change of the package intended for the next EPEL-8 minor release.</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">To build a package that will never be stable enough for EPEL-8, but still could be useful to some.</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 0pt;">
<span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">At minor RHEL releases (ie, 8.1, 8.2) people can pull in big changes from playground to the main EPEL-8 packages. Since people will be upgrading and paying more attention than usual anyhow at those points, it’s a great chance to do that change, but you can test beforehand in the playground to make sure these changes work.</span></div>
</li>
</ul>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">Consumers should be aware that packages in EPEL8-playground are without any Service Level Expectations. You may want to only cherry pick packages from the playground as needed.</span></div>
<span style="color: #434343; font-family: "arial"; font-size: 14pt; white-space: pre-wrap;"><br /></span>
<span style="color: #434343; font-family: "arial"; font-size: 14pt; white-space: pre-wrap;">New Architecture: s390x</span><br />
<span style="font-family: "arial"; font-size: 11pt; white-space: pre-wrap;"><br /></span>
<span style="font-family: "arial"; font-size: 11pt; white-space: pre-wrap;">We have added the s390x platform to builds. Some consumers have wanted this platform for many years but we did not have the time to integrate necessary changes. We have done this with EPEL-8, and hope to be able to do so for EPEL-7 if there are continued requests for it. </span><br />
<h2 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 18pt;">
<span style="font-family: "arial"; font-size: 16pt; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">What is next?</span></h2>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">The goal for EPEL-8.1 will be implementing modules into the repository, which allows builds for packages that depend on non-shipped devel packages. It also allows maintainers to supplement and replace other packages they could not under standard EPEL rules. </span></div>
<h2 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 18pt;">
<span style="font-family: "arial"; font-size: 16pt; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Known Issues:</span></h2>
<ol style="margin-bottom: 0; margin-top: 0;">
<li dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">EPEL-8.0 does not come with modules. Packages built for perl, python and other modules are only built against “default” modules. For example installing a perl library from EPEL will work with the perl-5.26 but not with the perl-5.24 module.</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">RHEL-8.0 and RHEL-8.1 beta do not come with the same packages in all architectures. There are 720 ‘desktop’ packages which were only shipped for x86_64 and ppc64le. Packagers looking to deliver GNOME, KDE, or other platforms will need to exclude s390x and aarch64 at this time.</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The dnf in RHEL-8.1 beta does not work with the EPEL repository due to zchunk code. This has been opened as an upstream bug as </span><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1719830" style="text-decoration-line: none;"><span style="color: #1155cc; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">https://bugzilla.redhat.com/show_bug.cgi?id=1719830</span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> </span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Until modularity and module builds are implemented in EPEL, there will be many packages which can not be built for EPEL. This is mainly due to RHEL-8 not shipping many -devel packages and the need for us to rebuild those packages in a module to make those -devel available to build against. When running into this please open a ticket with</span><a href="https://pagure.io/epel/new_issue" style="text-decoration-line: none;"><span style="color: black; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="color: #1155cc; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">https://pagure.io/epel/new_issue</span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> for us to put in a request for it to be added to Red Hat’s Code Ready Builder. Please list the package(s) which is blocked from being built because of its absence. We will collate these items into bugzilla tickets which will be reviewed by the Red Hat product groups to see if they will be added in future Code Ready Builder releases. Doing this will ensure that we do not have 70 requests for foo-devel but can have one with all the packages needing it.</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">/usr/bin/python does not exist. Developers should aim towards /usr/bin/python3 or /usr/bin/python2 and patch appropriately. Python2 packages are discouraged. RHEL-8 will contain python2.7 until probably the end of life of RHEL-7. However support upstream will only be minimal. When modularity occurs, we suggest that you make whatever python2 packages modules which can be pulled out when RHEL-8.N no longer has python2.</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">python2-sphinx is not shipped. Most packages should work with python3-sphinx, and if it doesn’t please open a bug. The python team has been good about making fixes for this.</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">When branching python packages, be aware that python in EL-8 is python36 and not the version currently in rawhide. This has come up with a couple of test packages where they assumed python37 or later.</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">While EL-8 comes with platform-python, it should NOT be used in Requires: unless absolutely necessary. python3 should be used instead. (Exceptions can be made but will be rare and need justification.) </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;"> [Accepted exception: </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Use python3.6dist(coverage) instead of python3-coverage. This package is not shipped but is needed in %check code.</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">]</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Sometimes RHEL8 only has a python3 package for a dependency you need for your build. (Example: python-bleach requires python2-html5lib, but RHEL8 provides only python3-html5lib). For EPEL-8.0 we recommend strongly to only focus on python3 subpackages.. </span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">RHEL-8 was built with packages which were not shipped. In general it is OK to branch these packages and build them in EPEL.</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">systemd-rpm-macros is not a separate packages. If needed, used BuildRequires: systemd</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 0pt;">
<span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">You will need to make sure you have a version of fedpkg greater than fedpkg-1.37-4 to work with both `epel8` and `epel8-playground`. Versions before that should work with just `epel8`.</span></div>
</li>
</ol>
<h2 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 18pt;">
<span style="font-family: "arial"; font-size: 16pt; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Developer requests for multiple branches</span></h2>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">Branching is handled the same way as requesting a branch using fedpkg request-branch. A maintainer can request an epel8 branch using fedpkg request-branch epel8 which will create a ticket in</span><a href="https://pagure.io/releng/fedora-scm-requests/issues" style="text-decoration-line: none;"><span style="color: black; font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="color: #1155cc; font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">https://pagure.io/releng/fedora-scm-requests/issues</span></a><span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"> and Release Engineering will process these requests.</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">To branch multiple packages please use this or a variant of this script:</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "roboto mono" , monospace; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">#!/usr/bin/sh</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "roboto mono" , monospace; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"># Reminder to get an updated pagure token for releng tickets</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "roboto mono" , monospace; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"># Usage: epel-8.sh package1 package2 package3 package4</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "roboto mono" , monospace; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">if [ $# -lt 1 ]</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "roboto mono" , monospace; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">then</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "roboto mono" , monospace; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"> echo "At least one package name should be provided"</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "roboto mono" , monospace; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">else</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "roboto mono" , monospace; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"> TMPDIR=`mktemp -d /tmp/epel8.XXXXXX`</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "roboto mono" , monospace; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"> pushd "$TMPDIR"</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "roboto mono" , monospace; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"> for pkg in "$@"</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "roboto mono" , monospace; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"> do</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "roboto mono" , monospace; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"> fedpkg clone "$pkg"</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "roboto mono" , monospace; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"> pushd "$pkg"</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "roboto mono" , monospace; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"> fedpkg request-branch epel8</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "roboto mono" , monospace; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-family: "roboto mono" , monospace; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"> fedpkg request-branch epel8-playground</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "roboto mono" , monospace; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"> popd</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "roboto mono" , monospace; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"> done</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "roboto mono" , monospace; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"> rm -rfv "$TMPDIR"</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "roboto mono" , monospace; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">fi</span></div>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">Releng will then work through the tickets in the system which is adding branches to the PDC and</span><a href="http://src.fedoraproject.org/" style="text-decoration-line: none;"><span style="color: black; font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="color: #1155cc; font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">src.fedoraproject.org</span></a><span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">.</span></div>
<h2 dir="ltr" style="line-height: 1.38; margin-bottom: 4pt; margin-top: 18pt;">
<span style="font-family: "arial"; font-size: 17pt; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Known RHEL-8 packages missing -devel</span></h2>
<ol style="margin-bottom: 0; margin-top: 0;">
<li dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 12pt;">
<span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">libblueray-devel</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">liba52-devel</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">libXvMC-devel</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">libdvdnav-devel</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">gfbgraph-devel</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">libuv-devel</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">rest-devel</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 0pt;">
<span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">qgpgme-devel</span></div>
</li>
</ol>
<h2 dir="ltr" style="line-height: 1.38; margin-bottom: 4pt; margin-top: 18pt;">
<span style="font-family: "arial"; font-size: 17pt; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Definitions</span></h2>
<ol style="margin-bottom: 0; margin-top: 0;">
<li dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 12pt;">
<span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Package maintainer. Person who has accepted responsibility to package and maintain software in the Fedora Project ecosystem. The main packager is usually someone focused on Fedora Linux, and secondary packagers may be focused on particular use cases like EPEL.</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Consumer. A person who has subscribed to EPEL for packages but is not a maintainer.</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 12pt; margin-top: 0pt;">
<span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">PDC. Product Definition Center. A tool to help list the lifetime and permissions that a product has so that branching and updates can be better managed.</span></div>
</li>
</ol>
Stephen Smoogenhttp://www.blogger.com/profile/17026786034163911165noreply@blogger.com1tag:blogger.com,1999:blog-9079308506459446172.post-68375915308673814582019-07-09T11:24:00.000-06:002019-07-09T11:24:02.798-06:00EPEL-8 Production Layout<h1 class="part" data-endline="1" data-startline="1" id="EPEL-8-Production-Layout">
EPEL-8 Production Layout</h1>
<h2 class="part" data-endline="3" data-startline="3" id="TL-DR">
<a class="anchor hidden-xs" href="https://hackmd.io/DIhzTXRDTe2CtgMNSdZHGA?both#TL-DR" smoothhashscroll="" title="TL-DR"></a></h2>
<h2 class="part" data-endline="3" data-startline="3" id="TL-DR">
TL; DR:</h2>
<ol class="part" data-endline="9" data-startline="4">
<li class="" data-endline="4" data-startline="4">EPEL-8 will have a multi-phase roll-out into production.</li>
<li class="" data-endline="5" data-startline="5">EPEL-8.0 will build using existing grobisplitter in order to use a ‘flattened’ build system without modules.</li>
<li class="" data-endline="6" data-startline="6">EPEL-8.1 will start in staging without grobisplitter and using default modules via mock.</li>
<li class="" data-endline="7" data-startline="7">The staging work will
allow for continual development changes in koji, ‘ursa-prime’, and MBS
functionality to work without breaking Fedora 31 or initial EPEL-8.0
builds.</li>
<li class="" data-endline="9" data-startline="8">EPEL-8.1 will look to
be ready by November 2019 after Fedora 31 around the time that RHEL-8.1
may release (if it uses a 6 month cadence.)</li>
</ol>
<h2 class="part" data-endline="10" data-startline="10" id="Multi-phase-rollout">
<a class="anchor hidden-xs" href="https://hackmd.io/DIhzTXRDTe2CtgMNSdZHGA?both#Multi-phase-rollout" smoothhashscroll="" title="Multi-phase-rollout"></a></h2>
<h2 class="part" data-endline="10" data-startline="10" id="Multi-phase-rollout">
Multi-phase roll-out</h2>
<div class="part" data-endline="12" data-startline="12">
<a href="https://smoogespace.blogspot.com/2019/06/epel-8.html" rel="noopener" target="_blank">As documented elsewhere</a>,
EPEL-8 has been slowly rolling out due to the many changes in RHEL and
in the Fedora build system since EPEL-7 was initiated in 2014. Trying to
roll out an EPEL-8 which was ‘final’ and thus the way it always will be
was too prone to failure as we find we have to constantly change plans
to match reality.</div>
<div class="part" data-endline="14" data-startline="14">
We
will be rolling out EPEL-8 in a multi-phase release cycle. Each cycle
will allow for hopefully greater functionality for developers and
consumers. On the flip side, we will find that we have to change
expectations of what can and can not be delivered inside of EPEL over
that time.</div>
<div class="part" data-endline="16" data-startline="16">
Phases:</div>
<ol class="part" data-endline="20" data-startline="17">
<li class="" data-endline="17" data-startline="17">8.0 will be a
‘minimal viability’. Due to un-shipped development libraries and the lack
of building replacement modules, not all packages will be able to
build. Instead only non-modular RPMs which can rely on only ‘default’
modules will work. Packages must also only rely on what is shipped in
RHEL-8.0 BaseOS/AppStream/CodeReadyBuilder channels versus any
‘unshipped -devel’ packages.</li>
<li class="" data-endline="18" data-startline="18">8.1 will add on
‘minimal modularity’. Instead of using a flattened build system, we will
look at updating koji to have basic knowledge of modularity, use a tool
to tag in packages from modules as needed, and possibly add in the
Module Build System (MBS) in order to ship modules.</li>
<li class="" data-endline="20" data-startline="19">8.2 will finish
adding in the Module Build System and will enable gating and CI into the
workflow so that packages can tested faster.</li>
</ol>
<div class="part" data-endline="21" data-startline="21">
Due to the
fact that the phases will change how EPEL is produced, there may be need
to be mass rebuilds between each one. There will also be changes in
policies about what packages are allowed to be in EPEL and how they
would be allowed.</div>
<h2 class="part" data-endline="24" data-startline="24" id="Problems-with-koji-modules-and-mock">
<a class="anchor hidden-xs" href="https://hackmd.io/DIhzTXRDTe2CtgMNSdZHGA?both#Problems-with-koji-modules-and-mock" smoothhashscroll="" title="Problems-with-koji-modules-and-mock"></a></h2>
<h2 class="part" data-endline="24" data-startline="24" id="Problems-with-koji-modules-and-mock">
Problems with koji, modules and mock</h2>
<div class="part" data-endline="26" data-startline="26">
If you are wanting to build packages in mock, you can set up a lot of controls in <code>/etc/mock/foo.cfg</code> which will turn on and off modules as needed so that you can enable the <code>javapackages-tools</code> or <code>virt-devel</code> module so that packages like <code>libssh2-devel</code> or <code>javapackages-local</code>
are available. However koji does not allow this control per channel
because it is meant to completely control what packages are brought into
a buildroot. Every build records what packages were used to build an
artifact and koji will create a special mock config file to pull in
those items. This allows for a high level of auditability and
confirmation that the package stored is the package built, and that what
was built used certain things.</div>
<div class="part" data-endline="28" data-startline="28">
For
building an operating system like Fedora or Red Hat Enterprise Linux
(RHEL), this works great because you can show how things were done 2-3
years later when trying to debug something else. However when koji does
not ‘own’ the life-cycle of packages this becomes problematic. In
building EPEL, the RHEL packages are given to the buildroot via external
repositories. This means that koji does not fully know the life-cycle of
the packages it ‘pulls’ in to the buildroot. In a basic mode it will
choose packages it has built/knows about first, then packages from the
buildroot, and if there is a conflict from external packages will try to
choose the one with the highest <code>epoch-version-release-timestamp</code> so that only the newest version is in. (If the timestamp is the same, it tends to refuse to use both packages).</div>
<div class="part" data-endline="30" data-startline="30">
An improvement to this was adding code to <code>mergerepo</code>
which allows for dnf to make a choice on which packages to use between
repositories. This allows for mock’s dnf to pull in modules without the
repositories having been mangled or ‘flattened’ as with grobisplitter.
However, it is not a complete story. For DNF to know which modules to
pull in it needs to set an environment variable for the platform (for
fedora releases it is something like f30 and for RHEL it is el8). Koji
doesn’t know how to do this so the solution would be to set it in the
build systems <code>/etc/mock/site-defaults.cfg</code> but that would affect all builds and would cause problems for building Fedora on the same build system.</div>
<h2 class="part" data-endline="32" data-startline="32" id="Grobisplitter">
<a class="anchor hidden-xs" href="https://hackmd.io/DIhzTXRDTe2CtgMNSdZHGA?both#Grobisplitter" smoothhashscroll="" title="Grobisplitter"></a></h2>
<h2 class="part" data-endline="32" data-startline="32" id="Grobisplitter">
Grobisplitter</h2>
<div class="part" data-endline="34" data-startline="34">
A
second initiative to deal with building with modules was to try and
take modules out of the equation completely. Since a module is a virtual
repository embedded in a real one, you should be able to pull them
apart and make new ones. <a href="https://pagure.io/puiterwijk/grobisplitter" rel="noopener" target="_blank">Grobisplitter</a>
was designed to do this to help get CentOS-8 ready and also allow for
EPEL to bootstrap using a minimal buildset. While working on this, we
found that we needed also parts of the ‘–bare’ koji work because certain
python packages have the same src.rpm name-version but different
releases which koji would kick out.</div>
<div class="part" data-endline="36" data-startline="36">
Currently
grobisplitter does not put in any information about the module it
‘spat’ out. This will affect building when dnf starts seeing metadata in
individual rpms which says ‘this is part of a module and needs to be
installed as such’.</div>
<h2 class="part" data-endline="39" data-startline="39" id="Production-plans">
<a class="anchor hidden-xs" href="https://hackmd.io/DIhzTXRDTe2CtgMNSdZHGA?both#Production-plans" smoothhashscroll="" title="Production-plans"></a></h2>
<h2 class="part" data-endline="39" data-startline="39" id="Production-plans">
Production plans</h2>
<div class="part" data-endline="41" data-startline="41">
We are trying to determine which tool will work better long term in order to make EPEL-8.0 and EPEL-8.1 work.</div>
<h3 class="part" data-endline="44" data-startline="44" id="EPEL-80">
<a class="anchor hidden-xs" href="https://hackmd.io/DIhzTXRDTe2CtgMNSdZHGA?both#EPEL-80" smoothhashscroll="" title="EPEL-80"></a></h3>
<h3 class="part" data-endline="44" data-startline="44" id="EPEL-80">
EPEL-8.0</h3>
<table class="part" data-endline="57" data-startline="46">
<thead>
<tr>
<th>Start Date</th>
<th>End Date</th>
<th>Work Planned</th>
<th>Party Involved</th>
</tr>
</thead>
<tbody>
<tr>
<td>2019-07-01</td>
<td>2019-07-05</td>
<td>Lessons Learned</td>
<td>Smoogen, Mohan</td>
</tr>
<tr>
<td>2019-07-01</td>
<td>2019-07-05</td>
<td>Documentation</td>
<td>Smoogen</td>
</tr>
<tr>
<td>2019-07-08</td>
<td>2019-07-12</td>
<td>Release Build work</td>
<td>Mohan, Fenzi</td>
</tr>
<tr>
<td>2019-07-08</td>
<td>2019-07-12</td>
<td>Call for packages</td>
<td>Smoogen</td>
</tr>
<tr>
<td>2019-07-15</td>
<td>2019-07-19</td>
<td>Initial branching</td>
<td>Mohan, Dawson</td>
</tr>
<tr>
<td>2019-07-22</td>
<td>2019-07-31</td>
<td>First branch/test</td>
<td>Dawson, et al</td>
</tr>
<tr>
<td>2019-08-01</td>
<td>2019-08-01</td>
<td>EPEL-8.0 GA</td>
<td>EPEL Steering Committee</td>
</tr>
<tr>
<td>2019-08-01</td>
<td>2019-08-08</td>
<td>Lessons Learned</td>
<td>Smoogen, et al</td>
</tr>
<tr>
<td>2019-08-01</td>
<td>2019-08-08</td>
<td>Revise documentation</td>
<td>Smoogen, et al</td>
</tr>
<tr>
<td>2019-09-01</td>
<td>2019-09-01</td>
<td>Bodhi gating turned on</td>
<td>Mohan</td>
</tr>
</tbody>
</table>
<h4 class="part" data-endline="59" data-startline="59" id="EPEL-80-Production-Breakout">
<a class="anchor hidden-xs" href="https://hackmd.io/DIhzTXRDTe2CtgMNSdZHGA?both#EPEL-80-Production-Breakout" smoothhashscroll="" title="EPEL-80-Production-Breakout"></a></h4>
<h4 class="part" data-endline="59" data-startline="59" id="EPEL-80-Production-Breakout">
EPEL-8.0 Production Breakout</h4>
<ol class="part" data-endline="69" data-startline="60">
<li class="" data-endline="60" data-startline="60"><strong>Lessons Learned.</strong>
Document the steps and lessons learned from the previous time frame.
Because previous EPEL spin-ups have been done multiple years apart, what
was known is forgotten and has to be relearned. By capturing it, we
hope that EPEL-9 does not take as long.</li>
<li class="" data-endline="61" data-startline="61"><strong>Documentation.</strong>
Write documents on what was done to set up the environment and what is
expected in the next section (how to branch to EPEL-8, how to build with
EPEL-8, dealing with unshipped packages, updated FAQ)</li>
<li class="" data-endline="62" data-startline="62"><strong>Call for Packages</strong> This will be going over the steps that packagers need to follow to get packages branched to EPEL-8.</li>
<li class="" data-endline="63" data-startline="63"><strong>Release Build Work.</strong>
This is setting up the builders and environment in production. Most of
the steps should be repeats of what was done in staging with additional
work done in bodhi to have signing and composes work</li>
<li class="" data-endline="64" data-startline="64"><strong>Initial Branching.</strong>
This where the first set of packages are needed to be branched and
built for EPEL-8: epel-release, epel-rpm-macros, fedpkg-minimal, fedpkg
(and all the things needed for it).</li>
<li class="" data-endline="65" data-startline="65"><strong>First Branch</strong>
Going over the various tickets for EPEL-8 packages, a reasonable sample
will be branched. Work will be done with the packagers on problems they
find. This will continue as needed.</li>
<li class="" data-endline="66" data-startline="66"><strong>EPEL-8.0 GA</strong> Branching can follow normal processes to get done.</li>
<li class="" data-endline="67" data-startline="67"><strong>Lessons Learned.</strong> Go over problems and feed into other groups backlogs.</li>
<li class="" data-endline="69" data-startline="68"><strong>Documentation</strong> Update previous documents and add any that were found to be needed.</li>
</ol>
<h3 class="part" data-endline="70" data-startline="70" id="EPEL-81">
<a class="anchor hidden-xs" href="https://hackmd.io/DIhzTXRDTe2CtgMNSdZHGA?both#EPEL-81" smoothhashscroll="" title="EPEL-81"></a></h3>
<h3 class="part" data-endline="70" data-startline="70" id="EPEL-81">
EPEL-8.1</h3>
<table class="part" data-endline="87" data-startline="72">
<thead>
<tr>
<th>Start Date</th>
<th>End Date</th>
<th>Work Planned</th>
<th>Party Involved</th>
</tr>
</thead>
<tbody>
<tr>
<td>2019-07-01</td>
<td>2019-07-05</td>
<td>Lessons Learned</td>
<td>Fenzi, Contyk, et al</td>
</tr>
<tr>
<td>2019-07</td>
<td>???</td>
<td>Groom Koji changes needed</td>
<td>???</td>
</tr>
<tr>
<td>2019-07</td>
<td>???</td>
<td>Write/Test Koji changes needed</td>
<td>???</td>
</tr>
<tr>
<td>2019-07</td>
<td>???</td>
<td>Non-modular RPM in staging</td>
<td>???</td>
</tr>
<tr>
<td>2019-07</td>
<td>???</td>
<td>MBS in staging</td>
<td>???</td>
</tr>
<tr>
<td>2019-08?</td>
<td>???</td>
<td>Implement Koji changes?</td>
<td>???</td>
</tr>
<tr>
<td>2019-08?</td>
<td>???</td>
<td>Implement bodhi compose in staging?</td>
<td>???</td>
</tr>
<tr>
<td>2019-09?</td>
<td>???</td>
<td>Close off 8.1 beta</td>
<td>???</td>
</tr>
<tr>
<td>2019-09?</td>
<td>???</td>
<td>Lessons learned</td>
<td>???</td>
</tr>
<tr>
<td>2019-09?</td>
<td>???</td>
<td>Begin changes in prod?</td>
<td>???</td>
</tr>
<tr>
<td>2019-10?</td>
<td>???</td>
<td>Open module builds in EPEL</td>
<td>???</td>
</tr>
<tr>
<td>2019-11?</td>
<td>???</td>
<td>EPEL-8.1 GA</td>
<td>EPEL Steering Committee</td>
</tr>
<tr>
<td>2019-11?</td>
<td>???</td>
<td>Lessons Learned</td>
<td>???</td>
</tr>
<tr>
<td>2019-11?</td>
<td>???</td>
<td>Revise documentation</td>
<td>???</td>
</tr>
</tbody>
</table>
<h4 class="part" data-endline="89" data-startline="89" id="EPEL-81-Production-Breakout">
<a class="anchor hidden-xs" href="https://hackmd.io/DIhzTXRDTe2CtgMNSdZHGA?both#EPEL-81-Production-Breakout" smoothhashscroll="" title="EPEL-81-Production-Breakout"></a></h4>
<h4 class="part" data-endline="89" data-startline="89" id="EPEL-81-Production-Breakout">
EPEL-8.1 Production Breakout</h4>
<div class="part" data-endline="91" data-startline="91">
This
follows the staging and production of the 8.0 with additional work in
order to make working with modules work in builds. Most of these dates
and layers need to be filled out in future meetings. The main work will
be adding in allowing a program code-named ‘Ursa-Prime’ to help build
non-modular rpms using modules as dependencies. This will allow for
grobisplitter to be replaced with a program that has long term maintenan</div>
Stephen Smoogenhttp://www.blogger.com/profile/17026786034163911165noreply@blogger.com0tag:blogger.com,1999:blog-9079308506459446172.post-8899583300092393262019-06-28T13:51:00.000-06:002019-06-28T13:52:31.529-06:00Update on EPEL-8 Status<h1 class="part" data-endline="1" data-startline="1" id="Update-on-EPEL-8-Status">
Update on EPEL-8 Status</h1>
<h2 class="part" data-endline="3" data-startline="3" id="Why-is-EPEL-8-Taking-So-Long-tldr">
<a class="anchor hidden-xs" href="https://hackmd.io/Cr1D9zaQQqC4Uiat5DVz4Q?both#Why-is-EPEL-8-Taking-So-Long-tldr" smoothhashscroll="" title="Why-is-EPEL-8-Taking-So-Long-tldr"></a></h2>
<h2 class="part" data-endline="3" data-startline="3" id="Why-is-EPEL-8-Taking-So-Long-tldr">
Where is EPEL-8? (tl;dr:)</h2>
<ol class="part" data-endline="4" data-startline="4">
<li class="" data-endline="4" data-startline="4">Getting koji to work smoothly with modules has been hard. A multi-level fix has had to be worked to get it working in staging.</li>
</ol>
<ul class="part" data-endline="9" data-startline="5">
<li class="" data-endline="5" data-startline="5">Needed a way to split out default modules to deal with koji merge options. <a href="https://github.com/puiterwijk/grobisplitter" rel="noopener" target="_blank">Grobisplitter</a> was written to do this</li>
<li class="" data-endline="6" data-startline="6"><a href="https://pagure.io/koji" rel="noopener" target="_blank">Koji</a> needed further patching to deal with src.rpms with same NVR but different targets (some python2 and python3 come from same src.rpm but were built in different times).</li>
<li class="" data-endline="8" data-startline="8">DNF reposync from RHEL-7 would delete the wrong files if you tried the <code>--newest</code> (fixed.)</li>
<li class="" data-endline="9" data-startline="9">DNF does not know how to reposync modules if it is not the local arch.</li>
</ul>
<ol class="part" data-endline="11" data-startline="10" start="2">
<li class="" data-endline="11" data-startline="10">Code Ready Builder is not always in sync with packages in main trees. If you need a -devel and it isn’t in CRB, then you have to wait until it is there to build something.</li>
<li class="" data-endline="11" data-startline="10">As a couple of fixes landed in mergerepo and koji, we are re-evaluating how we do builds in the next stage of building.</li>
</ol>
<h2 class="part" data-endline="12" data-startline="12" id="Introduction">
<a class="anchor hidden-xs" href="https://hackmd.io/Cr1D9zaQQqC4Uiat5DVz4Q?both#Introduction" smoothhashscroll="" title="Introduction"></a></h2>
<h2 class="part" data-endline="12" data-startline="12" id="Introduction">
Introduction</h2>
<div class="part" data-endline="14" data-startline="14">
In May of 2019, Red Hat released their 8.0 release of Red Hat Enterprise Linux (RHEL). Usually, the Extra Packages for Enterprise Linux (EPEL) group would have a beta available at that time or sooner. With RHEL-8, it has taken a lot longer to get things rolling.</div>
<h2 class="part" data-endline="16" data-startline="16" id="Repository-Changes">
<a class="anchor hidden-xs" href="https://hackmd.io/Cr1D9zaQQqC4Uiat5DVz4Q?both#Repository-Changes" smoothhashscroll="" title="Repository-Changes"></a></h2>
<h2 class="part" data-endline="16" data-startline="16" id="Repository-Changes">
Repository Changes</h2>
<div class="part" data-endline="18" data-startline="18">
EPEL
packages are built inside of the Fedora Projects’ build infrastructure.
This is done by downloading the packages from Red Hat’s public Content
Delivery Network (CDN), and then having the Fedora artifact build system
(koji) use the release as an external build channel. Koji looks at
packages in a different way than other build commands like ‘mock’ do.
Where mock is meant to just build packages, koji is designed about
auditing the entire lifecycle of a package. In other words, if you want
to know how a package in Fedora 12 was built and all its children
interacted over time in the buildroots… you can do that with enough work
and the koji databases. With mock you have a couple of log files which
tell you what was pulled into a buildroot but how those were built would
require you finding their log files, etc etc. A developer can also
download those packages and look at them to see what was in them and how
they were built.<br />
<br /></div>
<div class="part" data-endline="20" data-startline="20">
The
strength of koji is that you can have a credible chain of builds to
know where things came from. However this doesn’t work too well with
building packages for EPEL where koji doesn’t know where the RHEL kernel
came from. Koji uses mergerepo to look at the external packages
provided, determines the src.rpm they would come from and determines
what the latest version it would use from each. From this it creates a
‘buildroot’ which it will use to build packages from. This has worked
pretty well for building packages from RHEL-5,6, and 7. The major
downside has been where someone built a package with the same src.rpm
name which koji then decides is the master no matter if a newer version
shows up in RHEL.<br />
<br /></div>
<div class="part" data-endline="22" data-startline="22">
This
all changed with modularity. Koji really only has a rudimentary idea of
rpms and repositories… it has zero idea about modules and the rules it
has used to determine what an external package is are thrown out with
modules.</div>
<ol class="part" data-endline="26" data-startline="24">
<li class="" data-endline="24" data-startline="24">Packages with
different names may come with from the same src.rpm. In RHEL-8 many
python27 and python36 packages have the same parent src.rpm but were in
different build times. Koji’s standard repo comparison mode will choose
one or the other.</li>
<li class="" data-endline="26" data-startline="25">Packages may have the
same names-version-releases but were built in different module streams
(say perl-5.26 and perl-5.24) Koji would then choose a package depending
on whatever had the largest src.rpm which meant it could try to build a
buildroot with perl-5.24 perl modules but perl-5.26 as the master
perl.</li>
</ol>
<div class="part" data-endline="27" data-startline="27">
If a
developer uses mock to build a package with default repositories, mock
calls dnf which knows about modules and does the right thing. In the
case where you want it to do the ‘wrong’ thing you can also over-ride
mock to do that. With koji, further tools are needed to make this work.
If you are building a new module, then the Modular Build System (MBS)
sits on top of koji and tells koji what to do. It will look at the
module yaml file and turn on/off various modules so that it can build in
what is needed. To build non-modular packages, other fixes are needed.
One of these is called Ursa-Major which was a set of scripts to pull in
needed data from a third database and pull things in as needed. However,
this was not adopted in Fedora for general use so the EPEL group looked
for something different.<br />
<br /></div>
<div class="part" data-endline="29" data-startline="29">
The temporary solution written by Patrick Uiterwijk is called grobisplitter (<a href="https://github.com/puiterwijk/grobisplitter" rel="noopener" target="_blank">https://github.com/puiterwijk/grobisplitter</a>)
which relies on the fact that modules are virtual repositories embedded
in a master repository. Grobisplitter takes this fact, and uses it to
break out ‘real’ repositories for each module. So the RHEL-8 repository
will look like:<br />
<br /></div>
<pre class="part" data-endline="85" data-startline="30"><code>ant:1.10:820181213135032:5ea3b708:x86_64
container-tools:rhel8:8000020190416221845:2ffa3d27:x86_64
container-tools:rhel8:820190211172150:20125149:x86_64
freeradius:3.0:8000020190425181943:75ec4169:x86_64
freeradius:3.0:820190131191847:fbe42456:x86_64
gimp:2.8:820181213135540:77fc8825:x86_64
go-toolset:rhel8:820190208025401:b754926a:x86_64
httpd:2.4:8000020190405071959:55190bc5:x86_64
httpd:2.4:820190206142837:9edba152:x86_64
idm:client:820190227213458:49cc9d1b:x86_64
inkscape:0.92.3:820181213140018:77fc8825:x86_64
javapackages-runtime:201801:820181213140046:302ab70f:x86_64
javapackages-tools:201801:820181217165704:dca7b4a4:x86_64
llvm-toolset:rhel8:820190207221833:9edba152:x86_64
mailman:2.1:820181213140247:77fc8825:x86_64
mariadb:10.3:820190206164045:9edba152:x86_64
mariadb:10.3:820190314153642:9edba152:x86_64
maven:3.5:820181213140354:5ea3b708:x86_64
mercurial:4.8:820190108205035:77fc8825:x86_64
merged_repo
mysql:8.0:820190104140943:9edba152:x86_64
nginx:1.14:820181214004940:9edba152:x86_64
nodejs:10:820190108092226:9edba152:x86_64
non_modular
perl-App-cpanminus:1.7044:820181214184336:e5ce1481:x86_64
perl-DBD-MySQL:4.046:820181214121012:6bc6cad6:x86_64
perl-DBD-Pg:3.7:820181214121102:6fcea174:x86_64
perl-DBD-SQLite:1.58:820181214121133:6bc6cad6:x86_64
perl-DBI:1.641:820190116185335:fbe42456:x86_64
perl-FCGI:0.78:820181214153815:fbe42456:x86_64
perl-YAML:1.24:820181214175558:8652dbeb:x86_64
perl:5.26:820181219174508:9edba152:x86_64
php:7.2:820181215112050:76554e01:x86_64
postgresql:10:820190104140132:9edba152:x86_64
python27:2.7:8000020190410132513:c0efe978:x86_64
python27:2.7:820190212161047:43711c95:x86_64
python36:3.6:8000020190410133122:593c47b3:x86_64
python36:3.6:820190123171828:17efdbc7:x86_64
redis:5:820181217094919:9edba152:x86_64
rhn-tools:1.0:8000020190425124933:6ec19280:x86_64
rhn-tools:1.0:820190321094720:e122ddfa:x86_64
ruby:2.5:820190111110530:9edba152:x86_64
rust-toolset:rhel8:820181214214108:b09eea91:x86_64
satellite-5-client:1.0:820190204085912:9edba152:x86_64
scala:2.10:820181213143541:2b79a98f:x86_64
squid:4:820181213143653:9edba152:x86_64
subversion:1.10:820181215112250:a51370e3:x86_64
swig:3.0:820181213143944:9edba152:x86_64
varnish:6:820181213144015:9edba152:x86_64
virt-devel:rhel:820190226174025:9edba152:x86_64
virt:rhel:8000020190510171727:55190bc5:x86_64
virt:rhel:8000020190516125745:55190bc5:x86_64
virt:rhel:820190226174025:9edba152:x86_64
</code></pre>
<div class="part" data-endline="87" data-startline="87">
In the above, each of those names is the module name, and grobisplitter would then put the appropriate files in each sub repository. The problem with this version is that we end up with multiple repositories with some of them being ‘non-default’ modules. Building against a non-default module causes problems for someone trying to install that package. It
would replace packages from a different module than was wanted. Changes to grobisplitter were made at <a href="https://github.com/smooge/grobisplitter" rel="noopener" target="_blank">https://github.com/smooge/grobisplitter</a> to allow only default modules to be published.<br />
<br /></div>
<div class="part" data-endline="89" data-startline="89">
From this we were able to start deploying a devolved tree in the Fedora staging koji (<a href="https://koji.stg.fedoraproject.org/" rel="noopener" target="_blank">https://koji.stg.fedoraproject.org/</a>)
The first set of fixes needed was to make it so koji could work with
multiple artifacts coming from the same src.rpm. Instead of using the
standard mode for resolving differences, we import RHEL-8 repositories
with a bare mode which is supposed to use external repository data to
determine what should be pulled in. However, we found that koji still
gets confused if multiple versions of a package are in the repo data.
Say your repository contains both <code>glibc-*-2.1-2</code> and <code>glibc-*-2.2-2</code>. Koji would pull in <code>glibc-devel-2.1-2</code> and try to match it against <code>glibc-2.2-2</code>. This of course caused builds to fail.<br />
<br /></div>
<div class="part" data-endline="91" data-startline="91">
At first the fix looked to be having the reposync from the CDN pull only the latest data. However we ran into problems with either the RHEL-7 or
RHEL-8 reposync deleting data we wanted to keep depending on the options used. Part of this was due to module data and part of it was due to some bugs in dnf’s reposync with other architectures. At this point, it looked like one of two things needed to be done.<br />
<br />
One, grobisplitter needs to learn about package order and pull in just the latest package
into a non-modular repo. Two, another layer of indirection is needed where after we split out all the repositories we use reposync again to just pull from the grobisplit repositories. In this case we do so with a <code>-n</code> and only have the latest packages. The second option seemed easier to pursue as most of the grobisplitter toolkit should
become irrelevant when the next generation of Ursa-Major comes out.</div>
<h2 class="part" data-endline="93" data-startline="93" id="Code-Ready-Problems">
<a class="anchor hidden-xs" href="https://hackmd.io/Cr1D9zaQQqC4Uiat5DVz4Q?both#Code-Ready-Problems" smoothhashscroll="" title="Code-Ready-Problems"></a></h2>
<h2 class="part" data-endline="93" data-startline="93" id="Code-Ready-Problems">
Code Ready Problems</h2>
<div class="part" data-endline="95" data-startline="95">
We
ran into our next major problem with RHEL-8 repositories when we found
that -devel and -lib rpms in Code Ready Builder were not always in sync
with their parent packages in BaseOS/AppStream. This means that if your
build is wanting kernel-devel and the BaseOS is 4.9-11 but the CRB
version is 4.9-10 then koji has no way to supply the dependency for you.
The major culprit currently is that the virt module has had multiple
updates but the virt-devel module has not had any updates.</div>
<h2 class="part" data-endline="97" data-startline="97" id="Build-Over-View">
<a class="anchor hidden-xs" href="https://hackmd.io/Cr1D9zaQQqC4Uiat5DVz4Q?both#Build-Over-View" smoothhashscroll="" title="Build-Over-View"></a></h2>
<h2 class="part" data-endline="97" data-startline="97" id="Build-Over-View">
Build Over View</h2>
<ol class="part" data-endline="111" data-startline="98">
<li class="" data-endline="98" data-startline="98">RHEL-8 packages are reposync from cdn onto <a href="http://infrastructure.fedoraproject.org/" rel="noopener" target="_blank">infrastructure.fedoraproject.org</a> nfs directory.</li>
<li class="" data-endline="99" data-startline="99">grobisplitter runs on <a href="http://grobisplitter01.phx2.fedoraproject.org/" rel="noopener" target="_blank">grobisplitter01.phx2.fedoraproject.org</a> to break out each module into repositories in a <code>$date/$arch/$repos</code> layout.</li>
<li class="" data-endline="100" data-startline="100">createrepo is run on <code>$date/$arch</code></li>
<li class="" data-endline="101" data-startline="101">a symbolic link is set to <code>$date</code> staged</li>
<li class="" data-endline="102" data-startline="102"><code>reposync -n -d</code> is run against <code>staged/$arch</code> to <code>latest/$arch</code></li>
<li class="" data-endline="103" data-startline="103">createrepo is run on <code>latest/$arch</code></li>
<li class="" data-endline="104" data-startline="104">koji points to <code>latest/$arch</code></li>
<li class="" data-endline="105" data-startline="105">packages can be built</li>
<li class="" data-endline="106" data-startline="106">packages can be signed</li>
<li class="" data-endline="107" data-startline="107">bodhi and other items do their parts</li>
<li class="" data-endline="108" data-startline="108">we compose</li>
<li class="" data-endline="109" data-startline="109">…</li>
<li class="" data-endline="111" data-startline="110">profit?</li>
</ol>
<h2 class="part" data-endline="112" data-startline="112" id="What-Are-The-Next-Steps">
<a class="anchor hidden-xs" href="https://hackmd.io/Cr1D9zaQQqC4Uiat5DVz4Q?both#What-Are-The-Next-Steps" smoothhashscroll="" title="What-Are-The-Next-Steps"></a></h2>
<h2 class="part" data-endline="112" data-startline="112" id="What-Are-The-Next-Steps">
What Are The Next Steps?</h2>
<div class="part" data-endline="114" data-startline="114">
Currently we are looking to have our internal beta done by July 1st. At that point, we will work on documenting what we have done, and re-implementing the tool changes in production. At which point, developers will be able to make branch requests to releng to make packages available and builds should start flowing. From that we will probably find new things which will need fixes in either spec files or build infrastructure.<br />
<br /></div>
<div class="part" data-endline="116" data-startline="116">
A GANNT chart of our current production plan is provided below.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmQOi7QU7H7jB9DlgMzzze3gVCDaTqLhf8S5diei7-YSGw4dmkIEeEtxvkL-QmMesKyeqzqVkSx6jc5MQpq6rUmTCecT4iQghu3Iu8UFkfLpEsEmD2GGWLGWEoudon2LJv1Z90UPIw8ow/s1600/EPEL8+Plan.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="774" data-original-width="1600" height="307" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmQOi7QU7H7jB9DlgMzzze3gVCDaTqLhf8S5diei7-YSGw4dmkIEeEtxvkL-QmMesKyeqzqVkSx6jc5MQpq6rUmTCecT4iQghu3Iu8UFkfLpEsEmD2GGWLGWEoudon2LJv1Z90UPIw8ow/s640/EPEL8+Plan.png" width="640" /></a></div>
<br /></div>
Stephen Smoogenhttp://www.blogger.com/profile/17026786034163911165noreply@blogger.com1tag:blogger.com,1999:blog-9079308506459446172.post-17326826540400111862019-05-30T17:12:00.000-06:002019-05-31T09:40:54.701-06:00EPEL Proposal: Steve Gallagher's EPEL 8 Branch Strategy<h3>
Stephen Gallagher's Better Proposal for EPEL branching</h3>
<div>
So earlier this week I <a href="https://smoogespace.blogspot.com/2019/05/epel-proposal-epel-master-branch-aka.html">wrote up a proposal</a> for EPEL-rawhide which was to go over various ideas the EPEL steering committee has been kicking around for a bit. This was to try and work into how to branch for EPEL-8 and also how to deal with <a href="https://fedoraproject.org/wiki/EPEL/Changes/Minor_release_based_composes">https://fedoraproject.org/wiki/EPEL/Changes/Minor_release_based_composes</a> and <a href="https://fedoraproject.org/wiki/EPEL/Changes/Release_based_package_lifetimes">https://fedoraproject.org/wiki/EPEL/Changes/Release_based_package_lifetimes</a> During the meeting it was clear that my strawman didn't have much in it, and needed more thinking. Thankfully Stephen Gallagher looked into the meeting and came up with some ideas that he wrote up and <a href="https://lists.fedoraproject.org/archives/list/epel-devel@lists.fedoraproject.org/message/FYOOSAJOTDNCE4RLF32ORFO6WL2M5WD5/">proposed to the list.</a>. I recommend that you read the document and updates if you are interested in how branching in EPEL could work with EL7 and EL8.</div>
<div>
<br /></div>
<h3>
<br /></h3>
<div>
<br />
<br />
<br /></div>
Stephen Smoogenhttp://www.blogger.com/profile/17026786034163911165noreply@blogger.com0tag:blogger.com,1999:blog-9079308506459446172.post-30966265724675414912019-05-28T15:59:00.002-06:002019-05-30T16:44:56.028-06:00EPEL Proposal: EPEL Master branch AKA Rawhide<h1>
EPEL-rawhide</h1>
<h2>
Update:</h2>
<div>
This proposal has been superseded by Stephen Gallagher's excellent wagontrain post. I will put it as a separate post next.</div>
<h2>
tl; dr:</h2>
In order to allow for the ability for faster availability of packages, add rawhide branches for EPEL-7 and EPEL-8. These branches would allow developers to build new packages they aren't sure are ready for either EPEL-N or EPEL-N-testing, and would allow for faster rebuilds of newer features when RHEL has a large feature change.
<br />
<h2>
The Longer Story</h2>
In the past 6 months, EPEL has had to have two major changes in its builds which were made harder by the way EPEL is currently built. The first one was with changes in RHEL-7.6 which dropped some packages and changed some others ABI's. This required a rebuild of a lot of packages, but there was no way we could do a find and fix before we did a 'flag-week' of rebuilds with Troy Dawson and others doing lots of Proven Packager fixes and rebuilds.<br />
<br />
The second one was with the python36 move which also took a large amount of time and still has little problems showing up here and there. In a similar fashion, updates-testing had to be used as a rawhide for packages which made building and testing hard for things not doing this change.<br />
<br />
A third problem showed up when Troy was cleaning out packages in EPEL-6 and 7 testing repos which had been there for years. The packagers were using this for putting things they felt were too unstable for EPEL due to unstable API's so they could either iterate quicker or not break existing users. The problem is that these packages might accidentally get promoted by someone seeing that the packages are tested but wasn't pushed. Having a separate tree for these unstable packages needed a different thinking.<br />
<br />
While doing a review of these two exercises, the EPEL steering committee came up with various ideas.. and I believe Kevin Fenzi brought up adding a rawhide as an easier fix than some of my more convoluted branch every release (aka epel-7.6, epel-7.7, epel-7.8). In this new scheme, we would have the following branches: el6, epel7, epel7-master, epel8, epel8-master.<br />
<br />
A possible work flow could be the following:<br />
<ol>
<li>Packages when branched for EL7 or EL8 would get branched into the epel-M-master tree where they could have builds made against the latest RHEL. </li>
<li>When Red Hat released a new beta (RHEL-M.N-beta), Fedora Infrastructure would download it and set it up so koji could find it. as EPEL-M-Master (or properly bikeshed name). A mass update and rebuild would then be done against all packages in EPEL-M-Master. Breakfixes and testing can be done.</li>
<li>When the General Availability of RHEL-M.N occurs, EPEL will make a copy of EPEL-M.(N-1), EPEL-M.(N-1)-updates and EPEL-M.(N-1)-updates-testing in /pub/archive/epel/M/M.(N-1)/.</li>
<li>An <agreed amount="" of="" time="" upon=""> after Red Hat releases the General Availability of the RHEL-M.N release, </agreed></li>
<ol type="A">
<li>if the version in master is newer than the version in branch, the master version will be checked into the branch. (This step is probably the most problematic and needs more work and thinking by people).</li>
<li>packages which meet certain criteria will then be promoted to EPEL-M with a new compose of EPEL-M.N and an empty EPEL-M.N/updates and EPEL-M.N/updates-testing.</li>
</ol>
<li>The packager can do updates and fixes to packages in the EPEL-M branch </li>
<li>The finishing up of clean up the archives can occur.</li>
</ol>
<div>
This is a preliminary proposal which needs a lot more work and resource commitments in changes to tooling and documentation. I am bringing this up as something I would like to get done as part of revamping EPEL this summer, but I also need feedback and help.</div>
Stephen Smoogenhttp://www.blogger.com/profile/17026786034163911165noreply@blogger.com0tag:blogger.com,1999:blog-9079308506459446172.post-24609899928589724352019-05-28T14:55:00.001-06:002019-05-28T14:55:26.803-06:00EPEL Proposal: Removal of PPC64 (Not PPC64le) in 2019-06-01<h1>
TL; DR:</h1>
EPEL is looking to put its EL6 and EL7 branches of PPC64 into archives by 2019-06-01. This is due to the fact that Fedora no longer builds for the PPC64 big-endian architecture.<br />
<h2>
The long story</h2>
As of the EOL of Fedora 28, the Fedora Project no longer supports or builds packages for the big endian Power64 (or ppc64) architecture. Kevin Fenzi went <a href="https://www.scrye.com/wordpress/nirik/2019/05/23/attention-epel6-and-epel7-ppc64-users/"> over this in his blog article</a>, but I wanted to go over it again. I realize this is short notice so extra steps need to be done. <br />
<br />
The Fedora Project uses Fedora Linux on its builders which is useful for bringing on new architectures, and for getting new features which RHEL does not have yet. However it means that when an OS is End of Lifed, it no longer gets security updates, software improvements, or similar fixes. We could try and stand up an EL7 builder but it would require reworking both tools and scripts that are expecting an F28 world (python3, various newer libraries and scripts, different API's, etc). That would take a while to rework everything back and then continual work of keeping this builder in line with whatever EL8/F30+ world we move to in the coming months. Secondly, this would cut out a limited resource. We only have so many PPC8 systems which we can run PPC64 virtual machines on. The virtual machines can either build an EPEL package or a Fedora <29 be="" but="" down="" epel.="" just="" limiting="" p="" package="" this="" to="" we="" would=""><br />
In the end, the number of PPC64 users are not that great. We have an average of 90 systems per day checking in with many more PPC64LE systems. I think most of the PPC64 users would be able to get stuff from archives just as well.<br />
<h2>How do I get my stuff</h2>
The builds for EL6.10 and EL7.6 will be archived to /pub/archives/epel/7/7.6 and /pub/archives/epel/6/6.10 this week. We may need to roll out an updated epel-release which will point this architecture to that tree. We will then remove the builders from Fedora and stop building for it. In early July I will remove the remaining trees from /pub/epel and put in redirects to the archives.Stephen Smoogenhttp://www.blogger.com/profile/17026786034163911165noreply@blogger.com0tag:blogger.com,1999:blog-9079308506459446172.post-90017347372471922552019-03-14T13:18:00.001-06:002019-03-14T13:18:17.474-06:00Final 503 addendum<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgzaiJuKjKWNMq9BtlvYcgdq5-jdZl6M17tSrrfBrntTqotVdMETW-fSHFH3lZx9PVyew_NHb6eAiR03c_dmh6fJfbNsaazzMGot-eB4ZV5UxKt3UjWIOnSbGz7BKuN7SaJEDP5PcUWR1E/s1600/x503-latest.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1200" data-original-width="1600" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgzaiJuKjKWNMq9BtlvYcgdq5-jdZl6M17tSrrfBrntTqotVdMETW-fSHFH3lZx9PVyew_NHb6eAiR03c_dmh6fJfbNsaazzMGot-eB4ZV5UxKt3UjWIOnSbGz7BKuN7SaJEDP5PcUWR1E/s640/x503-latest.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">mirrorlist 503's for 2019</td></tr>
</tbody></table>
This is a graphical shape of the amount of 503's we have had in 2019. The earlier large growth in January/February have dropped down to just one web-server which is probably underpowered to run containers. We will look at taking it out of circulation in the coming weeks.Stephen Smoogenhttp://www.blogger.com/profile/17026786034163911165noreply@blogger.com0