2021-09-03

How to clone (a lot of) package spec files from CentOS 8 git.centos.org

Recently I have had to try and work out all the dependencies on a set of packages. I am writing this as a blog, as I needed to recreate work that I had done several times in the past, but past Smoogen had not fully documented. [I went looking and found I had 3 different copies of trees of packages from Fedora and CentOS but absolutely no notes and my bash_history had cycled over the 10k lines I normally keep. Past Smoogen was a BAD, BAD sysadmin.]

For general requires, I would do this by using dnf or dnf


$ dnf repoquery --requires bash
Last metadata expiration check: 2:44:24 ago on Fri 03 Sep 2021 10:14:22 EDT.
filesystem >= 3
libc.so.6(GLIBC_2.15)(64bit)
libdl.so.2()(64bit)
libdl.so.2(GLIBC_2.2.5)(64bit)
libtinfo.so.6()(64bit)
rtld(GNU_HASH)

However in this case, I needed to also work out all the buildrequires of the packages, and then the requires and buildrequires of those tools. Basically it is sort of building a buildroot for a set of leaf packages which usually means I need to get the spec files and parse them with something like spectool.

If I was working with Fedora, I would take the shallow git clone of the src.fedoraproject.org website which can be found at https://src.fedoraproject.org/lookaside/. Then I would start going down my list of 'known' software I need to work and clone out the usual 'buildroot' a fedpkg mockbuild of the package would give. However I am working with CentOS Stream 8 which is a slightly different repository layout, and does not have a prebuilt shallow clone.

Thankfully, the CentOS repository has a very useful sub-repository called https://git.centos.org/centos-git-common.git which contains all the tools to fetch the appropriate code and tools from the CentOS src repository. The first one I need work with is centos.git.repolist.py to query the pagure API and get a list of packages. I then need to clean up that list a bit because it contains some forks but the following will get me a complete list of the packages I want to parse:

[centos-git-common (master)]$ ./centos.git.repolist.py | grep -v '/forks/' > repolist-2021-09-03 [centos-git-common (master)]$ ./centos.git.repolist.py --namespace modules | grep -v '/forks/' >> repolist-2021-09-03 [centos-git-common (master)]$ sort -o repolist-2021-09-03 -u repolist-2021-09-03 [centos-git-common (master)]$ wc -l repolist-2021-09-03 8769 repolist-2021-09-03

That is a lot more packages than CentOS ships with as there are just over 2600 src.rpm packages in vault.centos.org for AppStream, BaseOS, and PowerTools. What is going on here?

It turns out there are two different events happening:

  1. Buildroot packages.
  2. SIG packages

Buildroot packages are the ones which are not shipped with EL8 but are needed to build EL8. [EL-8 was not meant to be build complete but just the packages that would be supportable by Red Hat.]. SIG packages are various ones which CentOS SIGs are supporting for projects like virtualization, hyperscale, and automotive. I may need to trace through all of these for my own package set, so I decided to clone them all first and then try to figure out what is needed afterwords.


#!/bin/bash
# A silly script to clone all the repositories from git.centos.org in
# order to work out things like buildorder and other tasks.
# Set up our local repo place. I have lots of space on /srv

CLONE_DIR=/srv/gitrepos/centos/
CLONE_LIST=repolist-2021-09-03

# loop over the namespaces we want to clone. It would have been nicer
# if there was a third namespace for sigs, but I don't think
# namespaces really happened when centos-7 was setting up git.centos.org.
for namespace in rpms modules; do
    mkdir -vp ${CLONE_DIR}/${namespace}
    cd ${CLONE_DIR}/${namespace}
    for repo in $( grep "/${namespace}/" ${CLONE_DIR}/${CLONE_LIST} ); do
        repodir=$( basename ${repo} )
        git clone ${repo}
        if [[ -d ${repodir} ]]; then
            pushd ${repodir} &> /dev/null
            X=`git branch --show-current`
            if [[ ${X} =~ 'c8s' ]]; then
                echo "${i} ${X}"
            else
                git branch -a | grep c8s &> /dev/null
                if [[ $? -eq 0 ]]; then
                    echo "${repodir} ${X} xxx"
                fi
            fi
            popd &> /dev/null
        fi
        sleep 1
    done
done

Running this takes a couple of hours, with a lot of errors about empty directories (for git repos which seem to have been created but never 'filled') and for non-existant branches (my script just looks for a c8s but some branches were called slightly differently than that.) In either case, I end up having the git repos I was trying to remember how I got earlier.

2021-06-21

Hello Rocky Linux 8.4 (and belated Hello to Alma Linux too)

So according to LWN.net, Rocky Linux 8.4 reached General Available (GA). This is great as it means that there are two* 'community rebuild' to move CentOS 8 systems to if CentOS Stream is not a good match.

Alma is built by the same people who have built Cloud Linux for years. From their downloads they focusing on x86_64 which is the most common in the cloud. 

Rocky Linux was founded by a person who worked on the pre-CentOS project of ChaOS. The wikipedia article on Rocky covers in more detail than me regurgitating it. My main reason for looking at it, is that it has a ARM port which will eventually run nicely on a raspberry pi for people interested in it.

My other reason was to remind people that if they are using CentOS Linux, that the end of life for 8 is December 31, 2021. Before that time it would be good to look at switching to Alma, CentOS Stream 8, Oracle Linux, Red Hat Enterprise Linux, or Rocky.  These transitions take time, and there are only ~190 days before it needs to be done.

Caveats

[*] There is a third rebuild which has been around for years called Springfield Linux made at Princeton University. However either due to the pandemic or other items, it does not seem to have had regular updates.

2021-06-16

Working with Raspberry PI4 systems

While my current work is aimed at ARM-64 hardware, many of the boards are not Server Ready Hardware and thus do not have things like UEFI to boot, ACPI to query the hardware stack, or various other things which are added later as firmware updates. They also end up having ‘developer kit boards’ of US$6000.00+ systems which having one at home is hard to justify. {Sorry kid, no college this semester… Dad bought himself a board that the customer may dump next week.}

In looking for proxy systems, my team has been focusing first on the classic ARM for small projects: The Raspberry Pi. The raspberry pi4 with 4 GB of ram works out as a proxy for all kinds of ‘low-end’ systems where you may need to play with a small GPU and try to make it work with a Server Ready operating system like CentOS Stream.

Getting the hardware

 

There are several places to get the hardware, and while I used to get things from Adafruit, but they did not have an IOT kit set up for the 4 series. I ended up going with CanaKit from Amazon just so I could get a bundle of parts together. Going from past experience with MMC cards burning out after a couple of days, I bought 3 32 gig cards to put the OS on. So far the cards have lasted longer than I expected but that just means they will burn out any day now.

When getting a raspberry pi, I highly recommend making sure you get the correct power supply, a USB 2 serial connector for the GPIO, and if you are using an external drive, a seperately powered disk drive. I have found that while the Raspberry Pi4 uses a larger power supply than the 1,2, or 3 series… attaching a non-powered USB drive can be problematic on boot (the ssd one I had would cause me to just have the rainbow picture of doom).

Setting up the hardware

 

For the raspberry pi4 if you are using it to compile or build things, make sure you have correctly sized heat dispensors for the CPU and possibly a fan (and maybe a replacement fan for when the first one dies). Then attach a serial cable to pins 6 (ground),8 (txd),10 (txd). Make sure you do not attach anything to 1,2,3 as you could be looking for a new pi or computer. The serial is useful for when you attempt to boot a new kernel config and the HDMI connector was not functional afterwords.

On another computer attach the USB connector and you can use the screen or minicom commands to see output from the system on boot. On my test system, I was able to capture the following:

 
$ screen -c /dev/null -L /dev/ttyUSB0 115200
recover4.elf not found (6)
recovery.elf not found (6)
Read start4x.elf bytes  2983784 hnd 0x000013b1 hash '3f7b34a64191a848'
Read fixup4x.dat bytes     8453 hnd 0x00000d3b hash '59e66162bed1b815'
0x00c03112 0x00000000 0x000003ff
MEM GPU: 32 ARM: 992 TOTAL: 1024
Starting start4x.elf @ 0xfec00200 partition 0

MESS:00:00:05.434998:0: arasan: arasan_emmc_open

Initial setup

 

Like any hardware setup, it is important to make sure the shipped hardware has up to date firmware and configs. For this I took one of the MMC cards, and burned the Raspian OS with recommended software Once this booted up, the tools did a firmware upgrade on the system from whatever had been on the box when it was stored in a depot. This OS is a 32 bit operating system but is maintained by the ‘manufacturer’ so is a good test for ‘did it break in shipment’

{Pro-tip: Once you have finished updating, shutdown the system, take this card out, and put it in a box/bag for later use. At some point things are going to go badly in an install and you won’t know if its you, your hardware, or something else. Having a known bootable backup that is supposed to work is good.}

After this it is time to focus on getting ‘bootable’ systems using the base OS’s we want to target:

  1. Fedora Linux
  2. CentOS Stream

Fedora 34 Initial Install

 

Fedora 34 came out as I started working on the project, so I decided to aim for that as an initial OS. The ARM work that the Fedora team is doing is aimed primarily at 64-bit and with Server Ready hardware. As such, they do try to make a raspberry pi4 work, but it is listed as possibly problematic. That said, the following worked mostly fine:

  1. Download the raw workstation image
  2. On an existing Fedora 33 system, install arm-image-installer via sudo dnf install arm-image-installer
  3. Insert an mmc into the computer using the appropriate adaptor, and find the disk name.
  4. GNOME will probably be overly helpful and have mounted partitions on the card for you. You will need to unmount them. df -a | grep mmc ; sudo umount /dev/mmcblk0p1; sudo umount /dev/...
  5. write the image to the mmc disk with image-installer:
fedora-arm-image-installer --image=./Fedora-Server-34-1.2.aarch64.raw.xz --media=/dev/mmcblk0 --addkey=a_key.pub --resizefs --showboot --target=rpi4 --addconsole=ttyAMA0,115200
  1. Move the mmc card over to the powered off raspberry pi4, and prepare to boot up the hardware.
  2. On my Fedora system I started a screen to watch the fireworks: screen /dev/ttyUSB0 115200
  3. Power on the raspberry pi and watch what happens. If you are lucky then you will get the system eventually booting into Fedora 34 graphical mode. If you aren’t, then it may stay in rainbow mode (like when I found that my SSD drive pulled too much power on boot.)
  4. Log into the system and play around a bit. This is a good time to do any updates and such. Getting an idea of how ‘fast’/‘slow’ the system with defaults is good here too.

Get serial working

 

At this point I wanted to make sure I could log into Fedora from the serial port. In order to do this, you need to edit the grub configs which is done in /etc/default/grub and then rebuilding the config. I moved the grub timeout to 10 seconds to give me a chance to choose different options, removed the rhgb and quiet, and added a console line.

 
$ sudo -i
# vi /etc/default/grub
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyAMA0,115200 "
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true
# grub2-mkconfig -o /etc/grub2-efi.cfg

A test reboot is good here to make sure that when you boot you get past the

EFI stub: Booting Linux Kernel...
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...

Fedora 34 with TianoCore

 

As stated before the Raspberry Pi systems are not System Ready and do not have a built in EFI or ACPI system for a kernel to boot from. Instead the default kernel boots usually with uboot mapping devices via device tree in order for hardware to be discovered. I am going to say that is about all I really know about the subject. There have been long threads in the Fedora ARM lists over the years on going over this versus ACPI, and I think it is better for an expert like Jon Masters or Peter Robinson to explain why APCI is preferred in Fedora Linux and Red Hat Enterprise Linux (RHEL) versus a novice like myself.

For the raspberry pi4, the current method to implement a UEFI interface is a port of the Tianocore by the Pi Firmware Task Force. TianoCore is an opensource implementation and extension of the Intel Extendable Firmware Interface which was written to replace the 16-bit BIOS used in personal computers since the 1980’s. A further extension was with the Open Virtual Machine Firmware which I believe was then used by the Pi Firmware Task Force for their version.

Easier than I thought

 

In reading various blogs and documentation on the status of the EFI support, I was prepared for a system that would only work via serial console or may not have networking or other utilities. Instead, I found the process to be fairly ‘painless’ and I only ended up with a non-booting system twice. The general steps were the following:

ssmoogen@fedora ~]$ mkdir RPi4
[ssmoogen@fedora ~]$ cd RPi4/
[ssmoogen@fedora RPi4]$ wget https://github.com/pftf/RPi4/releases/download/v1.27/RPi4_UEFI_Firmware_v1.27.zip
--2021-06-16 17:36:31--  https://github.com/pftf/RPi4/releases/download/v1.27/RPi4_UEFI_Firmware_v1.27.zip
Resolving github.com (github.com)... 140.82.114.3
Connecting to github.com (github.com)|140.82.114.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
...
RPi4_UEFI_Firmware_v1.27.zip                    100%[=====================================================================================================>]   2.92M  14.4MB/s    in 0.2s    

2021-06-16 17:36:32 (14.4 MB/s) - ‘RPi4_UEFI_Firmware_v1.27.zip’ saved [3064085/3064085]


$ wget https://github.com/pftf/RPi4/releases/download/v1.27/RPi4_UEFI_Firmware_v1.27.zip
$ unzip RPi4_UEFI_Firmware_v1.27.zip 
Archive:  RPi4_UEFI_Firmware_v1.27.zip
  inflating: RPI_EFI.fd              
  inflating: bcm2711-rpi-4-b.dtb     
  inflating: bcm2711-rpi-400.dtb     
  inflating: bcm2711-rpi-cm4.dtb     
  inflating: config.txt              
  inflating: fixup4.dat              
  inflating: start4.elf              
   creating: overlays/
  inflating: overlays/miniuart-bt.dtbo  
  inflating: Readme.md               
   creating: firmware/
  inflating: firmware/Readme.txt     
   creating: firmware/brcm/
  inflating: firmware/brcm/brcmfmac43455-sdio.clm_blob  
  inflating: firmware/brcm/brcmfmac43455-sdio.bin  
  inflating: firmware/brcm/brcmfmac43455-sdio.Raspberry  
  inflating: firmware/brcm/brcmfmac43455-sdio.txt  
  inflating: firmware/LICENCE.txt    

At this point, if you haven’t already, read the documentation. Our main tasks will be to setup the raspberry pi EFI boot partition to have the needed data in it. I had seen that this came with dtb files which I figured needed to be replaced. However as seen below, all of these are in a Fedora 34 and are of a similar timeframe. The only file we really need to work with is the RPI_EFI.fd and config.txt files.

 
[ssmoogen@fedora RPi4]$ sudo -i
[sudo] password for ssmoogen: 
[root@fedora ~]# cd /boot/efi/
[root@fedora efi]# cp config.txt config-orig.txt # always make a backup!
[root@fedora efi]# rpm -qf /boot/efi/bcm2711-rpi-4-b.dtb 
bcm2711-firmware-20210430-1.1a46874.fc34.aarch64
[root@fedora efi]# rpm -qf /boot/efi/overlays/miniuart-bt.dtbo 
bcm283x-overlays-20210430-1.1a46874.fc34.aarch64
[root@fedora efi]# cp ~ssmoogen/RPi4/RPI_EFI.fd /boot/efi/

At this point, it is time to replace the config.txt that came with the system with one which can be used for booting the UEFI program. This is where I caused my system to go into rainbow mode a couple of times. In the end, I put in the following file:

 

#boot in 64-bit mode
arm_64bit=1
# boot into the RPI_EFI firmware
armstub=RPI_EFI.fd
# turn on serial and keep it on in 2ndstage
enable_uart=1
uart_2ndstage=1
bootcode_delay=1
# using this from upstream UEFI config.txt
device_tree_address=0x1f0000
device_tree_end=0x200000
# set up the miniuart and the vc4
dtoverlay=miniuart-bt,vc4-fkms-v3d
disable_commandline_tags=1
disable_overscan=1
enable_gic=1
# set up the GPU ram and HDMI
gpu_mem=128
hdmi_ignore_cec_init=1
max_framebuffers=2
start_x=1

A couple of the commands in it are from a side trip on getting accelerated graphics working in CentOS Stream on Pi. I decided to document it for later as this is getting long. At this point I was able to get a booting system:

 
recover4.elf not found (6)
recovery.elf not found (6)
Read start4x.elf bytes  2983816 hnd 0x000032fe hash '210478ae179a91d0'
Read fixup4x.dat bytes     8451 hnd 0x00002c88 hash '7716af32619f0771'
0x00c03112 0x00000000 0x000003ff
MEM GPU: 256 ARM: 768 TOTAL: 1024
Starting start4x.elf @ 0xfec00200 partition 0

MESS:00:00:05.550101:0: arasan: arasan_emmc_open
MESS:00:00:05.698627:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:05.701548:0: brfs: File read: 342 bytes
MESS:00:00:05.821820:0: HDMI1:EDID error reading EDID block 0 attempt 0
MESS:00:00:05.831336:0: HDMI1:EDID error reading EDID block 0 attempt 1
MESS:00:00:05.840843:0: HDMI1:EDID error reading EDID block 0 attempt 2
MESS:00:00:05.850357:0: HDMI1:EDID error reading EDID block 0 attempt 3
MESS:00:00:05.859867:0: HDMI1:EDID error reading EDID block 0 attempt 4
MESS:00:00:05.869382:0: HDMI1:EDID error reading EDID block 0 attempt 5
MESS:00:00:05.878890:0: HDMI1:EDID error reading EDID block 0 attempt 6
MESS:00:00:05.888404:0: HDMI1:EDID error reading EDID block 0 attempt 7
MESS:00:00:05.897914:0: HDMI1:EDID error reading EDID block 0 attempt 8
MESS:00:00:05.907428:0: HDMI1:EDID error reading EDID block 0 attempt 9
MESS:00:00:05.911926:0: HDMI1:EDID giving up on reading EDID block 0
MESS:00:00:05.918043:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:06.995664:0: gpioman: gpioman_get_pin_num: pin DISPLAY_DSI_PORT not defined
MESS:00:00:07.002961:0: *** Restart logging
MESS:00:00:07.004382:0: brfs: File read: 342 bytes
MESS:00:00:07.072608:0: hdmi: HDMI1:EDID error reading EDID block 0 attempt 0
...
MESS:00:00:07.226856:0: dtb_file 'bcm2711-rpi-4-b.dtb'
MESS:00:00:07.235551:0: dtb_file 'bcm2711-rpi-4-b.dtb'
MESS:00:00:07.248011:0: brfs: File read: /mfs/sd/bcm2711-rpi-4-b.dtb
MESS:00:00:07.251301:0: Loading 'bcm2711-rpi-4-b.dtb' to 0x1f0000 size 0xc042
MESS:00:00:07.289344:0: brfs: File read: 49218 bytes
MESS:00:00:07.465024:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:07.467674:0: brfs: File read: 342 bytes
MESS:00:00:07.488759:0: brfs: File read: /mfs/sd/overlays/vc4-fkms-v3d.dtbo
MESS:00:00:07.535268:0: Loaded overlay 'vc4-fkms-v3d'
MESS:00:00:07.537248:0: dtparam: cma-256=true
MESS:00:00:07.541611:0: dtparam: miniuart-bt=true
MESS:00:00:07.552794:0: Unknown dtparam 'miniuart-bt' - ignored
MESS:00:00:07.654662:0: brfs: File read: 1446 bytes
MESS:00:00:07.658863:0: Failed to open command line file 'cmdline.txt'
MESS:00:00:08.953730:0: brfs: File read: /mfs/sd/RPI_EFI.fd
MESS:00:00:08.956193:0: Loading 'RPI_EFI.fd' to 0x0 size 0x1f0000
MESS:00:00:08.962019:0: No compatible kernel found
MESS:00:00:08.966520:0: Device tree loaded to 0x1f0000 (size 0xc622)
MESS:00:00:08.974158:0: uart: Set PL011 baud rate to 103448.300000 Hz
MESS:00:00:08.981676:0: uart: Baud rate change done...
MESS:00:00:08.983696:0:NOTICE:  BL31: v2.3():v2.3
NOTICE:  BL31: Built : 10:40:51, Apr 21 2020
UEFI firmware (version UEFI Firmware v1.27 built at 11:17:17 on May 25 2021)
3hESC (setup), F1 (shell), ENTER (boot)

On the HDMI monitor, you get a nice raspberry pi and a timeout to choose if you want to go into setup, shell or boot. If you hit ESC, you will get what looks like a fairly standard BIOS/EFI screen asking if you want to change various settings. At this point it is a good idea to make a change to allow for full memory usage (or you will be limited to 3GB and a slower system). Following the upstream README, Device ManagerRaspberry Pi ConfigurationAdvanced Configuration. At this point you select ‘Limit RAM to 3GB’ and disable it. Save the settings and escape up to the top menu. Choose the boot manager and you should be given the choices of the following operating systems:

 
Fedora
UEFI Shell
SD/MMC on Arasan SDHCI
UEFI PXEv4 (MAC:??)
UEFI PXEv6 (MAC:??)
UEFI HTTPv4 (MAC:??)
UEFI HTTPv6 (MAC:??)

 

Choose Fedora and after a short pause, you will get the grub config file. You shouldn’t need to change any defaults, but it is good in case you did. Once the kernel has been selected, the system will begin booting and the scary black screen occurs. This is a point where for some seconds nothing seems to be happening and a couple of times I was ready to power off and go back to other configs. Then you will see something similar to start scrolling:

 
[    0.000000] Linux version 5.12.10-300.fc34.aarch64 (mockbuild@buildvm-a64-03.iad2.fedoraproject.org) (gcc (GCC) 11.1.1 20210531 (Red Hat 11.1.1-3), GNU ld version 2.35.1-41.fc34) #1 SMP T
hu Jun 10 13:49:00 UTC 2021
[    0.000000] efi: EFI v2.70 by https://github.com/pftf/RPi4
[    0.000000] efi: ACPI 2.0=0x30720018 SMBIOS=0x33e00000 SMBIOS 3.0=0x33de0000 MEMATTR=0x321c2418 RNG=0x33fdb798 MEMRESERVE=0x30375118 
[    0.000000] efi: seeding entropy pool
[    0.000000] ACPI: Early table checksum verification disabled
[    0.000000] ACPI: RSDP 0x0000000030720018 000024 (v02 RPIFDN)
...

 

My belief is that the pause is the kernel and initial ramdisk are getting gunzipped in memory for usage. Reading from the MMC is slow, and uncompressing the files is slow. A future project may be to see if there is a sizable speedup of just doing this on the filesystem beforehand. In any case, the system will boot and be usable as a workstation.

End of File

This post has gotten on in size, and there were several other side tasks I worked on while doing it. Those will need to be seperate posts in the near future.

2021-04-16

Leaving Fedora Infrastructure

 In June 2009, I was given the opportunity to work in Fedora Infrastructure as Mike McGrath's assistant so that he could take some vacation. At the time I was living in New Mexico and had worked at the University of New Mexico for several years. I started working remote for the first time in my life, and had to learn all the nuances of IRC meetings and typing clearly and quickly. With the assistance of Seth Vidal, Luke Macken, Ricky Zhou, and many others I got quickly into 'the swing of things' with only 2 or 3 times taking all of Fedora offline because of a missed ; in a dns config file. 


For the last 4300+ days, I have worked with multiple great and wonderful system administrators and programmers to keep the Fedora Infrastructure running and growing so that the number of systems using 'deliverables' has grow into the millions of systems. I am highly indebted to everyone from volunteers to paid Red Hatters who has helped me grow. I want to especially thank Kevin Fenzi, Rick Elrod, and Pierre Yves-Chibon for the insights I have needed. 


Over the years, we have maintained a constantly spinning set of plates which allow for packagers to commit changes, build software, produce deliverables, and start all over again. We have moved our build systems physically at least 3 times, once across the North American continent. We have dealt with security breaches, mass password changes, and the undead project of replacing the 'Fedora Account System' which had been going on since before I started. [To the team which finished that monumental task in the last 3 months, we are all highly indebted. There may be pain-points but they did a herculean task.]


All in all, it has been a very good decade of working on a project that many have said would be 'gone' by next release. However, it is time for me to move onto other projects, and find new challenges that excite me. Starting next week, I will be moving to a group working with a strong focus on embedded hardware. I have been interested in embedded in some form or another since the 1970's. My first computer memories were of systems my dad showed me which would have been in an A-6 plane. From there I remember my dad taking me to see a friend who repaired PDP systems for textile mills and let me work on my first Unix running on a Dec Rainbow. Whenever I came home from those visits, I would have a smile and hum of excitement which would not leave me for days. I remember having that humm when in 1992, a student teacher showed me MCC Linux running on an i386 which we had been repairing from spare parts. I could do everything and anything on that box for a fraction of the price of the big Unix boxes I had to pay account time for. And recently I went to a set of talks on embedded projects and found myself with the same hum. It was a surprise for me but I found myself more and more interested in it as the weeks have gone by.


I was offered a chance to move over, and I decided to take it. I will still be in the Fedora community but will not be able to work much on Infrastructure issues. If I have tasks that you are waiting for, please let me know, and I will finish them either by myself or by doing a full handoff to someone else in Infrastructure. Thank you all for your help and patience over these last 11+ years. 

2020-12-18

Reading: Peopleware: Productive Projects and Team by Tom DeMarco and Timothy Lister

So last month I finished a re-read of "The Mythical Man-Month" by Fred Brooks.. and saw in the back of the book a reference to "Peopleware:..." by Tom DeMarco and Timothy Lister. I had heard of the book in the late 1990's (right before I joined Red Hat for the first time) from a fellow coder who was moving  into management and had wanted to do it right. Marshall had read the book, tried to put the ideas into place and was promptly let go by the company. [Within about 6 months, we were all let go also... for a reason given in the book..] In the years since I have seen the book brought  up several times, but I had never actually read it.

First off, the book is very ranty. Very very ranty in parts about open floor plans (and the Furniture police), open organizations, and a couple of other areas of middle management. It is not that I found their comments off the mark.. but you may have to soldier onto the next chapter through a couple of them. That said there are a lot of gems in the book which 'spoke to me'.

Parkinson's Law Doesn't Exist

Work expands to fill the time allocated to it. - C.  NorthCote Parkinson

I have had several managers quote this to teams I have been on in the last 30 years. People would bring up Parkinson's law about why we shouldn't add more time to a schedule but should cut it back. I, myself took it for certain that this was some sociological experiment which proved things about scheduling. I never realized until reading this book that a Mr. Parkinson was not a scientist, but a satirist and the things people have 'quoted' as being proven by it originally were all made up for a humorous article and books he wrote. 

In the extent that there may be places where people come up with busy work to meet a schedule... it usually is where people are not finding satisfaction in their current job. [chapter 5]

Open Floor Plans Don't Help Programmers Flow

This book was originally written in the 1980's when open floor plans were having 6 foot high cubicles versus offices with doors. Back then there was a lot of recorded data about how various 'knowledge' workers can't get into a flow state of thought because there are too many distractions. Today's open floor plans of no cubicle walls and brutalist concrete structures in order to get some sort of energy efficiency put those to shame. [This is where the rants on furniture police start and go on for quite some time.. ] The useful information I took from this is that one should measure their uninterrupted time and not their hours. Doing this can show where productivity is actually going and work out better planning on how efficient a project can meet its goals.

Hiring The Right People to Teamicide

This section of the book has a lot of useful parts, but one that stood out to me was the chapter "Happy to Be Here" with an excerpt from "Up the Organization" by Robert Townsend. It basically explains a method for dealing with an underperforming organization by moving it to a new location. The purported guaranteed results:
  1. the best people will move to the new location.
  2. the rest don't have to face the fact they have been fired. Instead they can say "The company moved town."
  3. your competition will move in and hire these less desirables thinking they are getting a bargain.
  4. the new people at your new site are infused with enthusiasm because all they know are your 'best' people. 
This was basically the strategy used by that company I worked for years before Red Hat. We were all told the company was moving town and we could reapply for our jobs at a different location. The problem they ran into is the one mentioned in the book.. most of the best people didn't go (they ended up being hired as contractors and paid 2.5x what they had been paid before so that deadlines could be met), and a lot of the people who did weren't the ones they wanted. Those of the best  people who did go left shortly after their moves. The psychological loyalty they had with the company had been broken so any problem at the new place was enough to send them packing. 

The next sections cover more about how to make a team of people "jell" which ends up being harder to describe than the ways to make a team break down. 
  1.  Defensive Management
  2. Bureaucracy
  3. Physical Seperation
  4. Fragmentation of people's time
  5. Quality of reduction of the product
  6. Phony deadlines
  7. Clique Control
  8. "Motivational" posters and awards
  9. Overtime causing undertime
  10. Internal team competition
  11. Annual salary/merit reviews
  12. Management by Objectives
I can see where some of the above are causes of team breakdowns but others seem to be a lot smaller concerns comparatively.

Making Change Possible

The chapter opens with a quote by Steve McMenamin, of the Atlantic Systems Guild (the organization the authors are members of).

People hate change... and that's because people hate change... I want you to be sure that you get my point. People really hate change. They really, really do. 

People don't react to change logically and very little 'logical argument' will help them. People react to change emotionally and the most likely result is going to be failure and ridicule from ones peers and sometimes by the people who instituted the change. The book also brings up the idea of Jerry Johnson, Menninger Foundation, that there is a continuum of ways people react to the change:

  1. Blindingly Loyal (Ask no questions.)
  2. Questioners
    1. skeptics (show me)
    2. passive (what's in it for me?)
    3. opposed due to fear of change
    4. opposed due to fear of loss of knowledge/power
  3. Militantly Opposed (Will Undermine and Destroy).
While some people think that the second 2 sets are 'non-supporters', the Blindingly loyal are also the ones to most likely jump ship at the next newest thing. The only people who will stick it through are the various questioners if they are helped through the emotional journey through the initial Chaos and failures before the change can be truly evaluated. [The rest of the chapter goes into parts of that while referencing books like "The Prince" by Machiavelli and Virginia Satir's model of therapy change. I actually wish this chapter had been a bit more in depth but I think I will reread Managing Transitions (which was referenced in different parts of the book) instead.

 


2020-12-14

CentOS-8 End of Life 2021-12-31

What is Happening

Red Hat announced changes to CentOS Linux release structure last week, and I like everyone else around is working through what that means. I have worked with and on CentOS since 2005, and saw a lot of my work in EPEL as helping people in that community able to get things done. This has been a real kick to the guts for a lot of people, and I do not have the words to express myself on this. That said, as a System Administrator, you have to take the Tango Charlie Foxtrot's as they are handed to you, plan what to do next, and execute as efficiently as possible.

Current End of Life Dates for CentOS Releases

The following seem to be the current dates to plan against:

CentOS Linux ReleaseCurrent End of Life
CentOS 62020-11-30
CentOS 72024-06-30
CentOS 82021-12-31
CentOS Stream ReleaseCurrent End of Life
CentOS Stream 82024-06-??

What Do I Need to do Next

Deal with the Grief

At this point, System Administrators need to start making a lot of assessments of what they are using CentOS for, why they are using it, and what are the alternatives to move to. This is hard to do when you are angry, pissed, depressed, betrayed, enraged, apoplectic, or the other dozen emotions that have been going through my head lately. So the first thing one has to do is try to work on those emotions as best you can. Talk them out with someone who can take one more rant from you, write them in a blog you never publish, and go for long walks away from computers for several hours. [There are a lot of other methods too and each person has their own way to deal with this.. but the point is that until you get the emotions out, you can not operate.]

Do an analytical assessment of why you are currently using CentOS Linux

There are a plethora of reasons why people use CentOS in their environments.. and those need to be examined to figure out what you need to do next. Sadly like most environments, I think most System Administrators rarely have time to document these and so when having to redeploy you end up with what seems like an impossible project. Like all Charlie Tango Foxtrot situations, one needs to eat the elephant one bite at a time. I find that the following can help make this easier. For each set of servers/services you answer:

  1. Who are the consumers of this service or server? 
  2. What do they need from their services?
  3. What is the amount of change that they can deal with on this service?
  4. How many systems do I have in this service? How can I break them into smaller chunks?
  5. Where are my systems? [How are
  6. What automation can I use to roll out these changes?
  7. When are upcoming deadlines that my consumers need to deal with
  8. When are upcoming deadlines I have to deal with?
  9. What are my alternatives? (This is a larger question needing a breakout sheet of why this alternative, how close is it to meeting my needs, what are the costs involved, when can I switch to it, where can i run it, what do I need to change from what I do now?, etc)
  10. How long would it take to work these alternatives?

If you have hundreds of servers, then this still looks like a giant load of dung to crawl through. I start with my central Infrastructure server and work my way out.

  1. Who are the consumers? [I am as use this server to run config scripts to outlying systems.]
  2. What do they need from their services? [Good uptime and stability so that when an outage happens elsewhere I have a good homebase to work from.]
  3. What is the amount of change they can deal with on this service? [If I use this as a learning experience, a lot. If I do not have time to learn.. then not a lot.]
  4. How many systems do I have in this service? 1
  5. Where are my systems? [Home base is under my desk in my office.]
  6. What automaton can I use to roll out these changes? Ansible looks like a good choice
  7. When are upcoming deadlines that my consumers need to deal with? [check with home calendar and mark off dates when I can actually futz with this.]
  8. When are upcoming deadlines that I have to deal with? 2020-12-31 
  9. What are my alternatives? [Only listing some alternatives.. rationale needs to be filled out]
  10. How long would it take to work these alternatives? ... unknown at this time.

This list is a starting point for getting the conversation going with some questions not needed and others needing to be added. Once you have worked out the questions then it is time to work out with each of the stakeholders of your services to answer the questions for themselves and then come to a conclusion. For deadlines to work this out, I would use the rule of thumb I found when dealing with change over of Fedora services with several hundred systems and also previous Tango Foxtrot Charlie changes:

StepSoft DeadlineHard deadline
Inventory current systems2021-01-312021-02-15
Evaluate alternatives2021-02-282021-03-15
Write automation rules for minimal services2021-03-312021-04-15
Roll out staged minimal new infra using different choices if possible2021-04-302021-05-15
Evaluate rollout and go with best choice2021-05-152020-05-31
Determine convergence planning for additional services2021-05-152020-05-31
Begin Rollout of New Infra2021-06-302021-07-15
A series of miracles occur sorry I mean lots of planned deliverables are met.


Last systems are moved over.2021-12-312020-12-31
Last updates for CentOS-82020-12-312020-12-31

Note: None of the above is to say 'golly gee, its no problem, you just need to throw away all your other work plans for 2021.' or any similar flipant response. It is only meant as a possible way to deal with this in a sane way. Some ways of converting/change are easier than others, but they all require some work and knowing what the tradeoffs are for each one.

2020-10-29

RHEL-6/CentOS-6/SciLin-6/EPEL-6 End Of Life Notice 2020-11-30

CentOS-6 End of Life Notice 2020-11-30

This is a short reminder that Red Hat Enterprise Linux (RHEL) version 6 will enter 'Extended Lifetime Support' in about 30 days from when I am writing this. Extended Lifetime Support (ELS) is a specific contract with Red Hat for them to cover certain security fixes for some extended time to allow sites some time for last minute transitions. 

RHEL-6 was released in November of 2010, and was the first RHEL I got to work with/on after I returned to Red Hat in 2009. The release has seen 10 minor releases (1 less than RHEL-5), and has been in 'extended' mode since the last 6.10 release in June 2018. 

What does this mean to me?

When RHEL-6 enters its version of ELS, then it is considered 'end of life' by its 'downstream' rebuilders CentOS and Scientific Linux (and by extension EPEL). I am not sure what Scientific Linux plans are, but for CentOS, they will follow a similar plan as they did with EL-4/EL-5 distributions end. First they will copy all the code from the master servers to the vault servers. Then they will turn off the main mirrorlist systems any references to EL-6. This will cause the 'yum' command to fail and usually causes all kinds of yelling and screaming as people realize they needed to have moved their servers to something else before December 1, 2020. 

For EPEL users it will be slightly different. All EPEL builds for EL-6 will be 'stopped' on December 1st.  [Packages will no longer be allowed to branch to EL-6, builders will not be able to build code for EL-6. Packagers will not be able to move software from testing to prod.] At that point all RPMS on the download servers will be hardlinked to /pub/archive on the master servers. After a week, the mirrorlists will point to the archive zone, and Fedora Infrastructure will remove the code from /pub/epel/6/ trees. 

What can I do to deal with this?

Primarily, if you are going to be affected by the end of EL-6 services, you either need to get an ELS contract, move to another OS, or move to self-support. In order to self-support, you will need to mirror the source code from your distribution provider and learn the basics of RPM building. If you are on CentOS and find your servers not able to do yum installs anymore.. you will need to mirror the EL-6 from the CentOS vault somewhere locally and use that as your new 'mirror'. Depending on time and energy, I will try to outline some of these steps in future blog posts.