2022-04-01

Compiling openldap for CentOS 8 Stream

Compiling OpenLDAP for EL8 systems

Steps to compile openldap-server for CentOS 8 Stream

The EL8 release did not ship an openldap-server like it did in previous releases. Instead only the client tools and some libraries are included for existing applications. Instead the focus from the upstream provider has been on other LDAP solutions.

This leaves a problem for various sites who have their data in an OpenLDAP system and do not have the time, energy, resources for moving to something else. There are several possible solutions to this:

  1. Continue to use EL5/EL6 even though it is at end of open maintenance.
  2. Continue to use EL7 until it is end of open maintenance around 2024-06-30.
  3. Move to a different distribution which does have working openldap
  4. Compile replacement tools using the Fedora src.rpm which may be closer to the ‘upstream’.
  5. Compile replacement tools using the upstream source.
  6. Compile using the upstream source from https://git.centos.org
  7. [Added after initial post] You can download them from https://koji.mbox.centos.org/koji/

In this tutorial we will work with number 5. At the end we will cover number 6.

Setting up a build environment.

For simplicity sake, we will assume you have a working but minimally installed Fedora 35 or EL8 system (Alma, Oracle, Rocky, etc) which you can do compiles in. If we are using an EL8 system are going to need to get mock and git installed.

$ sudo dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm

For Fedora and EL8 systems the following should work the same:

$ sudo dnf install git mock rpm-build
$ sudo usermod -a -G mock $USERNAME
$ newgrp mock

Answer yes to the questions about adding new keys and the packages should be installed to allow for a build to occur. We now need to set up a minimal .rpmmacros file for the next steps:

# uncomment if you want to build in standard homedirectory
#%_topdir %(echo $HOME)/rpmbuild
# comment if want to use standard home directory
%_topdir        %{getenv:PWD}
%_sourcedir     %{_topdir}/SOURCES
#%_sourcedir     %{_topdir}/SOURCES/%{name}-%{version}
%_specdir       %{_topdir}/SPECS
%_srcrpmdir     %{_topdir}/SRPMS
%_builddir      %{_topdir}/BUILD

%__arch_install_post \
    [ "%{buildarch}" = "noarch" ] || QA_CHECK_RPATHS=1 ; \
    case "${QA_CHECK_RPATHS:-}" in [1yY]*) /usr/lib/rpm/check-rpaths ;; esac \
    /usr/lib/rpm/check-buildroot

Once we have that in place, the following will get an openldap build going:

$ mkdir -vp ~/EL8-sources/ ~/output-packages/
$ cd ~/EL8-sources/
$ git clone https://git.centos.org/rpms/openldap.git
$ git clone https://git.centos.org/centos-git-common.git
$ cd openldap
$ ../centos-git-common/get_sources.sh
$ rpmbuild -bs SPECS/openldap.spec

Now depending on the host OS you are doing this on, you should see a file like SRPMS/openldap-2.4.46-18.fc35.src.rpm or SRPMS/openldap-2.4.46-18.el8.src.rpm having been created.

$ mock -r centos-stream+epel-next-8-x86_64 --chain --localrepo \
~/output-packages/ SRPMS/openldap-2.4.46-18.fc35.src.rpm

should then attempt to build the packages and will end up with a fully usable repo in ${HOMEDIR}/output-packages/results/centos-stream+epel-next-8-x86_64

If not, then there are probably some steps or problems I missed in this howto :(. At this point you can determine what to do with installing this -server package on the server needing it.

Downloading direct from CentOS.

This is the ‘feed the fisherman versus teaching how to fish’ part of the document.

If you are using CentOS Stream 8, you can download the build packages from the project koji. I expect similar steps can be done for other rebuilds.

  1. dnf list openldap to get which package you are looking for.
  2. Open a window to https://koji.mbox.centos.org/koji/
  3. Type in openldap in the Search box.
  4. Click on the build you would have installed. For this example, we will choose https://koji.mbox.centos.org/koji/buildinfo?buildID=18688 and then scroll down to the architecture you are using.
  5. Right click on the download button for openldap-servers like:https://koji.mbox.centos.org/pkgs/packages/openldap/2.4.46/18.el8/x86_64/openldap-servers-2.4.46-18.el8.x86_64.rpm
  6. Install this package in the package place you want.
  7. When dnf breaks because it can’t upgrade the package due to the upstream updating, go follow step 0 again.

2022-02-28

Dealing with RAID arrays

Dear Future Self,

 We have come to another letter where we are going to better document something PastSelf thought it knew, but clearly didn't. In this case we are going to start recovering from a RAID array after a reinstall. For reasons we won't get into, PastSelf had to reinstall the home server for the 2nd time this week. [Let us just say that PastSelf is no longer allowed to use sudo without supervision and move on.] In the reinstall, we could not get the /dev/sdb and /dev/sdc RAID array to be fully recognized and realized that we had also made the original ones too small for what we needed [which is what started the whole problem when we tried to grow a partition but forgot that the external backup always becomes /dev/sda for some reason and /dev/sdb was not the RAID drive but the / drive. Live and learn, live and learn.]

Due to some bad signatures we needed to clear the drives of their current data. This was done by booting from a USB stick (which also becomes /dev/sda in this hardware.... wtf?) and clearing each drive of its signatures. 

# wipefs -a /dev/sdb
# wipefs -a /dev/sdc
# wipefs -a /dev/sdd
# cat /proc/mdstat 
Personalities : 
md127 : inactive sdc1[1](S)
      1464851456 blocks super 1.2
       
unused devices: 

  

The above failed because the kernel and boot had tried to make them part of a RAID array /dev/md127 but was not able to sync them. I was also unable to

mdadm --stop /dev/md127
for some reason. At this point, PastSelf further broke his oath of primum non nocere by using dd on each of the disks.
# dd if=/dev/zero of=/dev/sdb bs=1024 count=1000000
# dd if=/dev/zero of=/dev/sdc bs=1024 count=1000000
# dd if=/dev/zero of=/dev/sdd bs=1024 count=1000000
A reboot and going into rescue mode still showed that some signatures were there which I realized was due these disks being formatted with GPT and being much more capable of surviving stupidity. However mdadm --stop now worked so I could use gdisk on the drives. I then reinstalled a minimal Alma8.5 onto the box and then did a manual creation of the RAID array:
# gdisk /dev/sdc
GPT fdisk (gdisk) version 1.0.3

Partition table scan:
  MBR: not present
  BSD: not present
  APM: not present
  GPT: not present

Creating new GPT entries.

Command (? for help): n
Partition number (1-128, default 1): 1
First sector (34-3907029134, default = 2048) or {+-}size{KMGTP}: 
Last sector (2048-3907029134, default = 3907029134) or {+-}size{KMGTP}: 
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300): fd00
Changed type of partition to 'Linux RAID'

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y

  

At this point we were able to get the system ready for creating the RAID partition.

# mdadm --create --verbose /dev/md1 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1 --force
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
mdadm: /dev/sdc1 appears to be part of a raid array:
       level=raid1 devices=2 ctime=Thu Dec 30 18:54:28 2021
mdadm: size set to 1953381440K
mdadm: automatically enabling write-intent bitmap on large array
Continue creating array? y
mdadm: Fail to create md1 when using /sys/module/md_mod/parameters/new_array, fallback to creation via node
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md1 started.
[root@xenadu ~]# cat /proc/mdstat 
Personalities : [raid1] 
md1 : active raid1 sdc1[1] sdb1[0]
      1953381440 blocks super 1.2 [2/2] [UU]
      [>....................]  resync =  0.7% (15491456/1953381440) finish=158.1min speed=204199K/sec
      bitmap: 15/15 pages [60KB], 65536KB chunk

unused devices: <none>
# mdadm --detail --scan
ARRAY /dev/md1 metadata=1.2 name=xenadu.int.smoogespace.com:1 UUID=c032f979:e8e4deda:a590ca5d:820a8548
# mdadm --detail --scan > /etc/mdadm.conf
# echo '/dev/md0 /srv xfs defaults 0 0' >> /etc/fstab

Now wait for the sync to be done, and then start the restore from backups... you know the ones that Past-PastSelf made just in case of this situation. Also Future-Self, could you please write up some ansible playbooks to do this from now on? Future-FutureSelf will appreciate it.

Yours Truly, PastSelf

Getting past EL-{8,9}'s limitations with toolbx

Dear Future Self,

One of the biggest issues with dealing with Enterprise Linux 8 (be it Rocky to Red Hat) is the lack of additional packages which you know are in Fedora. Trying to get them into EL-8 turns into a Sisyphean task of moving the boulder of multiple python/go/ruby/etc packages into EL8 only to find that the RPM macros and other software have changed so much in 2 to 3 Fedora releases you can't. Past self spent the weekend trying to get a simple GO package backported and found that he needed to touch at least 175 src.rpms to make this 'work'. That was just too much for trying to get something else working.

Thankfully, EL8 ships with a tool which will allow to get past most of these problems if you meet the following criteria:

  1. The package must not require any kernel feature not shipped in the EL-8 kernel.
  2. You have lots of disk space available to basically install a second OS. 
  3. You can deal with some of the limitations of containers.

The tool which does all this is Container Toolbx which uses podman to create an interactive shell using the runtime space of the OS you want.

$ sudo -i dnf install toolbox
Password:
$ toolbox create --distro fedora --release f35 f35
$ cat /etc/system-release
AlmaLinux release 8.5 (Arctic Sphynx)
$ ls
Ansible-smoogespace/  HUGO/  OLD/  Packages/  RPMS/  SSH-AGENT  Website-smoogespace/  go/  yadm-dotfiles/
$ toolbox enter f35
$ ls
Ansible-smoogespace/  HUGO/  OLD/  Packages/  RPMS/  SSH-AGENT  Website-smoogespace/  go/  yadm-dotfiles/
$ cat /etc/system-release
Fedora release 35 (Thirty Five)
$ sudo -i dnf update
< no password asked >
$ sudo -i dnf install {package I want}
$ {package_command}
  
As can be seen by the example above, toolbx basically puts the container in the home directory in the user but using the userspace of Fedora 35. This allowed me to have some newer commands which allowed for a compiled go package which I couldn't do in EL-8 at the moment. Since go is static, I can then use this package regularly in my EL-8 environment. [I was also able to get past some similar errors in emacs where I had used some package calls from newer emacs which compile elc which works with EL-8 emacs.]

Important!

This is not a cure-all. You are basically downloading basic containers and then using overlays to do updates and other magic to make this work. While it is quite likely possible one could make various daemons (say openvpn) work this way, I also expect that the network hell that comes with containers would make it fragile. However when needing fedpkg or some similar command it is easier to use this than try and port all the other 'packages' that it relies on if you have only a couple of hours free.

Anyway, this is the 2nd time I have had to re-discover this in the last 2 years so I figured I had better write a note to future me in 6 months or a year who has to do this again.

Yours truly, Past Self

2022-02-08

How to Install CentOS Stream 9 Cloud Image

Dear Future Self,

You have probably started to install a CentOS Stream 9 cloud image, and completely forgot all the things you learned this time around. No worries, past-self is going to write these down for your usage. 

First off, download the image you want. On the day we are writing this, the latest image is http://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-GenericCloud-9-20220207.0.x86_64.qcow2 but it will most likely be something much newer. They don't put a 'latest' in the directory, so open a browser, search for qcow2, and then instead of searching through 4000 entries from 2021-08-30, press the up-arrow and jump to the last entry on the web-page.

Next, we need to use virt-install to get the image imported to where virtual manager will use it. Older versions of CentOS had a default user, but CentOS Stream 9 relies on cloud-init in order set up the root user and password. This is done via the virt-install command IF you have a virt-install after version 3, so need to look at different command for EL8 and Ubuntu 18.

$ sudo virt-install --name guest-cs9 --memory 2048 \
  --vcpus 2 --disk ./CentOS-Stream-GenericCloud-9-20220207.0.x86_64.qcow2 \
  --import --os-type Linux --os-variant centos-stream9 \
  --network default --console pty,target_type=serial --graphics vnc \
  --cloud-init root-password-generate=on,disable=on,ssh-key=/home/ssmoogen/.ssh/id_ecdsa.pub
  

You can add more cloud init options by creating data-files for meta and user-data. Go to the cloud-init site for that.

Alternative method (ok the one most likely used).

In the case of trying to do this on EL8 or earlier Ubuntu editions, you will need to use the virt-customize command instead. First we have to make sure it is installed.

For Ubuntu:
  $ sudo apt install libguestfs-tools
  
For EL based distros:
  $ sudo dnf install guestfs-tools
  

The virt-customize command is meant to alter a non-running image. If you use it on a running one, you will probably have a very dead box afterwards. YOU HAVE BEEN WARNED.

 
$ virt-customize -v --uninstall cloud-init --selinux-relabel \
  -a CentOS-Stream-GenericCloud-9-20220207.0.x86_64.qcow2 \
  --ssh-inject root:file:/home/ssmoogen/.ssh/id_ecdsa.pub \
  --root-password random |& tee CSG.out
$ grep 'virt-customize.password' CSG.out # for the random set password.
  
Depending on the OS this may or may not need to be run with sudo. Check the CSG.out file for any extra errors. We sent both standard out and standard error there to make sure it all got captured. After this is done, you can import as working image with:
$ sudo virt-install --name guest-cs9 --memory 2048 \
  --vcpus 2 --disk ./CentOS-Stream-GenericCloud-9-20220207.0.x86_64.qcow2 \
  --import --os-type Linux --os-variant centos-stream9 \
  --network default --console pty,target_type=serial --graphics vnc
  

And with that, past self is done recording what is needed to be done.

2021-09-25

Growth of Fedora Distribution over Time

Growth of the Fedora Distribution over time

There was a conversation in IRC (libera.chat, #fedora-admin) on the amount of disk space that Fedora is using over time. It used to grow astronomically over time, but there was an idea that it might be slowing down.. and then the realization that no one had graphed it. Taking this challenge in hand I decided to look at it. Doing a complete mirror of the data would require me to have a very long time frame and 100+ TB of disk space, but luckily for me, the Fedora mirror system does a du every night and outputs this data to a file, https://dl.fedoraproject.org/pub/DIRECTORY_SIZES.txt

The file covers all the directories that the main download servers have including the archive trees which are where old releases go to live. It also puts it in a ‘human-readable’ format like

egrep '/rawhide$|/releases/[0-9]*$|/updates/[0-9]*$|/updates/testing/[0-9]*$' DIRECTORY_SIZES.txt | egrep -v '^8.0K|^12K|^4.0K|/pub/epel|/pub/alt' > /tmp/dirs
$ grep '/7' /tmp/dirs 
71G /pub/archive/fedora/linux/releases/7
55G /pub/archive/fedora/linux/updates/7
1.5G    /pub/archive/fedora/linux/updates/testing/7

The above takes all the directories we want to worry about and avoid /pub/alt which is a wild west of directories and data. I also want to avoid /pub/epel so I don’t get a mix between EPEL-7 and Fedora Linux 7. It also allows me to save that entire long grep into a file so I don’t repeat if for every time I do the next data manipulation which is:

# Thanks to https://gist.github.com/fsteffenhagen/e09b827430956d7f1de35140111e14c4
grep '/7' /tmp/dirs | numfmt --from=iec | awk 'BEGIN{sum=0} {sum=sum+$1} END{num=split($0,a,"/"); print sum,a[num]}' | numfmt --to=iec
128G 7

This uses a command numfmt that I wish I had known years before as I have ‘replicated’ it repeatedly poorly in awk and python. The first one converts it to an integer, then feeds it to awk which adds it, and then sums all that and prints the output. The conversion is lossy but ok for a quick blog post.

$ cat foobaz.sh 
#/bin/bash

for i in $( seq 7 35 ); do
     grep "/${i}$" /tmp/dirs | numfmt --from=iec | awk 'BEGIN{sum=0} {sum=sum+$1} END{num=split($0,a,"/"); print sum,a[num]}' | numfmt --to=iec | awk '{print $2","$1}'
done
$ bash foobaz.sh 
7,128G
8,153G
9,286G
10,207G
11,266G
12,267G
13,202G
14,229G
15,371G
16,388G
17,594G
18,669G
19,600G
20,639G
21,730G
22,804G
23,865G
24,816G
25,821G
26,971G
27,1.1T
28,1.2T
29,1.1T
30,1.3T
31,1.1T
32,1.2T
33,1.3T
34,1.3T
35,200G

This first run found a problem because 35 should be greater than 200G. However only /pub/fedora/linux/updates/35 and /pub/fedora/linux/updates/testing/ are publically readable. Getting some data from root and we correct this to 35 having 917G. Plotting this in openoffice with some magic we get:

This is ok for a textual map but how about a graph picture. For this we remove the conversion to human readable data (aka M,G,T) and put the data into openoffice for some simple bar graphs. And so here is our data:


 

After this we can also look at how someone mirroring the distributions over time need more disk space:


The total growth looks to be move from exponential to linear over time. If you wanted to break out into smaller archives, you could put release 1 to 25 on one 10 TB drive, and 26 to 32 on another 10 TB drive as the releases after 26 are usually 1.4 TB in size at the end of their release cycle.
 

2021-09-03

How to clone (a lot of) package spec files from CentOS 8 git.centos.org

Recently I have had to try and work out all the dependencies on a set of packages. I am writing this as a blog, as I needed to recreate work that I had done several times in the past, but past Smoogen had not fully documented. [I went looking and found I had 3 different copies of trees of packages from Fedora and CentOS but absolutely no notes and my bash_history had cycled over the 10k lines I normally keep. Past Smoogen was a BAD, BAD sysadmin.]

For general requires, I would do this by using dnf or dnf


$ dnf repoquery --requires bash
Last metadata expiration check: 2:44:24 ago on Fri 03 Sep 2021 10:14:22 EDT.
filesystem >= 3
libc.so.6(GLIBC_2.15)(64bit)
libdl.so.2()(64bit)
libdl.so.2(GLIBC_2.2.5)(64bit)
libtinfo.so.6()(64bit)
rtld(GNU_HASH)

However in this case, I needed to also work out all the buildrequires of the packages, and then the requires and buildrequires of those tools. Basically it is sort of building a buildroot for a set of leaf packages which usually means I need to get the spec files and parse them with something like spectool.

If I was working with Fedora, I would take the shallow git clone of the src.fedoraproject.org website which can be found at https://src.fedoraproject.org/lookaside/. Then I would start going down my list of 'known' software I need to work and clone out the usual 'buildroot' a fedpkg mockbuild of the package would give. However I am working with CentOS Stream 8 which is a slightly different repository layout, and does not have a prebuilt shallow clone.

Thankfully, the CentOS repository has a very useful sub-repository called https://git.centos.org/centos-git-common.git which contains all the tools to fetch the appropriate code and tools from the CentOS src repository. The first one I need work with is centos.git.repolist.py to query the pagure API and get a list of packages. I then need to clean up that list a bit because it contains some forks but the following will get me a complete list of the packages I want to parse:

[centos-git-common (master)]$ ./centos.git.repolist.py | grep -v '/forks/' > repolist-2021-09-03 [centos-git-common (master)]$ ./centos.git.repolist.py --namespace modules | grep -v '/forks/' >> repolist-2021-09-03 [centos-git-common (master)]$ sort -o repolist-2021-09-03 -u repolist-2021-09-03 [centos-git-common (master)]$ wc -l repolist-2021-09-03 8769 repolist-2021-09-03

That is a lot more packages than CentOS ships with as there are just over 2600 src.rpm packages in vault.centos.org for AppStream, BaseOS, and PowerTools. What is going on here?

It turns out there are two different events happening:

  1. Buildroot packages.
  2. SIG packages

Buildroot packages are the ones which are not shipped with EL8 but are needed to build EL8. [EL-8 was not meant to be build complete but just the packages that would be supportable by Red Hat.]. SIG packages are various ones which CentOS SIGs are supporting for projects like virtualization, hyperscale, and automotive. I may need to trace through all of these for my own package set, so I decided to clone them all first and then try to figure out what is needed afterwords.


#!/bin/bash
# A silly script to clone all the repositories from git.centos.org in
# order to work out things like buildorder and other tasks.
# Set up our local repo place. I have lots of space on /srv

CLONE_DIR=/srv/gitrepos/centos/
CLONE_LIST=repolist-2021-09-03

# loop over the namespaces we want to clone. It would have been nicer
# if there was a third namespace for sigs, but I don't think
# namespaces really happened when centos-7 was setting up git.centos.org.
for namespace in rpms modules; do
    mkdir -vp ${CLONE_DIR}/${namespace}
    cd ${CLONE_DIR}/${namespace}
    for repo in $( grep "/${namespace}/" ${CLONE_DIR}/${CLONE_LIST} ); do
        repodir=$( basename ${repo} )
        git clone ${repo}
        if [[ -d ${repodir} ]]; then
            pushd ${repodir} &> /dev/null
            X=`git branch --show-current`
            if [[ ${X} =~ 'c8s' ]]; then
                echo "${i} ${X}"
            else
                git branch -a | grep c8s &> /dev/null
                if [[ $? -eq 0 ]]; then
                    echo "${repodir} ${X} xxx"
                fi
            fi
            popd &> /dev/null
        fi
        sleep 1
    done
done

Running this takes a couple of hours, with a lot of errors about empty directories (for git repos which seem to have been created but never 'filled') and for non-existant branches (my script just looks for a c8s but some branches were called slightly differently than that.) In either case, I end up having the git repos I was trying to remember how I got earlier.

2021-06-21

Hello Rocky Linux 8.4 (and belated Hello to Alma Linux too)

So according to LWN.net, Rocky Linux 8.4 reached General Available (GA). This is great as it means that there are two* 'community rebuild' to move CentOS 8 systems to if CentOS Stream is not a good match.

Alma is built by the same people who have built Cloud Linux for years. From their downloads they focusing on x86_64 which is the most common in the cloud. 

Rocky Linux was founded by a person who worked on the pre-CentOS project of ChaOS. The wikipedia article on Rocky covers in more detail than me regurgitating it. My main reason for looking at it, is that it has a ARM port which will eventually run nicely on a raspberry pi for people interested in it.

My other reason was to remind people that if they are using CentOS Linux, that the end of life for 8 is December 31, 2021. Before that time it would be good to look at switching to Alma, CentOS Stream 8, Oracle Linux, Red Hat Enterprise Linux, or Rocky.  These transitions take time, and there are only ~190 days before it needs to be done.

Caveats

[*] There is a third rebuild which has been around for years called Springfield Linux made at Princeton University. However either due to the pandemic or other items, it does not seem to have had regular updates.