How to test an update for EPEL

Earlier this week the maintainer for clamav came onto the Freenode #epel channel asking for testers of EL-6. There was a security fix needing to be pushed to stable, but no one had given the package any karma in bodhi.

EPEL tries to straddle the slow and steady world of Enterprise Linux and the fast and furious world of Fedora. This means that packages are usually held in epel-testing for at least 14 days or until the package has been tested by at least 3 people who give a positive score in bodhi . Because EPEL is a 'Stone Soup' set of packages, it does not have dedicated QA which test every update, but instead relies on what people bring to the table in the form of testing things if they need it. This has its benefits, but it does lead to problems where someone who wants to get a CVE fix out right away having to find willing testers or wait 14 days for the package to auto-promote.

Since I had used clamav years ago, and I needed an article to publish on Wednesday.. I decided I would give it a go. My first step was to find a system to test with. My main website still runs happily on CentOS-6 and I saw that while I had configured spamassassin with postfix I had not done so with clamav. This would make a good test candidate because I could roll back to an older setup if the package did not work.

First step was to install the clamav updates. Unlike my desktop where I have epel-testing always on, I keep the setup rather conservative on the web server. So to get the testing version of clamav I needed to the following:

# yum list --enable=epel-testing clamav*
Available Packages
clamav.i686                     0.99.4-1.el6                 epel-testing
clamav-db.i686                  0.99.4-1.el6                 epel-testing
clamav-devel.i686               0.99.4-1.el6                 epel-testing         
clamav-milter.i686              0.99.4-1.el6                 epel-testing
...

I then realized I had only configured clamav with sendmail in the past (yes it was a long time ago.. I watched the moon landings too.. and I can mostly remember what I had for breakfast). I googled through various documents and decided that a document at vpsget was a useful one to follow (Thank you to vpsget). Next up was to see if the packages listed had changed which they had not. So it is time to do an install:

# yum install --enable=epel-testing clamav clamstmp clamd
Loaded plugins: fastestmirror
Setting up Install Process
Loading mirror speeds from cached hostfile
 * base: mirrordenver.fdcservers.net
 * epel-testing: mirror.texas3006.com
 * extras: repos-tx.psychz.net
 * updates: centos.mirror.lstn.net
Resolving Dependencies
--> Running transaction check
...
Is this ok [y/N]:

I didn't use a -y here because I wanted to confirm that no large number of dependencies or other things were pulled in. It all looked good so I hit y and the install happened. I then went through the other steps and saw that there was a change in setup from when the document was written.

[root@linode01 smooge]# chkconfig --list | grep clam
clamd           0:off   1:off   2:off   3:off   4:off   5:off   6:off
clamsmtp-clamd  0:off   1:off   2:off   3:on    4:on    5:on    6:off
clamsmtpd       0:off   1:off   2:off   3:on    4:on    5:on    6:off

I turned on clamsmtp-clamd instead of clamd, and continued through the configs. After this I emailed an eicar to myself and saw it get blocked in the logs. I then looked at the CVE and saw if I could trigger a test against it. It didn't look like I could so I skipped that part. I then repeated the setup in a EL-6 VM I have at home to see that it worked there also. At this point it was time to report my findings in bodhi. I opened the report the owner had pointed me to and logged into the bodhi system. I added a general comment that I had tested it on 2 systems, and then plus 1'd the parts I had tested. Other people joined in and this package was able to get pushed much earlier than it would have previously.

There are currently 326 packages in EPEL-testing for EL-6. Please take the time to test 1 or two packages if you can. [I did pax-utils in writing this because I wanted to replicate the steps I had done. It needs 2 more people to test and it is a security fix also.]

2.5 Year Warning: EPEL-6 will be archived in January 2021.

EPEL builds packages against Red Hat Enterprise Linux versions which are in Production Phases 1,2,3. Currently RHEL-6 will reach this on November 30, 2020. At this point, EPEL will follow the steps it did with RHEL-5 to end of life EPEL-6.

1. New builds will be stopped in the koji builders.
2. Branching into EL-6 will be stopped in the Fedora src mechanism
3. Packages in epel-6 testing will no longer be promoted to epel-6.
4. After about 2 months, we will archive the packages to fedora archives and have the mirrors point to that. 

What does this mean for users of EPEL-6 currently? Nothing much beyond the fact that you should start planning on moving to newer versions of (RH)EL in the next 2.5 years. [This includes me because my main website runs on CentOS-6.] If your EL-6 is looking to be run past December 1, 2020, then you need to look at getting extended software contracts from Red Hat (or some consultant who is mad enough to do so). [Red Hat Enterprise Linux 6 was initially released in 2010, so it will have had 10 years support by then.]

What does this mean for the EPEL Steering Committee? We need to work out a better mechanism than we had in EL-5 for various packages which were end of lifed. Currently the build system composes each EPEL tree like it was a completely new distribution of packages. When a package is retired by its maintainer, the only way for a user to get that copy is to get the last released build from koji.fedoraproject.org versus from a mirror. This puts a lot more load on koji and also on users who have to try and figure out how to keep an old box going. 

Using the Red Hat Developer Toolset (DTS) in EPEL-7

One of the problems developers find in supporting packages for any long lifed Enterprise Linux is that it is harder and harder to compile newer software. Packages may end requiring newer compilers and other tools in order to be made. Back-porting fixes or updating software become harder and harder because the tools are no longer available to make the newer code work.

In the past, this has been a problem with EPEL packages as various software upstreams focus on newer toolkits to meet their development needs. This has lead to many packages to either be removed or left to mummify at some level. The problem occurs outside of EPEL also which is why Red Hat has created a product called Developer Toolset (DTS) which contains newer gcc and other tools. This product uses software collections which have had a mixed history with Fedora and EPEL but was considered useful in this limited use.

How to Use DTS in spec files

In order to use DTS in a spec file you will need to do the following:

1. If you are not using mock and fedpkg to build packages, you will need to add either the Red Hat DTS channel to your system or if you are using CentOS/Scientific Linux, you can add the repository following these instructions.
2. If you are using mock/fedpkg, the scl.org repository should be available in the epel mock configs.
3. In the spec file add the following section to the top area:

   %if 0%{?rhel} == 7
   
   BuildRequires: devtoolset-7-toolchain, devtoolset-7-libatomic-devel
   
   %endif
   
   

   Then in the build section add the following:

   %build
   
   %if 0%{?rhel}
   
   . /opt/rh/devtoolset-7/enable
   
   %endif
   
   

4. Attempt to do a build using your favorite build tool (rpmbuild, mock -r , fedpkg mockbuild, etc).  

This should start finding what things you might need to add in to the buildrequires similar problems. We in the EPEL Steering Committee would like to get feedback on this and work out what additions are needed to get this working for other developers. 

Note:

There are several caveats to using the Developer ToolSet in EPEL.

1. Packages may only have a BuildRequire: on the packages in the DTS. If your package will need to Require: something in the DTS or Software-collections, it can NOT be in EPEL at this time as many users do not have this enabled or used.
2. This is only for EPEL-7. At the moment, I have not set up DTS for EL-6 because it was not asked for recently. The Steering Committee would like to hear from developers if they want it enabled in EL-6.
3. The architectures where DTS exists are: x86_64, ppc64le, and aarc64. There is no DTS for ppc64 and we do not currently have an EPEL for s390x.

Thanks:

Our thanks to Tom Callaway and many other developers for his patience on getting this working.

CORRECTIONS:

• Originally the article stated that the text %if 0%{?rhel} == 7 should be used. That fails. The correct code is %if 0%{?rhel}
• If you build with mock, you are restricted to only pulling in the DTS packages. Currently koji does not have this limitation which is being fixed.

Presents for System Administrators: Updated nagios/nagios-plugins/nrpe

It is the traditional US Thanksgiving time period, and I am thankful for the patience many sysadmins have had with me. After some delays updated packages for nagios, nagios-plugins, and nrpe have been made for the EPEL-6, EPEL-7, F25, F26, F27 and rawhide.

1. nagios is updated to 4.3.4 in all channels. I have also fixed some issues in EL-6.
2. nagios-plugins is built out of git with what I thought was going to be 2.2.2 this summer. I have cleaned out some old patches and added updates for it to work with compiles on rawhide. [I will have to update rawhide in December when I figure out the right maria-connector-c fix.]
3. nrpe has been updated to 3.2.1 which was released in September. 

Currently one problem I have to deal with is that moved an entry for the nagios status file at some point in the past. It was in /var/spool/nagios/ and looks to be in /var/log/nagios now which various configs might not have changed to.

Otherwise please look for the updates in the various updates channels and give feedback on what you find right or wrong.

EPEL mirror file layout changes

As several people have noted, the file directory structure of EPEL has changed recently. This layout may require changes in both (1) scripts written with hard-coded locations, and (2) mirrors which were unable to get daily updates from the main mirrors.  While the changes were communicated in meetings, I did not adequately comprehend their effects to let mirrors and EPEL users know about it. This meant this announcement was delayed over two weeks.

 What Happened

The updates in the build system were to add new features and make the release engineering code more manageable. The old release style used by EPEL in EL-6 and EL-7 was different from how all other releases were done and caused several problems for the release code and mirrors.

1.  Due to all the files of the release being in one directory, any code which needed to stat (2) the directory caused the server to go over thousands of files before returning. With EPEL being a large amount of downloads, this negatively impacted systems. Servers mirroring the data could find long delays in rsyncing the data down. 
2. The code that generated this was a 'special' case in the Fedora releng release process which was fragile and tended to cause problems for updates and releases in both EPEL and the normal release.
3. The layouts were different from the current Fedora release so that    people grabbing software from multiple places also had to special case their scripts.

During the updates to the release system with a new version of pungi, it was decided to remove this special case and have all software Fedora created laid out in the same structure by the build tools. This would hopefully make things much more maintainable and improve performance.

In order to safely transition, there would be a time where the old files would remain on the server in the old trees and also be hardlinked to their new location. This was intended to allow for mirrors to get the files with the minimum amount of bandwidth. However there were some problems which showed up.

1. As I said before, I didn't grasp that the change was going to affect EPEL and didn't communicate this to the lists. 
2. The transition time for removing the hardlinks was in days versus weeks. While most mirrors do daily updates, some only do weekly or  monthly rsync's. They missed the hardlinks completely and had to download data twice. 
3. In the usual rule of three, various top level mirrors (mirrors.kernel.org and some others) had un-related mirroring problems at the end of October. When these servers caught up with the new layout, the hardlinked files were gone. This meant that mirrors taking data from a couple of tier1 sites had large uploads.

 How to deal with current things

The current layout structure should be 'solid' for the next couple of years. With the break down of packages into alphabetical subtrees, the 'load' per server should not require a re-ordering in the near future.

If you have written scripts which downloaded a specific file from the mirrors, (aka http://dl.fedoraproject.org/pub/archive/epel/5/i386/epel-release-5-4.noarch.rpm or some similar link), you should instead use a  stable linked package like http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm The epel-release packages get updated regularly to get new macros or other changes so linking to a specific file is very error prone.

Otherwise one should use yum/dnf related commands to get the files from the mirrors. This is useful for mirror sites which may alter the directory structure themselves and thus only the repodata is 'safe' to figure out what to download.

Ansible RPMS are no longer in EPEL-7

Ansible packages are no longer shipped in EPEL-7 as they have been included in Red Hat Enterprise Linux Extras (and similarly in CentOS-7 and hopefully Scientific Linux 7.4).

Systems which are either using Amazon Linux or Red Hat Enterprise Linux EUS release of 7.2/7.3 will need to get packages from Ansible directly using

http://releases.ansible.com/ansible/rpm/

My thanks to the Ansible Maintainer Kevin Fenzi for having the package inside of EPEL for the last several years.

Nagios being updated to 4.3.4 in EPEL and Fedora

It took me longer than I wanted, but I have gotten a testing candidate for nagios-4.3.4 in EL6, EL7, F25, F26, F27 and rawhide. This will fix the security problem seen in CVE-2017-14312.

I have made a couple of changes in the RPM also as rpmlint pointed out that the libnagios.a should be in the -devel package and that various contrib items needed to be packaged up in a similar package. I expect I missed something so please test and let me know so I can get this published as soon as possible.