Canaries in a coal mine (apropos nothing)

[This post is brought to you by Matthew Inman. Reading http://theoatmeal.com/comics/believe made me realize I don't listen enough and Verisatium's https://www.youtube.com/watch?v=UBVV8pch1dM made me realize why thinking is hard. I am writing this to remind myself when I forget and jump on some phrase.]

Various generations ago, part of my family was coal miners and some of their lore was still passed down many many years later. One of those was about the proverbial canary. A lot of people like to think that they are being a canary when they bring up a problem that they believe will cause great harm.. singing louder because they have run out of air.

That isn't what a canary does. The birds in the mines go silent when the air runs out. They may have died or are on the verge of being dead. They got quieter and quieter and what the miners listened for was the lack of noise from birds versus more noise. Of course it is very very hard to hear the birds in the first place in a mine because they aren't quiet places. There is hammering, and shoveling and footsteps echoing down long tubes.. so you might think.. bring more birds.. that just added more distractions and miners would get into fights because the damn birds never shut up. So the birds were few and far between and people would have to check up on the birds every now and then to see if they were still kicking. Safer mines would have some old fellow stay near the bird and if it died/passed out they would begin ringing a bell which could be heard down the hole.

So if analogies were 1:1, the time to worry is not when people are complaining a lot on a mailing list about some change. In fact if everyone complains, then you could interpret that you have too many birds and not enough miners so go ahead. The time to worry would be when things have changed but no one complains. Then you probably really need to look at getting out of the mine (or most likely you will find it is too late).

However analogies are rarely 1:1 or even 1:20. People are not birds, and you should pay attention to when changes cause a lot of consternation. Listen to why the change is causing problems or pain. Take some time to process it, and see what can be done to either alter the change or find a way for the person who is in pain to get out of pain.


Moving EPEL-4 and EPEL-5 to archives

Today we say goodbye to the last parts of EPEL-5 (and also EPEL-4). The top level files in /pub/epel/4 and /pub/epel/5 were moved to /pub/archive/epel so that people who are still needing packages can get them from the archives. People using yum should not see any change in updates because mirrormanager had the changes to point to archives a couple of days previously.

For any kickstarts or scripts that used the main download servers all that needs to be done is change:




and you can have your kickstart scripts grab the epel rpm from


Thanks again to everyone who has helped with EPEL-5 over the years. It was a good crazy ride.


EPEL-5 article appearing on FedoraMagazine.org

So I thought I was not writing anything more about the EOL of EPEL-5, but I got asked by several people why no one had written anything about it 😐. The ability of my posts to reach the world was much smaller than I realized. In order to rectify that a bit, here is another article on the EOL of EPEL-5 this time at Fedora Magazine.


IMPORTANT REMINDER: EL 5 is EOL on March 31. 2017

This is probably my final reminder on this before April 3rd 2017. As listed at https://access.redhat.com/support/policy/updates/errata and https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux#Product_life_cycle Red Hat Enterprise Linux will be exiting "Production Phase 3", and CentOS will be archiving off old EL-5 releases.

At that point, all remaining EPEL-5 packages will be archived to /pub/archive/epel/5 for systems to get data from. No new updates or packages will be done after that.


Trying to get an idea about what packages are used


One of the questions I get asked a lot is "You provide various statistics for Fedora, can you show which packages are installed the most?"

To head off a lot of future requests, the answer is no, no I can't. We do not have any sort of popcorn database which shows what packages are popular. When a user requests the OS to install a package, there is no "Hey I am asking for Bob if I can install libfoobar" that gets sent to the Fedora servers. What yum, dnf, PackageKit, or Salt do is then request for the repo data, looks to see if there is a way to figure out what is wanted and then asks for any packages that it needs to get.

It is this data that I can sort of glean some sort of idea of most installed packages.. but I feel it is way past "Lies", "Damn Lies", and "Statistics" into regions like  "Political Promises" or "Half Life 3 confirmed". Looking over an entire month of requests, sorting the data, and ranking the requests, I find that a bunch of packages show up a lot while others fall off in a long tail. Things that make this data dirty are the fact that if 200 people ask for wordpress, 150 for mediawiki and 90 for nagios.. I will see various PHP trunk packages that all three want as a higher number. I can't simply tell if the person wanted that PHP package by itself or wanted wordpress. [I could possibly try and work out a transaction of requested packages and figure out what nodes and leafs there might be.. but I found that the tools don't always request from download.fedoraproject.org everything it is wanting because it possibly already 'knows' where something is.

In any case, here are the most requested packages to the download website for January.


  1. epel-release-7-9
  2. python2-pip-8
  3. python2-boto-2
  4. openvpn-2
  5. php-tcpdf-6
  6. php-tcpdf-dejavu-sans-fonts-6
  7. pdc-updater-0
  8. duplicity-0
  9. nagios-plugins-2 *lots of plugins show up here*
  10. ansible-2
  11. libopendkim-2
  12. opendkim-2
  13. cowsay-3
  14. python2-wikitcms-2
  15. pkcs11-helper-1
  16. fedmsg-0
  17. htop-2
  18. munin *lots of munin packages here
  19. awscli-1
  20. hdf5-1


  1. nagios-plugins-2 *lots of other nagios removed*
  2. libmcrypt-2
  3. nodejs-0 *lots of other nodejs removed*
  4. python2-boto-2
  5. GeoIP-1 *other GeoIP removed*
  6. geoipupdate-2
  7. nrpe-2
  8. libnet-1
  9. denyhosts-2
  10. eventlog-0
  11. syslog-ng-3
  12. epel-release-6-8
  13. php-pear-Auth-SASL-1
  14. php-pear-Net-SMTP-1
  15. php-pear-Net-Socket-1
  16. perl-Net-IDN-Encode-2
  17. perl-Net-Whois-Raw-2
  18. perl-Regexp-IPv6-0
  19. pwhois-2
  20. v8
EPEL-6 is our most popular distribution with a ratio of about 12 EPEL-6 : 7 EPEL-7: 1.5 Fedora 25 to 1 EPEL-5 request over the month of January. 


  1. R-core-3 *lots of other R packages removed*
  2. globus-gssapi-gsi-devel-12 *lots of other globus removed*
  3. nordugrid-arc-5
  4. xrootd-client-libs-4 *lots of other xrootd removed*
  5. pcp-libs-devel-3
  6. nordugrid-arc-devel-5
  7. libopendkim-2
  8. libopendmarc-1
  9. pcp-libs-3
  10. nordugrid-arc-plugins-globus-5
  11. libopendkim-devel-2
  12. libopendmarc-1
  13. ebtree-6
  14. myproxy-libs-6
  15. mosh-1
  16. lua-cyrussasl-1
  17. drupal7
  18. rear-2
  19. clustershell-1
  20. rsnapshot-1
I found it interesting that R was getting pulled in by a lot of computers on EPEL-5. This OS is almost end of lifed, but it looks like systems are still getting provisioned with it.

Fedora 25

  1. java-1
  2. vim-minimal-8
  3. kernel-core-4
  4. libX11-1
  5. perl-libs-5
  6. perl-5
  7. perl-IO-1
  8. perl-macros-5
  9. perl-Errno-1
  10. nss-3
  11. gdk-pixbuf2-2
  12. gtk3-3
  13. audit-libs-2
  14. nss-softokn-freebl-3
  15. libX11-common-1
  16. gdk-pixbuf2-modules-2
  17. libnl3-3
  18. gnutls-3
  19. pcre-8
  20. gtk-update-icon-cache-3
As can be seen from the Fedora 25, there is another problem with my trying to get an idea of packages.. a package getting updated that is installed on a lot of boxes will show up also. 


I really don't think any 'real' conclusions can come out of this other than people really want vim on their Fedora 25 desktops (emacs was way down the list). 😑 I also want to say that we should get an opt-in popcorn for Fedora :).

[Edited: I forgot this part]

This list of agents which get used to pull down packages for EPEL and Fedora was rather interesting. I combined all the yum together as the many different versions kind of polluted the numbers but here are the top agents:

  1. yum
  2. Salt
  3. dnf
  4. Artifactory
  5. python-requests
  6. Debian Apt-Cacher-NG
  7. PackageKit-hawkey
  8. Axel 2.4 (Linux)
  9. Wget
  10. libdnf
  11. curl
  12. urlgrabber
The Salt seems to come from a large number of amazon systems which are installing either epel-release-6 (80% of the time) or epel-release-7 (20% of the time). Nothing else seemed to be 'pulled' from download.fedoraproject.org so it is probably just a config artifact on bootup. 


Major update to Nagios in Fedora Rawhide and EPEL-7 [moving to 4.2.4]

After a couple months of work, I have put together an updated package for Fedora Rawhide and EPEL-7 today. I expect it will have some 'problems' and so have moved the needed karma to 4 and am looking for people to test and give it negative karma with feedback for items broken.

I will work on getting those done this week so we can try and have working versions of Nagios for Fedora Server 26 and EPEL. Currently I expect it to need changes to the selinux policies for both and may need some additional work there. I am working through the processes for getting those done.

EPEL-6 will need some more work because the rpmbuild is complaining that it can't make /var/run/nagios for some reason.

Creating a new update for  nagios-4.2.4-2.el7 
  Update ID: FEDORA-EPEL-2017-0f3297a19b
    Release: Fedora EPEL 7
     Status: pending
       Type: security
      Karma: 0
    Request: testing
       Bugs: 1288989 - None
           : 1289710 - None
           : 1299166 - None
           : 1322666 - None
           : 1329857 - None
           : 1330627 - None
           : 1341683 - None
           : 1405365 - None
           : 1411399 - None
      Notes: Major Update. Fixes various CVE and other issues.
  Submitter: smooge
  Submitted: 2017-02-07 23:46:16
   Comments: bodhi - 2017-02-07 23:46:17 (karma 0)
             This update has been submitted for testing by smooge.


Major update to Fedora/EPEL moving to nrpe-3.0.1

The version of nrpe in Fedora has been 2.15 for a very long time while the upstream Nagios group moved to a 3.0 series. With some work and a lot of help from my friends, I have put an updated nrpe into EPEL-testing for EPEL-6 and EPEL-7 and in Fedora Rawhide.


I have put the EPEL update karma for this to be 4 versus 3 as I would like some more testing done by people before it gets working. If it gets a lot of negative karma I will pull it and work with upstream to get a working version into EPEL.


This version of nrpe was the 'fun' one. This is due to the fact that the newer OpenSSL does not allow for introspection of various structures which it used to. Working with Tomas Mraz and Patrick Uiterwijk, I believe I have a semi-working version. [A secondary problem was that I had to pull some sslv2 code out because we do not ship with those libraries anymore. I am hoping upstream will come up with a better fix than my hacksaw method.]

Fedora 25

I have not put in an update to Fedora 25 because it is a major update and was not listed as a change request. I am looking through what needs to be done for this, and when I have gotten any approvals needed will publish it to Fedora 25 testing.