EPEL statistics: EL-7 and EL-6

The above is a 7 day moving average graph of checkins to mirror manager looking for EPEL updates. In general it shows that EPEL has been growing steadily since its inception with some some interesting peaks and dips.
In March of 2017, you see a drop off of EL-5 when CentOS end of lifed it from their mirror manager. I don't know if those clients are still just looking for updates and failing or if clusters all over were shut off. You will notice that the EL-7 growth becomes much steeper than it was before, so I am assuming it is people jumping from 5 to 7.  In January of 2018, you see another blip. This shows up in the normal data when the Spectre/Meltdown updates came out. It looks like a lot of hosts may only update every now and then, but they ALL updated on that week. 

The big news is that EL-6 is still growing by a lot more than anyone would expect. By this time in its lifetime, EL-5 was beginning to drop off, but it would seem that people are still deploying EL-6 in large numbers. That has made it a lot harder for EL-7 to 'catch' up, but it looks like it will finally do so in the next month or so. I realize that in todays 'move fast and break things' this seems crazy slow.. you can't code RUST on EL-6 or 7 without some serious hacks.. but this is actually a fast paced change in the 'ops' world. [I had to support Solaris 2.5 (release in 1995) in 2007. The system it was running had another 5 years before it was to be end of lifed. To replace it would have required recertifying a large amount of other systems and processes. ]


Things to be prepared when asking for support

It is fairly common to see someone drop into an IRC channel, ask a question and then quit 10-20 minutes later if they don't get an answer. [It isn't just IRC, I have seen this on other 'group' chat interfaces.. ] They may randomly jump from channel to channel trying to get anyone to answer their question. It is frustrating for the questioner and for anyone who comes online, writes up an answer and finds the person has left for some other channel right before they hit Enter.

Here are some useful guidelines if you are looking for help:

  1. The older the software, the longer it is going to take to get support. Most online communities go through a standard lifecycle. There is the first year or so where it seems like everyone is online all the time, and there are plenty of people to answer questions. However as time goes on there is a fall off as many people find other interesting things. It can look like the community is comatose, but there are usually people still around but with other jobs and problems taking up most of their time. [In other words, you will be able to find someone quicker to help on a nodejs problem than a Red Hat Linux 7.2 problem.]
  2. The more complicated the software, the more complicated the medium needed to support it. Communication tools like IRC and twitter are good for getting quick answers, but fall over quickly trying to go over a complicated configuration file. Especially when other people are trying to have conversations at the same time 😅. For really complicated problems, email is not going to be enough as more and more people need to answer things.
  3. FLOSS communities help those who help themselves. This means that a person needs to be willing to look up things first, read manuals, and have explored the entire system problem first. When someone has done this and can answer things like: "How much memory does the system have?", "Which version of the software is it?", "What extra components are there?" the problem can be solved quickly. Otherwise it can be 10 emails slowly having the person do all the exploration they should have done first.
  4. You can get online help quick, cheap, or easy.. pick 1. If someone tells you can get two of the three in software, they are usually selling you something. If they say you can get all three, it goes from selling to a con job.  There is usually a quick solution which is neither easy or cheap. There is usually an easy solution which is definitely not cheap or quick in the long run. And finally there is the expensive solution which will neither be quick or easy.
  5. Be prepared for contrary answers. As much as they call it "computer science", most software development and operations is part cargo-cult and part performance art. Some people will say you have to dance naked in front of the database server for the accounts receivable to work, and others will say that a chicken sacrificed is all that is required. We all know that the few times we didn't do it, bad stuff happened.. but we can't be sure which one gets us a paycheck or not. [This of course an exaggeration. Payroll completes because the plane gods flew over and no amount of dancing or chicken sacrifices will fix that.] 
  6. Be honest. Nothing makes a support session go bad faster than when someone feels the other side lied. If you have made configuration changes be open about it. If you don't remember what changes you made or that someone else made, just say that. The mistake many of us make is to say "Nothing" which is nearly always wrong. 

I think that list of metaphors got away from me somewhere around 3.. In any case, if you need to get help in most FLOSS online communities be patient, be prepared, and be kind. [This also goes for the person giving help.]


1.5 Year Warning: Python2 will be End of Lifed

The end of upstream Python 2.7 support will be January 1, 2020 (2020-01-01) and the Fedora Project is working out what to do with it. As Fedora 29 would be released in 2019-11 and would get 1.5 years of support, the last release which would be considered supportable would be the upcoming release of Fedora 28. This is why the current Python maintainers are looking to orphan python2. They have made a list of the packages that would be affected by this and have started a discussion on the Fedora development lists, but people who only see notes of this from blogs or LWN posts may not have seen it yet. 

Here is my current status on things:
  1. The epylog package will most likely be dead.package in rawhide. The package would need a complete rewrite to work with Python3 and it has not been touched by upstream, nor do I have time to do the port.
  2. I don't own the nagios-plugins-fts or nagios-plugins-lgcdm but if they ended up on my plate I would do the same thing.
  3. python2 is still supported in RHEL-7 and will be until 2024. epylog and any python scripts inside of nagios packages I own will be supported in EL7.
If your project currently requires python2, you need to either help your upstream move to python3, find a company or service that will maintain python2.7 for future years, or be prepared to spend years in the wilderness. From the many years and projects I supported Fortran IV, perl4, Java 1.2, and ADA83 code in the early 2000's... I expect that this will be where many people are going to be. [I will try to cover how to handle that in a future blog.]


Dealing with network hackers in 1995

Going back to early 1995, I was working for Los Alamos National Labs as a contractor systems administrator. I didn't have a security clearance so could not work 'behind the fence' as they said. Instead, I worked with a large number of similarly uncleared post-docs, graduate students, and college interns in a strip mall converted into offices. The offices ran from nearly one end of the strip mall to the other with a large selection of Unix, PC, and Mac systems spread through the building connected together with 10base2 (or thin-wire). To make things even more fun, most of the systems were disk-less SunOS Sparc ELC/SLC and IPC systems booting off a Sparc 10 which had 64 MB of RAM and I think 2 2 GB disk drives.

The first problem I had to deal with was my most of the systems would crash at different times during the day. I got a Digital network book my Dad had given me, and learned about common problems with networking as this was not something I had dealt with before. I found that the local network was connected to a T1 which ran back to the main campus about 2 miles away. The T1 went to a hub which had 7 thin-wire lines running out of it. That seemed fine until I traced the thin-wire out. I was worried there were bad connectors (there were) or kinks in the line (there were) but the real problem was that out of the 7 thin-wire lines 3 were used.  Most of the systems were on one line. 2 (my desktop and the Sparc 10) were on another one, and the Next and SGI's were on the third. The other lines were just laying under the carpets not used. I met with my new boss Dale, and showed him what I had found. I learned a lot from Dale. He got me a copy of the Unix System Administrators Handbook and told me to start reading it on networks.

So I spent the next week or so learning how to properly crimp and connect thin-wire. I also learned testing signals and ohm resistance as I found connectors which didn't work very well. I moved all the Windows 3.11 and Macintosh 6? systems over to one set of cables and then spread the disk-less stations over to other lines in order to try and diagnose the crashes. At first, I didn't have a crash for 2-3 days and I thought everything was licked. And then the next Monday, things started crashing again.

I started investigating, but we started getting reports in a different building of a similar problem. I found a server with no disk space because of a file in /usr/tmp which seemed to be random data. I believe this was late January 1995, and I had been putting in 60-80 hour weeks trying to get caught up with things. [I was an hourly and this was allowed overtime so I was paying off my college debts pretty quickly.] I hadn't watched TV or read USENET in weeks and had no idea that there was a massive computer hunt at the time. Now when I had worked at university, I would have probably deleted the file and moved on, but because I was supporting scientists I didn't want to delete some research. I contacted Dale, and he helped me work out that the file was being written to by the rshd or telnetd command. He had me check the Ethernet port and sure enough it was in promiscuous mode. Now I had been slow up until this point but I realized this was not a good thing.

Now back in 1995, nothing on these networks was encrypted. You used telnet or rsh to login into systems. You had fingerd running on every box because you used it to see who was logged in. Most networks were on hubbed networks so you could just listen on one system and hear everything in your /24. Dale started contacting the people in security and I started looking at what other boxes were bad. It turned out several had this but one of the ones in my area which had crashed still had the source code for the sniffer daemon on it. I looked at it and found that it linked to the equivalent to tcpdump and filtered for for anything typed after Password: so it caught telnetd, rshd, ftpd and also the su command.

It then kept whatever came afterwords til a NULL and xor that data to the file it opened. I think the xor was against  libc random() stream of data using a starting string as the seed. [The string was then freed I guess to make sure gdb couldn't find it.] To de-crypt the data you just ran the same command with I think a -u and the same password. You could set it up to log even more data at compile time so I was trying that out to see if there was anything I could see about what had been captured on different systems.

At this point I had gotten it running on the system and was looking at what it was capturing when along came the attacker. They logged into the system as a normal user and ran a program ... I had missed which was setuid. They then checked what they had captured and logged out. In doing so I had captured the password they were using. I also had found that they had been running finger against a bunch of systems looking for a particular login. It turned out that a prominent researcher had collaborated with a professor at Los Alamos and the systems with their accounts were being targeted. I turned this all over to my boss who asked me if any of that meant anything to me. It didn't. He told me to go read some particular Usenet groups and that we would have to reinstall everything. [I believe he had been trying to get everything installed, updated and secured for years but it had been a low priority in his management chain.]

For a while it was thought that the attacker may have been on particular person, but these days I doubt it very much. There were a lot of people trying to be 'helpful' at that time by doing various things, and I expect it was one of them. It was an eye opening experience and I wanted to capture it here for the amount of naivete we had back then:

  1. No firewall because it was meant to be an educational section where students and professors should just be able to telnet from their University to whatever computer they needed.
  2. Most of the passwords were stored in a centralized space anyone could look at the hashes for. The Sun yellow page system was useful in mass deployments but had its limits. All of the systems stored their passwords in /etc/password so if you got onto the system at all you could see the hash.
  3. Networks were generally hubbed so that anyone could listen to anyone else locally. This was considered less of a problem when only Unix systems were around because there was a 'separation' of root and user so you could 'control' who could look at the network. The growing number of Mac and PC's which allowed anyone to listen made this the next part hard.
  4. Network communication was mostly in the clear. This was due in part because encryption was expensive on CPUs but it was also that encryption was export controlled so no one wanted to use it in case they got in trouble for it 'leaking'. 
  5. Most systems came out of the box with many network services turned on which never needed to be. Just as these days, your Facebook or LinkedIN account starts off public to the universe, your computer would share if you were logged in, where you had logged in from, what your .plan might have in it and a dozen other things. 
  6. That security problems tend to travel along lines of social trust. The attackers were following along various people who had worked with one researcher and using each set of systems to jump to the next ones. One of the computers hacked was done via a talk session where someone asked someone else if they could help them. I think the victim thought they were helping their adviser with a temporary .rlogin and it escalated from there. 
  7. While Howard Tayler would put this better years later, I found that in security, failure is always going to happen. What is important is how to react to it. I made sure that I could always backup and restore systems en-mass if needed. I also made sure that I have a plan B and C for when the next security problem occurs. 
It would be great if that naivete was unique to that time frame, but we constantly reinvent ways to think we are completely safe.. only to find we covered ourselves in steak-sauce and laid down in the hungry lions pit. 

Addendum: I am recalling things from over 20 years ago and like a fish tale.. some things which make me look better will have grown in that time, and other things which would make me look like an idiot will have faded away.. 


How to install EPEL Packages

What is EPEL?

EPEL stands for Extra Packages for Enterprise Linux. As I have stated before, the Extra Packages are rebuilds of packages from various Fedora Project Linux releases with an aim to keep EPEL packages slower moving than what is in Fedora. The Enterprise Linux in this statement is aimed at the Red Hat Enterprise Linux and the many rebuilds of it (CentOS, Scientific Linux, Oracle Linux and sometimes Amazon Linux). [As opposed to the enterprise offerings from SuSE or Canonical.] 

How to set up EPEL?

In order to enable the EPEL repositories, you need to do the following things. 
  1. Determine which version of an Enterprise Linux you are using by looking in the /etc/system-release file. On my CentOS-6 system it looks like  
    [root@el-6 ~]# cat /etc/system-release
    CentOS release 6.9 (Final)
    and on my CentOS 7 system it currently looks like
    [root@el-7 ~]# cat /etc/system-release
    CentOS Linux release 7.4.1708 (Core) 
  2. If you are running CentOS or Scientific Linux you can now simply install and enable EPEL with a yum command:
    [root@el-7 ~]# yum --enablerepo=extras install epel-release
    [root@el-6 ~]# yum --enablerepo=extras install epel-release
    This will install the release keys and files with a package which has been signed by your distribution for you to have a chain of trust.
  3. If you are running Red Hat Enterprise Linux, you will need to do some extra steps. We will use EL-7 but they would be similar for EL-6.
    1. Read the page at https://getfedora.org/keys/faq/
    2. Download the RPM and GPG key and confirm they are valid.
      # wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
      # wget https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-7
      # rpm --import RPM-GPG-KEY-EPEL-7
      # rpm -K epel-release-latest-7.noarch.rpm
      # rpm -K epel-release-latest-7.noarch.rpm 
      epel-release-latest-7.noarch.rpm: rsa sha1 (md5) pgp md5 OK
  4. If you are more of a curl | su - type person then you can just install directly from the internet using the rpm command. I don't recommend this but it gets asked a lot.
  5. You can now list the packages using yum list which will give more packages. These can be installed with the normal install methods. If you did not already import the gpg keys, you will be prompted to trust these keys.
    [root@el-7 ~]# yum install pax-utils
    Loaded plugins: auto-update-debuginfo, fastestmirror
    Loading mirror speeds from cached hostfile
     * base: mirror.yellowfiber.net
     * epel: archive.linux.duke.edu
     * epel-testing: packages.oit.ncsu.edu
     * extras: mirror.yellowfiber.net
     * updates: mirror.yellowfiber.net
    Resolving Dependencies
    --> Running transaction check
    ---> Package pax-utils.x86_64 0:1.2.3-1.el7 will be installed
    --> Finished Dependency Resolution
    Dependencies Resolved
     Package          Arch         Version          Repository             Size
     pax-utils        x86_64       1.2.3-1.el7      epel-testing           96 k
    Transaction Summary
    Install  1 Package
    Total download size: 96 k
    Installed size: 249 k
    Is this ok [y/d/N]: y
    Downloading packages:
    pax-utils-1.2.3-1.el7.x86_64.rpm                        |  96 kB  00:00:00
    Running transaction check
    Running transaction test
    Transaction test succeeded
    Running transaction
      Installing : pax-utils-1.2.3-1.el7.x86_64                           1/1
      Verifying  : pax-utils-1.2.3-1.el7.x86_64                           1/1 
      pax-utils.x86_64 0:1.2.3-1.el7
  6. Sometimes you may need to install a package which has not made it to stable yet. Please see my earlier post on that.
  7. I have been told by various people in IRC that they were told by someone else that current version of Amazon Linux was based off EL-6 so they want to use the EL-6 EPEL for it. We have also had lots of reports of people finding things not working at times. I would recommend that any users of Amazon Linux to use what Amazon recommends first.

Side Note:

I have been told multiple times that the EPEL logo looks like a horse's tail swatting flies. I actually think it looks more like a full sail being blown as our original want was to have a trading ship  or even a container ship (this was years before containers) but none of the attempts looked good. This one was provided by the art team as a combination of an backwards E and a P/L. 



I have not done a general biography in a long time so figured I should put one out as a courtesy for people reading these blogs and emails I send out on various lists:

Who Am I?

My name is Stephen Smoogen, and I have been using computers for a very long time. According to a bug in a PHP website I was on years ago, I am over 400 years old which would mean I was born in Roanoke island with Virginia Dare. I think I slept a lot in the next 360 years as, according to my sister, my parents found me in a 'DO NOT RETURN TO SENDER' box outside their door. How she knew that when she is younger than me, I do not know.. but I have learned not to question her.

My first computer was a Heathkit Microcomputer Learning System ET-3400 that my Dad got at a swap meet when he was in the Navy in the 1970's. I had worked with my Dad on some other systems he fixed for coworkers but it was mostly being bored while watching an oscilloscope and moving probes around boards every now and then. When I wanted to get a computer in the early 1980's, he said I had to show that I could actually program it since an Apple ][ would have been a major investment for the family. I spent the summer learning binary, hexadecimal and doing the simple machine code that the book had in it. I also programmed a neighbour's Apple ][+ with every game I could in the public libraries Creative Computing 101 Basic Games. My mom and dad saved up for an Apple and we got an Apple ][e in 1983 which I then used through high school. The first thing I learned about the Apple ][e was how different it was with the ][+. The older systems came with complete circuit diagrams and chip layouts. It had been the reason my dad wanted to get an Apple because he knew he could fix it if a chip went bad. The ][e did not come with that and boy was Dad furious. "You don't buy a car with the engine welded shut. Don't buy a computer you can't work on." It seemed silly to me at the time, but would be a founding principle for what I do.

During those years, I went with my dad and his coworkers to various computer clubs where I learned how to play hack on a MicroVax running I think Ultrix or BSD. While I was interested in computers, I had decided I was going to university to get a degree in Astrophysics.. and the computers were just a hobby. Stubborn person that I am, I finally got the degree though I kept finding computers to be more enjoyable. I played nethack and learned more about Unix on a Vax 11/750 running BSD 4.1 and became a system administrator of a Prime 300 running a remote telescope project. I moved over to an early version of LynxOS on i386 and helped port various utilities like sendmail over to it for a short time.

After college I still tried to work in Astrophysics by being a satellite operator for an X-ray observation system at Los Alamos. However, I soon ended up administrating various systems to get them ready for an audit, and that turned into a full time job working on a vast set of systems. I got married, and we moved to Illinois where my wife worked on a graduate degree and I worked for a startup called Spyglass. I went to work for them because they had done scientific visualization which Los Alamos used.. but by the time I got there, the company had pivoted to being a browser company with Enhanced Mosaic.

For the next 2 years I learned what it is like to be a small startup trying to grow against Silicon Valley and Seattle. I got to administer even more Unix versions than I had before, and also see how Microsoft was going to take over the desktop. That was because Enhanced Mosaic was at the core of Internet Explorer. At the end of the two years, Spyglass had not gotten bought by Microsoft, and instead laid off the browser people to try and pivot once again as an embedded browser company at a different location. The company was about 15 years too soon for that as the smart items their plans had as the near future didn't start arriving until 2015 or so.

Without a job, I took a chance to work for another startup in North Carolina called Red Hat. At a Linux Conference, I had heard Bob Young give a talk about how you wouldn't buy a car with a welded bonnet and it brought back my dad's grumpiness with Apple decades ago. I realized that my work in closed source software had been one of continual grumpiness because I was welding shut the parts that other people needed open.

Because of that quote, I worked at Red Hat the next four years learning a lot about openness, startups and tech support. I found that the most important classes I had from my college were psychology and not computer science. I also learned that being a "smart mouthed know it all in" doesn't work when there are people who are much smarter and know a lot more. I think by the time I burned out on 4 years of 80 hour weeks, I was a wiser person than when I came.

I went to work elsewhere for the next 8 years, but came back to Red Hat in 2009, and have worked in the Fedora Project as a system administrator since then. I have seen 15 Fedora Linux releases go out the door, and come to really love working on the slowest part of Fedora, EPEL. I have also finally used some of the astrophysics degree as the thermodynamics and statistics have been useful with the graphs that various Fedora Project leaders have used to show how each release and how the community has continually changed.


Explaining disk speeds with straws

One of the most common user complaints in an Enterprise systems is 'why can't I have more disk space?' The idea is that they look at the costs of disks on Amazon or New Egg and see that they could get an 8 TB hard disk for $260.00 but the storage administrator says it will cost $26,000.00 for the same amount.

Years ago, I once even had someone buy me a disk and have it delivered to my desk to 'fix' the storage problem. They thought they were being funny so I thanked them for the paper weight. I then handed it back to them and then tried to explain to them why 1 drive was not going to help... I found that the developers eyes glistened over as I talked about RPM speeds of drives, cache sizes, amount of commands a ATA read/write use versus SCSI, etc. All of them are important but not terms useful for a person who just wants to never delete an email.

The best analogy I have is that you have a couple of 2 litre bottles of Coca Cola (fill in Pepsi, Fanta or Mr Pibb as needed) and a cocktail straw. You can only fill one Coke bottle with that straw. Sure the bottle is big enough but it takes a long time to move the soda from one to the other. That is what 1 SATA disk drive is like.

The next step is to add more disks and make a RAID array. Now you can get a bunch of empty coke bottles and empty out that one array through the multiple cocktail straws. Things are moving faster but it still takes a long time and you really can't use each of the large bottles as much as you like because emptying them out will be pretty slow via the cocktail straw.

The next sized solution is regular drinking straws with cans. The straws are bigger, but the cans are smaller.. you can fill the cans up or empty them without as much time waiting for a queue. However you need a lot more of them to equal the original bottle you are emptying. This is the SAS solution where the disks are smaller, faster, and much better throughput because of that. It is a tradeoff in that 15k drives use older technologies so store less data. They also have larger caches and smarter os's on the drive to make the straw bigger.

Finally there are the newest solution which would be the garden hose connected to a balloon to a coffee cup. This is the SAS SSD solution. The garden hose allows for a large amount of data to go up and down the pipe, the balloon is how much you can cache in case you are too fast somewhere in writes or reads. The coffee cup is because it is expensive and there isn't a lot of space. You need a LOT of coffee cups compared to soda cans or 2 litre bottles.

Most enterprise storage is some mixture of all of these to match the use case need.

  • SATA raid is useful for backups. You are going to sequentially read/ write large amounts of data to some other place. The straws don't need to be big per drive and you don't worry about how much is backed up. The cost per TB is of course the smallest.
  • SAS raid is useful for mixed user shared storage. The reads and writes to this need a larger straws because programs have different IO patterns. The cost per TB is usually an order or two of magnitude greater depending on other factors like how much redundancy you wanted etc.
  • SSD raid is useful for fast shared storage. It is still more expensive than SAS raid. 
And now to break the analogy completely. 

Software defined storage would be where you are using the cocktail straws with coke bottles but you have spread them around the building. Each time coke gets put on one, a hose spreads that coke around so each block of systems is equivalent. In this case the costs per system have gone down, but there needs to be a larger investment in the networking technology tying  the servers together. [A 1 gbit backbone network is like a cocktail straw between systems, A 10 gbit backbone is like a regular straw and the 40G/100G are the hoses.]

Now my question is .. has anyone done this in real life? It seems crazy enough that someone has done a video.. but my google-fu is not working tonight.


Ramblings about long ago and far away

My first job outside of college in 1994 was working at Los Alamos National Labs as a Graduate Research Assistant. It was supposed to be a post where I would use my bachelor's in Physics degree for a year until I became a graduate student somewhere. The truth was that I was burnt out of University and had little urge to go back. I instead used my time to learn much more about Unix system administration. It turned out the group I worked on had a mixture of SGI Irix's, Sun Sparcstations, HP, Convex, and I believe AIX. The systems had been run by graduate students for their professors and needed some central management. While I didn't work for the team that was doing that work, I spent more and more time working with them to get that in place. After a year, it was clear I was not going back to Physics, and my old job was ending. So the team I worked on gave me a reference to another place at the Lab where I began work. 

This network had even more Unix systems as they had NeXT cubes, old Sun boxes, Apollo, and some others I am sure to have forgotten. All of which needed a lot of love and care as they had been built for various Phd's and postdocs for various needs and then forgotten. My favorite box was one where the owner required that nearly every file was set 777. I had multiple emails which echo every comment people come up with Selinux in the last decade. If there was some problem on the system it was because it had a permission set.. and until it was shown it didn't work at 777 you could look at it being something else. [The owner was also unbelievably brilliant in other ways.. but hated arbitrary permission models.]

Any case, I got a lot of useful experience on all kinds of Unix systems, user needs, and user personalities. I also got to use Linux Softland Linux Systems (SLS) on a 486 with 4 MB of RAM running the linux kernel 0.99.4? and learn all kinds of things about PC hardware versus 'Real Computers'. The 486 was really an overclocked 386 with some added instructions that had been originally a Cyrix DX33 that had been relabeled with industrial whiteout as a 40MHz. It sort of worked at 40Mhz but was reliable only at 20Mhz. The issues with getting deals from Computer magazines.. sure the guy in the next apartment worked great.. mine was a dud.

I had originally run MCC (Manchester Computer Center Interim Linux) in college but when I moved it was easier to find a box of floppies with SLS so I had installed that on the 486. I would then download software source code from the internet and rebuild it for my own use using all the extra flags I could find in GCC to make my 20Mhz system seem faster. I instead learned that most of the options didn't do anything on i386 Linux at the time and most of my reports about it were probably met by eye-rolls with the people at Cygnus. My supposed goal was to try and set up a MUD so I could code up a text based virtual reality. Or to get a war game called Conquer working on Linux. Or maybe get xTrek working on my system. [I think I mostly was trying to become a game developer by just building stuff versus actually coding stuff. I cave-man debugged a lot of things using stuff I had learned in FORTRAN but it wasn't actually making new things.]

For years, I looked back on that time and thought it was a complete waste of time as I should have been 'coding' something. However I have come to realize I learned a lot about the nitty-gritty of hardware limitations. A 9600 baud Modem is not going to keep up with people on Ethernet playing xTrek. Moving it to a 56k modem later isn't going to keep up with a 56k partial T1. The numbers are the same but they are counting different things. A 5400 RPM IDE hard-drive is never going to be as good as 5400 RPM SCSI disks even if it is larger. 8 MB on a Sparc was enough for a MUD but on a PC it ran into problems because the CPU and MMU were not as fast or 'large'. 

All of this later became useful years later when I worked at Red Hat between 1997 and 2001. The customers at that time were people who had been using 'real Unix' hardware and were at times upset about how Linux didn't act the same way. In most cases it was the limitations of the hardware they had bought to put a system together, and by being able to debug that and recommend replacements, things improved. Being able to compare how a Convex used disks or an SGI graphics to the limitations of the old ISA and related buses helped show that you could redesign a problem to meet the hardware. [In many cases, it was cheaper to use N PC systems to replicate the behaviour of 1 Unix box but the problem needed to be broken in a way that it worked on N systems versus 1 box.] 

So what does this have to do with Linux today? Well mostly reminders to me to be less cranky with people who are 
  1. Having fun breaking things on their computers. People who want to tear apart their OS and rebuild it to something else are going to run into lots of hurdles. Don't tell them it was a stupid thing. The people at Cygnus may have rolled their eyes but they never told me to stop trying something else. Just read the documentation and see that it says 'undefined behavior' in a lot of places.
  2. Working with tiny computers to do stuff that you do on a bigger computer these days. It is very easy to think that because it is 'easier' and currently more maintainable to do a calculation on 1 large Linux box.. that you are wasting time on dozens of raspberry pis to do the same thing. But that is what the mainframers thought of the minicomputers, and the minicomputers thought of the Unix workstations, and the Unix thought of Linux on PC. 
  3. Seeming to spin around, not knowing what they are doing. I spent a decade doing that.. and while I could have been more focused.. I would have missed a lot of things that happened otherwise. Sometimes you need to do that to actually understand who you are. 


Please end Daylight Savings Time

This was going to be my 3rd article this week about something EPEL related, but I am having a hard time stringing any words together coherently. The following instead boiled out and I believe I have removed the profanity that leaked in.

So I like millions of other Americans (except those people blessed to be living in Arizona and Hawaii) are going through the week long jetlag when Daylight Savings Time starts. For the last 11? years the US has had DST start 2 weeks earlier than the rest of countries which observe this monstrosity, I think to show the rest of the  why it is a bad idea.  No one seems to learn and instead try to make it longer and longer.

I understand why it was begun during World War I in order to make electricity costs for lighting cheaper in factories. It just isn't solving that problem anymore. Instead I spend a week not really awake during the day, and for some reason as I get older not able to sleep at all during the night. And I get crankier and more sarcastic by the day. Finally sometime next Monday, I will conk out for 12-14 hours and be alright again. I would like to say I am anomaly, but this seems to happen to a lot of people around the world with higher numbers of heart attacks, strokes and accidents during the month of time changes.

So please, next time this comes up with your government (be it the EU, US, Canada Parliament, etc) write to your representatives that this needs to end. [For a fun read, the Wikipedia articles on various forms of daylight savings cover the political philandering to pay for this.]

Thank you for your patience while I whine about having only the lack of sleep when there are a hell of a lot worse things going on in the world.


How to test an update for EPEL

Earlier this week the maintainer for clamav came onto the Freenode #epel channel asking for testers of EL-6. There was a security fix needing to be pushed to stable, but no one had given the package any karma in bodhi.

EPEL tries to straddle the slow and steady world of Enterprise Linux and the fast and furious world of Fedora. This means that packages are usually held in epel-testing for at least 14 days or until the package has been tested by at least 3 people who give a positive score in bodhi . Because EPEL is a 'Stone Soup' set of packages, it does not have dedicated QA which test every update, but instead relies on what people bring to the table in the form of testing things if they need it. This has its benefits, but it does lead to problems where someone who wants to get a CVE fix out right away having to find willing testers or wait 14 days for the package to auto-promote.

Since I had used clamav years ago, and I needed an article to publish on Wednesday.. I decided I would give it a go. My first step was to find a system to test with. My main website still runs happily on CentOS-6 and I saw that while I had configured spamassassin with postfix I had not done so with clamav. This would make a good test candidate because I could roll back to an older setup if the package did not work.

First step was to install the clamav updates. Unlike my desktop where I have epel-testing always on, I keep the setup rather conservative on the web server. So to get the testing version of clamav I needed to the following:

# yum list --enable=epel-testing clamav*
Available Packages
clamav.i686                     0.99.4-1.el6                 epel-testing
clamav-db.i686                  0.99.4-1.el6                 epel-testing
clamav-devel.i686               0.99.4-1.el6                 epel-testing         
clamav-milter.i686              0.99.4-1.el6                 epel-testing
I then realized I had only configured clamav with sendmail in the past (yes it was a long time ago.. I watched the moon landings too.. and I can mostly remember what I had for breakfast). I googled through various documents and decided that a document at vpsget was a useful one to follow (Thank you to vpsget). Next up was to see if the packages listed had changed which they had not. So it is time to do an install:

# yum install --enable=epel-testing clamav clamstmp clamd
Loaded plugins: fastestmirror
Setting up Install Process
Loading mirror speeds from cached hostfile
 * base: mirrordenver.fdcservers.net
 * epel-testing: mirror.texas3006.com
 * extras: repos-tx.psychz.net
 * updates: centos.mirror.lstn.net
Resolving Dependencies
--> Running transaction check
Is this ok [y/N]:

I didn't use a -y here because I wanted to confirm that no large number of dependencies or other things were pulled in. It all looked good so I hit y and the install happened. I then went through the other steps and saw that there was a change in setup from when the document was written.

[root@linode01 smooge]# chkconfig --list | grep clam
clamd           0:off   1:off   2:off   3:off   4:off   5:off   6:off
clamsmtp-clamd  0:off   1:off   2:off   3:on    4:on    5:on    6:off
clamsmtpd       0:off   1:off   2:off   3:on    4:on    5:on    6:off

I turned on clamsmtp-clamd instead of clamd, and continued through the configs. After this I emailed an eicar to myself and saw it get blocked in the logs. I then looked at the CVE and saw if I could trigger a test against it. It didn't look like I could so I skipped that part. I then repeated the setup in a EL-6 VM I have at home to see that it worked there also. At this point it was time to report my findings in bodhi. I opened the report the owner had pointed me to and logged into the bodhi system. I added a general comment that I had tested it on 2 systems, and then plus 1'd the parts I had tested. Other people joined in and this package was able to get pushed much earlier than it would have previously.

There are currently 326 packages in EPEL-testing for EL-6. Please take the time to test 1 or two packages if you can. [I did pax-utils in writing this because I wanted to replicate the steps I had done. It needs 2 more people to test and it is a security fix also.]


2.5 Year Warning: EPEL-6 will be archived in January 2021.

EPEL builds packages against Red Hat Enterprise Linux versions which are in Production Phases 1,2,3. Currently RHEL-6 will reach this on November 30, 2020. At this point, EPEL will follow the steps it did with RHEL-5 to end of life EPEL-6.
  1. New builds will be stopped in the koji builders.
  2. Branching into EL-6 will be stopped in the Fedora src mechanism
  3. Packages in epel-6 testing will no longer be promoted to epel-6.
  4. After about 2 months, we will archive the packages to fedora archives and have the mirrors point to that. 
What does this mean for users of EPEL-6 currently? Nothing much beyond the fact that you should start planning on moving to newer versions of (RH)EL in the next 2.5 years. [This includes me because my main website runs on CentOS-6.] If your EL-6 is looking to be run past December 1, 2020, then you need to look at getting extended software contracts from Red Hat (or some consultant who is mad enough to do so). [Red Hat Enterprise Linux 6 was initially released in 2010, so it will have had 10 years support by then.]

What does this mean for the EPEL Steering Committee? We need to work out a better mechanism than we had in EL-5 for various packages which were end of lifed. Currently the build system composes each EPEL tree like it was a completely new distribution of packages. When a package is retired by its maintainer, the only way for a user to get that copy is to get the last released build from koji.fedoraproject.org versus from a mirror. This puts a lot more load on koji and also on users who have to try and figure out how to keep an old box going. 


Using the Red Hat Developer Toolset (DTS) in EPEL-7

One of the problems developers find in supporting packages for any long lifed Enterprise Linux is that it is harder and harder to compile newer software. Packages may end requiring newer compilers and other tools in order to be made. Back-porting fixes or updating software become harder and harder because the tools are no longer available to make the newer code work.

In the past, this has been a problem with EPEL packages as various software upstreams focus on newer toolkits to meet their development needs. This has lead to many packages to either be removed or left to mummify at some level. The problem occurs outside of EPEL also which is why Red Hat has created a product called Developer Toolset (DTS) which contains newer gcc and other tools. This product uses software collections which have had a mixed history with Fedora and EPEL but was considered useful in this limited use.

How to Use DTS in spec files

In order to use DTS in a spec file you will need to do the following:
  1. If you are not using mock and fedpkg to build packages, you will need to add either the Red Hat DTS channel to your system or if you are using CentOS/Scientific Linux, you can add the repository following these instructions.
  2. If you are using mock/fedpkg, the scl.org repository should be available in the epel mock configs.
  3. In the spec file add the following section to the top area:
    %if 0%{?rhel} == 7
    BuildRequires: devtoolset-7-toolchain, devtoolset-7-libatomic-devel
    Then in the build section add the following:
    %if 0%{?rhel}
    . /opt/rh/devtoolset-7/enable
  4. Attempt to do a build using your favorite build tool (rpmbuild, mock -r , fedpkg mockbuild, etc).  
This should start finding what things you might need to add in to the buildrequires similar problems. We in the EPEL Steering Committee would like to get feedback on this and work out what additions are needed to get this working for other developers. 


There are several caveats to using the Developer ToolSet in EPEL.
  1. Packages may only have a BuildRequire: on the packages in the DTS. If your package will need to Require: something in the DTS or Software-collections, it can NOT be in EPEL at this time as many users do not have this enabled or used.
  2. This is only for EPEL-7. At the moment, I have not set up DTS for EL-6 because it was not asked for recently. The Steering Committee would like to hear from developers if they want it enabled in EL-6.
  3. The architectures where DTS exists are: x86_64, ppc64le, and aarc64. There is no DTS for ppc64 and we do not currently have an EPEL for s390x.


Our thanks to Tom Callaway and many other developers for his patience on getting this working.


  • Originally the article stated that the text %if 0%{?rhel} == 7 should be used. That fails. The correct code is %if 0%{?rhel}
  • If you build with mock, you are restricted to only pulling in the DTS packages. Currently koji does not have this limitation which is being fixed.