Reminder: PSA: Enterprise Linux 5 End of Production on 2017-03-31 and EPEL.

So in less than 12 months (2017-03-31), Red Hat Enterprise Linux 5 will be leaving end of production level 3 . At that time EPEL support will be ending for EL-5 and all packages will be moved to the archives section of the fedoraproject download servers.

If you are relying on (CentOS|ScientificLinux|Oracle|Red Hat) Enterprise Linux 5 for your business needs, you should look at moving the to either a newer version of Enterprise Linux or at the extended support program that Red Hat offers as listed in the above article.

I will be publishing my next PSA on this in 6 months.


Attention Sites Mirroring Fedora

We are seeing a large number of partial rsync on the main download
servers which are causing problems for tier0/1 mirrors to get to the
servers. What we are seeing is that an ip address will start a rsync
of a large tree and then will drop the connection as the server takes
time to work through 1-10TB of disk space. The ip will then initiate a
new connection which another download server will try to fulfill but
again with a time-out and restart. This is adding a large amount of
IOPs onto our backend storage causing a cascading set of problems
through the infrastructure.

We are going to have to put a firewall rules to drop connections from
these systems so that the alpha and other work can get properly
mirrored onto the registered Tier 0 and Tier 1 mirrors. If that
doesn't work we will be putting firewall rules that only Tier 0 and
Tier 1 mirrors are allowed to connect to the download servers.

I will be trying to send emails to the registered abuse for the ips, but if you are getting dropped connections to the download servers or are a mirror administrator at one of the following domains:

  • wideopenwest.com
  • alshamil.net.ae
  • yandex.net
  • sl-reverse.com
  • univ-ubs.fr
  • ip.pt

please register in the Fedora mirror manager so that it can be whitelisted. 

Finally, to cut down mirroring problems please do the following:

1) Do not put in a cron job that you are going to do an rsync update
every 15 minutes as several of the above mirrors seem to do. We do not
update the trees that often and rsyncd has to stat every file.

2) Please review the tips and tricks at
Using the last-sync to schedule updates when they actually occur can
help lower rsync usage.

Thank you


EPEL Proposal #4: Build it all in CentOS promote to RHEL

So this one came up when we were going over problems with building alternative architectures in EPEL. One of the big problems is that currently everything is built against Red Hat Enterprise Linux. This has some bonuses for some people but it causes a problem when architectures are built against by CentOS but not Red Hat. [E.G. EL-7 for i386 and arm32] When a Red Hat release is done, the builders usually get updated as part of our repopull from the RHN servers. Usually there are some changes in added and subtracted packages which can cause packages to get new dependencies. While this is an inconvenience to customers.. it isn't a problem for the builders. When we start mixing distributions, it does cause a problem with the builders because a package may get compiled for EL-7 x86_64 and not be the same as arm32 or i386 due to the fact that buildroots for i386 are not as up2date as the RHEL ones. This can cause builds all across the system to break so it is frowned on.

A couple of proposed fixes were:

  1.  Compile everything against CentOS. This has the bonus that they all kind of get updated at the same time so breakages from going from 7.3 to 7.4 is smaller. However, a lot of business users have either ITIL or security plans which allow for EPEL packages to be used on their systems because they were built against RHEL versus a rebuild. Remove that and it causes various problems for users.
  2. Split off the builds for alternative architectures in a shadow koji which is turned off when these splits occur. This makes it partially easier to manage but brings in other problems.
  3. ... some other ways to fix this that I don't know how to describe correctly...
However while we were hashing on 1, someone else who had been asking about why don't we build everything came up with a different mix. Build everything against CentOS for various architectures and put it in some other repository (Extra Packages in CentOS came to mind). Packages which people really want and are willing to care for are branched off into EPEL packages which are built against RHEL. [Or if they really want it they get a business case to Red Hat to make it a product they will pay for it.]

There is a lot of problems with the above.. it requires a lot of work in many different moving parts to get done. It also could be even more confusing than what we have already. However, I did say I was going to write up all the proposals for people to look at. 

EPEL Proposal #3: EPEL Structured releases

This started off as a a rawhide post, but I realized it needs to be its own post

A staged/forked environment. This version is a lot more complicated and deals with extra release management. Currently, Red Hat does regular point releases (.1, .2, .3) of their main OS. In these point releases, various items get major updates or backports to bring new functionality to the base operating system. In this version, EPEL ties into this staged environment by using a similar method as Fedora does.

Using a directory tree structure like:
EPEL/archives/{5.x, 6.x, 7.x}
EPEL/archives/updates/{5.x, 6.x, 7.x}
EPEL/current/5 -> 5.11/
EPEL/current/6 -> 6.7/
EPEL/current/7 -> 7.2/
EPEL/updates/5 -> 5.11/
EPEL/updates/6 -> 6.7/
EPEL/updates/7 -> 7.2/
EPEL/updates/testing/{blah, blah}
EPEL/rawhide/{5, 6, 7}

Then packages would be built in rawhide first. When a new release (say 7.2 to 7.3) occurs, then the builders rawhide trees are updated to the code in the latest. The packages are built against that and hopefully tested to see that they still work. Packagers should have been building any large changes they needed to make in the rawhide tree already but a last minute notification date is done. The go-live for this tree will be either when CentOS gets its build out of that tree or 30 days if CentOS is able to release sooner. On that date the rawhide trees will be branched to /EPEL/current/{6,7}.n (eg 7.3) and updates to those packages will now be sent into EPEL/updates/7.3 . After another 30 days, the 7.2 tree will be moved to EPEL/archives/7.2 {probably really /pub/archives/epel/7.2 }

Pros: this gives a place where newer packages can be built and tested to see what may break in the next major build. it also gives a better release cycle that people can plan against.

Cons: this makes the previous post look simple to implement.

EPEL Proposal #2: EPEL Rolling Rawhide

An EPEL rawhide may sound like an oxymoron for having an always building main tree for a release that is known for its plodding slowness. However there are methods to the madness as people brought it up at FOSDEM Brussels.

Rolling Release Style. In this version there is a branch where many Fedora packages are rebuilt against EL-5/6/7. The packages in this release are always the latest that can be compiled against the release and all 'updates' and fixes are done upstream versus in the package. The epel file tree would look something like this:


Packages in EPEL/{5,6,7} are ones that are going to be kept stable until they can no longer be supported. Packages in EPEL/testing/ are were updates to the mainline packages are kept. The general idea is that Fedora packagers can freely fork to EPEL/rawhide with no care about whether it works or not. If people are interested enough to help fix it they can send it to the packager but if it doesn't build and no one fixes it.. it isn't there. If people do care about the package, they can sponsor it and those would be the ones in EPEL regular. These packages would follow the agreed upon rules about updates and lack of big changes.

Pros: Allows for more packages to be newer for more users.
Cons: More channels for release engineering and packagers to deal with.

PSA: Enterprise Linux 5 End of Production on 2017-03-31 and EPEL.

Red Hat Enterprise Linux builds its current releases around a 4 part lifecycle model.

  • Production 1. New drivers and features are added during this cycle along with security and production bug fixes.
  • Production 2. Only security and production bug fixes are done during this cycle.
  • Production 3. Only security and 'critical' bug fixes are done.
  • Extended Lifecycle Support. Security and critical bug fixes are only released to customers who are paying for this extra level of support. For rebuild Linuxes like CentOS and Scientific Linux the product is now End of Lifed.

The last time this happened was in 2012 with the end of life of Enterprise Linux 4. At that time, EPEL stopped supporting builds for EL-4 but no other changes were done. This wasn't a big deal because EL-4 had never gotten a large number of users. However in 1 year, 1 month, EL-5 will reach the end of its production run and move into Extended Life Support. Enterprise 5 is still over 20% of all EPEL installs so I wanted to give a long heads up that EL-5 will also be removed from the builders on April 1 2017 and no builds will be done after that. Depending on other proposals, EL-5 may also be moved over to an 'archive' like Fedora releases are so that it is clear that it is no longer under any production. If that happens there will be clear notice of where it has been moved to and how to keep at it.


EPEL proposal #1: rechartering


So my time at FOSDEM was pretty much the following. Walk into the hall where the Fedora/CentOS tables were. Get met by someone who worked on or in EPEL. Try to have a conversation and find that it was too packed and loud there. Walk over to the coffee area and buy the person a coffee and listen. Finish conversation and walk back over to the table.. [take a break every now and then to use the restroom from drinking a 16 oz tea every 30 minutes.] This was a lot of conversations and they did all blur together after a while. [Using a technique I learned from Centos' Karanbir Singh at FOSDEM, I will bring a small notebook to the next conference and write down a problem per page per person. Then I can go back the next year and see if the problems are still there.]


While they all did blur, there were many similar questions from people that came up:

  • What are the promises that EPEL makes to its users?
  • Do those promises make sense? [Do we need to keep a package the same for 5+ years?]
  • Do we actually keep those promises?
  • Who thinks we should be keeping these promises that various people didn't feel were needed?
  • What promises can we make that make sense and that after ~8 years we feel we can reasonably keep?
  • Who do we need to make these promises to?
As I said in the previous post, a lot of people felt that we had made promises of fixing and updating things in EPEL over the last 3 years but nothing really seemed to have changed. There are a lot of reasons for that (as in people say things and after a 20 hour flight have no idea what they said to looking at the promise and going "ok that is fine but who is going to do the work?" to "are we even allowed to change this?"

The last point is to me an undercurrent of a lot of conversations in and around EPEL in the last 3 years. There is the feeling that we were chartered nearly 10 years ago to do certain things, and we should keep doing them because people depend on that. And it is true that some people do depend on it, but it seems increasingly that they either don't show up to help or are a smaller set of people than we realize. 

These questions and the problems above was a long discussion between Brian Stinson and me on Sunday evening. One of the circles that kept coming up was "Does EPEL need to be rechartered?" EPEL is a very old Fedora SIG, and it has gone through several incarnations in the last 8 years from when it had yearly elections for board members to when it ran without any leadership to its recent experimentation with a new steering committee. However during that time there have been many rules which it has tried to keep:
  1. Do not do disruptive upgrades. Try to backport security fixes or do upgrades which do not change API/ABI issues.
  2. Be prepared to support the same package for the life of an Enterprise Linux release.
  3. Never replace or conflict with packages in the base Enterprise Linux.
  4. They should never require manual updates or changes between updates.
  5. Follow Fedora rules for bundling, licensing, and similar things.
There are some promises some people feel like we have made also:
  1. Packages will never disappear. [They don't disappear from Fedora 12 even if it is archived.. ]
  2. Packages will work from EL-X.0 to EL-X.N. [EPEL will rebuild packages regularly to deal with any ABI items]
  3. Packages are built against all of an Enterprise Linux 'base' (EG whatever is in CentOS/Scientific Linux base).

In many cases those promises aren't kept now or when they are attempted to be kept are impossible to do.
  • Most software which uses a database back end requires schema updates ranging from dump/reload to long running change processes. 
  • A lot of software rebases itself internally to the point that trying to backport a security fix usually requires a lot of coding skill in the package.
  • The Enterprise Linux lifetime has increased from 5 years to 10 years since EPEL started. While people were interested in supporting a package for 3-4 years.. doing so for 10 years is too much for a volunteer position.
  • The Enterprise Linux is not the same as it was in EL-3/4/5 where software versions were rarely 'rebased' to a newer release. Instead software would be recoded to work in the older software as much as possible. Currently in EL-7, the desktop and other parts of the OS were greatly updated between 7.1 and 7.2 with the idea that this will be the norm for all releases in order to keep the product relevant to the majority of users. 
From the above, 3 of the 5 promises are either too much to keep or haven't been kept in a long time. 

What to Do?

Because we haven't been able to keep many promises, but feel like we should there seem to be a couple of options ahead:
  1. Ignore and do nothing. I don't like the status-quo but I think it needs to be listed as something that can happen.
  2. Try to keep the original charter and be a lot more picky about what is in EPEL so that we can meet those promises. 
  3. Figure out what we can do, and go to the appropriate Fedora body to recharter ourselves to meet those more reasonable goals. 
Now number 3 may seem to be busy work, but I have found for the conservative minded enterprise maintainer, it is very hard to give up doing something even if you are failing until you are told you can stop trying. It also makes it clearer to future people working on EPEL in 2024 what they are trying to solve and how to change when the world of 2024 needs different solutions.

What would be in my charter

  1. EPEL is run by the EPEL Steering Committee. They act as sub-equivalents to the Fedora Extras Steering Committee, Fedora Packaging Committee and Fedora Release Engineering. 
  2. Packages in EPEL will never replace or conflict with packages in the base Enterprise Linux.
  3. Packages in EPEL will follow Fedora rules for bundling, licensing, and similar things. Exceptions to this will exist and will be documented in appropriate place.
  4. EPEL release engineering can set up systems to rebase packages in regular intervals. [There are multiple ways of doing this that I will try to outline in other proposals this week.]
  5. The only promise of a lifetime of a package is between dot releases of the Enterprise Linux. Maintainers should make an honest effort to support a package for the "life" of a sub-release (6.8->6.9) but no longer. 
  6. Packages in a dot release will not disappear during that time. There is no promise after that time. [If EPSco approves of a method that packages are regularly archived, then users can use that as a simpler method to get old packages.]
  7. EPEL only supports the current dot release. If a package in EL-6.now works on 6.0.z that is great. If it doesn't, it is up to the user (and not the maintainer or EPEL) to deal with it. [Again if an archive method is setup, then those older versions would just be in something like /pub/archives/epel/epel-6.8/ or some similar setup]
Those are my starting points for a rechartered EPEL. Please join in the discussion on the EPEL-devel mailing list.