EPEL Proposal #4: Build it all in CentOS promote to RHEL

So this one came up when we were going over problems with building alternative architectures in EPEL. One of the big problems is that currently everything is built against Red Hat Enterprise Linux. This has some bonuses for some people but it causes a problem when architectures are built against by CentOS but not Red Hat. [E.G. EL-7 for i386 and arm32] When a Red Hat release is done, the builders usually get updated as part of our repopull from the RHN servers. Usually there are some changes in added and subtracted packages which can cause packages to get new dependencies. While this is an inconvenience to customers.. it isn't a problem for the builders. When we start mixing distributions, it does cause a problem with the builders because a package may get compiled for EL-7 x86_64 and not be the same as arm32 or i386 due to the fact that buildroots for i386 are not as up2date as the RHEL ones. This can cause builds all across the system to break so it is frowned on.

A couple of proposed fixes were:

1.  Compile everything against CentOS. This has the bonus that they all kind of get updated at the same time so breakages from going from 7.3 to 7.4 is smaller. However, a lot of business users have either ITIL or security plans which allow for EPEL packages to be used on their systems because they were built against RHEL versus a rebuild. Remove that and it causes various problems for users.
2. Split off the builds for alternative architectures in a shadow koji which is turned off when these splits occur. This makes it partially easier to manage but brings in other problems.
3. ... some other ways to fix this that I don't know how to describe correctly...

However while we were hashing on 1, someone else who had been asking about why don't we build everything came up with a different mix. Build everything against CentOS for various architectures and put it in some other repository (Extra Packages in CentOS came to mind). Packages which people really want and are willing to care for are branched off into EPEL packages which are built against RHEL. [Or if they really want it they get a business case to Red Hat to make it a product they will pay for it.]

There is a lot of problems with the above.. it requires a lot of work in many different moving parts to get done. It also could be even more confusing than what we have already. However, I did say I was going to write up all the proposals for people to look at. 

EPEL Proposal #3: EPEL Structured releases

This started off as a a rawhide post, but I realized it needs to be its own post

A staged/forked environment. This version is a lot more complicated and deals with extra release management. Currently, Red Hat does regular point releases (.1, .2, .3) of their main OS. In these point releases, various items get major updates or backports to bring new functionality to the base operating system. In this version, EPEL ties into this staged environment by using a similar method as Fedora does.

Using a directory tree structure like:
EPEL/archives/{5.x, 6.x, 7.x}
EPEL/archives/updates/{5.x, 6.x, 7.x}
EPEL/current/5 -> 5.11/
EPEL/current/5.11/{x86_64,ppc64,x86_32}/
EPEL/current/6 -> 6.7/
EPEL/current/7 -> 7.2/
EPEL/updates/5 -> 5.11/
EPEL/updates/6 -> 6.7/
EPEL/updates/7 -> 7.2/
EPEL/updates/testing/{blah, blah}
EPEL/rawhide/{5, 6, 7}

Then packages would be built in rawhide first. When a new release (say 7.2 to 7.3) occurs, then the builders rawhide trees are updated to the code in the latest. The packages are built against that and hopefully tested to see that they still work. Packagers should have been building any large changes they needed to make in the rawhide tree already but a last minute notification date is done. The go-live for this tree will be either when CentOS gets its build out of that tree or 30 days if CentOS is able to release sooner. On that date the rawhide trees will be branched to /EPEL/current/{6,7}.n (eg 7.3) and updates to those packages will now be sent into EPEL/updates/7.3 . After another 30 days, the 7.2 tree will be moved to EPEL/archives/7.2 {probably really /pub/archives/epel/7.2 }

Pros: this gives a place where newer packages can be built and tested to see what may break in the next major build. it also gives a better release cycle that people can plan against.

Cons: this makes the previous post look simple to implement.

EPEL Proposal #2: EPEL Rolling Rawhide

An EPEL rawhide may sound like an oxymoron for having an always building main tree for a release that is known for its plodding slowness. However there are methods to the madness as people brought it up at FOSDEM Brussels.

Rolling Release Style. In this version there is a branch where many Fedora packages are rebuilt against EL-5/6/7. The packages in this release are always the latest that can be compiled against the release and all 'updates' and fixes are done upstream versus in the package. The epel file tree would look something like this:

EPEL/{5,6,7}
EPEL/testing/{5,6,7}
EPEL/rawhide/{5,6,7}

Packages in EPEL/{5,6,7} are ones that are going to be kept stable until they can no longer be supported. Packages in EPEL/testing/ are were updates to the mainline packages are kept. The general idea is that Fedora packagers can freely fork to EPEL/rawhide with no care about whether it works or not. If people are interested enough to help fix it they can send it to the packager but if it doesn't build and no one fixes it.. it isn't there. If people do care about the package, they can sponsor it and those would be the ones in EPEL regular. These packages would follow the agreed upon rules about updates and lack of big changes.

Pros: Allows for more packages to be newer for more users.
Cons: More channels for release engineering and packagers to deal with.

PSA: Enterprise Linux 5 End of Production on 2017-03-31 and EPEL.

Red Hat Enterprise Linux builds its current releases around a 4 part lifecycle model.

• Production 1. New drivers and features are added during this cycle along with security and production bug fixes.
• Production 2. Only security and production bug fixes are done during this cycle.
• Production 3. Only security and 'critical' bug fixes are done.
• Extended Lifecycle Support. Security and critical bug fixes are only released to customers who are paying for this extra level of support. For rebuild Linuxes like CentOS and Scientific Linux the product is now End of Lifed.

The last time this happened was in 2012 with the end of life of Enterprise Linux 4. At that time, EPEL stopped supporting builds for EL-4 but no other changes were done. This wasn't a big deal because EL-4 had never gotten a large number of users. However in 1 year, 1 month, EL-5 will reach the end of its production run and move into Extended Life Support. Enterprise 5 is still over 20% of all EPEL installs so I wanted to give a long heads up that EL-5 will also be removed from the builders on April 1 2017 and no builds will be done after that. Depending on other proposals, EL-5 may also be moved over to an 'archive' like Fedora releases are so that it is clear that it is no longer under any production. If that happens there will be clear notice of where it has been moved to and how to keep at it.

EPEL proposal #1: rechartering

Background

So my time at FOSDEM was pretty much the following. Walk into the hall where the Fedora/CentOS tables were. Get met by someone who worked on or in EPEL. Try to have a conversation and find that it was too packed and loud there. Walk over to the coffee area and buy the person a coffee and listen. Finish conversation and walk back over to the table.. [take a break every now and then to use the restroom from drinking a 16 oz tea every 30 minutes.] This was a lot of conversations and they did all blur together after a while. [Using a technique I learned from Centos' Karanbir Singh at FOSDEM, I will bring a small notebook to the next conference and write down a problem per page per person. Then I can go back the next year and see if the problems are still there.]

Promises

While they all did blur, there were many similar questions from people that came up:

• What are the promises that EPEL makes to its users?
• Do those promises make sense? [Do we need to keep a package the same for 5+ years?]
• Do we actually keep those promises?
• Who thinks we should be keeping these promises that various people didn't feel were needed?
• What promises can we make that make sense and that after ~8 years we feel we can reasonably keep?
• Who do we need to make these promises to?

As I said in the previous post, a lot of people felt that we had made promises of fixing and updating things in EPEL over the last 3 years but nothing really seemed to have changed. There are a lot of reasons for that (as in people say things and after a 20 hour flight have no idea what they said to looking at the promise and going "ok that is fine but who is going to do the work?" to "are we even allowed to change this?"

The last point is to me an undercurrent of a lot of conversations in and around EPEL in the last 3 years. There is the feeling that we were chartered nearly 10 years ago to do certain things, and we should keep doing them because people depend on that. And it is true that some people do depend on it, but it seems increasingly that they either don't show up to help or are a smaller set of people than we realize. 

These questions and the problems above was a long discussion between Brian Stinson and me on Sunday evening. One of the circles that kept coming up was "Does EPEL need to be rechartered?" EPEL is a very old Fedora SIG, and it has gone through several incarnations in the last 8 years from when it had yearly elections for board members to when it ran without any leadership to its recent experimentation with a new steering committee. However during that time there have been many rules which it has tried to keep:

1. Do not do disruptive upgrades. Try to backport security fixes or do upgrades which do not change API/ABI issues.
2. Be prepared to support the same package for the life of an Enterprise Linux release.
3. Never replace or conflict with packages in the base Enterprise Linux.
4. They should never require manual updates or changes between updates.
5. Follow Fedora rules for bundling, licensing, and similar things.

There are some promises some people feel like we have made also:

1. Packages will never disappear. [They don't disappear from Fedora 12 even if it is archived.. ]
2. Packages will work from EL-X.0 to EL-X.N. [EPEL will rebuild packages regularly to deal with any ABI items]
3. Packages are built against all of an Enterprise Linux 'base' (EG whatever is in CentOS/Scientific Linux base).

In many cases those promises aren't kept now or when they are attempted to be kept are impossible to do.

• Most software which uses a database back end requires schema updates ranging from dump/reload to long running change processes. 
• A lot of software rebases itself internally to the point that trying to backport a security fix usually requires a lot of coding skill in the package.
• The Enterprise Linux lifetime has increased from 5 years to 10 years since EPEL started. While people were interested in supporting a package for 3-4 years.. doing so for 10 years is too much for a volunteer position.
• The Enterprise Linux is not the same as it was in EL-3/4/5 where software versions were rarely 'rebased' to a newer release. Instead software would be recoded to work in the older software as much as possible. Currently in EL-7, the desktop and other parts of the OS were greatly updated between 7.1 and 7.2 with the idea that this will be the norm for all releases in order to keep the product relevant to the majority of users. 

From the above, 3 of the 5 promises are either too much to keep or haven't been kept in a long time. 

What to Do?

Because we haven't been able to keep many promises, but feel like we should there seem to be a couple of options ahead:

1. Ignore and do nothing. I don't like the status-quo but I think it needs to be listed as something that can happen.
2. Try to keep the original charter and be a lot more picky about what is in EPEL so that we can meet those promises. 
3. Figure out what we can do, and go to the appropriate Fedora body to recharter ourselves to meet those more reasonable goals. 

Now number 3 may seem to be busy work, but I have found for the conservative minded enterprise maintainer, it is very hard to give up doing something even if you are failing until you are told you can stop trying. It also makes it clearer to future people working on EPEL in 2024 what they are trying to solve and how to change when the world of 2024 needs different solutions.

What would be in my charter

1. EPEL is run by the EPEL Steering Committee. They act as sub-equivalents to the Fedora Extras Steering Committee, Fedora Packaging Committee and Fedora Release Engineering. 
2. Packages in EPEL will never replace or conflict with packages in the base Enterprise Linux.
3. Packages in EPEL will follow Fedora rules for bundling, licensing, and similar things. Exceptions to this will exist and will be documented in appropriate place.
4. EPEL release engineering can set up systems to rebase packages in regular intervals. [There are multiple ways of doing this that I will try to outline in other proposals this week.]
5. The only promise of a lifetime of a package is between dot releases of the Enterprise Linux. Maintainers should make an honest effort to support a package for the "life" of a sub-release (6.8->6.9) but no longer. 
6. Packages in a dot release will not disappear during that time. There is no promise after that time. [If EPSco approves of a method that packages are regularly archived, then users can use that as a simpler method to get old packages.]
7. EPEL only supports the current dot release. If a package in EL-6.now works on 6.0.z that is great. If it doesn't, it is up to the user (and not the maintainer or EPEL) to deal with it. [Again if an archive method is setup, then those older versions would just be in something like /pub/archives/epel/epel-6.8/ or some similar setup]

Those are my starting points for a rechartered EPEL. Please join in the discussion on the EPEL-devel mailing list. 

EPEL talks from CentOS-Brussels Dojo/FOSDEM and Configuration Management Camp EU

Apologies and So Forth

First, I would like to apologize for the delay in getting this post done. I really didn't realize the amount of energy the trip would take from me and how fuzzy brained I was for a week afterwords. Second, I would like to thank the Open Source and Standards (OSAS) group at Red Hat for sponsoring me that week. I believe I got a lot of things done and that we can work on getting Extra Packages for Enterprise Linux (EPEL) moving forward in some direction. Third I would like to thank everyone who took the time to talk to me at one of these places to tell me about what they were able to do with EPEL and what they were no longer able to use it for.

One of the items that people brought up a lot was that they felt conversations about fixing/growing/changing EPEL have become a broken record. Various things are said at many conferences by various people, but nothing ever gets done on this to the point that they feel they have been lied to. After succumbing to the fuzzy headed "what did I say yesterday?" for a week after flying back, I think that most of these broken promises have come up from jet-lag and people overload. If it hadn't been that I had people from IBM, CERN, and various other groups bring this up over and over again.. I may have forgotten just as well. In any case, I am going to try and outline every item that I wrote down on many pages of notes in this blog. If I have forgotten some point that you wanted me to bring up, please send me an email so that I can make sure it is dealt with.

Things EPEL is doing well

As can be seen in the following graph, EPEL has a large audience and a continually growing one. 

EPEL Growth from 2007 til 2016-01

The primary user of EPEL packages are on Red Hat Enterprise Linux (or a clone like CentOS or Scientific Linux) version 6 with a slowly shrinking set of EL-5 users. The audiences that use EPEL are very diverse. Everything from large organizations with strict audit controls to fast moving startups which have to work with the large organizations as their customers to the university trying to do some experiments to match things done in other organizations. 

Many users brought up the fact that the win of EPEL is that it is a low energy move to get packages from Fedora rebuilt for an Enterprise environment.  The fact that the package spec files had been vetted and tested in Fedora made for less chances of conflicts or configuration problems that crept into their own spec files or ones they got off the internet from various groups.  

Things EPEL is not doing well in

This is going to be a rather long list of items that came up. Some of them conflict with each other but that is mainly due to the fact that the customers are so diverse.

• The packaging guidelines for each EPEL version are not clear. This is mainly due to the fact that they are usually based off of older versions of Fedora (Fedora-6 for EPEL-5, Fedora-12 for EPEL-6, and Fedora-18 for EPEL-7) that may conflict with each other or with how things are being done in current EPEL. 
• EPEL does not have a regular release structure like Fedora and RHEL have. There isn't an EPEL-5.11 channel with an epel-updates-5.11 where updates are available. Because of various repository limitations, this means that directories aren't able to keep multiple old copies so downgrades when things do break aren't easy. 
• EPEL promises to keep things stable and only update for fixes, but this is only done on a few packages where others get upgraded to and fro. There does not seem to be much "steering" or "release engineering" of what is in the trees. 
• EPEL only covers part of Enterprise Linux (the Server product) but a lot of packages are for the Workstation but there is no way to see when things replace/conflict with them. [People believe that we build against the equivalent of CentOS-5/6/7 versus a subchannel.]
• EPEL sometimes has weird breaks between releases. The git in EPEL-5 is newer than what is in EL-6 in a way that was breaking repositories when pushes were done from EL-5 systems.  [People believe there is a promise that such changes are tested against.]
• EPEL packages disappear. If there is no maintainer packages are retired but if someone is re-building out the EL-5 hosts.. they need that package to be available.  [People believe there is a promise that packages will be always available.]
• From various people who were (or former) EPEL maintainers: packages in EPEL are the most complained about. If they can't update the package in EPEL, they get complaints about it being too old and they may end up having a newer version available for what they need to do anyway. If they do update the version, they get complaints that they broke someone because they needed some old version. Most of the complaints usually are the worst kinds ( off-hand death threats (why don't you die in a fire) etc etc). 

Things people wanted in EPEL

There were various things that people were wanting in EPEL that weren't really breakages because they don't exist yet.

• Enable 'alternative' architectures. There were requests from people about CentOS-i386 and CentOS-arm (for the Raspberry Pi2 in schools) with no vision on how those would be enabled. 
• Enable more packages with shorter lifetimes. One of the things that multiple sites were doing was rebuilding all of a Fedora release for their internal mirrors to supplement all of the things that EPEL didn't have in it. Why can't there be an EPEL-rawhide where all of these packages are built but no 'promises'?
• Is it possible to have a branch management system where packages get updated regularly? Say either whenever Fedora moves to a new release or Red Hat updates to the next branch (6.7->6.8, 7.2->7.3)
• Could packages in EPEL be tested in the CentOS continuous integration infrastructure as part of the autokarma testing?

Conclusions

I don't think anything above is new to people who have been contributing to EPEL in the last ~10 years. A lot of the problems are ones that were brought up in the beginning as we tried to square the circle of differing use cases. However, I wanted to catalogue them here and then make a promise that I will do my best to figure out ways to solve them by FOSDEM 2017 in some form or another. 

As I said at the beginning of the post, I believe that people had other complaints and suggestions. If I forgot or miswrote, my apologies and I will correct.