2019-05-30

EPEL Proposal: Steve Gallagher's EPEL 8 Branch Strategy

Stephen Gallagher's Better Proposal for EPEL branching

So earlier this week I wrote up a proposal for EPEL-rawhide which was to go over various ideas the EPEL steering committee has been kicking around for a bit. This was to try and work into how to branch for EPEL-8 and also how to deal with https://fedoraproject.org/wiki/EPEL/Changes/Minor_release_based_composes and https://fedoraproject.org/wiki/EPEL/Changes/Release_based_package_lifetimes During the meeting it was clear that my strawman didn't have much in it, and needed more thinking. Thankfully Stephen Gallagher looked into the meeting and came up with some ideas that he wrote up and proposed to the list.. I recommend that you read the document and updates if you are interested in how branching in EPEL could work with EL7 and EL8.





2019-05-28

EPEL Proposal: EPEL Master branch AKA Rawhide

EPEL-rawhide

Update:

This proposal has been superseded by Stephen Gallagher's excellent wagontrain post. I will put it as a separate post next.

tl; dr:

In order to allow for the ability for faster availability of packages, add rawhide branches for EPEL-7 and EPEL-8. These branches would allow developers to build new packages they aren't sure are ready for either EPEL-N or EPEL-N-testing, and would allow for faster rebuilds of newer features when RHEL has a large feature change.

The Longer Story

In the past 6 months, EPEL has had to have two major changes in its builds which were made harder by the way EPEL is currently built. The first one was with changes in RHEL-7.6 which dropped some packages and changed some others ABI's. This required a rebuild of a lot of packages, but there was no way we could do a find and fix before we did a 'flag-week' of rebuilds with Troy Dawson and others doing lots of Proven Packager fixes and rebuilds.

The second one was with the python36 move which also took a large amount of time and still has little problems showing up here and there. In a similar fashion, updates-testing had to be used as a rawhide for packages which made building and testing hard for things not doing this change.

A third problem showed up when Troy was cleaning out packages in EPEL-6 and 7 testing repos which had been there for years. The packagers were using this for putting things they felt were too unstable for EPEL due to unstable API's so they could either iterate quicker or not break existing users. The problem is that these packages might accidentally get  promoted by someone seeing that the packages are tested but wasn't pushed. Having a separate tree for these unstable packages needed a different thinking.

While doing a review of these two exercises, the EPEL steering committee came up with various ideas.. and I believe Kevin Fenzi brought up adding a rawhide as an easier fix than some of my more convoluted branch every release (aka epel-7.6, epel-7.7, epel-7.8). In this new scheme, we would have the following branches: el6, epel7, epel7-master, epel8, epel8-master.

A possible work flow could be the following:
  1. Packages when branched for EL7 or EL8 would get branched into the epel-M-master tree where they could have builds made against the latest RHEL. 
  2. When Red Hat released a new beta (RHEL-M.N-beta), Fedora Infrastructure would download it and set it up so koji could find it. as EPEL-M-Master (or properly bikeshed name). A mass update and rebuild would then be done against all packages in EPEL-M-Master. Breakfixes and testing can be done.
  3. When the General Availability of RHEL-M.N occurs, EPEL will make a copy of EPEL-M.(N-1), EPEL-M.(N-1)-updates and EPEL-M.(N-1)-updates-testing in /pub/archive/epel/M/M.(N-1)/.
  4. An after Red Hat releases the General Availability of the RHEL-M.N release, 
    1. if the version in master is newer than the version in branch, the master version will be checked into the branch. (This step is probably the most problematic and needs more work and thinking by people).
    2. packages which meet certain criteria will then be promoted to EPEL-M with a new compose of EPEL-M.N and an empty EPEL-M.N/updates and EPEL-M.N/updates-testing.
  5. The packager can do updates and fixes to packages in the EPEL-M branch 
  6. The finishing up of clean up the archives can occur.
This is a preliminary proposal which needs a lot more work and resource commitments in changes to tooling and documentation. I am bringing this up as something I would like to get done as part of revamping EPEL this summer, but I also need feedback and help.

EPEL Proposal: Removal of PPC64 (Not PPC64le) in 2019-06-01

TL; DR:

EPEL is looking to put its EL6 and EL7 branches of PPC64 into archives by 2019-06-01. This is due to the fact that Fedora no longer builds for the PPC64 big-endian architecture.

The long story

As of the EOL of Fedora 28, the Fedora Project no longer supports or builds packages for the big endian Power64 (or ppc64) architecture. Kevin Fenzi went over this in his blog article, but I wanted to go over it again. I realize this is short notice so extra steps need to be done.

The Fedora Project uses Fedora Linux on its builders which is useful for bringing on new architectures, and for getting new features which RHEL does not have yet. However it means that when an OS is End of Lifed, it no longer gets security updates, software improvements, or similar fixes. We could try and stand up an EL7 builder but it would require reworking both tools and scripts that are expecting an F28 world (python3, various newer libraries and scripts, different API's, etc). That would take a while to rework everything back and then continual work of keeping this builder in line with whatever EL8/F30+ world we move to in the coming months. Secondly, this would cut out a limited resource. We only have so many PPC8 systems which we can run PPC64 virtual machines on. The virtual machines can either build an EPEL package or a Fedora <29 be="" but="" down="" epel.="" just="" limiting="" p="" package="" this="" to="" we="" would="">
In the end, the number of PPC64 users are not that great. We have an average of 90 systems per day checking in with many more PPC64LE systems. I think most of the PPC64 users would be able to get stuff from archives just as well.

How do I get my stuff

The builds for EL6.10 and EL7.6 will be archived to /pub/archives/epel/7/7.6 and /pub/archives/epel/6/6.10 this week. We may need to roll out an updated epel-release which will point this architecture to that tree. We will then remove the builders from Fedora and stop building for it. In early July I will remove the remaining trees from /pub/epel and put in redirects to the archives.