EPEL Proposal: EPEL Master branch AKA Rawhide

EPEL-rawhide

Update:

This proposal has been superseded by Stephen Gallagher's excellent wagontrain post. I will put it as a separate post next.

tl; dr:

In order to allow for the ability for faster availability of packages, add rawhide branches for EPEL-7 and EPEL-8. These branches would allow developers to build new packages they aren't sure are ready for either EPEL-N or EPEL-N-testing, and would allow for faster rebuilds of newer features when RHEL has a large feature change.

The Longer Story

In the past 6 months, EPEL has had to have two major changes in its builds which were made harder by the way EPEL is currently built. The first one was with changes in RHEL-7.6 which dropped some packages and changed some others ABI's. This required a rebuild of a lot of packages, but there was no way we could do a find and fix before we did a 'flag-week' of rebuilds with Troy Dawson and others doing lots of Proven Packager fixes and rebuilds.

The second one was with the python36 move which also took a large amount of time and still has little problems showing up here and there. In a similar fashion, updates-testing had to be used as a rawhide for packages which made building and testing hard for things not doing this change.

A third problem showed up when Troy was cleaning out packages in EPEL-6 and 7 testing repos which had been there for years. The packagers were using this for putting things they felt were too unstable for EPEL due to unstable API's so they could either iterate quicker or not break existing users. The problem is that these packages might accidentally get  promoted by someone seeing that the packages are tested but wasn't pushed. Having a separate tree for these unstable packages needed a different thinking.

While doing a review of these two exercises, the EPEL steering committee came up with various ideas.. and I believe Kevin Fenzi brought up adding a rawhide as an easier fix than some of my more convoluted branch every release (aka epel-7.6, epel-7.7, epel-7.8). In this new scheme, we would have the following branches: el6, epel7, epel7-master, epel8, epel8-master.

A possible work flow could be the following:

1. Packages when branched for EL7 or EL8 would get branched into the epel-M-master tree where they could have builds made against the latest RHEL. 
2. When Red Hat released a new beta (RHEL-M.N-beta), Fedora Infrastructure would download it and set it up so koji could find it. as EPEL-M-Master (or properly bikeshed name). A mass update and rebuild would then be done against all packages in EPEL-M-Master. Breakfixes and testing can be done.
3. When the General Availability of RHEL-M.N occurs, EPEL will make a copy of EPEL-M.(N-1), EPEL-M.(N-1)-updates and EPEL-M.(N-1)-updates-testing in /pub/archive/epel/M/M.(N-1)/.
4. An after Red Hat releases the General Availability of the RHEL-M.N release, 
1. if the version in master is newer than the version in branch, the master version will be checked into the branch. (This step is probably the most problematic and needs more work and thinking by people).
2. packages which meet certain criteria will then be promoted to EPEL-M with a new compose of EPEL-M.N and an empty EPEL-M.N/updates and EPEL-M.N/updates-testing.
5. The packager can do updates and fixes to packages in the EPEL-M branch 
6. The finishing up of clean up the archives can occur.

This is a preliminary proposal which needs a lot more work and resource commitments in changes to tooling and documentation. I am bringing this up as something I would like to get done as part of revamping EPEL this summer, but I also need feedback and help.