EPEL-8 Production Layout
TL; DR:
- EPEL-8 will have a multi-phase roll-out into production.
- EPEL-8.0 will build using existing grobisplitter in order to use a ‘flattened’ build system without modules.
- EPEL-8.1 will start in staging without grobisplitter and using default modules via mock.
- The staging work will allow for continual development changes in koji, ‘ursa-prime’, and MBS functionality to work without breaking Fedora 31 or initial EPEL-8.0 builds.
- EPEL-8.1 will look to be ready by November 2019 after Fedora 31 around the time that RHEL-8.1 may release (if it uses a 6 month cadence.)
Multi-phase roll-out
As documented elsewhere,
EPEL-8 has been slowly rolling out due to the many changes in RHEL and
in the Fedora build system since EPEL-7 was initiated in 2014. Trying to
roll out an EPEL-8 which was ‘final’ and thus the way it always will be
was too prone to failure as we find we have to constantly change plans
to match reality.
We
will be rolling out EPEL-8 in a multi-phase release cycle. Each cycle
will allow for hopefully greater functionality for developers and
consumers. On the flip side, we will find that we have to change
expectations of what can and can not be delivered inside of EPEL over
that time.
Phases:
- 8.0 will be a ‘minimal viability’. Due to un-shipped development libraries and the lack of building replacement modules, not all packages will be able to build. Instead only non-modular RPMs which can rely on only ‘default’ modules will work. Packages must also only rely on what is shipped in RHEL-8.0 BaseOS/AppStream/CodeReadyBuilder channels versus any ‘unshipped -devel’ packages.
- 8.1 will add on ‘minimal modularity’. Instead of using a flattened build system, we will look at updating koji to have basic knowledge of modularity, use a tool to tag in packages from modules as needed, and possibly add in the Module Build System (MBS) in order to ship modules.
- 8.2 will finish adding in the Module Build System and will enable gating and CI into the workflow so that packages can tested faster.
Due to the
fact that the phases will change how EPEL is produced, there may be need
to be mass rebuilds between each one. There will also be changes in
policies about what packages are allowed to be in EPEL and how they
would be allowed.
Problems with koji, modules and mock
If you are wanting to build packages in mock, you can set up a lot of controls in
/etc/mock/foo.cfg
which will turn on and off modules as needed so that you can enable the javapackages-tools
or virt-devel
module so that packages like libssh2-devel
or javapackages-local
are available. However koji does not allow this control per channel
because it is meant to completely control what packages are brought into
a buildroot. Every build records what packages were used to build an
artifact and koji will create a special mock config file to pull in
those items. This allows for a high level of auditability and
confirmation that the package stored is the package built, and that what
was built used certain things.
For
building an operating system like Fedora or Red Hat Enterprise Linux
(RHEL), this works great because you can show how things were done 2-3
years later when trying to debug something else. However when koji does
not ‘own’ the life-cycle of packages this becomes problematic. In
building EPEL, the RHEL packages are given to the buildroot via external
repositories. This means that koji does not fully know the life-cycle of
the packages it ‘pulls’ in to the buildroot. In a basic mode it will
choose packages it has built/knows about first, then packages from the
buildroot, and if there is a conflict from external packages will try to
choose the one with the highest
epoch-version-release-timestamp
so that only the newest version is in. (If the timestamp is the same, it tends to refuse to use both packages).
An improvement to this was adding code to
mergerepo
which allows for dnf to make a choice on which packages to use between
repositories. This allows for mock’s dnf to pull in modules without the
repositories having been mangled or ‘flattened’ as with grobisplitter.
However, it is not a complete story. For DNF to know which modules to
pull in it needs to set an environment variable for the platform (for
fedora releases it is something like f30 and for RHEL it is el8). Koji
doesn’t know how to do this so the solution would be to set it in the
build systems /etc/mock/site-defaults.cfg
but that would affect all builds and would cause problems for building Fedora on the same build system.Grobisplitter
A
second initiative to deal with building with modules was to try and
take modules out of the equation completely. Since a module is a virtual
repository embedded in a real one, you should be able to pull them
apart and make new ones. Grobisplitter
was designed to do this to help get CentOS-8 ready and also allow for
EPEL to bootstrap using a minimal buildset. While working on this, we
found that we needed also parts of the ‘–bare’ koji work because certain
python packages have the same src.rpm name-version but different
releases which koji would kick out.
Currently
grobisplitter does not put in any information about the module it
‘spat’ out. This will affect building when dnf starts seeing metadata in
individual rpms which says ‘this is part of a module and needs to be
installed as such’.
Production plans
We are trying to determine which tool will work better long term in order to make EPEL-8.0 and EPEL-8.1 work.
EPEL-8.0
Start Date | End Date | Work Planned | Party Involved |
---|---|---|---|
2019-07-01 | 2019-07-05 | Lessons Learned | Smoogen, Mohan |
2019-07-01 | 2019-07-05 | Documentation | Smoogen |
2019-07-08 | 2019-07-12 | Release Build work | Mohan, Fenzi |
2019-07-08 | 2019-07-12 | Call for packages | Smoogen |
2019-07-15 | 2019-07-19 | Initial branching | Mohan, Dawson |
2019-07-22 | 2019-07-31 | First branch/test | Dawson, et al |
2019-08-01 | 2019-08-01 | EPEL-8.0 GA | EPEL Steering Committee |
2019-08-01 | 2019-08-08 | Lessons Learned | Smoogen, et al |
2019-08-01 | 2019-08-08 | Revise documentation | Smoogen, et al |
2019-09-01 | 2019-09-01 | Bodhi gating turned on | Mohan |
EPEL-8.0 Production Breakout
- Lessons Learned. Document the steps and lessons learned from the previous time frame. Because previous EPEL spin-ups have been done multiple years apart, what was known is forgotten and has to be relearned. By capturing it, we hope that EPEL-9 does not take as long.
- Documentation. Write documents on what was done to set up the environment and what is expected in the next section (how to branch to EPEL-8, how to build with EPEL-8, dealing with unshipped packages, updated FAQ)
- Call for Packages This will be going over the steps that packagers need to follow to get packages branched to EPEL-8.
- Release Build Work. This is setting up the builders and environment in production. Most of the steps should be repeats of what was done in staging with additional work done in bodhi to have signing and composes work
- Initial Branching. This where the first set of packages are needed to be branched and built for EPEL-8: epel-release, epel-rpm-macros, fedpkg-minimal, fedpkg (and all the things needed for it).
- First Branch Going over the various tickets for EPEL-8 packages, a reasonable sample will be branched. Work will be done with the packagers on problems they find. This will continue as needed.
- EPEL-8.0 GA Branching can follow normal processes to get done.
- Lessons Learned. Go over problems and feed into other groups backlogs.
- Documentation Update previous documents and add any that were found to be needed.
EPEL-8.1
Start Date | End Date | Work Planned | Party Involved |
---|---|---|---|
2019-07-01 | 2019-07-05 | Lessons Learned | Fenzi, Contyk, et al |
2019-07 | ??? | Groom Koji changes needed | ??? |
2019-07 | ??? | Write/Test Koji changes needed | ??? |
2019-07 | ??? | Non-modular RPM in staging | ??? |
2019-07 | ??? | MBS in staging | ??? |
2019-08? | ??? | Implement Koji changes? | ??? |
2019-08? | ??? | Implement bodhi compose in staging? | ??? |
2019-09? | ??? | Close off 8.1 beta | ??? |
2019-09? | ??? | Lessons learned | ??? |
2019-09? | ??? | Begin changes in prod? | ??? |
2019-10? | ??? | Open module builds in EPEL | ??? |
2019-11? | ??? | EPEL-8.1 GA | EPEL Steering Committee |
2019-11? | ??? | Lessons Learned | ??? |
2019-11? | ??? | Revise documentation | ??? |
EPEL-8.1 Production Breakout
This
follows the staging and production of the 8.0 with additional work in
order to make working with modules work in builds. Most of these dates
and layers need to be filled out in future meetings. The main work will
be adding in allowing a program code-named ‘Ursa-Prime’ to help build
non-modular rpms using modules as dependencies. This will allow for
grobisplitter to be replaced with a program that has long term maintenan