2019-01-16

NOTICE: Epylog has been retired for Fedora Rawhide/30

Epylog is a log analysis code written by Konstantin ("Icon") Ryabitsev, when he was working Duke University in the early 2000's. It was moved to FedoraHosted and then never got moved to other hosting afterwords. The code is written in early python2 syntax (maybe 2.2) and has been hacked to work with newer versions over time but has not seen any major development since 2008. I have been sort of looking after the package in Fedora with the hopes of a 'rewrite for Python3' that never got done by me. [This is on me as I have been licking the cookie here.]

Because it requires a lot of work, and Python 2's End of Life is coming up in a year, I retired it from rawhide so that it would not branch to Fedora 30. I would recommend that users of epylog look for newer replacements (we in Fedora infrastructure will be doing so and I will post any recommendations as time goes by).

2018-11-30

NOTICE: nagios-4.4.2 is heading towards updates

There are 2 CVE's for nagios which require an update to the latest version from the Nagios.com.




This is a major upgrade from 4.3 to 4.4, and will require extra testing (Karma has been made +4 versus +3). Other fixes seem to be a memory leak which had been seen in the 4.2 and 4.3 versions.

If you use nagios in Fedora or EPEL, please test and give karma to the builds:


2018-10-16

NOTICE: Major problem with nrpe-3.2.1-6 in EPEL-7

During the summer, I worked on updating nrpe to a newer version and made changes to the systemd startup to match the provided one. Part of this was adding PIDfile so that systemd could send signals and monitor the correct nrpe daemon as there had been bugs where systemctl was unable to restart the daemon.

I tested nrpe-3.2.1-6 on my systems and had no problems, and then put it in epel-testing for a couple of months waiting for some testing. This is where I made a mistake and forgot about it and also I did not thoroughly test nrpe updates from very old versions of nrpe. My tests of updates had been with more recent versions which had a line in the start up for


pid_file = /var/run/nrpe/nrpe.pid

which made sure that my tests worked fine. The daemon started up and it ran without problems, created the file in the correct place etc etc. However if you had a configuration management system with an older template for the file, or had touched your /etc/nagios/nrpe.cfg previously you have problems. yum update will fail to restart the nrpe and other errors will occur.

One fix would be to update the config file to the newer version in the 3.2.x series, but that is not going to work for a lot of people.

I have worked with Andrea Veri to work out a functional change which will allow for systemctl to work properly without needing the pid_file. This is by removing the PIDfile and making the startup a simple versus forking daemon. I have built nrpe-3.2.1-8 and it should show up in epel-testing in the next day or so.

Please, please test this and see if it works. If it really works (aka after 24 hours of an update it is still running, please add karma to it in

https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-7f7330f37a 

Thank you.

2018-07-25

NOTICE: EPEL/Fedora updates to nagios/nagios-plugins/nrpe

I have pushed out multiple updates to various nagios packages. They will be arriving in the various repository (f28, epel-7, epel-6) updates and in rawhide in the next 24 hours or so.


  1. nagios I tried making an updated package to 4.4.1 but our current spec file and patches need a lot more work than I currently have time for. I instead made minor changes to the 4.3.4 to deal with some permission errors and such.
  2. nrpe is mainly small fixes also but closes out some persistent bugzillas. 
  3. nagios-plugins is a fairly major update even though the version number has not changed. Many little fixes have been done in the upstream git tree's maint that needed to be groomed together so I have updated to that and marked the version so you can see which git commit it is. I have also fixed a FTBFS in rawhide and made openssl more granular so it is not getting added to every plugin.

Bodhi Links

These will be updated as I get them.

Nagios:

Nagios-plugins:

NRPE:

2018-07-18

When your software is used way after you EOL it.

One of my first jobs was working on a satellite project called ALEXIS at Los Alamos National Laboratory and had been part of a Congressional plan to explore making space missions faster and cheaper. This meant the project was a mix-mash of whatever computer systems were available at the time. Satellite tracking was planned on I think a Macintosh SE, the main uploads and capture were a combination of off the shelf hardware and a Sparc 10. Other analysis was done on spare Digital and SGI Irix systems. It was here I really learned a lot about system administration as each of those systems had their own 'quirks' and ways of doing things.

I worked on this for about a year as a Graduate Research Assistant, and learned a lot about how many projects in science and industrial controls get 'frozen' in place way longer than anyone writing the software expects. This is because at a certain point the device becomes cheaper to keep running than replace or even updating. So when I was watching this USGS video this morning,



I wasn't surprised to see old DEC computers with CRT screens intermixed with newer computers. The LANDSAT 7 was launched in 1999 when DEC no longer existed, but was designed in the early 1990's. The software for running specific hardware on the system was probably written on whatever system (I am guessing an Alpha but I am not sure). As long as that satellite is running, there will be some sort of team working to make sure that hardware has a giant box of spare parts and trying to make sure the software is still running.

Satellites may seem an extreme case, but the same goes for any large scientific studies and many things in the aerospace industry. You can still find inflight TV systems on major plane lines that will reboot themselves to some Red Hat Linux 7 logo.. an OS that was EOL over a decade ago. There are similar items in industrial controllers for making textiles, plastics, and other items.. the devices are large and expensive to replace so will run whatever software was in them for decades. They will also require software which interfaces with them to be 'locked' in place which can have a pile on effect where you find that you need to have some new computer system be able to run something written in Python 1.5.

I expect that a LOT of systems are currently written to work only with Python 2.7 and will be wanting software for it until the late 2030's. The problem is that very few of them are have plans or ability to pay for that maintenance support. While it is very late in the game, I would say that if you are relying on python for such a project, you need to start budgeting your 2020 and future budgets to take in account of paying some group to support those libraries somehow.

2018-05-16

Blue Sky Discussion: EPEL-next or EPIC

EPIC Planning Document

History / Background

Since 2007, Fedora Extra Packages for Enterprise Linux (EPEL) has been rebuilding Fedora Project Linux packages for Red Hat Enterprise Linux and its clones. Originally the goal was to compile packages that RHEL did not ship but were useful in the running of Fedora Infrastructure and other sites. Packages would be forked from the nearest Fedora release (Fedora 3 for EPEL-4, Fedora 6 for EPEL-5) with little updating or moving of packages in order to give similar lifetimes as the EL packages. Emphasis was made on back-porting fixes versus upgrading, and also not making large feature changes which would cause confusion. If a package could not longer be supported, it would be removed from the repository to eliminate security concerns. At the time RHEL lifetimes were thought to be only 5-6 years so back-porting did not look like a large problem.

As RHEL and its clones became more popular, Red Hat began to extend the lifetime of the Enterprise Linux releases from 6 years to 10 years of "active" support. This made trying to back-port fixes harder and many packages in EPEL would be "aged" out and removed. This in turn caused problems for consumers who had tied kick-starts and other scripts to having access to those packages. Attempts to fix this by pushing for release upgrade policies have run into resistance from packagers who find focusing on the main Fedora releases a full time job already and only build EPEL packages as one-offs. Other attempts to update policies have run into needing major updates and changes to build tools and scripting but no time to do so. Finally, because EPEL has not majorly changed in 10 years, conversations about changing fall into "well EPEL has always done it like this" from consumers, packagers, and engineering at different places.

In order to get around many of these resistance points with changing EPEL, I suggest that we frame the problems around a new project called Extra Packages for Inter Communities. The goal of this project would be to build packages from Fedora Project Linux releases to various Enterprise Linux whether they are Red Hat Enterprise Linux, CentOS, Scientific Linux or Oracle Enterprise Linux.

Problems and Proposals

Composer Limitations:

Problem:
Currently EPEL uses the Fedora build system to compose a release of packages every couple of days. Because each day creates a new compose, the only channels are the various architectures and a testing where future packages can be tested. Updates are not in a separate because EPEL does not track releases.
EPEL packagers currently have to support a package for the 10 year lifetime of an RHEL release. If they have to update a release, all older versions are no longer available. If they no longer want to support a package it is completely removed. While this sounds like it increases security of consumers, Fedora does not remove old packages from older releases.
Proposed Solution
EPIC will match the Enterprise Linux major/minor numbers for releases. This means that a set of packages will be built for say EL5 sub-release 11 (aka 5.11). Those packages would populate for each supported architecture a release, updates and updates-testing directory. This will allow for a set of packages to be composed when the sub-release occurs and then stay until the release is ended.
/pub/epic/releases/5/5.11/{x86_64,source,i386,aarch64,arm,ppc64}/
/pub/epic/updates/5/5.11/{x86_64,source,i386,aarch64,arm,ppc64}/
/pub/epic/updates/testing/5/5.11/{x86_64,source,i386,aarch64,arm,ppc64}/
/pub/epic/development/5/CR/

Once a minor release is done, the old tree will be hard linked to an appropriate archive directory.

/pub/archives/epic/releases/5/5.11/{x86_64,source,i386,aarch64,arm,ppc64}/
/pub/archives/epic/updates/5/5.11/{x86_64,source,i386,aarch64,arm,ppc64}/
/pub/archives/epic/updates/testing/5/5.11/{x86_64,source,i386,aarch64,arm,ppc64}/

A new one will be built and placed in appropriate sub directories. Hard links to the latest will point to the new one, and after some time the old-tree will be removed from the active directory tree.

Channel Limitations:

Problem
EPEL is built against a subset of channels that Red Hat Enterprise Linux has for customers, namely the Server, High Availability, Optional, and some sort of Extras. Effort is made to make sure that EPEL does not replace with newer packages anything in those channels. However this does not extend to packages which are in the Workstation, Desktop, and similar channels. This can cause problems where EPEL’s packages replace something in those channels.
Proposed Solution
EPIC will be built against the latest released CentOS minor release using the channels which are enabled by default in the CentOS-Base.repo. These packages are built from source code that Red Hat delivers via a git mechanism to the CentOS project in order to rebuild them for mass consumption. Packages will not be allowed to replace/update according to the standard RPM Name-Epoch-Version-Release (NEVR) mechanism. This will allow EPIC to actually service more clients

Build System Limitations

Problem
EPEL is built against Red Hat Enterprise Linux. Because these packages are not meant for general consumption, the Fedora Build-system does not import them but builds them similarly to a hidden build-root. This causes multiple problems:
  • If EPEL has a package with the same name, it supersedes the RHEL one even if the NEVR is newer. This means old packages may get built against and constant pruning needs to be done.
  • If the EPEL package has a newer NEVR, it will replace the RHEL one which may not be what the consumer intended. This may break other software requirements.
  • Because parts of the build are hidden the package build may not be as audit-able as some consumers would like.
Proposed Solution
EPIC will import into the build system the CentOS build it is building against. With this the build is not hidden from view. It also makes it easier to put in rules that an EPIC package will never replace/remove a core build package. Audits of how a build is done can be clearly shown.

Greater Frequency Rebasing

Problem
Red Hat Enterprise Linux have been split between competing customer needs. Customers wish to have some packages stay steady for 10 years with only some updates to them, but they have also found that they need rapidly updated software. In order to bridge this, recent RHEL releases have rebased many software packages during a minor release. This has caused problems because EPEL packages were built against older software ABI’s which no longer work with the latest RHEL. This requires the EPEL software to be rebased and rebuilt regularly. Conversely, because of how the Fedora build system sees Red Hat Enterprise Linux packages, it only knows about the latest packages. In the 2-4 weeks between various community rebuilds getting their minor release packages built, EPEL packages may be built against API’s which are not available.

Proposed Solution
The main EPIC releases will be built against specific CentOS releases versus the Continual Release (CR) channel. When the next RHEL minor is announced, the EPIC releng will create new git branch from the current minor version (aka 5.10 → 5.11). Packagers can then make major updates to versions or other needs done. When the CentOS CR is populated with the new rpms, CR will be turned on in koji and packages will be built in the new tree using those packages. After 2 weeks, the EPIC minor release will be frozen and any new packages or fixes will occur in the updates tree.

Guidelines

Packaging

EL-4

This release is no longer supported by CentOS and will not be supported by EPIC.

EL-5

This release is no longer supported by CentOS and will not be supported by EPIC.

EL-6

This release is supported until Nov 30 2020 (2020-11-30). The base packaging rules for any package would be those used by the Fedora Project during its 12 and 13 releases. Where possible, EPIC will make macros to keep packaging more in line with current packaging rules.

EL-7

This release is supported until Jun 30 2024 (2024-06-30). The base packaging rules for any package would be those used by the Fedora Project during its 18 and 19 releases. Because EL7 has seen major updates in certain core software, newer packaging rules from newer releases is possible to follow.

EL-next

Red Hat has not publicly announced what its next release will be, when it will be released, or what its lifetime is. When that occurs, it will be clearer which Fedora release packaging will be based off of.

GIT structure

Currently EPEL uses only 1 branch for every major RHEL release. In order to better match how current RHEL releases contain major differences, EPIC will have a branch for every major.minor release. This is to allow for people who need older versions for their usage to better snapshot and build their own software off of it. There are several naming patterns which need to be researched:

/<package_name>/epic/6/10/
/<package_name>/epic/6/11/
/<package_name>/epic/7/6/
/<package_name>/epic/7/7/
//epic-6/6.10/
/<package_name>/epic-6/6.11/
/<package_name>/epic-7/7.6/
/<package_name>/epic-7/7.7/

/<package_name>/epic-6.10/
/<package_name>/epic-6.11/
/<package_name>/epic-7.6/
/<package_name>/epic-7.7/
Git module patterns will need to match what upstream delivers for any future EL.

Continuous Integration (CI) Gating

EPIC-6

The EL-6 life-cycle is reaching its final sub releases with more focus and growth in EL-7 and the future. Because of this gating will be turned off EPIC-6. Testing of packages can be done at the packagers discretion but is not required.

EPIC-7

The EL-7 life-cycle is midstream with 1-2 more minor releases with major API changes. Due to this, it makes sense to research if gating can be put in place for the next minor release. If the time and energy to retrofit tools to the older EL are possible then it can be turned on.

EPIC-next

Because gating is built into current Fedora releases, there should be no problem with turning it on for a future release. Packages which do not pass testing will be blocked just as they will be in Fedora 29+ releases.

Modules

EPIC-6

Because EL-6’s tooling is locked at this point, it does not make sense to investigate modules.

EPIC-7

Currently EL-7 does not support Fedora modules and would require updates to yum, rpm and other tools in order to do so. If these show up in some form in a future minor release, then trees for modules can be created and builds done.

EPIC-next

The tooling for modules can match how Fedora approaches it. This means that rules for module inclusion will be similar to package inclusion. EPIC-next modules must not replace/conflict with CentOS modules. They may use their own name-space to offer newer versions than what is offered and those modules may be removed in the next minor release if CentOS offers them then.

Build/Update Policy

Major Release

In the past, Red Hat has released a public beta before it finalizes its next major version. If possible, the rebuilders have come out with their versions of this release in order to learn what gotchas they will have when the .0 release occurs. Once the packages for the beta are built, EPIC will make a public call for packages to be released to it. Because packagers may not want to support a beta or they know that there will be other problems, these packages will NOT be auto branched from Fedora.

Minor Release

The current method CentOS uses to build a minor release is to begin rebuilding packages, patching problems and then when ready put those packages in their /cr/ directory. These are then tested for by people while updates are built and ISOs for the final minor release is done. The steps for EPIC release engineering will be the following:
  1. Branch all current packages from X.Y to X.Y+1
  2. Make any Bugzilla updates needed
  3. Rebuild all branched packages against CR
  4. File FTBFS against any packages.
  5. Packagers will announce major updates to mailing list
  6. Packagers will build updates against CR.
  7. 2 weeks in, releng will cull any packages which are still FTBFS
  8. 2 weeks in, releng will compose and lock the X.Y+1 release
  9. symlinks will point to the new minor release.
  10. 4 weeks in, releng will finish archiving off the X.Y release

Between Releases

Updates and new packages between releases will be pushed to the appropriate /updates/X.Y/ tree. Packagers will be encouraged to only make minor non-api breaking updates during this time. Major changes are possible, but need to follow this work flow:
  1. Announce to the EPEL list that a change is required and why
  2. Open a ticket to EPIC steering committee on this change
  3. EPIC steering committee approves/disapproves change
  4. If approved change happens but packages are in updates
  5. If not approved it can be done next minor release.

Build System

Build in Fedora

Currently EPEL is built in Fedora using the Fedora Build system which integrates koji, bodhi, greenwave, other tools together. This could be still used with EPIC.

Build in CentOS

EPIC could be built in the CentOS BuildSystem (CBS) which also uses koji and has some integration to the CentOS Jenkins CI system.

Build in Cloud

Instead of using existing infrastructure, EPIC is built with newly stood up builders in Amazon or similar cloud environments. The reasoning behind this would be to see if other build systems can transition there eventually.

Definitions

Blue Sky Project
A project with a different name to help eliminate preconceptions with the existing project.
Customer
A person who pays for a service either in money, time or goods.
Consumer
Sometimes called a user. A person who is consuming the service without work put into it.
EPEL
Extra Packages for Enterprise Linux. A product name which was to be replaced years ago, but no one came up with a better one.
EPIC
Extra Packages Inter Community.
RHEL
Red Hat Enterprise Linux

Last updated 2018-05-16 19:10:17 EDT This document was imported from an adoc..

2018-05-11

EPEL Outage Report 2018-11-05

Problem Description:

On 2018-05-11 04:00 UTC reports started coming into centos IRC channels about EPEL being corrupted and causing breakages. These were then reported to #fedora-admin and #epel-devel. The problem would show up as something like:

 One of the configured repositories failed (Unknown),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

The problem was examined and turned out to be that an NFS problem on the backend systems causing the createrepo_c to create the repositories to create a corrupted SQL file. A program which was to catch this did not work for some reason still being investigated and the corrupted sqllite file was mirrored out.

Admins began filling up the #epel and #centos channel asking why their systems were broken. I would like to thank avij, tmz and others who worked on answering as many of the people as possible. I would also like to thank Kevin Fenzi for figuring out the problem, regenerating the builds and unstopping the NFS blockage.

Solution:

Because of the way mirroring works, this problem may affect clients for hours after the fix has been made on the server. There are three things a client can do:
  1. If you have a dedicated mirror, have the mirror update itself with the upstream mirrors.
  2. On client systems you may need to do a yum clean all in order to remove the bad sql in case yum thinks it is still good to cache from.
  3. You can skip yum on updates with:
    
    yum --disablerepo=epel update

Notes:

This will be filled out later as more information and future steps are taken.
  1. Mirrormanager did not have anything to do with this. It's job is to check that mirrors match the master site and in this case the master site was borked so it happily told people to go to mirrors which matched that.
  2. The problem showed up at 04:00 UTC because most servers are set up using GMT/UTC as their clock. At 04:00 the cron.daily starts up and many sites use a daily yum update which broke and mailed them.