How to archive a local copy of CentOS 7

How to archive a local copy of CentOS 7

How to archive a local copy of CentOS 7

This post is meant for the various junior system administrators who have been tasked with fixing problems with CentOS 7 systems after the software was removed from most of its mirrors. Most of these systems have probably been running fine for a decade, and now possible critical systems are generating failed cron jobs or other errors.

What happened?

On July 1st, 2024, CentOS Enterprise Linux 7 reached its end of life as its upstream, Red Hat Enterprise Linux 7, had moved to Extended Life(cycle) Support. As with previous releases, the operating system was moved from the main mirrors to the CentOS vault, and the mirrorlists were turned off. Once the software was removed from the main mirrors, many secondary mirrors also removed the software as rsync and similar scripts would see the old software was gone. At this point, some unknown number of CentOS Linux systems ended up ‘non-supported.’

The systems may have been extremely low maintenance for years running whatever tasks they had been without a problem. The people who initially set them up have probably moved to other jobs, and some new person is now finding out that things are broken. Maybe its a cron job which runs one a week to run updates, or the kickstart used to reinstall an old server now breaks. In most of these cases, there is little documentation on what is being used, why it is being used, or how big a problem this is going to be.

What is needed to be done?

Getting an infrastructure out of this place is really out of scope of a single blog post. It generally requires getting various levels of managements attention, and then long term planning on how to transform an infrastructure into something more manageable. However in the short term a site can make things workable by making a local mirror of content from an upstream vault.

The reason to use a local vault is that the existing upstream vaults are limited in bandwidth and scope. Plus as more sites try to use them, the services may be curtailed or removed. When dealing with ‘End of Life’ projects and software, it is better to assume that things will get worse before they get better.

Hardware and software requirements

In order to mirror CentOS 7 locally, you are going to need to set up a webserver with at least 500 GB of free space (if you don’t want to copy the out of date ‘atomic’ trees. ) The amount of memory and cpu cores needed is dependent on the number of servers you are going to be supporting. The more systems, the more memory and cores that might be needed. In any case, I was able to set up a system with 2 cores and 4 GB of ram to support 4 EL7 systems without problems.

Internet requirements

There are currently 3 major mirrors of the CentOS vault.

  • archive.kernel.org
  • linuxsoft.cern.ch
  • mirror.nsc.liu

It is best to find one which is ‘network’ close and set up scripts to rsync data from the site. I found that each server will be busier at different times, so expect that copying will take multiple hours.

Sample Rsync script

The script I used to do this was the following:



RSYNC_OPTS='-avSHP --delete-excluded'

## Mirror CentOS 7
mkdir -vp ${TREEDIR}/centos-vault/7.9/
rsync ${RSYNC_OPTS} ${EXCLUDE_ITEMS}  ${VAULT}/7.9.2009/ ${TREEDIR}/centos-vault/7.9/

Sample HTTPD config

In my /etc/httpd/conf.d/ I added the following config file:

Alias "/mirrors" "/srv/mirrors"
<Directory /srv/mirrors>
  AllowOverride None
  Require all granted
  Options +Indexes

Sample Yum repo config

Finally on the EL7 systems, I used a config like the following in /etc/yum.repos.d/CentOS-EOL.repo

name=CentOS-$releasever - Base

#released updates
name=CentOS-$releasever - Updates

#additional packages that may be useful
name=CentOS-$releasever - Extras

name=CentOS-$releasever - Plus

At this point, updates were possible and I was able to reinstall a system in order to rebuild some packages I needed. Similar work can be done to set up mirrors of CentOS Linux 6 or third party repositories like EPEL.


What happens to EPEL-7 when EL-7 goes EOL.

What happens to EPEL-7 when EL-7 goes EOL

What is happening to RHEL-7

The Red Hat Enterprise Linux release cycle is fairly straight forward with releases lasting about 10 years after their initial release.

RHEL ver Release Date EoMaint EoExtended
5 2007-03-15 2017-03-31 2020-11-30
6 2010-11-10 2020-11-30 2024-06-30
7 2014-06-10 2024-06-30 2028-06-30
8 2019-05-07 2029-05-?? 2031-05-??
9 2022-11-08 2032-05-?? 2034-05-??
10 2025-??-?? 2035-??-?? 2037-??-??

On June 30, 2024, Red Hat will stop doing general maintenance support of RHEL-7 and no more updates to that operating system will be available without purchasing ‘Extended Life-cycle Support’ contracts from Red Hat or similar contracts from SuSE’s Liberty Linux, Perforce’s OpenLogic, or various other consultants and companies offering various services.

What is happening to EPEL-7

The Fedora Extra Packages for Enterprise Linux (EPEL) is a repository of additional packages built against the Red Hat Enterprise Linux (RHEL) releases. When a RHEL release leaves maintenance mode and moves into ‘Extended Life-cycle Support’, the EPEL package set corresponding to that is closed and archived. No new builds or updates are done against that particular release, all bugs related to that release are closed as CANTFIX, and most mirrors will remove the content after the master mirror moves items to archives.

The usual process for closing out an EPEL release depends on how much work the central Fedora release engineering team have to deal with at the time of the RHEL EOL. In cases where a Fedora release is needing a high amount of focus, the only actions may be that new builds will be turned off on the day of the EOL, and then later other steps will be worked out. In other cases, all the actions occur fairly close together.

  1. A reminder email that the EOL is going to occur is sent to various Fedora mailing lists like: devel-announce@lists.fedoraproject.org and epel-announce@lists.fedoraproject.org. System administrators using EPEL should be following at least the second email address to see what issues may be occurring with the repository.
  2. The Fedora Build system is a complicated tool-set and so Release Engineering have to disable the building and composing of that particular EPEL release in multiple places. Because the release cycle is so long for EPEL compared to Fedora releases, there may be inconsistencies in how things were named or done. Basically it is a slower process than with Fedora releases to make sure that the tools continue to work after an EOL.
  3. All bugzilla.redhat.com bugs related to that release will be closed with a short stock answer tailored for that release as ‘CANTFIX’.
  4. The release is now copied into the Fedora down-loaders /pub/archive/epel tree with an appropriate name like /pub/archive/epel/7.9.2024-07-01 and then symlinks are made. A week is given for the archive mirrors to catch up with this new directory tree.
  5. The Fedora mirrormanager will be changed to point requests for ‘repo=epel-7’ to the mirrors which have /pub/archive.
  6. The content then will be removed from the main mirror locations of /pub/epel/ and mirrors who rsync will catch up to that in a day or so.

Usually about a week to a month after this happens, either the mailing lists or IRC will get people wondering where the content went. However in the current age of Continuous Integration and Container builds, changes in the EPEL-7 directory structure seem to cause reactions hours after they occur.

How to deal with this EOL.

The main advice I have for system administrators dealing with an End of Life operating system is the following:

  1. Mirror the content locally from whatever mirrors you can find. This lowers network bandwidth usage, and deals with the vagaries of archive sites removing content because they no longer can handle the load from large numbers of unexpected users.
  2. Work with your management on how to deal with the technical load that these systems are going to cause.
    1. Budget out what extended maintenance costs for critical security fixes to services like web-servers, databases, remote logins, etc are going to take.
    2. Budget out what mitigations are going to be required to keep these systems from causing longer term headaches. This may be extra firewalls or additional staff needed to take over software maintenance.
    3. Budget out what replacements of the hardware and software to move to an operating system which has long term maintenance support.
    4. Actually this is a very long list and probably should be its own blog post.

End Thoughts

Dealing with End of Life software is rarely an easy task. Just realize that you aren’t alone and there are tens of millions of systems which are going to be ‘unsupported’ on July 1st. Be patient with yourself and your infrastructure as you deal with this. Good luck.


Updates on the CentOS 7 EOL: 2024-06-30

I am writing this on 2024-06-20 which means according to the IRC centbot:

  centbot: CentOS 7 will go EOL on 30 June, 2024 -- in 1 week,
 10 hours, 22 minutes, and 41 seconds.

The CentOS Systems Administrator, Fabian Arrotin, has posted several updates about what to expect starting Monday Jul 1st, 2024. The first email covers what services will stop after the 'End of Life' occurs:

  1. mirrorlist.centos.org will be decommissioned. This will affect any system trying to do a `yum update` or similar command after that date.
  2. forums.centos.org
  3. will be turned off and decommissioned. This was a volunteer effort by various system administrators who are ending their work with CentOS at the end of 7.
  4. torrent.centos.org
  5. will be turned off and decommissioned. This was useful for putting out new isos, but was no longer done for the CentOS Stream project.

Removal of mirrorlist will probaby cause various cron jobs and similar tools to stop working. Manual work will be needed to either turn off the CentOS repos affected in /etc/yum.repos.d/ files or something similar. The loss of forums will mainly affect people looking for answers for problems with EL7 and before systems. Similar answers may be available at stackoverflow or similar sites.

Fabian's second email covers what systems mirroring CentOS Linux 7 can expect and when.

  1. First it reiterates that all EL7 content from mirror.centos.org will be removed. Also that the mirrorlists will be turned off and no longer answering.
  2. Second crawlers will be turned off to see what mirrors are 'up-to-date' with EL7 content as the release is over.
  3. Third, the head rsync mirrors will continue to exist, but will only have an 'empty' filelist on it. This will cause all mirrors which do an rsync with --delete flags to remove all EL7 content from their mirror. This is normal for end of life releases as disk space needs to be used for other content.
The biggest issues that system administrators with EL7 systems need to be ready to do are the following:
  1. Be prepared that existing scripts and tools pulling in EL7 content will break without work.
  2. If you are mirroring EL7 content, and need to keep it, you need to check that the rsync scripts do not delete content which is no longer on the mirror. If you are using reposync or similar commands you will need to look for similar options.
  3. You should look at mirroring the content local to your site until your organization has a transition plan to another operating system.
  4. You should work with your management and financial resources on a transition plan if you haven't already.


Where did 5 Million EPEL-7 systems come from starting in March?

ADDENDUM (2024-05-30T01:08+00:00): Multiple Amazon engineers reached out after I posted this and there is work on identifying what is causing this issue. Thank you to all the people who are burning the midnight oil on this.


So starting earlier this year, the Fedora mirror manager mailing list started getting reports about heavier usage from various mirrors. Looking at the traffic reported, it seemed to be a large number of EL-7 systems starting to check in a lot more than in the past. At first I thought it was because various domains were beginning to move their operating systems from EL-7 to a newer release using one of the transition tools like Red Hat LEAPP or Alma ELevate. However, the surge didn't seem to die down, and in fact the Fedora Project mirrors have had regular problems with load due to this surge. 

A couple of weeks ago, I finally had time to look at some graphs I had set up when I was in Fedora Infrastructure and saw this:

Cumulative EPEL releases since 2019


Basically the load is an additional 5 million systems starting to query both the Fedora webproxies for mirror data, and then mirrors around the world to get further information. Going through the logs, there seems to be a 'gradual' shift of additional servers starting to ask for content when they had not before. In looking at the logs, it is hard to see what the systems asking for this data are. EL-7 uses yum which doesn't report any user data beyond:

urlgrabber/3.10 yum/3.4.

That could mean the system is Red Hat Enterprise Linux 7, CentOS Linux 7, or even Amazon Linux 2 (which is sort of based on CentOS 7, but with various changes that using EPEL is probably not advised).

Because there wasn't a countme or any identifiers in the yum code, the older data-analysis program does a 'if I see an ip address 1 time a day, I count it once.. if I see it 1000 times, I count it once.' This had a problem of undercounting for various cloud and other networks behind a NAT router.. so normally maybe only 1 ip address would show up in a class C (/24) network space. What seemed to change is where we might only count one ip address in various networks, we were now seeing every ip address showing up in a Class C network. 

Doing some backtracking of the ip addresses to ASN numbers, I was able to show that the 'top 10' ASNs changed dramatically in March

January 27, 2024
Total  ASN 
1347016 16509_AMAZON-02,
219728 14618_AMAZON-AES,
11205 8560_IONOS-AS
10403 8987_AMAZON
8463 32244_LIQUIDWEB,
8019 54641_IMH-IAD,
7889 398101_GO-DADDY-COM-LLC,
7234 394303_BIGSCOOTS,

February 27, 2024
1871463 16509_AMAZON-02,
219545 14618_AMAZON-AES,
11021 8560_IONOS-AS
9016 8987_AMAZON
8208 32244_LIQUIDWEB,
7885 54641_IMH-IAD,
7618 398101_GO-DADDY-COM-LLC,
7383 394303_BIGSCOOTS,

March 27, 2024
2604768 16509_AMAZON-02,
276737 14618_AMAZON-AES,
10211 8560_IONOS-AS
8134 8987_AMAZON
7952 54641_IMH-IAD,
7677 32244_LIQUIDWEB,
7445 394303_BIGSCOOTS,
7250 398101_GO-DADDY-COM-LLC,

April 27, 2024
4247068 16509_AMAZON-02,
1807803 14618_AMAZON-AES,
65274 8987_AMAZON
51668 135629_WESTCLOUDDATA
41190 55960_BJ-GUANGHUAN-AP
7662 54641_IMH-IAD,
7561 394303_BIGSCOOTS,
6613 32244_LIQUIDWEB,
6425 8560_IONOS-AS

May 27, 2024
4186230 16509_AMAZON-02,
1775898 14618_AMAZON-AES,
62698 8987_AMAZON
50895 135629_WESTCLOUDDATA
38521 55960_BJ-GUANGHUAN-AP
7613 394303_BIGSCOOTS,
7531 54641_IMH-IAD,
6307 398101_GO-DADDY-COM-LLC,
6222 32244_LIQUIDWEB,

I am not sure what changed in Amazon in March, but it has had a tremendous impact on parts of Fedora Infrastructure and the volunteer mirror systems which use it.


CentOS Linux 7: End of Life 2024-06-30

CentOS 7 EOL

If this looks a lot like the CentOS Stream 8 EOL content.. well it is because they aren't too different. It is just that instead of doing this once every 4 years, we get to do this twice in one year.
The last CentOS Linux release, CentOS Linux 7, will reach its end of life on June 30th, 2024. When that happens, the CentOS Infrastructure will follow the standard practice it has been doing since the early days of CentOS Linux:
  1. Move all content to https://vault.centos.org/
  2. Stop the mirroring software from responding to EL-7 queries.  
  3. Additional software for EL-7 may also be removed from other locations.

The first change usually causes any mirror doing an rsync with delete options to remove the contents from their mirror. The second change will cause the yum commands to break with errors. The vault system is also a single server with few active mirrors of its contents. As such, it is likely to be overloaded and very slow to respond as additional requests are made of it. 

 At the same time, the EPEL software on https://dl.fedoraproject.org/pub/epel/7 will be moved to /pub/archive/epel/7 and the mirrormanager for that will be updated appropriately.

In total, if you are using CentOS Linux 7 in either production or a CI/CD system, you can expect a lot of errors on July 1st or shortly afterwords.

What you must do!

There are several steps you can do to get ahead of the tsunami of problems:
  1. You can convert to Red Hat Enterprise Linux 7 and see about getting a Extended LifeCycle Support contract with the system. This is a stop gap measure for you to move toward a newer release of a Linux operating system.
  2. You can move your system to a replacement Enterprise Linux distribution. The Alma Project, Rocky Linux, Oracle and Red Hat all offer tools which can transition an EL7 system to a version of the operating system they will support for several years.
  3. If you are not able to move your systems in the next 45 days, you should look at mirroring the CentOS Linux 7 operating system to a more local location and move your update configs to use your mirror. With the large size of systems which could potentially try to use the vault system, I would not expect this to be very useful. As you will probably need to reinstall, add software or do continuous CI/CD in other areas.. you should keep a copy of the operating system local to your networks.



CentOS Stream 8 END OF LIFE : 2024-05-31

CentOS Stream 8 EOL

CentOS Stream 8 will reach its end of life on May 31st, 2024. When that happens, the CentOS Infrastructure will follow the standard practice it has been doing since the early days of CentOS Linux:
  1. Move all content to https://vault.centos.org/
  2. Stop the mirroring software from responding to EL-8 queries. 

The first change usually causes any mirror doing an rsync with delete options to remove the contents from their mirror. The second change will cause the dnf or yum commands to break with errors. The vault system is also a single server with few active mirrors of its contents. As such, it is likely to be overloaded and very slow to respond as additional requests are made of it.

In total, if you are using CentOS Stream 8 in either production or a CI/CD system, you can expect a lot of errors on June 1st or shortly afterwords. 

What you can do!

There are several steps you can do to get ahead of a possible tsunami of problems:
  1. You can look to moving to a newer release of CentOS Stream before the end of the month. This usually will require deployment of new images or installs versus straight updates. 
  2. You can see if any of the 'move my Enterprise Linux' tools have added support for moving from CentOS Stream 8 to their EL8.10. For releases before 8.10, this was very hard because CentOS Stream 8 was usually the next release, but 8.10 is at a point where Alma, Oracle, Red Hat Enterprise Linux, or Rocky are at the same revisions or newer.
  3. You can start mirroring the CentOS Stream 8 content into your infrastructure and point any CI/CD or other systems to that mirror which will allow you to continue to function.

Mirroring CentOS Stream

Of the three options, I recommend the third. However in working out what is needed to mirror CentOS Stream, I realized I needed newer documentation and it would probably be a long post in itself. For a shorter version for self-starters, I recommend the documentation on the CentOS wiki,  https://wiki.centos.org/HowTos(2f)CreateLocalMirror.html  While the information was written for CentOS Linux 6 which was end-of-lifed in 2020, it covers most of the instructions needed. The parts which may need updating is the amount of disk space required for CentOS Stream which seems to be about 280 GB for everything and maybe around 120GB for any one architecture. 



Recent EPEL dnf problems with some EL8 systems

Last week there were several reports about various systems having problems with EPEL-8 repositories. The problems started shortly after various systems in Fedora had been updated from Fedora Linux 38 to Fedora 40, but the problems were not happening to all EL-8 systems. The problems were root-caused to the following:

  1. Fedora 40 had createrepo-1.0 installed which defaults to using zstd compression for various repositories.
  2. The EL8 systems which were working had some version of libsolv-0.7.20 installed which links against libzstd and so works.
  3. The EL8 systems which did not work either had versions of libsolv before 0.7.20 (and were any EL before 8.4) OR they had versions of libsolv-0.7.22

The newer version of libsolv was traced down to coming from repositories of either Red Hat Satellite or a related tool pulp, and was a rebuild of the EL9 package. However it wasn't a complete rebuild as the EL9 version has libzstd support, but the EL8 rebuild did not. 

A band-aid fix was made by Fedora Release Engineering to have EPEL repositories use the xz compression method for various files in /repodata/ versus libzstd. This is a slower compression method, but is possible to be used with all of the versions of libsolv reported in the various bugs and issue trackers.

This is a band-aid fix because users will run into this with any other repositories using the newer createrepo. A fuller fix will require affected systems to do one of the following:

  1. If the system is either Red Hat Enterprise Linux, Rocky Linux, Alma Linux, or Oracle Enterprise Linux and not running EL-8.9 or later, they need to upgrade to that OR point to an older version of a repository in https://dl.fedoraproject.org/pub/archive/epel/
  2. If the system has versions of libsolv-0.7.22 without libzstd support, they should contact the repository to see if a rebuild with libzstd support can be made. 
  3. If the system is CentOS Stream 8, then you should be making plans to upgrade to a different operating system as that version will be EOL on May 31st 2024.

External References