What happened to the CentOS Forums

Where did CentOS Forums Go

• What Happened
• Lessons (Re)-Learned.

What Happened

If you try to go to the CentOS Forums, you will instead be directed to the status page. The CentOS Forums is sadly one of the resources that went away the the end of CentOS Linux 7. The system was a classic PHP web-forum which had been run for over a decade by volunteers as a resource for the CentOS Community. Due to its age, and its usefulness, there have been several requests to get it back up and working. However, it looks highly unlikely for several reasons:

1. The service was not actually ‘owned’ by the CentOS Project. It was set up and run by various volunteers who were associated with the project but was never a ‘core’ part of the project. So when Red Hat ‘bought’ CentOS in 2014, this service was not part of that merger.
2. The volunteers who were running the service mostly left after CentOS Linux 8 was end of lifed earlier than thought on 2021-12-31. Many in the community had thought the operating system would have a 10 year lifetime like earlier releases, but Red Hat decided in 2021 to focus only on the CentOS Stream versions.
3. The remaining volunteer announced that they would only support the system until the end of CentOS Linux 7 in 2024. Announcements were put on the forums that the service would be going away.
4. The contents of the forums could not be transferred to the CentOS project (aka Red Hat) because the contents were never licensed under a license which would allow for it. Basically any post on the forums was under the copyright of the poster and possibly licensed to be viewed on that forum only. Moving the posts and documents on the forum would have required getting the permission of every individual poster where many of them may no longer be contactable.
5. Various parts of the database are also covered by the GDPR, ‘Right to Forget’ and similar laws around the world. Because again the licensing of the forum was fairly informal, it was thought that transfers would need further legal review.
6. Even putting the service in a ‘read-only’ format would have required continual maintenance for a PHP code base which would have needed porting to a newer operating system and toolkit. [Doing dumps and rewrites for forums tends to end up with unreadable messes unless a lot of extra time and effort is done.]

Since the remaining volunteer had done this for 4 years and didn’t want to continue, it was decided to shut off the forums when CentOS Linux 7 went away. For people who still wanted to discuss issues with CentOS Stream, the Fedora Project added a section to its discourse server: https://discussion.fedoraproject.org/c/neighbors/centos/71 which has seen some traffic (mostly people asking for howtos).

Lessons (Re)-Learned.

The first lesson I have re-learned from this EOL is that the Internet is ephemeral but operating system needs are forever. I have a very large list of bookmarks in my browser which are no longer useful but on checking, I realized that nearly every bookmark over 5 years was also no longer where it was ‘originally’ pointing to. In some places, some system administrator has made a redirect (which probably leads to another redirect and another).. but in many cases the posts are gone like so many tears in the rain.

The second lesson I have re-learned is that copyright licenses are much more important than we give them credence. When setting up a forum or other group, make sure that you ensure that others can reuse what is posted and the authors understand the permissions they are granting others before they post.

The third lesson is when-ever possible make a local copy of things you find important. If it is under some sort of copyleft, make a direct copy and keep it in markdown or some similar format for later updates. If it isn’t permissible, then rewrite it as best you can to make sure that you know how restore backups on a CentOS Linux 5 system even though the OS is now end-of-lifed for over 7 years.

How to archive a local copy of CentOS 7

• How to archive a local copy of CentOS 7
• What happened?
• What is needed to be done?
  • Hardware and software requirements
  • Internet requirements
  • Sample Rsync script
  • Sample HTTPD config
  • Sample Yum repo config

How to archive a local copy of CentOS 7

This post is meant for the various junior system administrators who have been tasked with fixing problems with CentOS 7 systems after the software was removed from most of its mirrors. Most of these systems have probably been running fine for a decade, and now possible critical systems are generating failed cron jobs or other errors.

What happened?

On July 1st, 2024, CentOS Enterprise Linux 7 reached its end of life as its upstream, Red Hat Enterprise Linux 7, had moved to Extended Life(cycle) Support. As with previous releases, the operating system was moved from the main mirrors to the CentOS vault, and the mirrorlists were turned off. Once the software was removed from the main mirrors, many secondary mirrors also removed the software as rsync and similar scripts would see the old software was gone. At this point, some unknown number of CentOS Linux systems ended up ‘non-supported.’

The systems may have been extremely low maintenance for years running whatever tasks they had been without a problem. The people who initially set them up have probably moved to other jobs, and some new person is now finding out that things are broken. Maybe its a cron job which runs one a week to run updates, or the kickstart used to reinstall an old server now breaks. In most of these cases, there is little documentation on what is being used, why it is being used, or how big a problem this is going to be.

What is needed to be done?

Getting an infrastructure out of this place is really out of scope of a single blog post. It generally requires getting various levels of managements attention, and then long term planning on how to transform an infrastructure into something more manageable. However in the short term a site can make things workable by making a local mirror of content from an upstream vault.

The reason to use a local vault is that the existing upstream vaults are limited in bandwidth and scope. Plus as more sites try to use them, the services may be curtailed or removed. When dealing with ‘End of Life’ projects and software, it is better to assume that things will get worse before they get better.

Hardware and software requirements

In order to mirror CentOS 7 locally, you are going to need to set up a webserver with at least 500 GB of free space (if you don’t want to copy the out of date ‘atomic’ trees. ) The amount of memory and cpu cores needed is dependent on the number of servers you are going to be supporting. The more systems, the more memory and cores that might be needed. In any case, I was able to set up a system with 2 cores and 4 GB of ram to support 4 EL7 systems without problems.

Internet requirements

There are currently 3 major mirrors of the CentOS vault.

• archive.kernel.org
• linuxsoft.cern.ch
• mirror.nsc.liu

It is best to find one which is ‘network’ close and set up scripts to rsync data from the site. I found that each server will be busier at different times, so expect that copying will take multiple hours.

Sample Rsync script

The script I used to do this was the following:

#!/bin/bash

VAULT=archive.kernel.org::centos-vault/
TREEDIR=/srv/mirrors/

RSYNC_OPTS='-avSHP --delete-excluded'

## Mirror CentOS 7
mkdir -vp ${TREEDIR}/centos-vault/7.9/
EXCLUDE_ITEMS="--exclude=atomic/"
rsync ${RSYNC_OPTS} ${EXCLUDE_ITEMS}  ${VAULT}/7.9.2009/ ${TREEDIR}/centos-vault/7.9/

Sample HTTPD config

In my /etc/httpd/conf.d/ I added the following config file:

Alias "/mirrors" "/srv/mirrors"
<Directory /srv/mirrors>
  AllowOverride None
  Require all granted
  Options +Indexes
</Directory>

Sample Yum repo config

Finally on the EL7 systems, I used a config like the following in /etc/yum.repos.d/CentOS-EOL.repo

[base]
name=CentOS-$releasever - Base
baseurl=http://192.168.1.150/mirrors/centos-vault/7.9/os/$basearch/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#released updates
[updates]
name=CentOS-$releasever - Updates
baseurl=http://192.168.1.150/mirrors/centos-vault/7.9/updates/$basearch/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras
baseurl=http://192.168.1.150/mirrors/centos-vault/7.9/extras/$basearch/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

[centosplus]
name=CentOS-$releasever - Plus
baseurl=http://192.168.1.150/mirrors/centos-vault/7.9/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

At this point, updates were possible and I was able to reinstall a system in order to rebuild some packages I needed. Similar work can be done to set up mirrors of CentOS Linux 6 or third party repositories like EPEL.

What happens to EPEL-7 when EL-7 goes EOL

• What is happening to RHEL-7
• What is happening to EPEL-7
• How to deal with this EOL.
• End Thoughts

What is happening to RHEL-7

The Red Hat Enterprise Linux release cycle is fairly straight forward with releases lasting about 10 years after their initial release.

RHEL ver	Release Date	EoMaint	EoExtended
5	2007-03-15	2017-03-31	2020-11-30
6	2010-11-10	2020-11-30	2024-06-30
7	2014-06-10	2024-06-30	2028-06-30
8	2019-05-07	2029-05-??	2031-05-??
9	2022-11-08	2032-05-??	2034-05-??
10	2025-??-??	2035-??-??	2037-??-??

On June 30, 2024, Red Hat will stop doing general maintenance support of RHEL-7 and no more updates to that operating system will be available without purchasing ‘Extended Life-cycle Support’ contracts from Red Hat or similar contracts from SuSE’s Liberty Linux, Perforce’s OpenLogic, or various other consultants and companies offering various services.

What is happening to EPEL-7

The Fedora Extra Packages for Enterprise Linux (EPEL) is a repository of additional packages built against the Red Hat Enterprise Linux (RHEL) releases. When a RHEL release leaves maintenance mode and moves into ‘Extended Life-cycle Support’, the EPEL package set corresponding to that is closed and archived. No new builds or updates are done against that particular release, all bugs related to that release are closed as CANTFIX, and most mirrors will remove the content after the master mirror moves items to archives.

The usual process for closing out an EPEL release depends on how much work the central Fedora release engineering team have to deal with at the time of the RHEL EOL. In cases where a Fedora release is needing a high amount of focus, the only actions may be that new builds will be turned off on the day of the EOL, and then later other steps will be worked out. In other cases, all the actions occur fairly close together.

1. A reminder email that the EOL is going to occur is sent to various Fedora mailing lists like: devel-announce@lists.fedoraproject.org and epel-announce@lists.fedoraproject.org. System administrators using EPEL should be following at least the second email address to see what issues may be occurring with the repository.
2. The Fedora Build system is a complicated tool-set and so Release Engineering have to disable the building and composing of that particular EPEL release in multiple places. Because the release cycle is so long for EPEL compared to Fedora releases, there may be inconsistencies in how things were named or done. Basically it is a slower process than with Fedora releases to make sure that the tools continue to work after an EOL.
3. All bugzilla.redhat.com bugs related to that release will be closed with a short stock answer tailored for that release as ‘CANTFIX’.
4. The release is now copied into the Fedora down-loaders /pub/archive/epel tree with an appropriate name like /pub/archive/epel/7.9.2024-07-01 and then symlinks are made. A week is given for the archive mirrors to catch up with this new directory tree.
5. The Fedora mirrormanager will be changed to point requests for ‘repo=epel-7’ to the mirrors which have /pub/archive.
6. The content then will be removed from the main mirror locations of /pub/epel/ and mirrors who rsync will catch up to that in a day or so.

Usually about a week to a month after this happens, either the mailing lists or IRC will get people wondering where the content went. However in the current age of Continuous Integration and Container builds, changes in the EPEL-7 directory structure seem to cause reactions hours after they occur.

How to deal with this EOL.

The main advice I have for system administrators dealing with an End of Life operating system is the following:

1. Mirror the content locally from whatever mirrors you can find. This lowers network bandwidth usage, and deals with the vagaries of archive sites removing content because they no longer can handle the load from large numbers of unexpected users.
2. Work with your management on how to deal with the technical load that these systems are going to cause.
   1. Budget out what extended maintenance costs for critical security fixes to services like web-servers, databases, remote logins, etc are going to take.
   2. Budget out what mitigations are going to be required to keep these systems from causing longer term headaches. This may be extra firewalls or additional staff needed to take over software maintenance.
   3. Budget out what replacements of the hardware and software to move to an operating system which has long term maintenance support.
   4. Actually this is a very long list and probably should be its own blog post.

End Thoughts

Dealing with End of Life software is rarely an easy task. Just realize that you aren’t alone and there are tens of millions of systems which are going to be ‘unsupported’ on July 1st. Be patient with yourself and your infrastructure as you deal with this. Good luck.

Updates on the CentOS 7 EOL: 2024-06-30

I am writing this on 2024-06-20 which means according to the IRC centbot:

  centbot: CentOS 7 will go EOL on 30 June, 2024 -- in 1 week,

 10 hours, 22 minutes, and 41 seconds.
  

The CentOS Systems Administrator, Fabian Arrotin, has posted several updates about what to expect starting Monday Jul 1st, 2024. The first email covers what services will stop after the 'End of Life' occurs:

1. mirrorlist.centos.org will be decommissioned. This will affect any system trying to do a `yum update` or similar command after that date.
2. forums.centos.org
will be turned off and decommissioned. This was a volunteer effort by various system administrators who are ending their work with CentOS at the end of 7.
3. torrent.centos.org
will be turned off and decommissioned. This was useful for putting out new isos, but was no longer done for the CentOS Stream project.

Removal of mirrorlist will probaby cause various cron jobs and similar tools to stop working. Manual work will be needed to either turn off the CentOS repos affected in /etc/yum.repos.d/ files or something similar. The loss of forums will mainly affect people looking for answers for problems with EL7 and before systems. Similar answers may be available at stackoverflow or similar sites.

Fabian's second email covers what systems mirroring CentOS Linux 7 can expect and when.

1. First it reiterates that all EL7 content from mirror.centos.org will be removed. Also that the mirrorlists will be turned off and no longer answering.
2. Second crawlers will be turned off to see what mirrors are 'up-to-date' with EL7 content as the release is over.
3. Third, the head rsync mirrors will continue to exist, but will only have an 'empty' filelist on it. This will cause all mirrors which do an rsync with --delete flags to remove all EL7 content from their mirror. This is normal for end of life releases as disk space needs to be used for other content.

The biggest issues that system administrators with EL7 systems need to be ready to do are the following:

1. Be prepared that existing scripts and tools pulling in EL7 content will break without work.
2. If you are mirroring EL7 content, and need to keep it, you need to check that the rsync scripts do not delete content which is no longer on the mirror. If you are using reposync or similar commands you will need to look for similar options.
3. You should look at mirroring the content local to your site until your organization has a transition plan to another operating system.
4. You should work with your management and financial resources on a transition plan if you haven't already.

Where did 5 Million EPEL-7 systems come from starting in March?

ADDENDUM (2024-05-30T01:08+00:00): Multiple Amazon engineers reached out after I posted this and there is work on identifying what is causing this issue. Thank you to all the people who are burning the midnight oil on this.

ORIGINAL ARTICLE:

So starting earlier this year, the Fedora mirror manager mailing list started getting reports about heavier usage from various mirrors. Looking at the traffic reported, it seemed to be a large number of EL-7 systems starting to check in a lot more than in the past. At first I thought it was because various domains were beginning to move their operating systems from EL-7 to a newer release using one of the transition tools like Red Hat LEAPP or Alma ELevate. However, the surge didn't seem to die down, and in fact the Fedora Project mirrors have had regular problems with load due to this surge. 

A couple of weeks ago, I finally had time to look at some graphs I had set up when I was in Fedora Infrastructure and saw this:

Cumulative EPEL releases since 2019

 

Basically the load is an additional 5 million systems starting to query both the Fedora webproxies for mirror data, and then mirrors around the world to get further information. Going through the logs, there seems to be a 'gradual' shift of additional servers starting to ask for content when they had not before. In looking at the logs, it is hard to see what the systems asking for this data are. EL-7 uses yum which doesn't report any user data beyond:

urlgrabber/3.10 yum/3.4.

That could mean the system is Red Hat Enterprise Linux 7, CentOS Linux 7, or even Amazon Linux 2 (which is sort of based on CentOS 7, but with various changes that using EPEL is probably not advised).

Because there wasn't a countme or any identifiers in the yum code, the older data-analysis program does a 'if I see an ip address 1 time a day, I count it once.. if I see it 1000 times, I count it once.' This had a problem of undercounting for various cloud and other networks behind a NAT router.. so normally maybe only 1 ip address would show up in a class C (/24) network space. What seemed to change is where we might only count one ip address in various networks, we were now seeing every ip address showing up in a Class C network. 

Doing some backtracking of the ip addresses to ASN numbers, I was able to show that the 'top 10' ASNs changed dramatically in March

January 27, 2024
Total  ASN 
1347016 16509_AMAZON-02,
219728 14618_AMAZON-AES,
53500 396982_GOOGLE-CLOUD-PLATFORM,
11205 8560_IONOS-AS
10403 8987_AMAZON
8463 32244_LIQUIDWEB,
8019 54641_IMH-IAD,
7965 8075_MICROSOFT-CORP-MSN-AS-BLOCK,
7889 398101_GO-DADDY-COM-LLC,
7234 394303_BIGSCOOTS,

February 27, 2024
1871463 16509_AMAZON-02,
219545 14618_AMAZON-AES,
51511 396982_GOOGLE-CLOUD-PLATFORM,
11021 8560_IONOS-AS
9016 8987_AMAZON
8208 32244_LIQUIDWEB,
7885 54641_IMH-IAD,
7768 8075_MICROSOFT-CORP-MSN-AS-BLOCK,
7618 398101_GO-DADDY-COM-LLC,
7383 394303_BIGSCOOTS,

March 27, 2024
2604768 16509_AMAZON-02,
276737 14618_AMAZON-AES,
34674 396982_GOOGLE-CLOUD-PLATFORM,
10211 8560_IONOS-AS
9560 135629_WESTCLOUDDATA
8134 8987_AMAZON
7952 54641_IMH-IAD,
7677 32244_LIQUIDWEB,
7445 394303_BIGSCOOTS,
7250 398101_GO-DADDY-COM-LLC,

April 27, 2024
4247068 16509_AMAZON-02,
1807803 14618_AMAZON-AES,
65274 8987_AMAZON
51668 135629_WESTCLOUDDATA
41190 55960_BJ-GUANGHUAN-AP
9799 396982_GOOGLE-CLOUD-PLATFORM,
7662 54641_IMH-IAD,
7561 394303_BIGSCOOTS,
6613 32244_LIQUIDWEB,
6425 8560_IONOS-AS

May 27, 2024
4186230 16509_AMAZON-02,
1775898 14618_AMAZON-AES,
62698 8987_AMAZON
50895 135629_WESTCLOUDDATA
38521 55960_BJ-GUANGHUAN-AP
9059 396982_GOOGLE-CLOUD-PLATFORM,
7613 394303_BIGSCOOTS,
7531 54641_IMH-IAD,
6307 398101_GO-DADDY-COM-LLC,
6222 32244_LIQUIDWEB,

I am not sure what changed in Amazon in March, but it has had a tremendous impact on parts of Fedora Infrastructure and the volunteer mirror systems which use it.

CentOS Linux 7: End of Life 2024-06-30

CentOS 7 EOL

If this looks a lot like the CentOS Stream 8 EOL content.. well it is because they aren't too different. It is just that instead of doing this once every 4 years, we get to do this twice in one year.

 

The last CentOS Linux release, CentOS Linux 7, will reach its end of life on June 30th, 2024. When that happens, the CentOS Infrastructure will follow the standard practice it has been doing since the early days of CentOS Linux:

1. Move all content to https://vault.centos.org/
2. Stop the mirroring software from responding to EL-7 queries.  
3. Additional software for EL-7 may also be removed from other locations.
   

The first change usually causes any mirror doing an rsync with delete options to remove the contents from their mirror. The second change will cause the yum commands to break with errors. The vault system is also a single server with few active mirrors of its contents. As such, it is likely to be overloaded and very slow to respond as additional requests are made of it. 

 At the same time, the EPEL software on https://dl.fedoraproject.org/pub/epel/7 will be moved to /pub/archive/epel/7 and the mirrormanager for that will be updated appropriately.

In total, if you are using CentOS Linux 7 in either production or a CI/CD system, you can expect a lot of errors on July 1st or shortly afterwords.

What you must do!

There are several steps you can do to get ahead of the tsunami of problems:

1. You can convert to Red Hat Enterprise Linux 7 and see about getting a Extended LifeCycle Support contract with the system. This is a stop gap measure for you to move toward a newer release of a Linux operating system.
   
2. You can move your system to a replacement Enterprise Linux distribution. The Alma Project, Rocky Linux, Oracle and Red Hat all offer tools which can transition an EL7 system to a version of the operating system they will support for several years.
   
3. If you are not able to move your systems in the next 45 days, you should look at mirroring the CentOS Linux 7 operating system to a more local location and move your update configs to use your mirror. With the large size of systems which could potentially try to use the vault system, I would not expect this to be very useful. As you will probably need to reinstall, add software or do continuous CI/CD in other areas.. you should keep a copy of the operating system local to your networks.
   

 References:

• https://www.redhat.com/en/blog/announcing-4-years-extended-life-cycle-support-els-red-hat-enterprise-linux-7
  
• https://wiki.centos.org/HowTos(2f)CreateLocalMirror.html
• https://lists.centos.org/hyperkitty/list/devel@lists.centos.org/thread/DS7Q6NQWYD3YXCECJPSAXFFSRSDIJG2Q/
• https://lists.centos.org/hyperkitty/list/devel@lists.centos.org/thread/BIWCCSDWOPCX5DMDEYJVTTEXHNYHSJZ3/
• https://data-analysis.fedoraproject.org/csv-reports/images/epel-stacked.png
   

CentOS Stream 8 END OF LIFE : 2024-05-31

CentOS Stream 8 EOL

CentOS Stream 8 will reach its end of life on May 31st, 2024. When that happens, the CentOS Infrastructure will follow the standard practice it has been doing since the early days of CentOS Linux:

1. Move all content to https://vault.centos.org/
2. Stop the mirroring software from responding to EL-8 queries. 

The first change usually causes any mirror doing an rsync with delete options to remove the contents from their mirror. The second change will cause the dnf or yum commands to break with errors. The vault system is also a single server with few active mirrors of its contents. As such, it is likely to be overloaded and very slow to respond as additional requests are made of it.

In total, if you are using CentOS Stream 8 in either production or a CI/CD system, you can expect a lot of errors on June 1st or shortly afterwords. 

What you can do!

There are several steps you can do to get ahead of a possible tsunami of problems:

1. You can look to moving to a newer release of CentOS Stream before the end of the month. This usually will require deployment of new images or installs versus straight updates. 
2. You can see if any of the 'move my Enterprise Linux' tools have added support for moving from CentOS Stream 8 to their EL8.10. For releases before 8.10, this was very hard because CentOS Stream 8 was usually the next release, but 8.10 is at a point where Alma, Oracle, Red Hat Enterprise Linux, or Rocky are at the same revisions or newer.
3. You can start mirroring the CentOS Stream 8 content into your infrastructure and point any CI/CD or other systems to that mirror which will allow you to continue to function.

Mirroring CentOS Stream

Of the three options, I recommend the third. However in working out what is needed to mirror CentOS Stream, I realized I needed newer documentation and it would probably be a long post in itself. For a shorter version for self-starters, I recommend the documentation on the CentOS wiki,  https://wiki.centos.org/HowTos(2f)CreateLocalMirror.html  While the information was written for CentOS Linux 6 which was end-of-lifed in 2020, it covers most of the instructions needed. The parts which may need updating is the amount of disk space required for CentOS Stream which seems to be about 280 GB for everything and maybe around 120GB for any one architecture. 

 References:

• https://wiki.centos.org/HowTos(2f)CreateLocalMirror.html
• https://lists.centos.org/hyperkitty/list/devel@lists.centos.org/thread/DS7Q6NQWYD3YXCECJPSAXFFSRSDIJG2Q/
• https://lists.centos.org/hyperkitty/list/devel@lists.centos.org/thread/BIWCCSDWOPCX5DMDEYJVTTEXHNYHSJZ3/