2024-07-27

What happened to the CentOS Forums

Where did CentOS Forums Go

What Happened

If you try to go to the CentOS Forums, you will instead be directed to the status page. The CentOS Forums is sadly one of the resources that went away the the end of CentOS Linux 7. The system was a classic PHP web-forum which had been run for over a decade by volunteers as a resource for the CentOS Community. Due to its age, and its usefulness, there have been several requests to get it back up and working. However, it looks highly unlikely for several reasons:

  1. The service was not actually ‘owned’ by the CentOS Project. It was set up and run by various volunteers who were associated with the project but was never a ‘core’ part of the project. So when Red Hat ‘bought’ CentOS in 2014, this service was not part of that merger.
  2. The volunteers who were running the service mostly left after CentOS Linux 8 was end of lifed earlier than thought on 2021-12-31. Many in the community had thought the operating system would have a 10 year lifetime like earlier releases, but Red Hat decided in 2021 to focus only on the CentOS Stream versions.
  3. The remaining volunteer announced that they would only support the system until the end of CentOS Linux 7 in 2024. Announcements were put on the forums that the service would be going away.
  4. The contents of the forums could not be transferred to the CentOS project (aka Red Hat) because the contents were never licensed under a license which would allow for it. Basically any post on the forums was under the copyright of the poster and possibly licensed to be viewed on that forum only. Moving the posts and documents on the forum would have required getting the permission of every individual poster where many of them may no longer be contactable.
  5. Various parts of the database are also covered by the GDPR, ‘Right to Forget’ and similar laws around the world. Because again the licensing of the forum was fairly informal, it was thought that transfers would need further legal review.
  6. Even putting the service in a ‘read-only’ format would have required continual maintenance for a PHP code base which would have needed porting to a newer operating system and toolkit. [Doing dumps and rewrites for forums tends to end up with unreadable messes unless a lot of extra time and effort is done.]

Since the remaining volunteer had done this for 4 years and didn’t want to continue, it was decided to shut off the forums when CentOS Linux 7 went away. For people who still wanted to discuss issues with CentOS Stream, the Fedora Project added a section to its discourse server: https://discussion.fedoraproject.org/c/neighbors/centos/71 which has seen some traffic (mostly people asking for howtos).

Lessons (Re)-Learned.

XKCD

The first lesson I have re-learned from this EOL is that the Internet is ephemeral but operating system needs are forever. I have a very large list of bookmarks in my browser which are no longer useful but on checking, I realized that nearly every bookmark over 5 years was also no longer where it was ‘originally’ pointing to. In some places, some system administrator has made a redirect (which probably leads to another redirect and another).. but in many cases the posts are gone like so many tears in the rain.

The second lesson I have re-learned is that copyright licenses are much more important than we give them credence. When setting up a forum or other group, make sure that you ensure that others can reuse what is posted and the authors understand the permissions they are granting others before they post.

The third lesson is when-ever possible make a local copy of things you find important. If it is under some sort of copyleft, make a direct copy and keep it in markdown or some similar format for later updates. If it isn’t permissible, then rewrite it as best you can to make sure that you know how restore backups on a CentOS Linux 5 system even though the OS is now end-of-lifed for over 7 years.

2024-07-16

How to archive a local copy of CentOS 7

How to archive a local copy of CentOS 7

How to archive a local copy of CentOS 7

This post is meant for the various junior system administrators who have been tasked with fixing problems with CentOS 7 systems after the software was removed from most of its mirrors. Most of these systems have probably been running fine for a decade, and now possible critical systems are generating failed cron jobs or other errors.

What happened?

On July 1st, 2024, CentOS Enterprise Linux 7 reached its end of life as its upstream, Red Hat Enterprise Linux 7, had moved to Extended Life(cycle) Support. As with previous releases, the operating system was moved from the main mirrors to the CentOS vault, and the mirrorlists were turned off. Once the software was removed from the main mirrors, many secondary mirrors also removed the software as rsync and similar scripts would see the old software was gone. At this point, some unknown number of CentOS Linux systems ended up ‘non-supported.’

The systems may have been extremely low maintenance for years running whatever tasks they had been without a problem. The people who initially set them up have probably moved to other jobs, and some new person is now finding out that things are broken. Maybe its a cron job which runs one a week to run updates, or the kickstart used to reinstall an old server now breaks. In most of these cases, there is little documentation on what is being used, why it is being used, or how big a problem this is going to be.

What is needed to be done?

Getting an infrastructure out of this place is really out of scope of a single blog post. It generally requires getting various levels of managements attention, and then long term planning on how to transform an infrastructure into something more manageable. However in the short term a site can make things workable by making a local mirror of content from an upstream vault.

The reason to use a local vault is that the existing upstream vaults are limited in bandwidth and scope. Plus as more sites try to use them, the services may be curtailed or removed. When dealing with ‘End of Life’ projects and software, it is better to assume that things will get worse before they get better.

Hardware and software requirements

In order to mirror CentOS 7 locally, you are going to need to set up a webserver with at least 500 GB of free space (if you don’t want to copy the out of date ‘atomic’ trees. ) The amount of memory and cpu cores needed is dependent on the number of servers you are going to be supporting. The more systems, the more memory and cores that might be needed. In any case, I was able to set up a system with 2 cores and 4 GB of ram to support 4 EL7 systems without problems.

Internet requirements

There are currently 3 major mirrors of the CentOS vault.

  • archive.kernel.org
  • linuxsoft.cern.ch
  • mirror.nsc.liu

It is best to find one which is ‘network’ close and set up scripts to rsync data from the site. I found that each server will be busier at different times, so expect that copying will take multiple hours.

Sample Rsync script

The script I used to do this was the following:

#!/bin/bash

VAULT=archive.kernel.org::centos-vault/
TREEDIR=/srv/mirrors/

RSYNC_OPTS='-avSHP --delete-excluded'

## Mirror CentOS 7
mkdir -vp ${TREEDIR}/centos-vault/7.9/
EXCLUDE_ITEMS="--exclude=atomic/"
rsync ${RSYNC_OPTS} ${EXCLUDE_ITEMS}  ${VAULT}/7.9.2009/ ${TREEDIR}/centos-vault/7.9/

Sample HTTPD config

In my /etc/httpd/conf.d/ I added the following config file:

Alias "/mirrors" "/srv/mirrors"
<Directory /srv/mirrors>
  AllowOverride None
  Require all granted
  Options +Indexes
</Directory>

Sample Yum repo config

Finally on the EL7 systems, I used a config like the following in /etc/yum.repos.d/CentOS-EOL.repo

[base]
name=CentOS-$releasever - Base
baseurl=http://192.168.1.150/mirrors/centos-vault/7.9/os/$basearch/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#released updates
[updates]
name=CentOS-$releasever - Updates
baseurl=http://192.168.1.150/mirrors/centos-vault/7.9/updates/$basearch/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras
baseurl=http://192.168.1.150/mirrors/centos-vault/7.9/extras/$basearch/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

[centosplus]
name=CentOS-$releasever - Plus
baseurl=http://192.168.1.150/mirrors/centos-vault/7.9/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

At this point, updates were possible and I was able to reinstall a system in order to rebuild some packages I needed. Similar work can be done to set up mirrors of CentOS Linux 6 or third party repositories like EPEL.