2016-07-04

Fedora PSA: Why is my new Fedora account listed as spamcheck_waiting or spamcheck_denied?


The Fedora Project is currently undergoing a massive spam account creation and insertion problem. In the month of June, we saw over 14,000 accounts created just to put in various Quickbook and Antivirus scam web pages in order to get higher SEO search rankings in various search engines. There have also been some malicious pdf and docx files uploaded which would have potentially harmed people. The groups behind these campaigns have shown themselves to be organized and are using teams of people to solve various captchas and other "I am a human" tests to create more accounts.

The amount of spam caused several parts of the project to actually get delisted from various search engines with warnings that we needed to clean up the pages. However the amount of spam being created was more than our usual scripted cleanups as various pages would not be found until the next google or bing crawl. [I spent this weekend cleaning up pages from late 2015 and early 2016 which finally got 'used' and showed up in weblogs.]

Due to the nature of this, Fedora Infrastructure has had to implement multiple circuit breakers to slow down registration and web page creation to try and cut down the amount of bad accounts and webpages being inserted.  Like any diagnostics test, it is not infallible and produces both  false positives and false negatives.

In the case of false negatives, we eventually will get alerted by someone looking through the wikis and finding the spam we weren't able to find. In the case of false positives, we have closed an account for a bad review of the account.

If you have gotten spamchecked, please email admin@fedoraproject.org, and we will review the logs to see why this happened. This will take time however as we have found that most of the requests for review are now coming from the spam account openers.

I am sorry for the delay and problems this last 2 month's have become a "Tragedy of the Commons" where a small set of people have taken up a large amount of resources for their own benefit.

[For more details on the statistics and the various circuit breaker programs written, please look forward to various future blogs from various members of the Fedora Infrastructure.]

2016-04-06

Reminder: PSA: Enterprise Linux 5 End of Production on 2017-03-31 and EPEL.

So in less than 12 months (2017-03-31), Red Hat Enterprise Linux 5 will be leaving end of production level 3 . At that time EPEL support will be ending for EL-5 and all packages will be moved to the archives section of the fedoraproject download servers.

If you are relying on (CentOS|ScientificLinux|Oracle|Red Hat) Enterprise Linux 5 for your business needs, you should look at moving the to either a newer version of Enterprise Linux or at the extended support program that Red Hat offers as listed in the above article.

I will be publishing my next PSA on this in 6 months.

2016-03-25

Attention Sites Mirroring Fedora

We are seeing a large number of partial rsync on the main download
servers which are causing problems for tier0/1 mirrors to get to the
servers. What we are seeing is that an ip address will start a rsync
of a large tree and then will drop the connection as the server takes
time to work through 1-10TB of disk space. The ip will then initiate a
new connection which another download server will try to fulfill but
again with a time-out and restart. This is adding a large amount of
IOPs onto our backend storage causing a cascading set of problems
through the infrastructure.

We are going to have to put a firewall rules to drop connections from
these systems so that the alpha and other work can get properly
mirrored onto the registered Tier 0 and Tier 1 mirrors. If that
doesn't work we will be putting firewall rules that only Tier 0 and
Tier 1 mirrors are allowed to connect to the download servers.

I will be trying to send emails to the registered abuse for the ips, but if you are getting dropped connections to the download servers or are a mirror administrator at one of the following domains:

  • wideopenwest.com
  • alshamil.net.ae
  • yandex.net
  • sl-reverse.com
  • univ-ubs.fr
  • ip.pt

please register in the Fedora mirror manager so that it can be whitelisted. 


Finally, to cut down mirroring problems please do the following:

1) Do not put in a cron job that you are going to do an rsync update
every 15 minutes as several of the above mirrors seem to do. We do not
update the trees that often and rsyncd has to stat every file.

2) Please review the tips and tricks at
Using the last-sync to schedule updates when they actually occur can
help lower rsync usage.

Thank you

2016-02-24

EPEL Proposal #4: Build it all in CentOS promote to RHEL

So this one came up when we were going over problems with building alternative architectures in EPEL. One of the big problems is that currently everything is built against Red Hat Enterprise Linux. This has some bonuses for some people but it causes a problem when architectures are built against by CentOS but not Red Hat. [E.G. EL-7 for i386 and arm32] When a Red Hat release is done, the builders usually get updated as part of our repopull from the RHN servers. Usually there are some changes in added and subtracted packages which can cause packages to get new dependencies. While this is an inconvenience to customers.. it isn't a problem for the builders. When we start mixing distributions, it does cause a problem with the builders because a package may get compiled for EL-7 x86_64 and not be the same as arm32 or i386 due to the fact that buildroots for i386 are not as up2date as the RHEL ones. This can cause builds all across the system to break so it is frowned on.

A couple of proposed fixes were:

  1.  Compile everything against CentOS. This has the bonus that they all kind of get updated at the same time so breakages from going from 7.3 to 7.4 is smaller. However, a lot of business users have either ITIL or security plans which allow for EPEL packages to be used on their systems because they were built against RHEL versus a rebuild. Remove that and it causes various problems for users.
  2. Split off the builds for alternative architectures in a shadow koji which is turned off when these splits occur. This makes it partially easier to manage but brings in other problems.
  3. ... some other ways to fix this that I don't know how to describe correctly...
However while we were hashing on 1, someone else who had been asking about why don't we build everything came up with a different mix. Build everything against CentOS for various architectures and put it in some other repository (Extra Packages in CentOS came to mind). Packages which people really want and are willing to care for are branched off into EPEL packages which are built against RHEL. [Or if they really want it they get a business case to Red Hat to make it a product they will pay for it.]

There is a lot of problems with the above.. it requires a lot of work in many different moving parts to get done. It also could be even more confusing than what we have already. However, I did say I was going to write up all the proposals for people to look at. 

EPEL Proposal #3: EPEL Structured releases

This started off as a a rawhide post, but I realized it needs to be its own post

A staged/forked environment. This version is a lot more complicated and deals with extra release management. Currently, Red Hat does regular point releases (.1, .2, .3) of their main OS. In these point releases, various items get major updates or backports to bring new functionality to the base operating system. In this version, EPEL ties into this staged environment by using a similar method as Fedora does.

Using a directory tree structure like:
EPEL/archives/{5.x, 6.x, 7.x}
EPEL/archives/updates/{5.x, 6.x, 7.x}
EPEL/current/5 -> 5.11/
EPEL/current/5.11/{x86_64,ppc64,x86_32}/
EPEL/current/6 -> 6.7/
EPEL/current/7 -> 7.2/
EPEL/updates/5 -> 5.11/
EPEL/updates/6 -> 6.7/
EPEL/updates/7 -> 7.2/
EPEL/updates/testing/{blah, blah}
EPEL/rawhide/{5, 6, 7}

Then packages would be built in rawhide first. When a new release (say 7.2 to 7.3) occurs, then the builders rawhide trees are updated to the code in the latest. The packages are built against that and hopefully tested to see that they still work. Packagers should have been building any large changes they needed to make in the rawhide tree already but a last minute notification date is done. The go-live for this tree will be either when CentOS gets its build out of that tree or 30 days if CentOS is able to release sooner. On that date the rawhide trees will be branched to /EPEL/current/{6,7}.n (eg 7.3) and updates to those packages will now be sent into EPEL/updates/7.3 . After another 30 days, the 7.2 tree will be moved to EPEL/archives/7.2 {probably really /pub/archives/epel/7.2 }

Pros: this gives a place where newer packages can be built and tested to see what may break in the next major build. it also gives a better release cycle that people can plan against.

Cons: this makes the previous post look simple to implement.

EPEL Proposal #2: EPEL Rolling Rawhide

An EPEL rawhide may sound like an oxymoron for having an always building main tree for a release that is known for its plodding slowness. However there are methods to the madness as people brought it up at FOSDEM Brussels.

Rolling Release Style. In this version there is a branch where many Fedora packages are rebuilt against EL-5/6/7. The packages in this release are always the latest that can be compiled against the release and all 'updates' and fixes are done upstream versus in the package. The epel file tree would look something like this:

EPEL/{5,6,7}
EPEL/testing/{5,6,7}
EPEL/rawhide/{5,6,7}

Packages in EPEL/{5,6,7} are ones that are going to be kept stable until they can no longer be supported. Packages in EPEL/testing/ are were updates to the mainline packages are kept. The general idea is that Fedora packagers can freely fork to EPEL/rawhide with no care about whether it works or not. If people are interested enough to help fix it they can send it to the packager but if it doesn't build and no one fixes it.. it isn't there. If people do care about the package, they can sponsor it and those would be the ones in EPEL regular. These packages would follow the agreed upon rules about updates and lack of big changes.

Pros: Allows for more packages to be newer for more users.
Cons: More channels for release engineering and packagers to deal with.


PSA: Enterprise Linux 5 End of Production on 2017-03-31 and EPEL.

Red Hat Enterprise Linux builds its current releases around a 4 part lifecycle model.

  • Production 1. New drivers and features are added during this cycle along with security and production bug fixes.
  • Production 2. Only security and production bug fixes are done during this cycle.
  • Production 3. Only security and 'critical' bug fixes are done.
  • Extended Lifecycle Support. Security and critical bug fixes are only released to customers who are paying for this extra level of support. For rebuild Linuxes like CentOS and Scientific Linux the product is now End of Lifed.


The last time this happened was in 2012 with the end of life of Enterprise Linux 4. At that time, EPEL stopped supporting builds for EL-4 but no other changes were done. This wasn't a big deal because EL-4 had never gotten a large number of users. However in 1 year, 1 month, EL-5 will reach the end of its production run and move into Extended Life Support. Enterprise 5 is still over 20% of all EPEL installs so I wanted to give a long heads up that EL-5 will also be removed from the builders on April 1 2017 and no builds will be done after that. Depending on other proposals, EL-5 may also be moved over to an 'archive' like Fedora releases are so that it is clear that it is no longer under any production. If that happens there will be clear notice of where it has been moved to and how to keep at it.