2017-06-22

Problems with EPEL and Fedora mirroring: Many Root Cause Analysis

There was a problem with EPEL and Fedora mirrors for the last 24 hours where people getting updates would get various errors like:

Updateinfo file is not valid XML:

The problem was caused by a problem in the compose which output the XML file not as xml but as sqllite. The problem was fixed within a couple of hours on the Fedora side, but it has taken a lot longer to fix further downstream.

  • Some of the Fedora mirror containers were not updating correctly. We use a docker container on each proxy to keep the data fresh. 4? of the 14 proxies said they were updating but seem to not do so. These servers were our main ipv6 servers so people getting updates from these were more affected than other users. 
  • Some mirrors only update 1 or 2 times a day (or even slower). This means that your favourite mirror may keep the data for 12 to 48 hours. 
  • Some client plugins like to peg to a quickest mirror to try and keep downloads fast. While we may tell you that there are 20 mirrors up to date, the plugin will use the one it got stuff fastest from in the past. This means you can end up with going to a 'broken' mirror for a lot longer.
  • Some yum/dnf systems seem to have other options set to keep the bad xml file until it 'ages' out. This means that while an updated xml is there, some systems are still complaining because their box already has it.
The fixes on the Fedora side are to put in better tests to try and see that this does not happen again. The client side fixes are currently to do either one of the following:

  • yum clean all
  • yum clean metadata
Thank you all for your patience on this problem.

2017-06-07

Call for Papers: Flock to Fedora 2017

In summer, an old engineer's fancy turns to writing paper proposals. For it is time for people to submit papers to https://flocktofedora.org/. This year, Flock is being held in Cape Cod Massachusetts from August 29 to September 01. Flock is also focusing on being a 'get-er-done' conference where workshops on getting software problems worked on by many people will have focus. So do you have something you have wanted to get done in Fedora that you needed to have a bunch of people from around the US and Europe to focus on? Put together a short proposal and submit it to https://register.flocktofedora.org/  [Oh and make sure that the people who you need to work with know about it.. and agree that they want to do it also. Surprise is the opposite of consensus.]

The CFP ends on July 15th 2017. Good luck. I am putting in a proposal for a fast moving EPEL workshop. For a more complete post on FLOCK talk/workshop requirements please see http://blog.linuxgrrl.com/2017/06/08/propose-a-talk-for-flock/

2017-05-30

The steam roller of life

Some days it really feels like you are the last man standing as the zombie horde rolls in, and sometimes it feels like people just seem to scream stop at every little thing. However, a lot of times it just looks like this to everyone else:


The security guard is doing his job and is the hero of his own story (in fact has an extra on DVD about his family.) He is trying to get the 'villians' to stop. Austin Powers is the hero in his story because he is just trying to get to the other side of the room to stop Doctor Evil. The vast gulf between the two is just how far apart and how little danger there really is. It is also a story about how avoidable the inevitable crunch at the end is.

  1. The guard could have stood to the left or right and let the steamroller go by. [The guard could have also shot Austin or something else.]
  2. Austin could have 'swerved to the left or right' just a little and missed the guard. [Or he could have gotten out and gotten there faster.]
OK so you are thinking "Yes Captain Obvious that is exactly the humour being shown here.. thank you for breaking it down for us..." The point I am looking at is how often this mirrors our online community problems. Someone is trying to accomplish something, and someone for whatever reason yells stop. (Or someone is meant to keep something stable, and someone is ramming through a new paradigm). Those of us in the moment get caught up in all the energy, and  we forget that to most people outside that all they see is how avoidable the whole confrontation was. 
Sometimes we feel that it is better to get run over by the steamroller than take a step left or right. Sometimes we feel that putting the pedal to the metal on the steamroller is going to make this so much faster, and we can't move it to the right or left for a small change. 

2017-05-24

Canaries in a coal mine (apropos nothing)


[This post is brought to you by Matthew Inman. Reading http://theoatmeal.com/comics/believe made me realize I don't listen enough and Verisatium's https://www.youtube.com/watch?v=UBVV8pch1dM made me realize why thinking is hard. I am writing this to remind myself when I forget and jump on some phrase.]

Various generations ago, part of my family was coal miners and some of their lore was still passed down many many years later. One of those was about the proverbial canary. A lot of people like to think that they are being a canary when they bring up a problem that they believe will cause great harm.. singing louder because they have run out of air.

That isn't what a canary does. The birds in the mines go silent when the air runs out. They may have died or are on the verge of being dead. They got quieter and quieter and what the miners listened for was the lack of noise from birds versus more noise. Of course it is very very hard to hear the birds in the first place in a mine because they aren't quiet places. There is hammering, and shoveling and footsteps echoing down long tubes.. so you might think.. bring more birds.. that just added more distractions and miners would get into fights because the damn birds never shut up. So the birds were few and far between and people would have to check up on the birds every now and then to see if they were still kicking. Safer mines would have some old fellow stay near the bird and if it died/passed out they would begin ringing a bell which could be heard down the hole.

So if analogies were 1:1, the time to worry is not when people are complaining a lot on a mailing list about some change. In fact if everyone complains, then you could interpret that you have too many birds and not enough miners so go ahead. The time to worry would be when things have changed but no one complains. Then you probably really need to look at getting out of the mine (or most likely you will find it is too late).

However analogies are rarely 1:1 or even 1:20. People are not birds, and you should pay attention to when changes cause a lot of consternation. Listen to why the change is causing problems or pain. Take some time to process it, and see what can be done to either alter the change or find a way for the person who is in pain to get out of pain.

2017-04-11

Moving EPEL-4 and EPEL-5 to archives

Today we say goodbye to the last parts of EPEL-5 (and also EPEL-4). The top level files in /pub/epel/4 and /pub/epel/5 were moved to /pub/archive/epel so that people who are still needing packages can get them from the archives. People using yum should not see any change in updates because mirrormanager had the changes to point to archives a couple of days previously.

For any kickstarts or scripts that used the main download servers all that needs to be done is change:


http://dl.fedoraproject.org/pub/epel/5/

to

http://dl.fedoraproject.org/pub/archive/epel/5/

and you can have your kickstart scripts grab the epel rpm from

http://dl.fedoraproject.org/pub/archive/epel/epel-release-latest-5.noarch.rpm

Thanks again to everyone who has helped with EPEL-5 over the years. It was a good crazy ride.

2017-03-17

EPEL-5 article appearing on FedoraMagazine.org

So I thought I was not writing anything more about the EOL of EPEL-5, but I got asked by several people why no one had written anything about it 😐. The ability of my posts to reach the world was much smaller than I realized. In order to rectify that a bit, here is another article on the EOL of EPEL-5 this time at Fedora Magazine.

2017-02-15

IMPORTANT REMINDER: EL 5 is EOL on March 31. 2017

This is probably my final reminder on this before April 3rd 2017. As listed at https://access.redhat.com/support/policy/updates/errata and https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux#Product_life_cycle Red Hat Enterprise Linux will be exiting "Production Phase 3", and CentOS will be archiving off old EL-5 releases.

At that point, all remaining EPEL-5 packages will be archived to /pub/archive/epel/5 for systems to get data from. No new updates or packages will be done after that.