Fedora/EPEL Mirrormanager problems in Asia Pacific countries.

We have been getting a lot of reports of people unable to get updates for EPEL or Fedora at various times. What people are seeing is that they will do a 'yum update' and it will give a long list of failures and quit. At this moment we seem to have pinpointed that most of the people having this problem are in various Asia Pacific nations (primarily Australia and Japan). The problem for both of these seems to be a lack of cross connects between networks.

In the US, if you are on Comcast in say New Mexico and going to a server on Time Warner in North Carolina, your route is usually pretty direct. You will go from one network to various third party providers who will then send the packets the quickest path to the eventual server. If you use a visual grapher of locations, you even find that the path usually follows a linear path. [You might end up going to say California or Seattle first but that is only when Texas and Colorado cross connects are full.] Similarly in most European countries you also see a similar routing algorithm.

In the various Pacific and Indian Ocean countries, you do not see similar interconnects. You can watch a system in Sydney on one network send packets all the way to San Francisco and back again to a server in Sydney because the two telecoms do not 'talk' with each other. This seems to happen also in Japan for a couple of telecom networks. The result of this is that it is much more expensive to mirror data in those countries than you would think. For users it might be faster to get data from mainland China or the United States than it is to get it from a server only hundreds of miles away.

The problem is that mirrormanager is currently not coded to deal with that. It makes an optimistic assumption that you are in Adele and the nearest server is in Sydney.. you should go to that. The mirror in Sydney though is still catching up with data from pulling things in the mainland US (or if the mirror admin made the assumption that an asia pacific mirror is the one to go to.. may be pulling data from a server 20 miles physically and several tens of thousand miles away by network.) The mirrormanager developers are trying to figure out ways to deal with this without making servers and clients having to send each other network maps with throughput charts to figure out things.. [And no the fastest mirror yum plugin doesn't fix this for all/most people. It uses a very very simple 'works for me' test to figure out what mirrors might be a good match at one point in time. You could end up with using a poor mirror 90% of the time but the one time it set itself up.. it also just uses the "hey ping is fast" dynamic which breaks for people on various networks. Improving the fastest mirror plugin would be useful if someone did it.]

So what to do? For EPEL, the current fix is to edit your /etc/yum.repos.d/epel.repo files by adding a '&country=global'

name=Extra Packages for Enterprise Linux 7 - $basearch

This will cause yum to ask for the global versus 'local' and you will get all the mirrors. This usually will give a few servers which are in sync even if they are 'not' local.


Why are EPEL python packages getting renamed?

Someone joined the irc.freenode.net #epel channel on early Saturday (2016-12-10) and asked:

[2016-12-10-07:33 UTC]  hi, it seems that python-pip was renamed to python2-pip, is there any documentation available why that choice was made?

By the time I got to IRC (12 hours later), they had left so I am going to try and answer them here. . Python packages in EPEL are made like every other package from Fedora. Currently Fedora is working on finishing making Python3 their default python and moving the older python to python2 naming. In doing so python packages are either being named python2 or python3 depending on which chain they are built against. This is going to have an effect on the names of Python packages in EPEL for EL6 and EL7 as the changes occur. For more information on Fedora Python packaging guidelines please see the following link.


Puppet not coming back to EPEL EL-6

I made a grievous mistake in my previous blog post about the removal of Puppet Software and would it be coming back to EL-6. At the time, I misread someone's IRC comment and thought it was. However after a couple of days of not seeing any updated builds in Koji, I decided to check and found out I was wrong. The maintainers (and upstream) will not be putting puppet back in EL-6 for a couple of good reasons:

  1. The version that was in EL-6 was actually a very old version no longer maintained or updated by PuppetLabs. 
  2. Puppetlabs maintains a set of packages which will work on EL-6 distributions and focuses their attention there for problems and bug fixes.
So what to do? The upstream recommends that users of puppet use the one available from from Puppetlabs called puppet collections. The documentation on setting up the repository and using it are pretty clear, but for security sakes I will add a couple of steps.

$ sudo -i
# mkdir /root/puppet; cd /root/puppet
# wget https://yum.puppetlabs.com/puppetlabs-release-pc1-el-6.noarch.rpm

if you want to try and check the signature before installing you can add the following steps:

# wget https://yum.puppetlabs.com/RPM-GPG-KEY-puppetlabs
# rpm --import RPM-GPG-KEY-puppetlabs
# rpm -K puppetlabs-release-pc1-el-6.noarch.rpm 
however this does mean altering the rpm database before you check to see if it is ok.

# yum localinstall puppetlabs-release-pc1-el-6.noarch.rpm

if you are wanting a local copy of the repository for other boxes to install from you can use 'reposync'. For more on configuration management, you might also want to look at or join the CentOS Configuration Management Special Interest Group

[UPDATE: The old packages for puppet are still in Fedora Koji at http://koji.fedoraproject.org/koji/buildinfo?buildID=609117 ]
[UPDATE2: For better instructions on how to carefully check GPG headers of packages before you install them willy nilly, try the following:
http://orcorc.blogspot.com/2008/08/gnupg-few-minutes-on-using-detached-and.html ]


Abiword for EL-7

Over Thanksgiving break, I decided to go through a long list of emails that were marked "when you have a spare moment". I really didn't have one but I realized that many of those emails were crufty and old. One was some people asking about getting abiword together for EL-7. This looked like a straightforward enough task so I got into it and started working out all the packages that would need to be branched to say EPEL and what would be needed to compile them.
At first the packages were pretty easy to bring over from Fedora:

  • ./aiksaurus/aiksaurus-1.2.1-34.el7.centos.src.rpm
  • ./gtkmathview/gtkmathview-0.8.0-19.el7.centos.src.rpm 
  • ./libwps/libwps-0.4.4-1.el7.centos.src.rpm 
  • ./wv/wv-1.2.9-13.el7.centos.src.rpm 
  • ./loudmouth/loudmouth-1.5.3-1.el7.centos.src.rpm  
  • ./ots/ots-0.5.0-12.el7.centos.src.rpm
Then we got to the ones which didn't compile so cleanly.
  • ./minisat/minisat2-2.2.0-5.fc18.src.rpm
  • ./link-grammar/link-grammar-5.0.8-3.el7.centos.src.rpm
  • ./abiword/abiword-3.0.1-12.el7.centos.src.rpm

The latest minisat2 does compile but I was getting all kinds of compilation errors with link-grammar-5.3 in using it. Backing down to an older version of minisat2 and link-grammar I could get other parts to compile but I could not get abiword to compile. Through some pointers from Yaakov  Selkowitz who had dealt with this package in Cygwin, I was able to get it compiled. I then opened an email to the link-grammar people who figured out that there was a bug in their code when compiling on CentOS or Scientific Linux.  They haven't spun a new release with the fix yet but when they do I can update the packages.  At the moment, the abiword package has to be compiled with the --disable-introspection flag which probably limits some functionality.
In the end, we have a set of packages now available for testing a copr:

name=Copr repo for Abiword owned by smooge

Good luck and let me know if it works for you.

Where has puppet gone in EPEL-6

This week various people using EPEL on RHEL and CentOS 6 have found that the puppet package is no longer provided by EPEL. The reason for this is due to the way EPEL packages are built and kept inside the repository. A package needs a sponsor so that we can hopefully get bug fixes and security updates to it. In the case of puppet this package is sponsored by the user kanarip. However, most packages aren't whole pieces, they rely on other software.. in this case the package puppet relies on a lot of different ruby gems of which one of them was called ruby-shadow. This package was orphaned 30 weeks ago and while it did have other people watching it, none of them took over the package.

Depending on: ruby-shadow (3), status change: 2016-04-13 (30 weeks ago)
        puppet (maintained by: kanarip, domcleal, gchamoul, georgiou, jehane, jpo, lzap, mmagr, moses, rharrison, skottler, stahnma)
                puppet-2.7.26-2.el6.noarch requires ruby-shadow = 1.4.1-13.el6

        puppet-gluster (maintained by: averi, purpleidea)
                puppet-gluster-0.0.3-1.el6.noarch requires puppet = 2.7.26-2.el6

        puppetlabs-stdlib (maintained by: averi, gchamoul, purpleidea, shlomizadok)
                puppetlabs-stdlib-4.5.1-2.20150121git7a91f20.el6.noarch requires puppet = 2.7.26-2.el6

Last week a large cleanup was done to clean out orphaned packages from EPEL which removed ruby-shadow. Once that was removed, then all of the other packages depending on ruby-shadow were also removed. Today various people reinstalling systems found puppet wasn't around and came onto #epel to ask.. which seems to have gotten the packages responsored and hopefully they will be back in the EPEL release in a day or so.

This problem has been happening a lot lately. I think it shows quite a few problems with how EPEL is set up and managed. For this, I take responsibility as I said I would try to clean it up after FOSDEM 2016 and it is still happening.

[Correction 2016-12-01 added. Please see my updated post.]


Another tale, apropos nothing again

When Nema got back to her village she found that her house had nothing in it for planting the next year. Her parents were dead, and her younger brothers were used to drinking and gambling at the tavern house with what was left of the inheritance.

Now the Duke's fields had been cleared recently, and it was the law that peasants could gather any seed left over from the harvest. Nema asked her brothers if they would come help her do so. 

"Not I" they said, for they had games they wanted to play.

So Nema went through the fields and gathered as much seed as she could get. 

When the spring came, she needed to plant the seeds. She asked her brothers if they would help.

"Not I" they said, for they had ale they wanted to drink.

The grain grew and it needed to be harvested. Nema asked her brothers again if they would help.

"Not I" they said for their was both ale and games to think about.

The grain was harvested and milled at the miller. It was time to bake the bread, but none of the brothers were around to help out. Yet when the bread was just out of the oven.. who were to show up? All the brothers asking for bread.

Now Nema could have said something like "Thems that works gets to eat." but she was kind hearted and liked to share. So she gave them loaves from the oven and what did they cry out? "We wanted pumpernickel and this is just plain rye". 

A tale apropos nothing

Once long ago, there was a war where many soldiers went off to war. In those days, win or lose if the war was over you were dropped out of the army and walked your way back to your home countless miles away.

Now there was one formerly conscripted soldier named Nema who walked through the forests and she came upon a village. Now Nema was very hungry but had no money as whatever payment was either long gone. She begged from house but found that the villagers did not like soldiers and told her to go somewhere else. In fact, the villagers were tired of soldiers and fighting and looting and always being the ones trodden on. They didn't even like each other that much because they were all sure that someone had been a collaborator sending soldiers to their house to take their last food.

When the villager had gone to all the houses, Nema sat in the square wondering what to do. Her grumbling stomach told her that she couldn't walk to another village. She had nothing to trade as the only weapon the army had let her take back, an axe, had broken when she had cut wood for her meal to villages back. Looking at the head of the axe an idea sprang into her mind. Nema remembered that one of the villagers had a large cauldron in her front yard.. probably to clean clothes or something..

Going to the villager hut, Nema asked again if she could borrow the pot to make a soup. She would clean it out as a service but she had a magical axe which just needed some water and a fire to make the tastiest soup ever. The old villager eyed Nema as crazy, but because the cauldron was outside and it did need a good cleaning thought it would be an even score. Nema cleaned the pot and gathered water from the village well. She got a fire going and put her axe in it.

The villagers started to gather around this crazy soldier and with much laughter asking how the water soup tasted. Nema would spoon out a bit and say "hmm it is OK but it just needs something to flavor it up a bit." One of the villagers thought and said "Oh I have some old beans.. maybe that will do it." They got the beans from where ever they were hidden and the soldier put it in the pot. Again the villagers asked "oh how is the soup". The soldier stirred some more and said "Oh the beans helped a lot but now it needs a balance." Another villager remembered some old carrots and onions they had. These were added to the pot. This went on for a while and shortly afterwords there was a wondrous soup for everyone to share and eat.

After all the soup had been divvied up, Nema took her magic axe head out of the bottom and put it back in her pouch. She would make it back to her own village many miles away, and this villagers would talk with each other and make many soups from what they had through the long winter.