2017-01-07

Fedora/EPEL Mirrormanager problems in Asia Pacific countries.

We have been getting a lot of reports of people unable to get updates for EPEL or Fedora at various times. What people are seeing is that they will do a 'yum update' and it will give a long list of failures and quit. At this moment we seem to have pinpointed that most of the people having this problem are in various Asia Pacific nations (primarily Australia and Japan). The problem for both of these seems to be a lack of cross connects between networks.

In the US, if you are on Comcast in say New Mexico and going to a server on Time Warner in North Carolina, your route is usually pretty direct. You will go from one network to various third party providers who will then send the packets the quickest path to the eventual server. If you use a visual grapher of locations, you even find that the path usually follows a linear path. [You might end up going to say California or Seattle first but that is only when Texas and Colorado cross connects are full.] Similarly in most European countries you also see a similar routing algorithm.

In the various Pacific and Indian Ocean countries, you do not see similar interconnects. You can watch a system in Sydney on one network send packets all the way to San Francisco and back again to a server in Sydney because the two telecoms do not 'talk' with each other. This seems to happen also in Japan for a couple of telecom networks. The result of this is that it is much more expensive to mirror data in those countries than you would think. For users it might be faster to get data from mainland China or the United States than it is to get it from a server only hundreds of miles away.

The problem is that mirrormanager is currently not coded to deal with that. It makes an optimistic assumption that you are in Adele and the nearest server is in Sydney.. you should go to that. The mirror in Sydney though is still catching up with data from pulling things in the mainland US (or if the mirror admin made the assumption that an asia pacific mirror is the one to go to.. may be pulling data from a server 20 miles physically and several tens of thousand miles away by network.) The mirrormanager developers are trying to figure out ways to deal with this without making servers and clients having to send each other network maps with throughput charts to figure out things.. [And no the fastest mirror yum plugin doesn't fix this for all/most people. It uses a very very simple 'works for me' test to figure out what mirrors might be a good match at one point in time. You could end up with using a poor mirror 90% of the time but the one time it set itself up.. it also just uses the "hey ping is fast" dynamic which breaks for people on various networks. Improving the fastest mirror plugin would be useful if someone did it.]

So what to do? For EPEL, the current fix is to edit your /etc/yum.repos.d/epel.repo files by adding a '&country=global'



[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/7/$basearch
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=$basearch&country=global
failovermethod=priority
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7


This will cause yum to ask for the global versus 'local' and you will get all the mirrors. This usually will give a few servers which are in sync even if they are 'not' local.

No comments: