EPEL mirror file layout changes

As several people have noted, the file directory structure of EPEL has changed recently. This layout may require changes in both (1) scripts written with hard-coded locations, and (2) mirrors which were unable to get daily updates from the main mirrors.  While the changes were communicated in meetings, I did not adequately comprehend their effects to let mirrors and EPEL users know about it. This meant this announcement was delayed over two weeks.

 What Happened

The updates in the build system were to add new features and make the release engineering code more manageable. The old release style used by EPEL in EL-6 and EL-7 was different from how all other releases were done and caused several problems for the release code and mirrors.

1.  Due to all the files of the release being in one directory, any code which needed to stat (2) the directory caused the server to go over thousands of files before returning. With EPEL being a large amount of downloads, this negatively impacted systems. Servers mirroring the data could find long delays in rsyncing the data down. 
2. The code that generated this was a 'special' case in the Fedora releng release process which was fragile and tended to cause problems for updates and releases in both EPEL and the normal release.
3. The layouts were different from the current Fedora release so that    people grabbing software from multiple places also had to special case their scripts.

During the updates to the release system with a new version of pungi, it was decided to remove this special case and have all software Fedora created laid out in the same structure by the build tools. This would hopefully make things much more maintainable and improve performance.

In order to safely transition, there would be a time where the old files would remain on the server in the old trees and also be hardlinked to their new location. This was intended to allow for mirrors to get the files with the minimum amount of bandwidth. However there were some problems which showed up.

1. As I said before, I didn't grasp that the change was going to affect EPEL and didn't communicate this to the lists. 
2. The transition time for removing the hardlinks was in days versus weeks. While most mirrors do daily updates, some only do weekly or  monthly rsync's. They missed the hardlinks completely and had to download data twice. 
3. In the usual rule of three, various top level mirrors (mirrors.kernel.org and some others) had un-related mirroring problems at the end of October. When these servers caught up with the new layout, the hardlinked files were gone. This meant that mirrors taking data from a couple of tier1 sites had large uploads.

 How to deal with current things

The current layout structure should be 'solid' for the next couple of years. With the break down of packages into alphabetical subtrees, the 'load' per server should not require a re-ordering in the near future.

If you have written scripts which downloaded a specific file from the mirrors, (aka http://dl.fedoraproject.org/pub/archive/epel/5/i386/epel-release-5-4.noarch.rpm or some similar link), you should instead use a  stable linked package like http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm The epel-release packages get updated regularly to get new macros or other changes so linking to a specific file is very error prone.

Otherwise one should use yum/dnf related commands to get the files from the mirrors. This is useful for mirror sites which may alter the directory structure themselves and thus only the repodata is 'safe' to figure out what to download.