We are seeing a large number of partial rsync on the main download
servers which are causing problems for tier0/1 mirrors to get to the
servers. What we are seeing is that an ip address will start a rsync
of a large tree and then will drop the connection as the server takes
time to work through 1-10TB of disk space. The ip will then initiate a
new connection which another download server will try to fulfill but
again with a time-out and restart. This is adding a large amount of
IOPs onto our backend storage causing a cascading set of problems
through the infrastructure.
We are going to have to put a firewall rules to drop connections from
these systems so that the alpha and other work can get properly
mirrored onto the registered Tier 0 and Tier 1 mirrors. If that
doesn't work we will be putting firewall rules that only Tier 0 and
Tier 1 mirrors are allowed to connect to the download servers.
I will be trying to send emails to the registered abuse for the ips, but if you are getting dropped connections to the download servers or are a mirror administrator at one of the following domains:
- wideopenwest.com
- alshamil.net.ae
- yandex.net
- sl-reverse.com
- univ-ubs.fr
- ip.pt
please register in the Fedora mirror manager so that it can be whitelisted.
Finally, to cut down mirroring problems please do the following:
1) Do not put in a cron job that you are going to do an rsync update
every 15 minutes as several of the above mirrors seem to do. We do not
update the trees that often and rsyncd has to stat every file.
2) Please review the tips and tricks at
Using the last-sync to schedule updates when they actually occur can
help lower rsync usage.
Thank you
No comments:
Post a Comment