2016-09-08

PSA: Fedorahosted.org will be end of lifed in 2017.

Earlier this week Kevin Fenzi posted to the Fedora announce list that fedorahosted.org would be going away in early 2017. Fedora started offering FedoraHosted in  2007 years before GitHub and such were around. It was based around using Trac from https://trac.edgewall.org/ which fit with the pre-git use of SVN  that many projects used for source code control.  [Trac also had plugins for many other revision control systems as this was a time frame where there were many and people had very strong opinions of which ones needed to be supported.]

In any case, Fedorahosted was very useful for a long time, but in the last 2-3 years has seen a fall off in usage with many projects moving to GitHub. Many of the other projects had become nesting grounds for various spam and some malware. This became very evident during the summer Spam Storm when 27000+ spam pages were created in a couple of tracs. In cleaning those out, I found that around 40 other tracs had various 'tickets' with spam in them that were regularly 'updated' by the spammers for years but the project owners had been '/dev/nulling' any tickets so weren't aware that it was happening.

As outlined Kevin outlines in the email, what will happen is the following:

  1. No new fedorahosted projects will be created.
  2. Owners of projects will be sent an email letting them know that the sunset is going to occur on February 29, 2017.
  3. The owners can then move the project to another source repository system. Examples of these would be:
  4. On February 29, 2017, the files will be put into read-only mode and no further updates will be done. 
As with all such end of life projects, I expect that after number 4 there will be "Hey, no one told me that this was going to happen." emails which will require some sort of 5.. but I am hoping that regular blogs and posts and emails will keep that to a minimum.

2016-08-18

PSA: How to unsubscribe from a mailing list

So you have gotten onto some mailing list which was entertaining at first, but is now a drag, time sink or just filling up your mailbox with gigabits of unwanted emails. How do you unsubscribe?

What you do not do is send an email to the list with the word unsubscribe in the body. 

From: "Bob Smith" bob@smith.mail 
Date: Thu, 18 Aug 2016 01:56:16 +0200
To: A Mailing List mailing-list@lists.place.org 
Subject: Re: some subject
unsubscribe

This email does not unsubscribe you from the vast majority of mailing list software out there but instead sends your email to unknown hundreds or thousands of people who are all going to get this email and are now going to fill your mail box with emails that will tell you to not do that but never really explain how to unsubscribe. [They may also tell you do things with yourself or farm animals which are anatomically impossible or potentially life dangerous. People are weird in thinking that is helpful.]

A long, long, long, time ago as in before 1988.. many mailing list software programs would unsubscribe you if you did exactly what Bob did above. When an email went to the list software it would look through all the text and if it found the words unsubscribe it would do that. Of course if you sent an email using that word for some other reason you ended up off the list with little idea why. The next version of the mailing list software would just do this if the unsubscribe was by itself like Bob did above. This worked "ok" in the world of RFC822  where what you saw is what you sent and nothing special was added. It didn't work for in the world of MIME, HTML or other formats (say not an ARPA email but a BITNET email) as the Internet became "mainstream" around 1994 or so. At this point a lot of people were accidentally unsubscribing or being maliciously unsubscribed by trolls who would forge emails saying they were bob@foo.org but weren't.

By the time the third generation of mailing list software came around in 1998, most software does allow for a naked unsubscribe to get you off the list. Instead you either have to go to a specific webpage and unsubscribe or have to email a specific alias that will start the process of un-subscription. For the web it is usually a multi step process of you go to link, you log in, you unsubscribe and you get a confirmation email that you were unsubscribed. For the email address you will usually get an email sent to you asking if you really want to unsubscribe and if you do email back with a special one time code included in the email. This is all because of the ever present malicious troll problem of people who think it is loads of fun to harass people either because that person exists or for the 'lols' which they seem to trade under their bridges for self-validation.

Which brings us back to how do I know how to really unsubscribe from a list. Every email has a bunch of 'hidden' headers that are included which tell various software where the email maybe came from and where it is going. It will also tell software extra data like how to display it and can contain things like how to subscribe, find an archive or unsubscribe. The current standard way this is done is with List-Id: headers. Here are some examples:


Mailing-List: contact cygwin-help@cygwin.com; run by ezmlm
Precedence: bulk
List-Id: cygwin.cygwin.com
List-Unsubscribe: mailto:cygwin-unsubscribe-BOB=bob.mail@cygwin.com
List-Subscribe: mailto:cygwin-subscribe@cygwin.com
List-Archive: http://sourceware.org/ml/cygwin/
List-Post: mailto:cygwin@cygwin.com
List-Help: mailto:cygwin-help@cygwin.com, http://sourceware.org/ml/#faqs

or

Precedence: list
Reply-To: server@lists.fedoraproject.org
List-Id: server.lists.fedoraproject.org
Archived-At: https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject.org/message/457OVFFRFAC7ASPHWZ5OTCH7GACAUYYE/
List-Archive: https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject.org/
List-Help: mailto:server-request@lists.fedoraproject.org?subject=help
List-Post: mailto:server@lists.fedoraproject.org
List-Subscribe: https://lists.fedoraproject.org/admin/lists/server@lists.fedoraproject.org,
 mailto:server-join@lists.fedoraproject.org
List-Unsubscribe: https://lists.fedoraproject.org/admin/lists/server@lists.fedoraproject.org,
 mailto:server-leave@lists.fedoraproject.org

So from the above we have an unsubscribe for ezmlm mailing list system which is to send an email to a specific email address. For the Fedora lists we use Mailman 3 which has both a mailing list address and a web page to go to start the unsubscribe process.

How do you find these hidden headers? It depends on the mail software you are using. Most of them have some sort of "Show original" or "Show Source" which will pop up a new window which will show a lot of headers. In other mailing list software, you will see a button which says "unsubscribe" which will look for the headers and then fire off the things it says in there.

[Edited 2016-08-18] I realized I completely forgot the obvious way to unsubscribe from Mailman lists. At the bottom of every email from a Mailman email list contains the follow:




--
devel mailing list
devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org


That link at the bottom is all you need to click on to get on the road to un-subscription. You will be directed to a page that will ask you to login. You can do so through one of several methods which will get you to a page that looks like this

Click on unsubscribe, and you should be unsubscribed from the list. It is a lot more steps than just typing "unsubscribe" in an email, but in a self-service world that has a lot of trolls.. it is what you end up doing.

PSA: How to unsubscribe from a mailing list

So you have gotten onto some mailing list which was entertaining at first, but is now a drag, time sink or just filling up your mailbox with gigabits of unwanted emails. How do you unsubscribe?

What you do not do is send an email to the list with the word unsubscribe in the body. 

From: "Bob Smith" bob@smith.mail 
Date: Thu, 18 Aug 2016 01:56:16 +0200
To: A Mailing List mailing-list@lists.place.org 
Subject: Re: some subject
unsubscribe

This email does not unsubscribe you from the vast majority of mailing list software out there but instead sends your email to unknown hundreds or thousands of people who are all going to get this email and are now going to fill your mail box with emails that will tell you to not do that but never really explain how to unsubscribe. [They may also tell you do things with yourself or farm animals which are anatomically impossible or potentially life dangerous. People are weird in thinking that is helpful.]

A long, long, long, time ago as in before 1988.. many mailing list software programs would unsubscribe you if you did exactly what Bob did above. When an email went to the list software it would look through all the text and if it found the words unsubscribe it would do that. Of course if you sent an email using that word for some other reason you ended up off the list with little idea why. The next version of the mailing list software would just do this if the unsubscribe was by itself like Bob did above. This worked "ok" in the world of RFC822  where what you saw is what you sent and nothing special was added. It didn't work for in the world of MIME, HTML or other formats (say not an ARPA email but a BITNET email) as the Internet became "mainstream" around 1994 or so. At this point a lot of people were accidentally unsubscribing or being maliciously unsubscribed by trolls who would forge emails saying they were bob@foo.org but weren't.

By the time the third generation of mailing list software came around in 1998, most software does allow for a naked unsubscribe to get you off the list. Instead you either have to go to a specific webpage and unsubscribe or have to email a specific alias that will start the process of un-subscription. For the web it is usually a multi step process of you go to link, you log in, you unsubscribe and you get a confirmation email that you were unsubscribed. For the email address you will usually get an email sent to you asking if you really want to unsubscribe and if you do email back with a special one time code included in the email. This is all because of the ever present malicious troll problem of people who think it is loads of fun to harass people either because that person exists or for the 'lols' which they seem to trade under their bridges for self-validation.

Which brings us back to how do I know how to really unsubscribe from a list. Every email has a bunch of 'hidden' headers that are included which tell various software where the email maybe came from and where it is going. It will also tell software extra data like how to display it and can contain things like how to subscribe, find an archive or unsubscribe. The current standard way this is done is with List-Id: headers. Here are some examples:


Mailing-List: contact cygwin-help@cygwin.com; run by ezmlm
Precedence: bulk
List-Id: cygwin.cygwin.com
List-Unsubscribe: mailto:cygwin-unsubscribe-BOB=bob.mail@cygwin.com
List-Subscribe: mailto:cygwin-subscribe@cygwin.com
List-Archive: http://sourceware.org/ml/cygwin/
List-Post: mailto:cygwin@cygwin.com
List-Help: mailto:cygwin-help@cygwin.com, http://sourceware.org/ml/#faqs

or

Precedence: list
Reply-To: server@lists.fedoraproject.org
List-Id: server.lists.fedoraproject.org
Archived-At: https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject.org/message/457OVFFRFAC7ASPHWZ5OTCH7GACAUYYE/
List-Archive: https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject.org/
List-Help: mailto:server-request@lists.fedoraproject.org?subject=help
List-Post: mailto:server@lists.fedoraproject.org
List-Subscribe: https://lists.fedoraproject.org/admin/lists/server@lists.fedoraproject.org,
 mailto:server-join@lists.fedoraproject.org
List-Unsubscribe: https://lists.fedoraproject.org/admin/lists/server@lists.fedoraproject.org,
 mailto:server-leave@lists.fedoraproject.org

So from the above we have an unsubscribe for ezmlm mailing list system which is to send an email to a specific email address. For the Fedora lists we use Mailman 3 which has both a mailing list address and a web page to go to start the unsubscribe process.

How do you find these hidden headers? It depends on the mail software you are using. Most of them have some sort of "Show original" or "Show Source" which will pop up a new window which will show a lot of headers. In other mailing list software, you will see a button which says "unsubscribe" which will look for the headers and then fire off the things it says in there.

[Edited 2016-08-18] I realized I completely forgot the obvious way to unsubscribe from Mailman lists. At the bottom of every email from a Mailman email list contains the follow:




--
devel mailing list
devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org


That link at the bottom is all you need to click on to get on the road to un-subscription. You will be directed to a page that will ask you to login. You can do so through one of several methods which will get you to a page that looks like this

Click on unsubscribe, and you should be unsubscribed from the list. It is a lot more steps than just typing "unsubscribe" in an email, but in a self-service world that has a lot of trolls.. it is what you end up doing.

2016-08-17

EPEL PSA: Mock builds broken

If you are having problems with building packages with a target for EPEL, it is not because the EPEL gpg keys have been tampered with, but because the mock developer made a mistake in the packages

The problem occurs because mock by default builds EPEL packages with CentOS and the mock developer put the CentOS keys as being the keys to compare EPEL packages with. He is working on a fixed set of packages and the problem should go away with mock-1.2.20 [The broken version is 1.2.19]

2016-08-05

Fedora: How a release is archived. [Fedora 22 moving to archives]

This week has been the week of Fedora Flock in beautiful Krakow, Poland. Due to other travels, I was unable to attend which meant that I have been focusing this week on getting some outstanding tasks done that had been delayed on my part. One of those tasks was getting Fedora 22 ready for being permanently archived.

A Fedora release has many stages in its life.

  1. An embryo in rawhide
  2. A larva in alpha/beta/rc modes
  3. A pupa at release time 
  4. A short mature life as an imago while the next release  is in its pupil stage. 
  5. Then like all things, the release shuffles off this mortal coil and gets moved to the afterlife in the Fedora Archives.   
The Fedora Archives are a hidden resource for people needing to find some older package or to try and track a history of packages. I use it quite extensively for building packages for EPEL ( a post for later ) or to try and find the 'last' workable version of some software as various things in Fedora evolved past the Enterprise Linux release.

The steps for archiving a release are pretty straight forward:
  1.  Before version N+2 goes into beta, hardlink copy with cp -l the version N from /pub/fedora/linux/releases/N into /pub/archive/fedora/linux/N. This allows for any mirrors who do slow updates of the archive tree to start getting the data moved over in their trees.
  2. A couple of weeks after version N+2 has been released, version N is End of Lifed and no more updates will be done. At this point we can do a hardlink copy of /pub/fedora/linux/updates/N and /pub/fedora/linux/updates/testing/N to /pub/archive
  3. A week later we log into the mirrormanager tool and change the point where F-N updates are pointing to the archive version. This means that systems still looking for updates will not get 404's but will be directed to a mirror of the archives.
  4. We then remove the data from the main Fedora disk space.
We can do this because /pub/fedora, /pub/archive and /pub/epel are all on one disk which allows for us to hardlink them together and if a mirror is doing a rsync -avSHP of the trees they should not have to transfer as much data because of the hardlinks. 

In any case, I have started the archive of Fedora 22 and next Friday will complete the archival by moving the pointers in mirrormanager. Thank you Fedora 22 for your service.
Cicada, shell, upper marlboro, md 2014-07-10-19.57.12 ZS PMax
An end of life release

2016-08-04

Fedora: Fixing fonts

Someone on twitter commented that I should write an article about how to fix fonts on Fedora. Sadly this is not something I am able to do. That doesn't mean there isn't a problem, just that I don't have the font specialist eye where they can look at a screen and go "OMG THERE IS SOMETHING WRONG HERE!!!!" in the way that someone who is slightly flat or sharp drives me bonkers. [I on the other hand can happily read a webpage in Comic Sans or Papyrus and wonder why everyone looking over my shouldr is cringing and hissing as I do so.]

That said, I did some research on why fonts look like 'crap' compared to Ubuntu and Windows. It looks like the reason is that there is an option in many fonts called "subpixel rendering" which are not enabled in Fedora but is enabled in other distributions. This looks to be done due to Microsoft having various software patents on ClearType which do not end until 2019. [Other patents ended earlier but those only cover the older TrueType fonts.]

There seem to be fixes that will work for people outside of areas that are covered under current software patents (which for all I know is nowhere because of some super secret trade agreement).

  1. Look at ask.fedoraproject.org for steps to use. 
  2. Look at installing and enabling RPM fusion on your system.
  3. Follow the steps in 1.
Or you can look at something like fedy which I am going to point to the source code for. I don't recommend using the method listed on their website because using a blind su "curl" is a recipe for disaster. Instead I would recommend looking at the code, seeing if you would want to run that on your system and then doing it by hand. 

[I don't care that all the cool kids use a bash -c 'su - "curl ... && run" thing with their nodes and their rubies. I would end with a rant about emacs being good enough for anyone and that the kids are on my lawn.. but you all see that coming anyway.]

2016-08-03

Fedora Statistics: Questions and answers.

I work for Red Hat as a system administrator for the Fedora Project where I get to do a lot of neat and interesting things daily. One of the tasks I have is gathering various statistics for the Fedora Project Leader's state of the hat speeches like the one he has just given at Flock. This means I also get to help answer regular questions on mailing lists and irc channels like "How many users does Fedora have?", "How many downloads of Fedora are there?", "How does this compare to ?"

All of these questions are where Matthew Miller pulls out a clip from a dinosaur movie eating a lawyer on a toilet or similar comedic point. Why? Because raptors live here and will eat you if you do not have a proper escape policy.

The question "How does this compare to ?" is the easiest to answer: "Nothing I give you can be compared to what any other distribution probably says."  This doesn't mean I or they are lying... it just means the terms being used aren't well defined and we are probably using slightly different ones. 

What is a user? Someone who created a Fedora account, did something and never logged in again? Someone who created a Fedora account and logs into FAS daily? weekly? monthly? yearly? Maybe that person who never logs in just answers things on IRC or bugzilla or mailing lists? Maybe the person who logs in daily is just a script that does that because a developer decided to test a cron script and then forgot about it.

What is a download? If I go by raw "GET .*iso .*" in various logs I could say we have had millions of downloads weekly. But anyone who knows weblogs knows that is as fishy as the 1990 website counters and being told "We got N million hits". Those downloads are inflated because of several reasons:
  1. Some people mirror everything. They may not install any of it.. but just in case they ever need it.. they have it.
  2. Some people try to be helpful in promoting their OS by downloading the OS over and over again so that any count of downloads will be larger than the next guys. There are multiple IP addresses which download Fedora isos hourly. No one needs 24 copies of Fedora 8 every day... especially when they had just downloaded 24 copies of Fedora Zod. 
  3. Some "web companies" do the same and then send us mailers about how we can see how they have increased our downloads and if we used them we could see even further growth. 
  4. A lot of people use specialized download tools which try to torrent downloads via http. What they do is ask 20-100 times for a mirror and then use each one to download a bit of the file as a speed booster of some sort. This shows up as even more downloads.
  5. Then there are the people who are stuck on NAT over NAT or satellite links. They may show up as a dozen or so IP addresses in their attempt to grab a single IP address. 
On the other side there are multiple people behind a single NAT so that the 200 downloads from IBM are probably not the same system.. but maybe they are? There are ways to get clearer results via various 'fingerprinting' techniques but I don't think that they really can help when you have companies whose basic job is to bump up numbers so you can lie about how good your product is doing on the web. [I am going to avoid commenting on the morality of deep fingerprinting because I am in a rather cranky mood today.] Depending on how one looks at the data, you can keep discounting stuff further and further down until your only answer is that you know that you had more than 1 download and less than whatever your max amount of non spider hits were. 

So what does that leave us with so many "unknown unknowns" and too few "known knowns"? What we have been using has been not looking at downloads at all and instead focus on actual users who have installed the OS and are using yum or dnf to update their systems. This can give us a rough lower bound number of 'active' systems. Going through the mirror logs we create a large amount of tuples of (date, ip address, hardware arch, fedora release). We then unique this list to deal with the various users who have cron set up to do a yum update every 10 minutes. It has a bad effect of making the thousands of systems behind the Red Hat NAT be counted as 1 system, but we hope that is made up for by the person doing a yum update over their Verizon phone and showing up as 20 ips as they get ip shifted every now and then. 
Fedora OS yum checkins (oldest highest)

Now again these are 'reasonable' minimum numbers. The N thousands of ARM IOT systems with Fedora on them do not do yum updates so aren't counted. The systems which are hard configured to use a local mirror aren't counted. And so on and so on. 

[Caveats:
  1. The above graph is a hack job I created using awk and gnuplot with a 7 day moving average calculated via python pandas. I expect that it could be made prettier or cleaner through various ways. 
  2. The drop off in late 2008 was due to the webservers being mostly off during the security incident of that time.
  3. The hockey stick drop off in late 2014 is due to Fedora dropping support for RC4 encryption and various systems hard coded to use it not being able to check in any more. All of these systems were way past End of Life for each release so they were not getting any updates.
  4. Most releases have a growth pattern of continually growing until the day the next release comes out. That pattern changes for Fedora 8, 14 and 22 which seem to have continued to grow even after they were end of lifed (or close to it). These seem to be due to some VPS, cloud or similar provider continually basing images off of these releases.
]

I think somewhere in this post I lost my original focus. Sorry about that.. I looked at the picture and got all oooh I need to talk about that.