PSA: How to unsubscribe from a mailing list

So you have gotten onto some mailing list which was entertaining at first, but is now a drag, time sink or just filling up your mailbox with gigabits of unwanted emails. How do you unsubscribe?

What you do not do is send an email to the list with the word unsubscribe in the body. 

From: "Bob Smith" bob@smith.mail 
Date: Thu, 18 Aug 2016 01:56:16 +0200
To: A Mailing List mailing-list@lists.place.org 
Subject: Re: some subject
unsubscribe

This email does not unsubscribe you from the vast majority of mailing list software out there but instead sends your email to unknown hundreds or thousands of people who are all going to get this email and are now going to fill your mail box with emails that will tell you to not do that but never really explain how to unsubscribe. [They may also tell you do things with yourself or farm animals which are anatomically impossible or potentially life dangerous. People are weird in thinking that is helpful.]

A long, long, long, time ago as in before 1988.. many mailing list software programs would unsubscribe you if you did exactly what Bob did above. When an email went to the list software it would look through all the text and if it found the words unsubscribe it would do that. Of course if you sent an email using that word for some other reason you ended up off the list with little idea why. The next version of the mailing list software would just do this if the unsubscribe was by itself like Bob did above. This worked "ok" in the world of RFC822  where what you saw is what you sent and nothing special was added. It didn't work for in the world of MIME, HTML or other formats (say not an ARPA email but a BITNET email) as the Internet became "mainstream" around 1994 or so. At this point a lot of people were accidentally unsubscribing or being maliciously unsubscribed by trolls who would forge emails saying they were bob@foo.org but weren't.

By the time the third generation of mailing list software came around in 1998, most software does allow for a naked unsubscribe to get you off the list. Instead you either have to go to a specific webpage and unsubscribe or have to email a specific alias that will start the process of un-subscription. For the web it is usually a multi step process of you go to link, you log in, you unsubscribe and you get a confirmation email that you were unsubscribed. For the email address you will usually get an email sent to you asking if you really want to unsubscribe and if you do email back with a special one time code included in the email. This is all because of the ever present malicious troll problem of people who think it is loads of fun to harass people either because that person exists or for the 'lols' which they seem to trade under their bridges for self-validation.

Which brings us back to how do I know how to really unsubscribe from a list. Every email has a bunch of 'hidden' headers that are included which tell various software where the email maybe came from and where it is going. It will also tell software extra data like how to display it and can contain things like how to subscribe, find an archive or unsubscribe. The current standard way this is done is with List-Id: headers. Here are some examples:

Mailing-List: contact cygwin-help@cygwin.com; run by ezmlm
Precedence: bulk
List-Id: cygwin.cygwin.com
List-Unsubscribe: mailto:cygwin-unsubscribe-BOB=bob.mail@cygwin.com
List-Subscribe: mailto:cygwin-subscribe@cygwin.com
List-Archive: http://sourceware.org/ml/cygwin/
List-Post: mailto:cygwin@cygwin.com
List-Help: mailto:cygwin-help@cygwin.com, http://sourceware.org/ml/#faqs

or

Precedence: list
Reply-To: server@lists.fedoraproject.org
List-Id: server.lists.fedoraproject.org
Archived-At: https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject.org/message/457OVFFRFAC7ASPHWZ5OTCH7GACAUYYE/
List-Archive: https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject.org/
List-Help: mailto:server-request@lists.fedoraproject.org?subject=help
List-Post: mailto:server@lists.fedoraproject.org
List-Subscribe: https://lists.fedoraproject.org/admin/lists/server@lists.fedoraproject.org,
 mailto:server-join@lists.fedoraproject.org
List-Unsubscribe: https://lists.fedoraproject.org/admin/lists/server@lists.fedoraproject.org,
 mailto:server-leave@lists.fedoraproject.org

So from the above we have an unsubscribe for ezmlm mailing list system which is to send an email to a specific email address. For the Fedora lists we use Mailman 3 which has both a mailing list address and a web page to go to start the unsubscribe process.

How do you find these hidden headers? It depends on the mail software you are using. Most of them have some sort of "Show original" or "Show Source" which will pop up a new window which will show a lot of headers. In other mailing list software, you will see a button which says "unsubscribe" which will look for the headers and then fire off the things it says in there.

[Edited 2016-08-18] I realized I completely forgot the obvious way to unsubscribe from Mailman lists. At the bottom of every email from a Mailman email list contains the follow:

--
devel mailing list
devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

That link at the bottom is all you need to click on to get on the road to un-subscription. You will be directed to a page that will ask you to login. You can do so through one of several methods which will get you to a page that looks like this

Click on unsubscribe, and you should be unsubscribed from the list. It is a lot more steps than just typing "unsubscribe" in an email, but in a self-service world that has a lot of trolls.. it is what you end up doing.

PSA: How to unsubscribe from a mailing list

So you have gotten onto some mailing list which was entertaining at first, but is now a drag, time sink or just filling up your mailbox with gigabits of unwanted emails. How do you unsubscribe?

What you do not do is send an email to the list with the word unsubscribe in the body. 

From: "Bob Smith" bob@smith.mail 
Date: Thu, 18 Aug 2016 01:56:16 +0200
To: A Mailing List mailing-list@lists.place.org 
Subject: Re: some subject
unsubscribe

This email does not unsubscribe you from the vast majority of mailing list software out there but instead sends your email to unknown hundreds or thousands of people who are all going to get this email and are now going to fill your mail box with emails that will tell you to not do that but never really explain how to unsubscribe. [They may also tell you do things with yourself or farm animals which are anatomically impossible or potentially life dangerous. People are weird in thinking that is helpful.]

A long, long, long, time ago as in before 1988.. many mailing list software programs would unsubscribe you if you did exactly what Bob did above. When an email went to the list software it would look through all the text and if it found the words unsubscribe it would do that. Of course if you sent an email using that word for some other reason you ended up off the list with little idea why. The next version of the mailing list software would just do this if the unsubscribe was by itself like Bob did above. This worked "ok" in the world of RFC822  where what you saw is what you sent and nothing special was added. It didn't work for in the world of MIME, HTML or other formats (say not an ARPA email but a BITNET email) as the Internet became "mainstream" around 1994 or so. At this point a lot of people were accidentally unsubscribing or being maliciously unsubscribed by trolls who would forge emails saying they were bob@foo.org but weren't.

By the time the third generation of mailing list software came around in 1998, most software does allow for a naked unsubscribe to get you off the list. Instead you either have to go to a specific webpage and unsubscribe or have to email a specific alias that will start the process of un-subscription. For the web it is usually a multi step process of you go to link, you log in, you unsubscribe and you get a confirmation email that you were unsubscribed. For the email address you will usually get an email sent to you asking if you really want to unsubscribe and if you do email back with a special one time code included in the email. This is all because of the ever present malicious troll problem of people who think it is loads of fun to harass people either because that person exists or for the 'lols' which they seem to trade under their bridges for self-validation.

Which brings us back to how do I know how to really unsubscribe from a list. Every email has a bunch of 'hidden' headers that are included which tell various software where the email maybe came from and where it is going. It will also tell software extra data like how to display it and can contain things like how to subscribe, find an archive or unsubscribe. The current standard way this is done is with List-Id: headers. Here are some examples:

Mailing-List: contact cygwin-help@cygwin.com; run by ezmlm
Precedence: bulk
List-Id: cygwin.cygwin.com
List-Unsubscribe: mailto:cygwin-unsubscribe-BOB=bob.mail@cygwin.com
List-Subscribe: mailto:cygwin-subscribe@cygwin.com
List-Archive: http://sourceware.org/ml/cygwin/
List-Post: mailto:cygwin@cygwin.com
List-Help: mailto:cygwin-help@cygwin.com, http://sourceware.org/ml/#faqs

or

Precedence: list
Reply-To: server@lists.fedoraproject.org
List-Id: server.lists.fedoraproject.org
Archived-At: https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject.org/message/457OVFFRFAC7ASPHWZ5OTCH7GACAUYYE/
List-Archive: https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject.org/
List-Help: mailto:server-request@lists.fedoraproject.org?subject=help
List-Post: mailto:server@lists.fedoraproject.org
List-Subscribe: https://lists.fedoraproject.org/admin/lists/server@lists.fedoraproject.org,
 mailto:server-join@lists.fedoraproject.org
List-Unsubscribe: https://lists.fedoraproject.org/admin/lists/server@lists.fedoraproject.org,
 mailto:server-leave@lists.fedoraproject.org

So from the above we have an unsubscribe for ezmlm mailing list system which is to send an email to a specific email address. For the Fedora lists we use Mailman 3 which has both a mailing list address and a web page to go to start the unsubscribe process.

How do you find these hidden headers? It depends on the mail software you are using. Most of them have some sort of "Show original" or "Show Source" which will pop up a new window which will show a lot of headers. In other mailing list software, you will see a button which says "unsubscribe" which will look for the headers and then fire off the things it says in there.

[Edited 2016-08-18] I realized I completely forgot the obvious way to unsubscribe from Mailman lists. At the bottom of every email from a Mailman email list contains the follow:

--
devel mailing list
devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

That link at the bottom is all you need to click on to get on the road to un-subscription. You will be directed to a page that will ask you to login. You can do so through one of several methods which will get you to a page that looks like this

Click on unsubscribe, and you should be unsubscribed from the list. It is a lot more steps than just typing "unsubscribe" in an email, but in a self-service world that has a lot of trolls.. it is what you end up doing.

EPEL PSA: Mock builds broken

If you are having problems with building packages with a target for EPEL, it is not because the EPEL gpg keys have been tampered with, but because the mock developer made a mistake in the packages

The problem occurs because mock by default builds EPEL packages with CentOS and the mock developer put the CentOS keys as being the keys to compare EPEL packages with. He is working on a fixed set of packages and the problem should go away with mock-1.2.20 [The broken version is 1.2.19]

Fedora: How a release is archived. [Fedora 22 moving to archives]

This week has been the week of Fedora Flock in beautiful Krakow, Poland. Due to other travels, I was unable to attend which meant that I have been focusing this week on getting some outstanding tasks done that had been delayed on my part. One of those tasks was getting Fedora 22 ready for being permanently archived.

A Fedora release has many stages in its life.

1. An embryo in rawhide, 
2. A larva in alpha/beta/rc modes, 
3. A pupa at release time 
4. A short mature life as an imago while the next release  is in its pupil stage. 
5. Then like all things, the release shuffles off this mortal coil and gets moved to the afterlife in the Fedora Archives.   

The Fedora Archives are a hidden resource for people needing to find some older package or to try and track a history of packages. I use it quite extensively for building packages for EPEL ( a post for later ) or to try and find the 'last' workable version of some software as various things in Fedora evolved past the Enterprise Linux release.

The steps for archiving a release are pretty straight forward:

1.  Before version N+2 goes into beta, hardlink copy with cp -l the version N from /pub/fedora/linux/releases/N into /pub/archive/fedora/linux/N. This allows for any mirrors who do slow updates of the archive tree to start getting the data moved over in their trees.
2. A couple of weeks after version N+2 has been released, version N is End of Lifed and no more updates will be done. At this point we can do a hardlink copy of /pub/fedora/linux/updates/N and /pub/fedora/linux/updates/testing/N to /pub/archive
3. A week later we log into the mirrormanager tool and change the point where F-N updates are pointing to the archive version. This means that systems still looking for updates will not get 404's but will be directed to a mirror of the archives.
4. We then remove the data from the main Fedora disk space.

We can do this because /pub/fedora, /pub/archive and /pub/epel are all on one disk which allows for us to hardlink them together and if a mirror is doing a rsync -avSHP of the trees they should not have to transfer as much data because of the hardlinks. 

In any case, I have started the archive of Fedora 22 and next Friday will complete the archival by moving the pointers in mirrormanager. Thank you Fedora 22 for your service.

An end of life release

Fedora: Fixing fonts

Someone on twitter commented that I should write an article about how to fix fonts on Fedora. Sadly this is not something I am able to do. That doesn't mean there isn't a problem, just that I don't have the font specialist eye where they can look at a screen and go "OMG THERE IS SOMETHING WRONG HERE!!!!" in the way that someone who is slightly flat or sharp drives me bonkers. [I on the other hand can happily read a webpage in Comic Sans or Papyrus and wonder why everyone looking over my shouldr is cringing and hissing as I do so.]

That said, I did some research on why fonts look like 'crap' compared to Ubuntu and Windows. It looks like the reason is that there is an option in many fonts called "subpixel rendering" which are not enabled in Fedora but is enabled in other distributions. This looks to be done due to Microsoft having various software patents on ClearType which do not end until 2019. [Other patents ended earlier but those only cover the older TrueType fonts.]

There seem to be fixes that will work for people outside of areas that are covered under current software patents (which for all I know is nowhere because of some super secret trade agreement).

1. Look at ask.fedoraproject.org for steps to use. 
2. Look at installing and enabling RPM fusion on your system.
3. Follow the steps in 1.

Or you can look at something like fedy which I am going to point to the source code for. I don't recommend using the method listed on their website because using a blind su "curl" is a recipe for disaster. Instead I would recommend looking at the code, seeing if you would want to run that on your system and then doing it by hand. 

[I don't care that all the cool kids use a bash -c 'su - "curl ... && run" thing with their nodes and their rubies. I would end with a rant about emacs being good enough for anyone and that the kids are on my lawn.. but you all see that coming anyway.]

Fedora Statistics: Questions and answers.

I work for Red Hat as a system administrator for the Fedora Project where I get to do a lot of neat and interesting things daily. One of the tasks I have is gathering various statistics for the Fedora Project Leader's state of the hat speeches like the one he has just given at Flock. This means I also get to help answer regular questions on mailing lists and irc channels like "How many users does Fedora have?", "How many downloads of Fedora are there?", "How does this compare to ?"

All of these questions are where Matthew Miller pulls out a clip from a dinosaur movie eating a lawyer on a toilet or similar comedic point. Why? Because raptors live here and will eat you if you do not have a proper escape policy.

The question "How does this compare to ?" is the easiest to answer: "Nothing I give you can be compared to what any other distribution probably says."  This doesn't mean I or they are lying... it just means the terms being used aren't well defined and we are probably using slightly different ones. 

What is a user? Someone who created a Fedora account, did something and never logged in again? Someone who created a Fedora account and logs into FAS daily? weekly? monthly? yearly? Maybe that person who never logs in just answers things on IRC or bugzilla or mailing lists? Maybe the person who logs in daily is just a script that does that because a developer decided to test a cron script and then forgot about it.

What is a download? If I go by raw "GET .*iso .*" in various logs I could say we have had millions of downloads weekly. But anyone who knows weblogs knows that is as fishy as the 1990 website counters and being told "We got N million hits". Those downloads are inflated because of several reasons:

1. Some people mirror everything. They may not install any of it.. but just in case they ever need it.. they have it.
2. Some people try to be helpful in promoting their OS by downloading the OS over and over again so that any count of downloads will be larger than the next guys. There are multiple IP addresses which download Fedora isos hourly. No one needs 24 copies of Fedora 8 every day... especially when they had just downloaded 24 copies of Fedora Zod. 
3. Some "web companies" do the same and then send us mailers about how we can see how they have increased our downloads and if we used them we could see even further growth. 
4. A lot of people use specialized download tools which try to torrent downloads via http. What they do is ask 20-100 times for a mirror and then use each one to download a bit of the file as a speed booster of some sort. This shows up as even more downloads.
5. Then there are the people who are stuck on NAT over NAT or satellite links. They may show up as a dozen or so IP addresses in their attempt to grab a single IP address. 

On the other side there are multiple people behind a single NAT so that the 200 downloads from IBM are probably not the same system.. but maybe they are? There are ways to get clearer results via various 'fingerprinting' techniques but I don't think that they really can help when you have companies whose basic job is to bump up numbers so you can lie about how good your product is doing on the web. [I am going to avoid commenting on the morality of deep fingerprinting because I am in a rather cranky mood today.] Depending on how one looks at the data, you can keep discounting stuff further and further down until your only answer is that you know that you had more than 1 download and less than whatever your max amount of non spider hits were. 

So what does that leave us with so many "unknown unknowns" and too few "known knowns"? What we have been using has been not looking at downloads at all and instead focus on actual users who have installed the OS and are using yum or dnf to update their systems. This can give us a rough lower bound number of 'active' systems. Going through the mirror logs we create a large amount of tuples of (date, ip address, hardware arch, fedora release). We then unique this list to deal with the various users who have cron set up to do a yum update every 10 minutes. It has a bad effect of making the thousands of systems behind the Red Hat NAT be counted as 1 system, but we hope that is made up for by the person doing a yum update over their Verizon phone and showing up as 20 ips as they get ip shifted every now and then. 

Fedora OS yum checkins (oldest highest)

Now again these are 'reasonable' minimum numbers. The N thousands of ARM IOT systems with Fedora on them do not do yum updates so aren't counted. The systems which are hard configured to use a local mirror aren't counted. And so on and so on. 

[Caveats:

1. The above graph is a hack job I created using awk and gnuplot with a 7 day moving average calculated via python pandas. I expect that it could be made prettier or cleaner through various ways. 
2. The drop off in late 2008 was due to the webservers being mostly off during the security incident of that time.
3. The hockey stick drop off in late 2014 is due to Fedora dropping support for RC4 encryption and various systems hard coded to use it not being able to check in any more. All of these systems were way past End of Life for each release so they were not getting any updates.
4. Most releases have a growth pattern of continually growing until the day the next release comes out. That pattern changes for Fedora 8, 14 and 22 which seem to have continued to grow even after they were end of lifed (or close to it). These seem to be due to some VPS, cloud or similar provider continually basing images off of these releases.

]

I think somewhere in this post I lost my original focus. Sorry about that.. I looked at the picture and got all oooh I need to talk about that.

Fedora PSA: How to know you are seeing a SPAM web page.

So one of the things that I really really hated about the current crop of SPAM on Fedora was that it completely lacked any originality or usefulness. The pages were usually hundreds of repeating lines saying something like:

customer support
technical support 
customer service
1-8[0-9][0-9]-xxx-xxxx 

over and over again seems to be a useless waste of electrons for anyone landing on the page. Especially when the web page says something like

c*u*s*t*o*m*e*r s*e*r*v*i*c*e

or some similar text. My first guess was that these are either used as landing text pages for some sort of malware to pop up some sub-section of the webpage as a problem alert. You have a problem with your . However that made less and less sense in that the pages did not seem to have any encoding for the malware to say 'this is the string to use today'. It also didn't make sense in that it was clear that this was being inserted many times by a cut and paste versus program insertion (someone would double paste the text into a web page but with the text intermingled). 

My next guess was that it was probably related to SEO ratings for some other search engines. Sure enough there was a pattern of various search engines that seemed to be triggered shortly after a set of webpages were created. They would specifically add the various pages and shortly afterwords a search at the search engine for " help" would show up the usual one word line like:

Call Support 

at the top of the search list linked to our website. Most of the searches that I found through the webpage logs were either for google.com or redirects from bit.do websites (actually the only bit.do redirects for over a month were for these pages.)

In any case, back to the subject title. If you are seeing a large page of text with various small variations on 'search terms' to get you technical support for product.. and a phone number DO NOT CALL IT. Especially if the page you are looking at is NOT for the product you are looking for.

Repeat after me:

• For actual QuickBooks support go to https://quickbooks.com
• For actual Cisco support go to https://cisco.com
• For actual Microsoft product support go to https://microsoft.com

If you are ending on some other website, it is most likely some sort of scam.

Thank you

Fedora Inactive/Disabled Accounts

In a recent #fedora-admin IRC conversation, it was asked if Fedora Infrastructure ever makes accounts inactive. We have done this a couple of times in the past,  usually after a security breach where password hashes may have been seen. While happily we have not had a security event happen in a while.. we did have a large amount of spam accounts created which have taken up a bunch of names that people may want to use (if you have wanted vipin1 -> vipin357, nagar1 -> nagar240, or many others, you are currently out of luck). We also have many users who created accounts, and either never logged into FAS again or haven't for over a year or more.

Now for some projects you would regularly set these accounts to inactive or clear them out for reuse. Free software projects usually do not have the luxury for doing this for several reasons:

1. The Norwegian Blue Parrot Problem.
   In this case, we find that many users have not been deceased, but like a true Norwegian Blue have been pining for the fnords. This means that even a year after you have set accounts to inactive there is a large amount of "why is my account locked?" emails and general grumpiness of people finding that someone had locked their account without their permission. [It also leads into the argument sketch and other too much silliness.]
2. This account was used for something in the past and we need to keep track of things for licensing reasons. While someone might want the account thegreatandmightyoz and that account is no longer active.. if that account put in patches or other items into Fedora.. those inclusions may be tracked via the userid in one of the many Fedora sub-systems. [Yes they should be tracked by the UID or some other non-changing ID, but that is 10 years too late for some inclusions.] Because of this if someone else gets the 'thegreatandmightyoz' they could they could get additional responsibilities (what do you mean I need to fix this bug? I am just a circus man from Kansas not a real wizard!)

For the second problem, we are looking at making changes in the new FAS system so that we can give users an 'EPOCH' where 0:thegreatandmightyoz is different from 1:thegreatandmightyoz in various systems trying to figure out what is linked to whom. For the first problem, we will work out a system of saying that a user who has not logged in for N days (where N is greater than say 270 days ) will be set for inactive. We would then advertise quite a bit before doing it and then get ready for a long list of arguments and complaints :).