Note to mailing list admins: Large scale auto-subscribers as revenge seems to be happening.

[In case what I am outlining is a known thing and I just found out about it.. my apologies.. I have been thankfully away from this.]

A couple of days ago, a couple of the list admins on fedora lists were noticing an uptick in subscription attempts for an address @mms.att.com which is basically an email to MMS gateway. The reason they were seeing this was because the ATT was blocking and sending back a helpful email saying "This person does not accept unsolicited text emails." Doing some investigation it turns out that this person was actually one of 20 or so people who were getting attempts to auto-subscribe them to every Fedora list. For people at gmail.com and other email addresses they used the "filtering" technique of +@gmail.com so that each subscription looked unique to the mailing list software. Thankfully none of the lists allow for auto subscriptions or some people would have received 10,000 copies of every email. Still each attempt got them sent an email saying "Hey someone tried to subscribe you to XYZ list. Please click this link to confirm." They use some sort of botnet to distribute out the subscription attempts as over 8000 ips were used in 24 hours for 160,000 attempted subscriptions.

Doing a simple google search of a couple of the emails and looking at some of the posts related to them in Facebook and Twitter, the only common thread was that they had all gotten involved in messy breakups. So my weak conjecture is that someone has created a "Revenge Spam" site where you pay them, give them an email address and they sign that email address to every mailing list etc to make their lives miserable. This type of things isn't much different from the old "sign up my ex to Columbia House Tapes/Records/CD collection" and every magazine with a free insert for N magazines before being billed. It is a pain and petty.

In any case, if you run into this and are using mailman2, you can add filtering per list using the amazing Mark Shapiro's ban module which will allow for various regexp so that when they change from to or some similar tactic these attempts are silently dropped.

[Thanks to Kevin Fenzi for finding this link http://www.mail-archive.com/mailman-users@python.org/msg67312.html as google and I were having a disagreement on what I meant by filtering and this showed up on page 4 of my google search and page 1 of his.]

In any case, you should check the /var/log/mailman/subscribe for a large growth in subscription attempts of the same email address from multiple ips. I am going to see what we need to do to block this on the mailman3 side of things and will hopefully post a followup with that info.

[To the people who got a ton of email requests to subscribe to N mailing lists.. I apologize for not catching this earlier.]


Where is my Fedora 15 (or how to deal with the disconnect between Fedora and your textbook).

The Problem

Every year, Fedora Infrastructure will get an influx of requests to admin@ or webmaster@ or some similar email address from someone who is looking for an old version of Fedora. Most of the time, we don't know why they are looking for this and ask them to use the recent version of Fedora instead of the one they asked for. The reason we do this is because those versions are old, insecure, and problems they run into will not be fixed or probably answered.

I have run into the problem myself this month as a class I am taking is using a Networking book that came out in 2013 and asks the students to use Fedora 15 (or later) to complete various examples. The "or later" sounds like it should cover the problems until I got into the actual examples and tools being used. The commands being referenced are the older network tools commands: ifconfig, netstat, route, etc. Those have been considered legacy for many years but are still the ones that a student will find in most textbooks and many online tutorials. The commands that replaced them ip and ss. [For more information the wikipedia article was good.] To add further injury to the student, the graphical commands all referenced no longer are bundled or look completely different from what the books reference.

The disconnect only gets greater as most of the textbooks do not get refreshed at a rate greater than once every 5 years. So the book which was written in 2011 (when Fedora 15 was being released) and didn't get finished through the editing system until 2013 (when Fedora 15 was end of lifed) will still be in use until 2018 to 2021 and what is Fedora 30+ will probably not look anything like the textbook or now.

Solutions for Students:

  1. Download Fedora 15 from the Fedora Archives. {1}{2}
  2. Use CentOS-6 for the problems in your text book. While CentOS-6 is based off of Fedora 12, the command line options are probably the same and the graphical utilities are also the same. The reason to use CentOS-6 is that it is supported til 2020 (which should match the lifetime of the textbook).
  3. If you do download and use a later version of Fedora, you can currently (as of Fedora 22/23) install the net-tools RPM which will give you access to the command line tools used in most of these textbooks. Do this with the following steps:
    1. Open a terminal window.
    2. sudo dnf install net-tools
    3. Follow the prompts and then use the commands as in the book.
  • {1} People ask if there are torrents for these older releases, but that relies on the idea that torrents are always faster. They are only 'faster' if you have many people sharing the copy and enough of them are close to you that getting little bits from them versus a further away mirror. In the case of old releases where there aren't many people in the torrent group.. it is much slower to use torrents.
  • {2} Use a live image and do not run this OS for long after use.

Solutions for Publishers

This one is probably going to get me in trouble, but if you are making a printed textbook do not use a short-lifed Linux OS like Fedora for the examples. For online coursework where you can update the examples and problems, Fedora will keep the students up to date but in printed form it quickly makes the textbook 'useless'. I am not even sure that an LTS from Ubuntu would live long enough for printed books. At the time the book was being written, Ubuntu 10.4  LTS would have been the one to make examples from. However it went End of Life earlier this year so a student would still be needing a version from the archives. The OS which seems to meet the lifetime of textbooks better is CentOS or Scientific Linux which matches the Red Hat Enterprise Linux lifetime.

Solutions for Fedora?

I am not sure what the solution for Fedora is. An LTS version would need to have an 7-10 year lifetime to match the publishing industries norms. It would be nice if we could work with publishers to have updated tutorials. They explain what things they are wanting in various textbooks and we put online and update things that meet those needs.  However, I don't see the will to do this because these textbooks are costing students $150+ and this is 'free' work from contributors without seeing that cost go down.  


Note to Future Self: Random problems with anaconda installs after a new release

So  I have run into the following problem several times in the past, and never documented it to remember the next time it happens (I seem to do this about once every 3 years..)

Story behind the problem:

Tonight I was starting to reinstall our test cloud system so that msuchy could run his playbooks against the top node and other systems. I have been doing this very regularly, and it is pretty much a

  1. Log into Dell iDrac on the cloud systems
  2. Fire up the remote console.
  3. Click on 'Next Boot' -> 'PXE'
  4. Click on 'Warm Reboot'
  5. Let the PXE menu come up and pick the mode for the particular type of hardware I am rebuilding.
  6. Let it run for 10 minutes. Move to the next system
  7. .... ansible stuff goes here ....
Tonight as I was doing it, it died at 

ValueError: new value non-existant xfs filesystem is not valid as a default fs type

I then used the tmux Control-B 2 to get to the shell and started looking at the log files in /tmp

01:04:30,510 INFO program: Running... modprobe xfs

storage.log has [typed from remote console]

01:04:30,509 DEBUG blivet: trying to set new default fstype to 'xfs'
01:03:30,510 DEBUG blivet:         XFS.supported: supported: True;
01:04:30,521 ERR blivet: Could not load kernel module xfs
01:04:30,521 DEBUG blivet: getFormat('xfs') returning XFS instance with object id 1
01:04:30,523 DEBUG blivet:                    XFS.supported: supported: False ;
01:04:30,521 DEBUG blivet: invalid default fstype: XFS instance (0x7fe6987362d0) objet id 1--
  type = xfs name = xfs status = False
  device = None uuid = None exists = None
  options = defaults supported = False formattable = True resizable = False
  mountpoint = None mountopts = None
  label = None size = 0 B targetSize = 0 B

01:04:30,525 DEBUG blivet:                    XFS.supported: supported: False ;
01:04:30,527 DEBUG blivet:                    XFS.supported: supported: False ;

I then tried a manual insmod of the xfs module and looked in dmesg

[  760.625372] xfs: module verification failed: signature and/or required key missing - tainting kernel
[  760.625484] xfs: disagrees about version of symbol ftrace_raw_output_prep

Aha! That tells me that my kernels don't match. Now since no one else is complaining about not being able to install xfs on EL-7.1 that tells me I have a mismatched kernel somewhere.

Now for a lot of our systems we use https://infrastructure.fedoraproject.org/infra/docs/kickstarts.txt and the grub-boot method to rebuild systems. But as mentioned above, we use PXE for the cloud systems and when we updated to the new RHEL-7.1 images we forgot to update the PXE kernel items. Update the vmlinuz and initrd.img on the PXE server and tada! it all works.


FLOCK Rochester NY 2015: 3 things you should do...

This is cribbed from Ruth Suehle's email to announce and other lists:

  1. Register! Only 70 people have registered so far. What are you waiting for? https://register.flocktofedora.org/
  2. Submit a talk!  We've extended the deadline to May 2. Get those talk proposals in! https://register.flocktofedora.org/ 
  3. Reserve your hotel room! The deadline is July 16, but that doesn't mean you should wait until the last minute. It doesn't cost anything to reserve. https://resweb.passkey.com/go/FLOCK2015 

The FLOCK committee is exploring some really fun options for evening events right now, so Ruth hopes to be able to tell you about those soon. Keep an eye on the website, which in the next few days Ruth and others will be updating with information about Rochester.


FLOCK 2015 is in Rochester NY

This is a short blog post to get over my writer's block. For the last 2 years, Fedora has had a computer festival called FLOCK in either Europe or North America. The first FLOCK happened in beautiful Charleston SC in 2013. The second FLOCK was in the wonderful capital of the Czech Republic, Prague. This years FLOCK is to be held from August 12-15 in Rochester New York. The main website is having some issues (various links aren't pointing to the correct places because Wordpress is being obstinate) but these are being worked on as I write and hopefully will be fixed soon.

To register for the Rochester FLOCK, please go to https://register.flocktofedora.org/ and add your name and data to the list.

[Edited to fix a STUPID mistake on my part of somehow confusing Prague with Austria. Once again, mea culpa. I know better.. and I should have edited before I hit send. I was reading a history of Prague and there was a section on when it has been used in the past to stand in for ancient Vienna or other cities. PIBKAC error.]


Mea Maxima Culpa

I would like to apologize for my last blog post. My original intention was to make an absurd point by proposing to drop 32 bit architectures from being primary in Fedora. I didn't communicate clearly that this was meant to be absurd. It also did not clearly state that the problem I am worried about is that with many core developers only focusing on x86_64 and hardware that is less than 4 years old that people using x86_32 and ARM32 are in effect on borrowed time.

I made things worse by then trying to defend why I was making the absurd proposal in the first place. In doing so, I muddled things further and pissed off people versus making things better.


Devil's Proposal: Moving Fedora to 64 bit only in Fedora 23

[Edited: 2014-01-22 This post is meant to have been absurd with a level of Jonathan Swift Modest Proposal. Lesson learned I am no Jonathan Swift.. if I see a problem I will write about it directly instead of as what only I saw as satire. Please see the next post for my formal apology to those I wronged.]

I am going to make the uncomfortable and ugly proposal to drop 32 bit in Fedora 23 and only look at 64 bit architectures as primary architectures. All 32 bit architectures (arm7hl, i386) would be moved to being secondary architectures that would require their own build teams and 'koji' to maintain builds in future releases. At the moment that would make the only 64 bit primary architecture x86_64 with arm64 and ppc64 possible candidates for mainstream support in F24 (if they aren't ready by Fedora 23).

The main reasons I am proposing this are:

  1. i386 yum usage and usage of Fedora have not grown and actually fallen slightly over the last 2 years.  [Harder to show in this picture is that a larger number of the i386 are older releases no longer supported by Fedora] 
  2. Multiple proposed changes in builder options for security work 'faster', 'better', or at all only in 64 bit.
  3. Development and developers have been focusing on 64 bit for quite a while leading to 32 bit to being an afterthought or a "I don't have a Pentium III to try and replicate your problem with." Many active developers and projects are looking to support only 4 year old hardware that they have. 
I think that those point to a picture where people who are using 32 bit architectures are not getting well served by the distribution ( and have not been for a while). Trying to force developers to focus on them usually ends up with a passive-aggressive relationship where things are only fixed after a long 'fight' of blame versus someone who is interested in older hardware taking active charge of the problem.

Some people running 32 bit currently may feel that they are not capable of running a build system. The job at this point would be to look for the people capable and willing to do so. Maybe a patreon or some similar program will be needed. I think it will be better to start an active community towards 32 bit Linux versus alternatives where Fedora X is 64 bit only and no avenues for 32 bit are left.

This may also mean that people with older hardware end up dropping Fedora altogether and going to Debian or Arch. I would actually say that the people doing so are being active and taking control of their destinies which is better than waiting for hand-scraps. If it means that Arch has a strong 32 bit presence and users of that hardware can get better support then the users win.