Whither EPEL (Talk at FOSDEM 2016)

If you are going to FOSDEM 2016 and use CentOS, Scientific Linux, Red Hat Enterprise Linux or even Oracle Enterprise Linux.. you should be interested in a round-table talk we are having on Sunday to talk about the Fedora Project's Extra Packages for Enterprise Linux (EPEL) . EPEL is a repository which has a curated set of Fedora packages rebuilt for EL-5, EL-6, and EL-7. 

Things We Will Talk About:
  1. Who are our consumers? A consumer is someone who uses EPEL but doesn't have the time, knowledge, etc to help build, test, or curate packages. [This is a place for audience members to bring things they do (or don't do) with EPEL.]
  2. Who are our contributors? A contributor is someone who builds, tests, writes documnts, etc for the EPEL SIG. [This is a place for people who build and test packages in EPEL can speak up.]
  3. What are the "promises" that we as a SIG believe that we are making to our contributors and consumers? What are the "promises" that contributors and consumers think we are giving?
  4. Do our rules and guidelines still make sense? [When the project was set up, there were many debates about what people wanted and the projects rules were built out of a general consensus of what people would put up with.]
  5. What about other ways to get these packages? Atomic, docker, software collections, COPRs, etc. Are these meeting peoples needs better and what can we learn to meet those inside of EPEL or branch out to meet them in other ways.
In general tradition, we will have a reenactment of West Side Stories meetings of the Sharks and the Jets while we discuss repotags. [I kid, I kid.. it will be more of Montague's and Capulet's.

Brussels CentOS Dojo

This Friday (2016-01-29) there will be a great CentOS dojo in Brussels Belgium. I am going mostly to help out run the cameras and announce different talks. I will also be wanting to listen to people use who use EPEL about what they use it for and what they are wanting to see in growth. This will help me prepare for the talk on Sunday at FOSDEM which I will be moderating.

If you have questions please try and find me. I will probably be the one fellow in a tie and sports coat. [And because I am from the desert.. no rain coat which sounds like a minus currently in Belgium].

This is my first time in Belgium and I will be trying to see a little of Brussels on Saturday and a bit of Ghent on Monday. If I had a day or two more I would have been going to visit Veere in the Netherlands so that I could try and see some of the places written by favorite author Hendrik Willem van Loon. I will just have to do this next time I am visiting western Europe. 

[I would like to write a bit longer, but I am stuck in the Atlanta airport blaring some inane news channel talking about the 'latest' election non-crisis. ]


Note to mailing list admins: Large scale auto-subscribers as revenge seems to be happening.

[In case what I am outlining is a known thing and I just found out about it.. my apologies.. I have been thankfully away from this.]

A couple of days ago, a couple of the list admins on fedora lists were noticing an uptick in subscription attempts for an address @mms.att.com which is basically an email to MMS gateway. The reason they were seeing this was because the ATT was blocking and sending back a helpful email saying "This person does not accept unsolicited text emails." Doing some investigation it turns out that this person was actually one of 20 or so people who were getting attempts to auto-subscribe them to every Fedora list. For people at gmail.com and other email addresses they used the "filtering" technique of +@gmail.com so that each subscription looked unique to the mailing list software. Thankfully none of the lists allow for auto subscriptions or some people would have received 10,000 copies of every email. Still each attempt got them sent an email saying "Hey someone tried to subscribe you to XYZ list. Please click this link to confirm." They use some sort of botnet to distribute out the subscription attempts as over 8000 ips were used in 24 hours for 160,000 attempted subscriptions.

Doing a simple google search of a couple of the emails and looking at some of the posts related to them in Facebook and Twitter, the only common thread was that they had all gotten involved in messy breakups. So my weak conjecture is that someone has created a "Revenge Spam" site where you pay them, give them an email address and they sign that email address to every mailing list etc to make their lives miserable. This type of things isn't much different from the old "sign up my ex to Columbia House Tapes/Records/CD collection" and every magazine with a free insert for N magazines before being billed. It is a pain and petty.

In any case, if you run into this and are using mailman2, you can add filtering per list using the amazing Mark Shapiro's ban module which will allow for various regexp so that when they change from to or some similar tactic these attempts are silently dropped.

[Thanks to Kevin Fenzi for finding this link http://www.mail-archive.com/mailman-users@python.org/msg67312.html as google and I were having a disagreement on what I meant by filtering and this showed up on page 4 of my google search and page 1 of his.]

In any case, you should check the /var/log/mailman/subscribe for a large growth in subscription attempts of the same email address from multiple ips. I am going to see what we need to do to block this on the mailman3 side of things and will hopefully post a followup with that info.

[To the people who got a ton of email requests to subscribe to N mailing lists.. I apologize for not catching this earlier.]


Where is my Fedora 15 (or how to deal with the disconnect between Fedora and your textbook).

The Problem

Every year, Fedora Infrastructure will get an influx of requests to admin@ or webmaster@ or some similar email address from someone who is looking for an old version of Fedora. Most of the time, we don't know why they are looking for this and ask them to use the recent version of Fedora instead of the one they asked for. The reason we do this is because those versions are old, insecure, and problems they run into will not be fixed or probably answered.

I have run into the problem myself this month as a class I am taking is using a Networking book that came out in 2013 and asks the students to use Fedora 15 (or later) to complete various examples. The "or later" sounds like it should cover the problems until I got into the actual examples and tools being used. The commands being referenced are the older network tools commands: ifconfig, netstat, route, etc. Those have been considered legacy for many years but are still the ones that a student will find in most textbooks and many online tutorials. The commands that replaced them ip and ss. [For more information the wikipedia article was good.] To add further injury to the student, the graphical commands all referenced no longer are bundled or look completely different from what the books reference.

The disconnect only gets greater as most of the textbooks do not get refreshed at a rate greater than once every 5 years. So the book which was written in 2011 (when Fedora 15 was being released) and didn't get finished through the editing system until 2013 (when Fedora 15 was end of lifed) will still be in use until 2018 to 2021 and what is Fedora 30+ will probably not look anything like the textbook or now.

Solutions for Students:

  1. Download Fedora 15 from the Fedora Archives. {1}{2}
  2. Use CentOS-6 for the problems in your text book. While CentOS-6 is based off of Fedora 12, the command line options are probably the same and the graphical utilities are also the same. The reason to use CentOS-6 is that it is supported til 2020 (which should match the lifetime of the textbook).
  3. If you do download and use a later version of Fedora, you can currently (as of Fedora 22/23) install the net-tools RPM which will give you access to the command line tools used in most of these textbooks. Do this with the following steps:
    1. Open a terminal window.
    2. sudo dnf install net-tools
    3. Follow the prompts and then use the commands as in the book.
  • {1} People ask if there are torrents for these older releases, but that relies on the idea that torrents are always faster. They are only 'faster' if you have many people sharing the copy and enough of them are close to you that getting little bits from them versus a further away mirror. In the case of old releases where there aren't many people in the torrent group.. it is much slower to use torrents.
  • {2} Use a live image and do not run this OS for long after use.

Solutions for Publishers

This one is probably going to get me in trouble, but if you are making a printed textbook do not use a short-lifed Linux OS like Fedora for the examples. For online coursework where you can update the examples and problems, Fedora will keep the students up to date but in printed form it quickly makes the textbook 'useless'. I am not even sure that an LTS from Ubuntu would live long enough for printed books. At the time the book was being written, Ubuntu 10.4  LTS would have been the one to make examples from. However it went End of Life earlier this year so a student would still be needing a version from the archives. The OS which seems to meet the lifetime of textbooks better is CentOS or Scientific Linux which matches the Red Hat Enterprise Linux lifetime.

Solutions for Fedora?

I am not sure what the solution for Fedora is. An LTS version would need to have an 7-10 year lifetime to match the publishing industries norms. It would be nice if we could work with publishers to have updated tutorials. They explain what things they are wanting in various textbooks and we put online and update things that meet those needs.  However, I don't see the will to do this because these textbooks are costing students $150+ and this is 'free' work from contributors without seeing that cost go down.  


Note to Future Self: Random problems with anaconda installs after a new release

So  I have run into the following problem several times in the past, and never documented it to remember the next time it happens (I seem to do this about once every 3 years..)

Story behind the problem:

Tonight I was starting to reinstall our test cloud system so that msuchy could run his playbooks against the top node and other systems. I have been doing this very regularly, and it is pretty much a

  1. Log into Dell iDrac on the cloud systems
  2. Fire up the remote console.
  3. Click on 'Next Boot' -> 'PXE'
  4. Click on 'Warm Reboot'
  5. Let the PXE menu come up and pick the mode for the particular type of hardware I am rebuilding.
  6. Let it run for 10 minutes. Move to the next system
  7. .... ansible stuff goes here ....
Tonight as I was doing it, it died at 

ValueError: new value non-existant xfs filesystem is not valid as a default fs type

I then used the tmux Control-B 2 to get to the shell and started looking at the log files in /tmp

01:04:30,510 INFO program: Running... modprobe xfs

storage.log has [typed from remote console]

01:04:30,509 DEBUG blivet: trying to set new default fstype to 'xfs'
01:03:30,510 DEBUG blivet:         XFS.supported: supported: True;
01:04:30,521 ERR blivet: Could not load kernel module xfs
01:04:30,521 DEBUG blivet: getFormat('xfs') returning XFS instance with object id 1
01:04:30,523 DEBUG blivet:                    XFS.supported: supported: False ;
01:04:30,521 DEBUG blivet: invalid default fstype: XFS instance (0x7fe6987362d0) objet id 1--
  type = xfs name = xfs status = False
  device = None uuid = None exists = None
  options = defaults supported = False formattable = True resizable = False
  mountpoint = None mountopts = None
  label = None size = 0 B targetSize = 0 B

01:04:30,525 DEBUG blivet:                    XFS.supported: supported: False ;
01:04:30,527 DEBUG blivet:                    XFS.supported: supported: False ;

I then tried a manual insmod of the xfs module and looked in dmesg

[  760.625372] xfs: module verification failed: signature and/or required key missing - tainting kernel
[  760.625484] xfs: disagrees about version of symbol ftrace_raw_output_prep

Aha! That tells me that my kernels don't match. Now since no one else is complaining about not being able to install xfs on EL-7.1 that tells me I have a mismatched kernel somewhere.

Now for a lot of our systems we use https://infrastructure.fedoraproject.org/infra/docs/kickstarts.txt and the grub-boot method to rebuild systems. But as mentioned above, we use PXE for the cloud systems and when we updated to the new RHEL-7.1 images we forgot to update the PXE kernel items. Update the vmlinuz and initrd.img on the PXE server and tada! it all works.


FLOCK Rochester NY 2015: 3 things you should do...

This is cribbed from Ruth Suehle's email to announce and other lists:

  1. Register! Only 70 people have registered so far. What are you waiting for? https://register.flocktofedora.org/
  2. Submit a talk!  We've extended the deadline to May 2. Get those talk proposals in! https://register.flocktofedora.org/ 
  3. Reserve your hotel room! The deadline is July 16, but that doesn't mean you should wait until the last minute. It doesn't cost anything to reserve. https://resweb.passkey.com/go/FLOCK2015 

The FLOCK committee is exploring some really fun options for evening events right now, so Ruth hopes to be able to tell you about those soon. Keep an eye on the website, which in the next few days Ruth and others will be updating with information about Rochester.


FLOCK 2015 is in Rochester NY

This is a short blog post to get over my writer's block. For the last 2 years, Fedora has had a computer festival called FLOCK in either Europe or North America. The first FLOCK happened in beautiful Charleston SC in 2013. The second FLOCK was in the wonderful capital of the Czech Republic, Prague. This years FLOCK is to be held from August 12-15 in Rochester New York. The main website is having some issues (various links aren't pointing to the correct places because Wordpress is being obstinate) but these are being worked on as I write and hopefully will be fixed soon.

To register for the Rochester FLOCK, please go to https://register.flocktofedora.org/ and add your name and data to the list.

[Edited to fix a STUPID mistake on my part of somehow confusing Prague with Austria. Once again, mea culpa. I know better.. and I should have edited before I hit send. I was reading a history of Prague and there was a section on when it has been used in the past to stand in for ancient Vienna or other cities. PIBKAC error.]