Living in the Past

Read through Bradley Kuhn's article on Living in the Past. I think it pretty much says many of the things I have said before, though maybe more eloquently. While I have pretty much converted to XFCE in the last 3 months, I can appreciate that GNOME is working towards a different audience set. I also agree with Bradley that ranting about it doesn't make anything better.. it just makes people more entrenched which clearly makes things work less. So I am happy to use XFCE and will work in whatever way I can to "make it a stepup from GNOME-2.32" for Linus and the like.

Moving from rawhide to F16alpha

Well the test laptop was stable until today when updates seem to have caused all kinds of issues.. and then I realized the issue was that rawhide moved from F16->F17. I think I am going to reinstall the laptop to being F16 alpha later this week and see how it progresses over the span of the week. Once we get to the betas I will jump back to rawhide to see if I can cause the laptop to burn out again :).

Currently the 3.0 kernels and systemd are fighting over system locks. As can be seen by this bugzilla, I am not sure who is winning.

Boot Times are Important.. sort of

Adam W was at CLS and has been going over ideas he ran into. One of them was a talk by Dan Allen of Mojave Linux on speeding up boot times of Jboss.

While I agree that starting times do matter for laptops and such.. for servers they are really a joke. Mainly because all of Linux could start up in 10 seconds, but the real wait is all the BIOS/EFI startups which Linux has no power over. Two weeks ago, I spent the week with Kevin Fenzi inventorying old hardware and installing new hardware at the Red Hat/Fedora Phoenix bunker. Most of our time was spent waiting.. (average of 4 minutes per server.. the newer the server, the longer the time) for a tiny period of time when we could enter a keystroke to enter some BIOS menu. Go too soon and they keyboard buffer is overrun, wait to long and you have 1-3 more minutes waiting until you can press it. For one server, we spent close to an hour trying to get all the menus gone through so we could actually get to the point of running Linux on it. And even after going into the menus and turning on "fast" boot options, the servers still take 2-4 minutes to get to a grub prompt.

And yes during those times we were going on yak shaving expeditions or trying to figure out how many prime numbers were below a googol.. but for the most part an even long RHEL-5 boot time with all the trimmings is stunningly fast compared to getting the server to booting. Until that problem is really solved, I think most boot time arguments are going to just go over server administrator heads.

This slowness is a primary reason why Infrastructure uses virtual machines everywhere. We can reboot a virtual machine in far less time than it takes for the real hardware version to think about rebooting. In the virtual server case, slow services are more detectable. I can no longer reboot a box, get in the car, go the coffee house, come back wait for the coffee to cool and then be at a Grub prompt. I might get to the door when the prompt is waiting for me.

Updating to rawhide (part trois)

The title turns out to be an unintentional pun. I spent the weekend at my first comic convention ever, the Albuquerque Comic Expo, and met Marina Sirtis who played Deana Troi on Star Trek: The Next Generation. But I digress.

After spending a lot of Friday and some of Monday trying to figure out how to capture the boot problem with kernel-3.0, I finally got lucky and saw that the watchdog occurred right as systemd mentioned that smartd was starting. Aha Data (who wasn't at the comic expo). Turning off smartd with a systemctl --disable smartd.service was able to allow for a boot to occur. Now I need to figure out how to capture the oops correctly and we should be go for opening a bugzilla report.

The other issues dealing with gdm etc are still being worked on by Adam Williamson's blog so I will wait to move out of runlevel 3 to runlevel 5. [Then again I am wondering.. why did I really ever stop using startx?]

[On a side note, the comic expo was fun. I got to meet my favourite Ghostbuster, Ernie Hudson, and geeked out speechless at meeting Jill Thompson (of various Endless comics and Scary Godmother). I also got to buy a comic I had been looking for 36 years... Thanks to all the people who were there. ]

Updating to rawhide (part deux)

Continuing from yesterday's travails of having a desktop running rawhide.

1. Let the system cool down for a couple of hours and try to boot again.
2. Download and run the Lenovo diagnostics for the system. Everything passes except for the battery and printer port (turned off in BIOS). Both were known issues. Fans and memory say they are ok.
3. Boot into rescue mode. The rpm database seems rather hosed that even doing the standard chicken foot fixes of rm /var/lib/rpm/__*; rpm --rebuilddb does not fix. Decide that trying to recover the database at that point was not worth it.
4. Nuke from orbit (its the only way to be sure.) Make tar file backups of /home (but forget to backup /etc/ssh and other files.) and then reinstall
5. Decide that installing Fedora 15 i686 makes more sense than x86_64. The box only has 2 gigabytes of ram so not gaining much for 64 bit.
6. Do a custom install of the system and then afterwords install various fonts, eclipse, javascript, and emacs packages I want. Do not install everygame this time, just the few I know will get played.
7. Put the system into runlevel 3. This will get past any yum update issues to rawhide that kill X etc (ok more chickenfoot voodoo, but been burned too many times. Plus being in run level 3 seems to make the system a lot cooler which I figure might have been problem also.
8. Do a yum --disablerepo="*" --enablerepo="rawhide" --skip-broken update. After 3 minutes of dep-checking, there are 1651 packages out of 3536 installed packages to update.
9. After about 2 hours of downloads and updating, box finishes updates. Commands work and its time to see what breaks.
10. Found first breakage: yum. The first time I ran it I got a traceback.. by doing a yum clean all this removed that issue Doing a yum update again though just gave me a long list of packages that supposedly needed to be updated but conflicted with the packages that I just had updated. 
11. Time to reboot and see if I have something workable. systemd and plymouthd seemed to spend a long time trying to shut the box down. (I could see this via a shell I had on the box via ssh.. but after 4 minutes it finally killed the sshd daemon.)
12. Go for a walk, come back and reboot the system the old fashioned way. System reboots into plymouth and stops halfway through the building the icon. Oook gets out hammer.. and boots system into rescue mode again. 
13. Remove quiet rhgb from grub.conf. Turn off hiddenmenu so I can type in options if needed. Make the timeout longer than 0 seconds.
14. Boot my first 3.0.0 kernel.. and watch the bootup output like God intended us to. See a long pause due to the fact that I ran rescue mode on the system and a restorecon is needed. See a string of errors from the bind mounted directories. Watch a looooong relabel happen. Go for another walk.
15. Found what the freeze was. A watchdog reset on the kernel which crashes it at "Starting Configure read-only root support". What part of the oops I can get onto the screen has to do with modprobe.
16. Ok lets see if a single gets past this. If it does then we can work around with an update afterwords... nope. This is before single comes into play. 
17. Now its time to try the pre 3.0-rc3 kernel. The box does boot up but gdm dies for some reason leaving just the login background of the doves and the mechanical bird. Logging into a console gives me the ability to set the system into init 3. Oh boy we can go old school with startx.
18. Next bug.. startx doesn't. Ooooh gnome doesn't like things. Looking at the log file, I see a lot of javascript exceptions "FIXME: Only supporting fixed size ARRAYs".
19. yum update still doesn't work but in new weird ways:
    

    Error: Package: ghc-tar-0.3.1.0-11.fc16.i686 (rawhide)
               Requires: libHSold-locale-1.0.0.2-ghc7.0.2.so
               Removing: ghc-old-locale-1.0.0.2-16.3.fc15.i686 (@updates/15)
                   libHSold-locale-1.0.0.2-ghc7.0.2.so
               Updated By: ghc-old-locale-1.0.0.2-26.fc16.i686 (rawhide)
                   Not found
    

    but it is there.. so I am confused. A --skip-broken doesn't fix the issue either.
20. Ok lets go ask Seth Vidal. And yes.. somehow asking the expert makes it work when he asks me to repeat the steps I have done before. We have a system with console and if I pull back into the hinterland of knowledge before gdm and such.. there was .xinitrc. Set that to
    

    exec startxfce4
    

    and I have a working X session after a startx.
21. Change the run level so when it reboots to runlevel3.target and time for bed. Wait a reboot sits around trying to stop syslogd.
    

    Unit systemd-kmsg-syslogd.service entered failed state.
    systemd-kmsg-syslogd.service: main process exited, code=exited, status=218
    

    Over and over again... looks like the reboot is killing syslogd, and systemd thinks it should still be up so it starts it up only to kill it again.. or something like that.. it is the reason for a reboot needing a power-off versus working.
    

Well after a lot of steps.. I have a system that boots. I relearned a lot of tech support skills and various things like rhgb is bad for testing systems when they dont work :). Now to work with Lennart on why syslogd is whacked and with others on why kernel-3.0rc3 kills my box.

Updating to rawhide is harder than I thought.

So with my work done with the Fedora Board, I have decided to spend more time on things I can fix (technical things) versus those I can not (human interactions). One of those things, was to use more of the Rawhide release because if I can earn LWN.net weekly quotes of the week, I should endure the pain that Jonathan Corbet goes through every day. Hopefully it will also get F16 to be better.

Ok steps to do this:

1. Install a laptop with Fedora 15.
   This was pretty easy as I am just using the laptop I built for SELF that just has a bunch of development things and a lot of games for the kid (ok me) to play when bored.
2. Update to rawhide.
   Not as easy as I thought. I figured, hey lets just do a
   
   yum --enablerepo=rawhide update
   
   Which gave me after 2 minutes a long list of broken dependencies and such. Hey no problem.. I ran into this with the Fedora 15 betas.. use --skip-broken.
   
   yum --enablerepo=rawhide --skip-broken update
   
   This went along a lot further, but died due to packages with the same name installed multiple times. Ok let us get around that.
   
   package-cleanup --cleandupes
   
   Tada.. we can now get to our next error: "Protected multilib versions". For some reason various packages are protected from being updated even when the update is showing x86_64 to x86_64. Ok.. maybe this is an error fixed in yum.
   
   yum --enablerepo=rawhide --skip-broken update y*
   
   Oh lookie.. I have updated yum.. that was easy. Too easy.. I forgot to update python also. Running yum now gave me a huge traceback:
   
   
--> Processing Dependency: libstartup-notification-1.so.0()(64 bit) for gnome-desktop3-...
Traceback (most recent call last):
...
pkgtup = (nevra_dict['name'], nevra_dict['arch'],
TypeError: tuple indices must be integers, not str

   
   Ok maybe an updated python will fix this. Turns out that yum update python worked.. but it didn't fix the issue.. ok time to see if I can work around it in parts. Caveman Oook says to do a
   
   yum --enablerepo=rawhide --skip-broken update [xyz]*
   
   and back away down the alphabet over time. Why? Because it is a magic chicken foot that used to work in the days before yum when rpm -Uvh *rpm was used.
   
3. Watch laptop catch on fire.
   Well not exactly but the /proc/acpi/ibm/thermal showed the temps over 100C which is quite hot. The system no longer was responsive and trying to log in as root would not work with messages about audit backlog limit exceeded. Turning off the system was about the only hope and turning it back on resulted in a blinking cursor in the left hand corner.

An interesting experiment.. a bit worse than my "Doing and Everything Install" but interesting. Will see if the box will come back after an hour cooling period.

Things I learned at SELF this year.

I would like to thank the people at SouthEast LinuxFest for allowing me to speak about "How to Train Your Users". It was a last minute addition to replace my talk on "How to Fork Fedora for Fun and Profit" since I ran out of Fun and Profit in getting that ready. I would also like to thank the people who sat through my presentation. It was the first one I have given in years and really need to work on it. I do apologize for not bringing a true User Attention Reset Tool, but after being told by several people that many staff were packing ... bringing a knife to a gun fight was ill-advised :).

I had intended to spend Friday at the talk, but spent the day working on the last parts of my paper and getting Fedora 15's Libre Office working in presentation mode. The Friday night dinner was fun to swap .com war stories with people.. learning the sins that Spot had to pay for by working Red Hat Technical Support was quite interesting. Saturday was a lot of fun as my dad and I spent the entire day there going to various talks and seeing how much swag he could fit in a tiny bag.

Once I got to my talk.. I learned the first lesson of presentation: USE the equiptment they set up versus your own. All my work with Libre Office failed and people got a tiny little slide on the screen. However my going over proper methods of tasers and quicklime with users seemed to cover that. On Sunday, we came again but most of the places were shut down for Sunday services and I couldn't find the drupal class I wanted to sit on.. so we went to Kings Mountain which was a nice hike in 95 degree heat. Monday was supposed to be a trip to Pisgah mountain but I ended up doing tech support and the next thing I knew it was Wednesday on a plane.

Other things I learned:

1. Navy people will swap stories on the drop of a dime. I would like to thank Sparks and various other Navy guys on keeping my dad entertained for the weekend. He had a great time and will be going next year. I also want to thank the Marine for laughing along nicely when father told "Ship-going bellhop" stories.
2. 3-d printers are really cool. The maker lab from Greensboro or  Winston-Salem made multiple penguins which made for my wife to want to build her own printer, and my kid to want to make a board game from the figurines.
3. Sudo make coffee makes good coffee, but it takes a special grind to make it work in a peculator. I learned that you can make a fair imitation of Turkish coffee though.
4. GNOME 3 has one major issue that needs to be worked out. It makes a video card very hot which in a laptop means I ended up with some interesting lap red marks. [Pictures redacted for the sake of sanity.] I ended up switching to XFCE which cooled the laptop down quite a bit.
5. The Ubuntu Women's table could have been better placed, but I learned how to spin wool and read a presentation on a computer screen I enjoyed.
6. I got a lot of DVD's and my dad is installing them on his laptop this week to play with. He is learning Ubuntu, Fedora, OpenSuSE and PC-BSD. That should keep him out of Mom's hair for a week or so.

Anyway, it was a good week and I learned some new things I hope to use in the next year. I also hope to improve my talk and see how it goes down at other talks. Thank you all for feeding and putting up with me.

Say goodnight, gracie.
Goodnight gracie.