2014-07-10

CentOS-7: Full install

So CentOS-7 was released last week, and I finally was able to download it yesterday. One of the reasons for doing so was to have a machine that I could check for conflicts between EL-7 and EPEL. I do this via a very very caveman hack way (warning, there are better ways of doing this.. there must be :)).

So I create a VM and do an everything install of CentOS-7. This usually takes a couple of times as I figure out how much disk space is required. My first attempt I estimated that 24GB would be enough, but turned out I was wrong as various POST and other items would require more space. I ended up making a 32 GB VM and then getting it all to install in steps.

To do the install I mount the CentOS-7-Everything ISO as a cdrom in the VM machine. I then cd in /mnt/Packages and try a


yum localinstall *

This runs for a while and then spits out that there a bunch of conflicts that can't be resolved.

Error: php-mysql conflicts with php-mysqlnd-5.4.16-21.el7.x86_64
Error: libpng12-devel conflicts with 2:libpng-devel-1.5.13-5.el7.i686
Error: libcmpiCppImpl0 conflicts with 2:tog-pegasus-libs-2.12.1-16.el7.i686
Error: libcmpiCppImpl0 conflicts with 2:tog-pegasus-libs-2.12.1-16.el7.x86_64
Error: tog-pegasus-libs conflicts with libcmpiCppImpl0-2.0.3-5.el7.x86_64
Error: tog-pegasus-libs conflicts with libcmpiCppImpl0-2.0.3-5.el7.i686
Error: libpng12-devel conflicts with 2:libpng-devel-1.5.13-5.el7.x86_64


At this point I can do a --skip-broken which leads to the following almost working and then running into some bugs in rpms.

Transaction check error:
  file /usr/include/freeradius/radpaths.h from install of freeradius-devel-3.0.1-6.el7.i686 conflicts with file from package freeradius-devel-3.0.1-6.el7.x86_64
  file /usr/include/bacula/src/host.h from install of bacula-devel-5.2.13-18.el7.i686 conflicts with file from package bacula-devel-5.2.13-18.el7.x86_64
  file /usr/include/ImageMagick/magick/version.h from install of ImageMagick-devel-6.7.8.9-10.el7.i686 conflicts with file from package ImageMagick-devel-6.7.8.9-10.el7.x86_64
  file /usr/include/event2/event-config.h from install of libevent-devel-2.0.21-4.el7.i686 conflicts with file from package libevent-devel-2.0.21-4.el7.x86_64

[root@localhost Packages]# yum -y localinstall *rpm --skip-broken --exclude=freeradius-devel-3.0.1-6.el7.i686 --exclude=bacula-devel-5.2.13-18.el7.i686 --exclude=ImageMagick-devel-6.7.8.9-10.el7.i686 --exclude=libevent-devel-2.0.21-4.el7.i686


Eventually I am able to get everything installed and have

[root@localhost Packages]# rpm -qa | wc -l
8447
[root@localhost Packages]# df
Filesystem              1K-blocks     Used Available Use% Mounted on
/dev/mapper/centos-root  31728500 25849976   5878524  82% /
devtmpfs                  1932608        0   1932608   0% /dev
tmpfs                     1941760        0   1941760   0% /dev/shm
tmpfs                     1941760    16740   1925020   1% /run
tmpfs                     1941760        0   1941760   0% /sys/fs/cgroup
/dev/vda1                 1020588   192696    827892  19% /boot
/dev/sr0                  6896194  6896194         0 100% /mnt


My hat is off to the developers of xz as that is phenomenal compression from a 6+ GB ISO to 25G diskspace. It has been a week since release.. how many updates have been issued so far?

Transaction Summary
=============================================================================================================================
Install    4 Packages
Upgrade  245 Packages

Total size: 659 M
Total download size: 420 M


A short while later we have everything updated and installed and I can see what epel has in it currently:

[root@localhost ~]# wget http://dl.fedoraproject.org/pub/epel/beta/7/x86_64/epel-release-7-0.2.noarch.rpm
[root@localhost ~]# yum -y localinstall epel-release-7-0.2.noarch.rpm
[root@localhost ~]# yum --disablerepo=base --disablerepo=updates --disablerepo=extras install '*' 

...
--> Finished Dependency Resolution
Error: php-pecl-http1 conflicts with php-pecl-http-2.0.6-1.el7.x86_64
Error: Package: gedit-beesu-plugin-0.4-19.el7.x86_64 (epel)
           Requires: python3
Error: php-pecl-http1 conflicts with php-pecl-event-1.10.2-1.el7.x86_64
Error: Package: lyx-2.1.1-1.el7.x86_64 (epel)
           Requires: /usr/bin/chktex
Error: Package: globus-gram-job-manager-pbs-1.6-7.el7.x86_64 (epel)
           Requires: torque-client
Error: php-pecl-http1-devel conflicts with php-pecl-http-devel-2.0.6-1.el7.x86_64
Error: Package: wine-desktop-1.7.21-1.el7.x86_64 (epel)
           Requires: wine-core(x86-32) = 1.7.21-1.el7
Error: Package: tlp-0.5-2.el7.noarch (epel)
           Requires: wireless-tools
Error: Package: lyx-2.1.1-1.el7.x86_64 (epel)
           Requires: tetex-dvipost
Error: compat-qpid-cpp-server-xml conflicts with qpid-cpp-server-xml-0.28-4.el7.x86_64
Error: Package: pyhoca-cli-0.4.0.2-1.el7.noarch (epel)
           Requires: notify-python
Error: Package: rabbitvcs-thunar-0.16-1.el7.x86_64 (epel)
           Requires: thunar
Error: Package: oz-0.12.0-2.el7.noarch (epel)
           Requires: libvirt-daemon-qemu
Error: Package: rubygem-simplecov-0.7.1-8.el7.noarch (epel)
           Requires: rubygem(multi_json) >= 1.0
Error: firebird-classic-common conflicts with firebird-superserver-2.5.2.26539.0-8.el7.x86_64
Error: Package: amavisd-new-2.9.1-1.el7.noarch (epel)
           Requires: cabextract
Error: Package: TurboGears-1.1.3-8.el7.noarch (epel)
           Requires: python-turbokid >= 1.0.5
Error: compat-qpid-cpp-server conflicts with qpid-cpp-server-0.28-4.el7.x86_64
Error: compat-qpid-cpp-client-devel-docs conflicts with qpid-cpp-client-devel-docs-0.28-4.el7.noarch
Error: Package: rubygem-term-ansicolor-1.2.2-3.el7.noarch (epel)
           Requires: rubygem(tins) >= 0.8
Error: Package: TurboGears-1.1.3-8.el7.noarch (epel)
           Requires: python-turbojson >= 1.3
Error: Package: lyx-2.1.1-1.el7.x86_64 (epel)
           Requires: /usr/bin/dv2dt
Error: Package: 1:python-flask-0.10.1-3.el7.noarch (epel)
           Requires: python-itsdangerous
Error: Package: inxi-2.1.28-1.el7.noarch (epel)
           Requires: hddtemp
Error: tomcatjss conflicts with tomcat-native-1.1.30-1.el7.x86_64
Error: Package: TurboGears-1.1.3-8.el7.noarch (epel)
           Requires: python-turbocheetah >= 1.0
Error: Package: wine-1.7.21-1.el7.x86_64 (epel)
           Requires: wine-cms(x86-32) = 1.7.21-1.el7
Error: Package: lxc-templates-0.9.0-3.el7.x86_64 (epel)
           Requires: dpkg
Error: compat-qpid-cpp-client-devel conflicts with qpid-cpp-client-devel-0.28-4.el7.x86_64
Error: Package: amavisd-new-2.9.1-1.el7.noarch (epel)
           Requires: altermime
Error: Package: datagrepper-0.4.1-3.el7.noarch (epel)
           Requires: python-backports-lzma
Error: Package: wine-1.7.21-1.el7.x86_64 (epel)
           Requires: wine-openal(x86-32) = 1.7.21-1.el7
Error: Package: amavisd-new-2.9.1-1.el7.noarch (epel)
           Requires: perl(Convert::TNEF)
Error: Package: rubygem-rspec-expectations-2.14.5-2.el7.1.noarch (epel)
           Requires: rubygem(diff-lcs)
Error: libev-libevent-devel conflicts with libevent-devel-2.0.21-4.el7.x86_64
Error: Package: perl-Protocol-WebSocket-0.17-1.el7.noarch (epel)
           Requires: perl(AnyEvent::Handle)
Error: Package: fldigi-3.21.83-1.el7.x86_64 (epel)
           Requires: perl(RPC::XML::Client)
Error: Package: 2ping-2.0-2.el7.noarch (epel)
           Requires: perl(Digest::CRC)
Error: Package: openpgpkey-milter-0.3-1.el7.noarch (epel)
           Requires: python-pymilter
Error: Package: amavisd-new-2.9.1-1.el7.noarch (epel)
           Requires: perl(Unix::Syslog)
Error: Package: rubygem-rspec-2.14.1-1.el7.noarch (epel)
           Requires: rubygem(rspec-core) >= 2.14.1
Error: compat-qpid-cpp-server-ha conflicts with qpid-cpp-server-ha-0.28-4.el7.x86_64
Error: Package: nagios-plugins-openmanage-3.7.11-1.el7.x86_64 (epel)
           Requires: perl(Crypt::Rijndael)
Error: compat-qpid-cpp-client-rdma conflicts with qpid-cpp-client-rdma-0.28-4.el7.x86_64
Error: Package: caja-beesu-manager-1.8-19.el7.noarch (epel)
           Requires: /usr/bin/pluma
Error: Package: rubygem-mizuho-0.9.20-2.el7.noarch (epel)
           Requires: rubygem(sqlite3)
Error: compat-qpid-cpp-server-store conflicts with qpid-cpp-server-store-0.28-4.el7.x86_64
Error: compat-qpid-cpp-client conflicts with qpid-cpp-client-0.28-4.el7.x86_64
Error: Package: wine-1.7.21-1.el7.x86_64 (epel)
           Requires: wine-core(x86-32) = 1.7.21-1.el7
Error: Package: globus-gram-job-manager-sge-1.7-2.el7.x86_64 (epel)
           Requires: gridengine
Error: Package: openpgpkey-milter-0.3-1.el7.noarch (epel)
           Requires: python-gnupg
Error: Package: dislocker-0.3.1-2.20140423git.el7.x86_64 (epel)
           Requires: libpolarssl.so.5()(64bit)
Error: Package: lyx-2.1.1-1.el7.x86_64 (epel)
           Requires: tex-simplecv
Error: GtkAda3-devel conflicts with GtkAda-devel-2.24.2-8.el7.x86_64
Error: Package: TurboGears-1.1.3-8.el7.noarch (epel)
           Requires: python-cherrypy2
Error: Package: TurboGears-1.1.3-8.el7.noarch (epel)
           Requires: python-peak-rules
Error: firebird-superclassic conflicts with firebird-classic-2.5.2.26539.0-8.el7.x86_64
Error: Package: wine-1.7.21-1.el7.x86_64 (epel)
           Requires: wine-ldap(x86-32) = 1.7.21-1.el7
Error: Package: fuse-dislocker-0.3.1-2.20140423git.el7.x86_64 (epel)
           Requires: libpolarssl.so.5()(64bit)
Error: Package: perl-Protocol-WebSocket-0.17-1.el7.noarch (epel)
           Requires: perl(AnyEvent::Socket)
Error: Package: rabbitvcs-core-0.16-1.el7.noarch (epel)
           Requires: pysvn
Error: Package: lyx-2.1.1-1.el7.x86_64 (epel)
           Requires: wv
Error: Package: zarafa-search-7.1.10-1.el7.x86_64 (epel)
           Requires: catdoc
Error: Package: amavisd-new-2.9.1-1.el7.noarch (epel)
           Requires: unzoo
Error: Package: tor-0.2.4.22-2.el7.x86_64 (epel)
           Requires: torsocks
Error: Package: yumex-3.0.15-1.el7.noarch (epel)
           Requires: udisks
Error: Package: wine-1.7.21-1.el7.x86_64 (epel)
           Requires: wine-twain(x86-32) = 1.7.21-1.el7
Error: Package: perl-Protocol-WebSocket-0.17-1.el7.noarch (epel)
           Requires: perl(AnyEvent)
Error: Package: zarafa-search-7.1.10-1.el7.x86_64 (epel)
           Requires: w3m
Error: Package: RemoteBox-1.8.1-1.el7.noarch (epel)
           Requires: rdesktop
Error: Package: rubygem-gssapi-1.1.2-3.el7.noarch (epel)
           Requires: rubygem(ffi) >= 1.0.1
Error: Package: pyhoca-gui-0.4.0.9-2.el7.noarch (epel)
           Requires: notify-python
Error: libev-libevent-devel conflicts with libevent-devel-2.0.21-4.el7.i686
Error: php-pecl-uuid conflicts with uuid-php-1.6.2-26.el7.x86_64
Error: Package: amavisd-new-snmp-2.9.1-1.el7.noarch (epel)
           Requires: perl(Unix::Syslog)
Error: Package: amavisd-new-2.9.1-1.el7.noarch (epel)
           Requires: clamav-server-systemd
Error: Package: qt4pas-2.5-3.el7.x86_64 (epel)
           Requires: fpc-src
Error: Package: freight-0.3.5-4.el7.noarch (epel)
           Requires: dpkg
Error: Package: lxc-templates-0.9.0-3.el7.x86_64 (epel)
           Requires: busybox
Error: Package: dpm-xrootd-3.3.4-1.el7.x86_64 (epel)
           Requires: libXrdUtils.so.1()(64bit)
Error: Package: ewftools-20130416-1.el7.x86_64 (epel)
           Requires: fuse-python >= 0.2
Error: mariadb-galera-server conflicts with 1:mariadb-server-5.5.37-1.el7_0.x86_64
Error: Package: nf3d-0.8-2.el7.noarch (epel)
           Requires: python-visual
Error: Package: rabbitvcs-thunar-0.16-1.el7.x86_64 (epel)
           Requires: thunarx-python
Error: php-pecl-http conflicts with php-pecl-http1-1.7.6-4.el7.x86_64
Error: wxGTK3-devel conflicts with wxGTK-devel-2.8.12-8.el7.x86_64
Error: compat-qpid-tools conflicts with qpid-tools-0.28-4.el7.x86_64
Error: Package: yumex-3.0.15-1.el7.noarch (epel)
           Requires: python-pexpect
Error: Package: pluma-beesu-plugin-0.4-19.el7.x86_64 (epel)
           Requires: pluma
Error: zabbix20 conflicts with zabbix-2.2.3-1.el7.x86_64
Error: Package: yumex-3.0.15-1.el7.noarch (epel)
           Requires: pyxdg
Error: Package: freecad-0.13-10.el7.x86_64 (epel)
           Requires: python-collada
Error: Package: rubygem-term-ansicolor-1.2.2-3.el7.noarch (epel)
           Requires: rubygem(tins) < 1
Error: Package: trac-blackmagictickettweaks-plugin-0.12.2-2.20140425svn9962.el7.noarch (epel)
           Requires: trac >= 0.12
Error: Package: wine-1.7.21-1.el7.x86_64 (epel)
           Requires: wine-pulseaudio(x86-32) = 1.7.21-1.el7
Error: Package: TurboGears-1.1.3-8.el7.noarch (epel)
           Requires: python-paste-script >= 1.7
Error: firebird-superserver conflicts with firebird-classic-common-2.5.2.26539.0-8.el7.x86_64
Error: Package: rabbitvcs-nautilus-0.16-1.el7.x86_64 (epel)
           Requires: nautilus-python
Error: Package: amavisd-new-2.9.1-1.el7.noarch (epel)
           Requires: clamav-server
Error: firebird-classic conflicts with firebird-superclassic-2.5.2.26539.0-8.el7.x86_64
Error: compat-qpid-cpp-server-rdma conflicts with qpid-cpp-server-rdma-0.28-4.el7.x86_64
Error: Package: spectrwm-2.5.0-1.el7.x86_64 (epel)
           Requires: xlockmore
Error: php-pecl-rrd conflicts with rrdtool-php-1.4.8-8.el7.x86_64
Error: Package: fldigi-3.21.83-1.el7.x86_64 (epel)
           Requires: perl(RPC::XML)
Error: Package: php-phpseclib-crypt-aes-0.3.5-2.el7.noarch (epel)
           Requires: php-pear(phpseclib.sourceforge.net/Crypt_Rijndael) >= 0.3.0
Error: Package: TurboGears-1.1.3-8.el7.noarch (epel)
           Requires: python-elixir >= 0.6.1
Error: Package: wine-1.7.21-1.el7.x86_64 (epel)
           Requires: wine-capi(x86-32) = 1.7.21-1.el7
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest



Some of those are purposeful problems, but others are ones about packages needed which may have been in the beta but not anymore and vice versa. Adding a --skip-broken gets me a lot more packages (4503) to install... now to see if I have enough disk space. NOPE. We're gonna need a bigger boat.
Again there are better ways of doing this. I am mostly doing it so I can try out parts of the CentOS environment and how EPEL 7 is looking. My next post will be on how to do this correctly. [Edited after post because I forgot about pre tags which made it unreadable.]

2014-04-24

Useless things to ask for: Please delete my name from your mailing list and archives.

It seems to have become a common occurrence lately for people to send requests to email or website administrators to delete some sort of data they emailed a mailing list about. Many times it is because of some personal data that they posted accidentally like a phone number or a home address. Other times they want to completely scrub their email address or real-name from the posts.

I expect that the majority of them have real world reasons for doing so. Maybe someone is stalking them, maybe they have gotten a load of phishing and spam because of it, maybe there house got broken into. Yes, there are probably a lot of other reasons that might seem silly, but I am going to focus on the ones that worry me at night. I worry about them because I can't do what they ask in a way that is helpful to them.

I can quite probably delete the mailing from say our online archives, but I can't remove other people's archives, I can't remove the post from the many mailing list mirrors on the internet, and in most cases I can't be assured that the person asking me to remove their data is really that person. [Too many cases of ex-lovers taking over the others account and sending out weird requests.]

The best solution I can come up with is some training before a person is allowed to join a mailing list... the worst solution is a EULA which they have to click through saying "You are aware that posting data to the internet is a one-way occurence. Whatever you send will be mirrored, reshared, stored, collated, and kept by multiple entities now and in the future. Any rights to remove that data in the future are impossible due to the many parties outside of this mailing lists control. "

Beyond that I don't know... and it can keep me up at night.

2014-04-15

Why I am not worried about the lack of a default firewall in F21 Workstation

So one of the proposals for Fedora 21 is that the Workstation Product will not ship with a firewall. Normally I would be up in arms about something like this (I expect someone can find my emails in the past) but not this time. It might be the mai-tais and my vacation talking, but I look at many of these changes to the Workstation as product differentiation points. If Fedora Workstation does X, Y, and Z then the Xedora product can aim at not doing those.

Maybe Xedora is an OS for people who are tired old grumpy system administrators who the world has passed by. Maybe it will come with E19 and FVWM2 desktops with a firewall and a E-toolkit configurator for firewalld, maybe it will be KDE and QT configuration tools for items that the Workstation isn't aiming at. Then both groups can get what they want without a lot of squabbling and wasted Email trying to convince each other items that the other side feels are strawmen arguments.

Anyway, my mai-tai has arrived. Have fun.

2014-03-13

SSH Key Magic for pkgs and fedorahosted (or how to not cause false logins)

So every couple of hours, I check the Fedora Project's servers logs to see if we have had failed logins, bad logins, etc. Sometimes people decide that they really really want to see if they can log in as someone else using '123456' or something. Its all fun and games until your atmosphere gets sucked into space (or something). One of the problems I see a lot though is that developers may get denials getting into fedorahosted.org or pkgs.fedoraproject.org due to the fact that they have multiple SSH public private keys.

Unless told otherwise, most ssh clients do not have a heuristic to know which public/private key to use for which site.. and so will have to play 20 questions to see if any of them work. If you have a lot of keys, this can result in you being denied access because your client tried 4 keys and didn't get the right one. Those 4 keys might get logged as seperate failed attempts which can make it look like someone is trying to break into an account, and then I need to send an email to make sure it was X really trying to log into fedorahosted.org at 4 am in the morning.

There is a way to avoid this problem by editing your .ssh/config file to know the appropriate key for each server (or set of servers). I use a variant of the following to cut down the problems.



Host  *.fedorahosted.org *fedorapeople.org *.fedoraproject.org
    User X
    IdentityFile ~/.ssh/id_fedora_rsa.pub
ForwardAgent no ForwardX11 no Port 22 KeepAlive yes HashKnownHosts no GSSAPIAuthentication no VerifyHostKeyDNS yes ControlMaster no

To explain the lines:

  1. The Host configuration option says for the following hosts the following settings are to be used.
  2. Set the account name to X. [EG change this to match the account you use.
  3. Use the specific public key in this file for this system. This is actually the most important line and should cut down the failed attempts per user.
  4. Do not forward my ssh credentials. I do this to cut off possible forwarding attacks where an malevolent host can leapfrog to other machines that id_fedora_rsa would be trusted.
  5. Do not forward X11. The boxes I log into don't normally run X11 so this is more about cutting down a "hey can I run X11?" question from my client to the server.
  6. Use port 22. I am being pedantic here because I have it set to other ports for some other boxes in my .ssh/config.
  7. KeepAlive is turned on because I am on wireless and sometime things quit talking.
  8. Don't hash my known hosts.. mainly because I find I need to read where I have been as much as someone who might break into my account.
  9. None of these systems use kerberos so turning off GSSAPI means its anotehr set of "Hey can I?" questions not asked during login.
  10. If possible verify the hosts public key in ssh. Not really useful without a signed DNS.. but someday :).
  11. Don't use controlmaster for this host. Multiplexing is good when you need it, but I don't generally need it. I have it here as another 'Can I?' which may slow down login for some connections.
Anyway, if you connect up your hosts with your keys, you can make sure your client isn't trying to authenticate your Fedora account with your GNOME, KDE, School, Home, etc etc keys.

Fell off the Internet.

Well that was fun and exciting.. I fell off the Internet for 8 months. Nothing broken, just a bunch of little things which took up my time.


  1. CentOS and Red Hat are now co-habitating. This was a project that came out of the blue early last summer and I was told to treat with utmost secrecy. So that cut down what I could say about anything. 
  2. Dog attacks aren't fun. I had an unfortunate case where my dog and another dog got into a fight and I tried to break it up. Scariest 10 minutes of my life. I am glad I only got out of it with a couple of bites and no one else was hurt. That took much of November out of my life.
  3. If you are over 35 years old, get your flu shot. There are multiple versions of the flu which affect people over 35 much worse than people under that age. Also anti-viral drugs work better than I thought and I didn't come down with pneumonia versus the other people I knew who came down with this version.
  4. Losing a best friend takes a lot out of you. You are missed Seth.

Anyway, I am back and should have a couple of posts in me before I fall off the internet again.

2013-08-19

Danger, Will Robinson Danger!!! Don't miss the Fedora 20 naming elections

With all the summer conferences going on (FLOCK, FOSDEM, GUADEC, KDE-academy, etc etc) people are running behind on announcing things. The Naming election for Fedora 20 is now live. Vote for one of the following names:

Chateaubriand
Cherry Ice Cream
Eigenstate
FĂ©licette
Heisenbug
Ă–sterreich
Santa Claus
Superego 

I am voting for Santa Claus!

2013-07-12

Remembering Seth Vidal

I have been trying to write a memory of Seth, but like everyone else have been going through periods anger, denial, and depression. At this point I am at the depression side so figured I had better just post what I can and try to move on.

Seth was my best friend online. He would call and make sure I was handling stress, and he always seemed to be able to look at a bad situation and find a way we could make it better. That is something I really will miss in the months ahead.

I first met Seth Vidal in late 2001 except I didn't realize it until he told me years later. I had left Red Hat in May after 4 years of startup work and had taken a couple of months off to learn that I wasn't a freelance consultant (especially if I couldn't ask people to pay me). I needed a job but the dotCom crash had happened and no one was hiring for Linux or Unix administrators. Duke University had an opening in their physics department and since I had an Astrophysics degree.. I figured it was a good match. I got called in and sat in a room where people were being marched in and out at 15 minute intervals. When it was finally my turn, I walked into a room where a very weary Seth and someone else were sitting there looking like they had been doing interviews for days. Turned out that wasn't too far off. Seth's first words were pretty much:
"I will be totally honest with you. We had over a thousand applicants for this job, and after the University filtering process got it down to 400 interviews. We have been doing this for days. You like a lot of people are waaaay overqualified for this gig. So in 5 minutes or less, why should we hire you?"

Completely cut to the chase. I could tell that the other person in the interview wasn't comfortable with the "truthiness" of it all but was way too tired from doing interviews back to back to put up a fight on it anymore. Seth then went on about what the job really covered over what was advertised, and that mostly it was to be a buffer between various Phd's, grad students, and post-docs and scarce resources. "I can see you have done this before, but quite frankly you could get paid a lot more doing it somewhere else."

I don't remember much about the interview than that.. in fact Seth remembered more of what I said and what I did than I could when he told me about it years later. What I did remember was that by the end of the interview I realized that I wasn't being led around. I had gotten the straight dope and that I was probably not going to be happy in the position. [I further realized this when I went to work for a different University years later and how "layered" stories could be about conditions.]

Years later I would run into Seth in various places and would finally work with him on the Fedora Infrastructure Team. I learned a lot from him even when I infuriated him at times... we had our good times working together, we had our "Lets just agree to disagree and come back later on this." times, and we had out "Are me? Fan--tastic. Want me to quit now or after you finish that sentence." times. Always to the point which I will miss terribly.

That is all.