Story behind the problem:
Tonight I was starting to reinstall our test cloud system so that msuchy could run his playbooks against the top node and other systems. I have been doing this very regularly, and it is pretty much a
- Log into Dell iDrac on the cloud systems
- Fire up the remote console.
- Click on 'Next Boot' -> 'PXE'
- Click on 'Warm Reboot'
- Let the PXE menu come up and pick the mode for the particular type of hardware I am rebuilding.
- Let it run for 10 minutes. Move to the next system
- .... ansible stuff goes here ....
Tonight as I was doing it, it died at
ValueError: new value non-existant xfs filesystem is not valid as a default fs type
I then used the tmux Control-B 2 to get to the shell and started looking at the log files in /tmp
01:04:30,510 INFO program: Running... modprobe xfs
storage.log has [typed from remote console]
01:04:30,509 DEBUG blivet: trying to set new default fstype to 'xfs'
01:03:30,510 DEBUG blivet: XFS.supported: supported: True;
01:04:30,521 ERR blivet: Could not load kernel module xfs
01:04:30,521 DEBUG blivet: getFormat('xfs') returning XFS instance with object id 1
01:04:30,523 DEBUG blivet: XFS.supported: supported: False ;
01:04:30,521 DEBUG blivet: invalid default fstype: XFS instance (0x7fe6987362d0) objet id 1--
type = xfs name = xfs status = False
device = None uuid = None exists = None
options = defaults supported = False formattable = True resizable = False
mountpoint = None mountopts = None
label = None size = 0 B targetSize = 0 B
01:04:30,525 DEBUG blivet: XFS.supported: supported: False ;
01:04:30,527 DEBUG blivet: XFS.supported: supported: False ;
I then tried a manual insmod of the xfs module and looked in dmesg
[ 760.625372] xfs: module verification failed: signature and/or required key missing - tainting kernel
[ 760.625484] xfs: disagrees about version of symbol ftrace_raw_output_prep
Aha! That tells me that my kernels don't match. Now since no one else is complaining about not being able to install xfs on EL-7.1 that tells me I have a mismatched kernel somewhere.
Now for a lot of our systems we use https://infrastructure.fedoraproject.org/infra/docs/kickstarts.txt and the grub-boot method to rebuild systems. But as mentioned above, we use PXE for the cloud systems and when we updated to the new RHEL-7.1 images we forgot to update the PXE kernel items. Update the vmlinuz and initrd.img on the PXE server and tada! it all works.