Note to Future Self: Random problems with anaconda installs after a new release

So  I have run into the following problem several times in the past, and never documented it to remember the next time it happens (I seem to do this about once every 3 years..)

Story behind the problem:

Tonight I was starting to reinstall our test cloud system so that msuchy could run his playbooks against the top node and other systems. I have been doing this very regularly, and it is pretty much a

1. Log into Dell iDrac on the cloud systems
2. Fire up the remote console.
3. Click on 'Next Boot' -> 'PXE'
4. Click on 'Warm Reboot'
5. Let the PXE menu come up and pick the mode for the particular type of hardware I am rebuilding.
6. Let it run for 10 minutes. Move to the next system
7. .... ansible stuff goes here ....

Tonight as I was doing it, it died at 

ValueError: new value non-existant xfs filesystem is not valid as a default fs type

I then used the tmux Control-B 2 to get to the shell and started looking at the log files in /tmp

01:04:30,510 INFO program: Running... modprobe xfs

storage.log has [typed from remote console]

01:04:30,509 DEBUG blivet: trying to set new default fstype to 'xfs'
01:03:30,510 DEBUG blivet:         XFS.supported: supported: True;
01:04:30,521 ERR blivet: Could not load kernel module xfs
01:04:30,521 DEBUG blivet: getFormat('xfs') returning XFS instance with object id 1
01:04:30,523 DEBUG blivet:                    XFS.supported: supported: False ;
01:04:30,521 DEBUG blivet: invalid default fstype: XFS instance (0x7fe6987362d0) objet id 1--
  type = xfs name = xfs status = False
  device = None uuid = None exists = None
  options = defaults supported = False formattable = True resizable = False
  mountpoint = None mountopts = None
  label = None size = 0 B targetSize = 0 B

01:04:30,525 DEBUG blivet:                    XFS.supported: supported: False ;
01:04:30,527 DEBUG blivet:                    XFS.supported: supported: False ;

I then tried a manual insmod of the xfs module and looked in dmesg

[  760.625372] xfs: module verification failed: signature and/or required key missing - tainting kernel
[  760.625484] xfs: disagrees about version of symbol ftrace_raw_output_prep

Aha! That tells me that my kernels don't match. Now since no one else is complaining about not being able to install xfs on EL-7.1 that tells me I have a mismatched kernel somewhere.

Now for a lot of our systems we use https://infrastructure.fedoraproject.org/infra/docs/kickstarts.txt and the grub-boot method to rebuild systems. But as mentioned above, we use PXE for the cloud systems and when we updated to the new RHEL-7.1 images we forgot to update the PXE kernel items. Update the vmlinuz and initrd.img on the PXE server and tada! it all works.