In looking for proxy systems, my team has been focusing first on the classic ARM for small projects: The Raspberry Pi.
The raspberry pi4 with 4 GB of ram works out as a proxy for all kinds
of ‘low-end’ systems where you may need to play with a small GPU and try
to make it work with a Server Ready operating system like CentOS
Stream.
Getting the hardware
There are several places to get the hardware, and while I used to get things from Adafruit, but they did not have an IOT kit set up for the 4 series. I ended up going with CanaKit from Amazon
just so I could get a bundle of parts together. Going from past
experience with MMC cards burning out after a couple of days, I bought 3
32 gig cards to put the OS on. So far the cards have lasted longer than
I expected but that just means they will burn out any day now.
When
getting a raspberry pi, I highly recommend making sure you get the
correct power supply, a USB 2 serial connector for the GPIO, and if you
are using an external drive, a seperately powered disk drive. I have
found that while the Raspberry Pi4 uses a larger power supply than the
1,2, or 3 series… attaching a non-powered USB drive can be problematic
on boot (the ssd one I had would cause me to just have the rainbow
picture of doom).
Setting up the hardware
For
the raspberry pi4 if you are using it to compile or build things, make
sure you have correctly sized heat dispensors for the CPU and possibly a
fan (and maybe a replacement fan for when the first one dies). Then
attach a serial cable to pins 6 (ground),8 (txd),10 (txd). Make sure you
do not attach anything to 1,2,3 as you could be looking for a new pi or
computer. The serial is useful for when you attempt to boot a new
kernel config and the HDMI connector was not functional afterwords.
On another computer attach the USB connector and you can use the screen
or minicom
commands to see output from the system on boot. On my test system, I was able to capture the following:
$ screen -c /dev/null -L /dev/ttyUSB0 115200
recover4.elf not found (6)
recovery.elf not found (6)
Read start4x.elf bytes 2983784 hnd 0x000013b1 hash '3f7b34a64191a848'
Read fixup4x.dat bytes 8453 hnd 0x00000d3b hash '59e66162bed1b815'
0x00c03112 0x00000000 0x000003ff
MEM GPU: 32 ARM: 992 TOTAL: 1024
Starting start4x.elf @ 0xfec00200 partition 0
MESS:00:00:05.434998:0: arasan: arasan_emmc_open
Initial setup
Like
any hardware setup, it is important to make sure the shipped hardware
has up to date firmware and configs. For this I took one of the MMC
cards, and burned the Raspian OS with recommended software
Once this booted up, the tools did a firmware upgrade on the system
from whatever had been on the box when it was stored in a depot. This OS
is a 32 bit operating system but is maintained by the ‘manufacturer’ so
is a good test for ‘did it break in shipment’
{Pro-tip:
Once you have finished updating, shutdown the system, take this card
out, and put it in a box/bag for later use. At some point things are
going to go badly in an install and you won’t know if its you, your
hardware, or something else. Having a known bootable backup that is
supposed to work is good.}
After this it is time to focus on getting ‘bootable’ systems using the base OS’s we want to target:
- Fedora Linux
- CentOS Stream
Fedora 34 Initial Install
Fedora
34 came out as I started working on the project, so I decided to aim
for that as an initial OS. The ARM work that the Fedora team is doing is
aimed primarily at 64-bit and with Server Ready hardware. As such, they
do try to make a raspberry pi4 work, but it is listed as possibly
problematic. That said, the following worked mostly fine:
- Download the raw workstation image
- On an existing Fedora 33 system, install arm-image-installer via
sudo dnf install arm-image-installer
- Insert an mmc into the computer using the appropriate adaptor, and find the disk name.
- GNOME will probably be overly helpful and have mounted partitions on the card for you. You will need to unmount them.
df -a | grep mmc ; sudo umount /dev/mmcblk0p1; sudo umount /dev/...
- write the image to the mmc disk with image-installer:
fedora-arm-image-installer --image=./Fedora-Server-34-1.2.aarch64.raw.xz --media=/dev/mmcblk0 --addkey=a_key.pub --resizefs --showboot --target=rpi4 --addconsole=ttyAMA0,115200
- Move the mmc card over to the powered off raspberry pi4, and prepare to boot up the hardware.
- On my Fedora system I started a screen to watch the fireworks:
screen /dev/ttyUSB0 115200
- Power on the raspberry pi and watch what happens. If you are
lucky then you will get the system eventually booting into Fedora 34
graphical mode. If you aren’t, then it may stay in rainbow mode (like
when I found that my SSD drive pulled too much power on boot.)
- Log into the system and play around a bit. This is a good time
to do any updates and such. Getting an idea of how ‘fast’/‘slow’ the
system with defaults is good here too.
Get serial working
At
this point I wanted to make sure I could log into Fedora from the
serial port. In order to do this, you need to edit the grub configs
which is done in /etc/default/grub
and then
rebuilding the config. I moved the grub timeout to 10 seconds to give me
a chance to choose different options, removed the rhgb and quiet, and
added a console line.
$ sudo -i
# vi /etc/default/grub
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="console=tty0 console=ttyAMA0,115200 "
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true
# grub2-mkconfig -o /etc/grub2-efi.cfg
A test reboot is good here to make sure that when you boot you get past the
EFI stub: Booting Linux Kernel...
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
Fedora 34 with TianoCore
As
stated before the Raspberry Pi systems are not System Ready and do not
have a built in EFI or ACPI system for a kernel to boot from. Instead
the default kernel boots usually with uboot mapping devices via device tree
in order for hardware to be discovered. I am going to say that is about
all I really know about the subject. There have been long threads in
the Fedora ARM lists over the years on going over this versus ACPI, and I
think it is better for an expert like Jon Masters or Peter Robinson to explain why APCI is preferred in Fedora Linux and Red Hat Enterprise Linux (RHEL) versus a novice like myself.
For the raspberry pi4, the current method to implement a UEFI interface is a port of the Tianocore by the Pi Firmware Task Force. TianoCore is an opensource implementation and extension of the Intel Extendable Firmware Interface which was written to replace the 16-bit BIOS used in personal computers since the 1980’s. A further extension was with the Open Virtual Machine Firmware which I believe was then used by the Pi Firmware Task Force for their version.
Easier than I thought
In
reading various blogs and documentation on the status of the EFI
support, I was prepared for a system that would only work via serial
console or may not have networking or other utilities. Instead, I found
the process to be fairly ‘painless’ and I only ended up with a
non-booting system twice. The general steps were the following:
ssmoogen@fedora ~]$ mkdir RPi4
[ssmoogen@fedora ~]$ cd RPi4/
[ssmoogen@fedora RPi4]$ wget https://github.com/pftf/RPi4/releases/download/v1.27/RPi4_UEFI_Firmware_v1.27.zip
--2021-06-16 17:36:31-- https://github.com/pftf/RPi4/releases/download/v1.27/RPi4_UEFI_Firmware_v1.27.zip
Resolving github.com (github.com)... 140.82.114.3
Connecting to github.com (github.com)|140.82.114.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
...
RPi4_UEFI_Firmware_v1.27.zip 100%[=====================================================================================================>] 2.92M 14.4MB/s in 0.2s
2021-06-16 17:36:32 (14.4 MB/s) - ‘RPi4_UEFI_Firmware_v1.27.zip’ saved [3064085/3064085]
$ wget https://github.com/pftf/RPi4/releases/download/v1.27/RPi4_UEFI_Firmware_v1.27.zip
$ unzip RPi4_UEFI_Firmware_v1.27.zip
Archive: RPi4_UEFI_Firmware_v1.27.zip
inflating: RPI_EFI.fd
inflating: bcm2711-rpi-4-b.dtb
inflating: bcm2711-rpi-400.dtb
inflating: bcm2711-rpi-cm4.dtb
inflating: config.txt
inflating: fixup4.dat
inflating: start4.elf
creating: overlays/
inflating: overlays/miniuart-bt.dtbo
inflating: Readme.md
creating: firmware/
inflating: firmware/Readme.txt
creating: firmware/brcm/
inflating: firmware/brcm/brcmfmac43455-sdio.clm_blob
inflating: firmware/brcm/brcmfmac43455-sdio.bin
inflating: firmware/brcm/brcmfmac43455-sdio.Raspberry
inflating: firmware/brcm/brcmfmac43455-sdio.txt
inflating: firmware/LICENCE.txt
At this point, if you haven’t already, read the
documentation. Our main tasks will be to setup the raspberry pi EFI boot
partition to have the needed data in it. I had seen that this came with
dtb files which I figured needed to be replaced. However as seen below,
all of these are in a Fedora 34 and are of a similar timeframe. The
only file we really need to work with is the RPI_EFI.fd
and config.txt
files.
[ssmoogen@fedora RPi4]$ sudo -i
[sudo] password for ssmoogen:
[root@fedora ~]# cd /boot/efi/
[root@fedora efi]# cp config.txt config-orig.txt # always make a backup!
[root@fedora efi]# rpm -qf /boot/efi/bcm2711-rpi-4-b.dtb
bcm2711-firmware-20210430-1.1a46874.fc34.aarch64
[root@fedora efi]# rpm -qf /boot/efi/overlays/miniuart-bt.dtbo
bcm283x-overlays-20210430-1.1a46874.fc34.aarch64
[root@fedora efi]# cp ~ssmoogen/RPi4/RPI_EFI.fd /boot/efi/
At this point, it is time to replace the
config.txt that came with the system with one which can be used for
booting the UEFI program. This is where I caused my system to go into
rainbow mode a couple of times. In the end, I put in the following file:
#boot in 64-bit mode
arm_64bit=1
# boot into the RPI_EFI firmware
armstub=RPI_EFI.fd
# turn on serial and keep it on in 2ndstage
enable_uart=1
uart_2ndstage=1
bootcode_delay=1
# using this from upstream UEFI config.txt
device_tree_address=0x1f0000
device_tree_end=0x200000
# set up the miniuart and the vc4
dtoverlay=miniuart-bt,vc4-fkms-v3d
disable_commandline_tags=1
disable_overscan=1
enable_gic=1
# set up the GPU ram and HDMI
gpu_mem=128
hdmi_ignore_cec_init=1
max_framebuffers=2
start_x=1
A couple of the commands in it are from a side
trip on getting accelerated graphics working in CentOS Stream on Pi. I
decided to document it for later as this is getting long. At this point I
was able to get a booting system:
recover4.elf not found (6)
recovery.elf not found (6)
Read start4x.elf bytes 2983816 hnd 0x000032fe hash '210478ae179a91d0'
Read fixup4x.dat bytes 8451 hnd 0x00002c88 hash '7716af32619f0771'
0x00c03112 0x00000000 0x000003ff
MEM GPU: 256 ARM: 768 TOTAL: 1024
Starting start4x.elf @ 0xfec00200 partition 0
MESS:00:00:05.550101:0: arasan: arasan_emmc_open
MESS:00:00:05.698627:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:05.701548:0: brfs: File read: 342 bytes
MESS:00:00:05.821820:0: HDMI1:EDID error reading EDID block 0 attempt 0
MESS:00:00:05.831336:0: HDMI1:EDID error reading EDID block 0 attempt 1
MESS:00:00:05.840843:0: HDMI1:EDID error reading EDID block 0 attempt 2
MESS:00:00:05.850357:0: HDMI1:EDID error reading EDID block 0 attempt 3
MESS:00:00:05.859867:0: HDMI1:EDID error reading EDID block 0 attempt 4
MESS:00:00:05.869382:0: HDMI1:EDID error reading EDID block 0 attempt 5
MESS:00:00:05.878890:0: HDMI1:EDID error reading EDID block 0 attempt 6
MESS:00:00:05.888404:0: HDMI1:EDID error reading EDID block 0 attempt 7
MESS:00:00:05.897914:0: HDMI1:EDID error reading EDID block 0 attempt 8
MESS:00:00:05.907428:0: HDMI1:EDID error reading EDID block 0 attempt 9
MESS:00:00:05.911926:0: HDMI1:EDID giving up on reading EDID block 0
MESS:00:00:05.918043:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:06.995664:0: gpioman: gpioman_get_pin_num: pin DISPLAY_DSI_PORT not defined
MESS:00:00:07.002961:0: *** Restart logging
MESS:00:00:07.004382:0: brfs: File read: 342 bytes
MESS:00:00:07.072608:0: hdmi: HDMI1:EDID error reading EDID block 0 attempt 0
...
MESS:00:00:07.226856:0: dtb_file 'bcm2711-rpi-4-b.dtb'
MESS:00:00:07.235551:0: dtb_file 'bcm2711-rpi-4-b.dtb'
MESS:00:00:07.248011:0: brfs: File read: /mfs/sd/bcm2711-rpi-4-b.dtb
MESS:00:00:07.251301:0: Loading 'bcm2711-rpi-4-b.dtb' to 0x1f0000 size 0xc042
MESS:00:00:07.289344:0: brfs: File read: 49218 bytes
MESS:00:00:07.465024:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:07.467674:0: brfs: File read: 342 bytes
MESS:00:00:07.488759:0: brfs: File read: /mfs/sd/overlays/vc4-fkms-v3d.dtbo
MESS:00:00:07.535268:0: Loaded overlay 'vc4-fkms-v3d'
MESS:00:00:07.537248:0: dtparam: cma-256=true
MESS:00:00:07.541611:0: dtparam: miniuart-bt=true
MESS:00:00:07.552794:0: Unknown dtparam 'miniuart-bt' - ignored
MESS:00:00:07.654662:0: brfs: File read: 1446 bytes
MESS:00:00:07.658863:0: Failed to open command line file 'cmdline.txt'
MESS:00:00:08.953730:0: brfs: File read: /mfs/sd/RPI_EFI.fd
MESS:00:00:08.956193:0: Loading 'RPI_EFI.fd' to 0x0 size 0x1f0000
MESS:00:00:08.962019:0: No compatible kernel found
MESS:00:00:08.966520:0: Device tree loaded to 0x1f0000 (size 0xc622)
MESS:00:00:08.974158:0: uart: Set PL011 baud rate to 103448.300000 Hz
MESS:00:00:08.981676:0: uart: Baud rate change done...
MESS:00:00:08.983696:0:NOTICE: BL31: v2.3():v2.3
NOTICE: BL31: Built : 10:40:51, Apr 21 2020
UEFI firmware (version UEFI Firmware v1.27 built at 11:17:17 on May 25 2021)
3hESC (setup), F1 (shell), ENTER (boot)
On the HDMI monitor, you get a nice raspberry pi
and a timeout to choose if you want to go into setup, shell or boot. If
you hit ESC, you will get what looks like a fairly standard BIOS/EFI
screen asking if you want to change various settings. At this point it
is a good idea to make a change to allow for full memory usage (or you
will be limited to 3GB and a slower system). Following the upstream
README, Device Manager
→ Raspberry Pi Configuration
→ Advanced Configuration
.
At this point you select ‘Limit RAM to 3GB’ and disable it. Save the
settings and escape up to the top menu. Choose the boot manager and you
should be given the choices of the following operating systems:
Fedora
UEFI Shell
SD/MMC on Arasan SDHCI
UEFI PXEv4 (MAC:??)
UEFI PXEv6 (MAC:??)
UEFI HTTPv4 (MAC:??)
UEFI HTTPv6 (MAC:??)
Choose Fedora and after a short pause, you will
get the grub config file. You shouldn’t need to change any defaults, but
it is good in case you did. Once the kernel has been selected, the
system will begin booting and the scary black screen occurs. This is a
point where for some seconds nothing seems to be happening and a couple
of times I was ready to power off and go back to other configs. Then you
will see something similar to start scrolling:
[ 0.000000] Linux version 5.12.10-300.fc34.aarch64 (mockbuild@buildvm-a64-03.iad2.fedoraproject.org) (gcc (GCC) 11.1.1 20210531 (Red Hat 11.1.1-3), GNU ld version 2.35.1-41.fc34) #1 SMP T
hu Jun 10 13:49:00 UTC 2021
[ 0.000000] efi: EFI v2.70 by https:
[ 0.000000] efi: ACPI 2.0=0x30720018 SMBIOS=0x33e00000 SMBIOS 3.0=0x33de0000 MEMATTR=0x321c2418 RNG=0x33fdb798 MEMRESERVE=0x30375118
[ 0.000000] efi: seeding entropy pool
[ 0.000000] ACPI: Early table checksum verification disabled
[ 0.000000] ACPI: RSDP 0x0000000030720018 000024 (v02 RPIFDN)
...
My belief is that the pause is the kernel and
initial ramdisk are getting gunzipped in memory for usage. Reading from
the MMC is slow, and uncompressing the files is slow. A future project
may be to see if there is a sizable speedup of just doing this on the
filesystem beforehand. In any case, the system will boot and be usable
as a workstation.
End of File
This
post has gotten on in size, and there were several other side tasks I
worked on while doing it. Those will need to be seperate posts in the
near future.
In looking for proxy systems, my team has been focusing first on the classic ARM for small projects: The Raspberry Pi. The raspberry pi4 with 4 GB of ram works out as a proxy for all kinds of ‘low-end’ systems where you may need to play with a small GPU and try to make it work with a Server Ready operating system like CentOS Stream.
Getting the hardware
There are several places to get the hardware, and while I used to get things from Adafruit, but they did not have an IOT kit set up for the 4 series. I ended up going with CanaKit from Amazon just so I could get a bundle of parts together. Going from past experience with MMC cards burning out after a couple of days, I bought 3 32 gig cards to put the OS on. So far the cards have lasted longer than I expected but that just means they will burn out any day now.
When getting a raspberry pi, I highly recommend making sure you get the correct power supply, a USB 2 serial connector for the GPIO, and if you are using an external drive, a seperately powered disk drive. I have found that while the Raspberry Pi4 uses a larger power supply than the 1,2, or 3 series… attaching a non-powered USB drive can be problematic on boot (the ssd one I had would cause me to just have the rainbow picture of doom).
Setting up the hardware
For the raspberry pi4 if you are using it to compile or build things, make sure you have correctly sized heat dispensors for the CPU and possibly a fan (and maybe a replacement fan for when the first one dies). Then attach a serial cable to pins 6 (ground),8 (txd),10 (txd). Make sure you do not attach anything to 1,2,3 as you could be looking for a new pi or computer. The serial is useful for when you attempt to boot a new kernel config and the HDMI connector was not functional afterwords.
On another computer attach the USB connector and you can use the
screen
orminicom
commands to see output from the system on boot. On my test system, I was able to capture the following:Initial setup
Like any hardware setup, it is important to make sure the shipped hardware has up to date firmware and configs. For this I took one of the MMC cards, and burned the Raspian OS with recommended software Once this booted up, the tools did a firmware upgrade on the system from whatever had been on the box when it was stored in a depot. This OS is a 32 bit operating system but is maintained by the ‘manufacturer’ so is a good test for ‘did it break in shipment’
{Pro-tip: Once you have finished updating, shutdown the system, take this card out, and put it in a box/bag for later use. At some point things are going to go badly in an install and you won’t know if its you, your hardware, or something else. Having a known bootable backup that is supposed to work is good.}
After this it is time to focus on getting ‘bootable’ systems using the base OS’s we want to target:
Fedora 34 Initial Install
Fedora 34 came out as I started working on the project, so I decided to aim for that as an initial OS. The ARM work that the Fedora team is doing is aimed primarily at 64-bit and with Server Ready hardware. As such, they do try to make a raspberry pi4 work, but it is listed as possibly problematic. That said, the following worked mostly fine:
sudo dnf install arm-image-installer
df -a | grep mmc ; sudo umount /dev/mmcblk0p1; sudo umount /dev/...
screen /dev/ttyUSB0 115200
Get serial working
At this point I wanted to make sure I could log into Fedora from the serial port. In order to do this, you need to edit the grub configs which is done in
/etc/default/grub
and then rebuilding the config. I moved the grub timeout to 10 seconds to give me a chance to choose different options, removed the rhgb and quiet, and added a console line.A test reboot is good here to make sure that when you boot you get past the
Fedora 34 with TianoCore
As stated before the Raspberry Pi systems are not System Ready and do not have a built in EFI or ACPI system for a kernel to boot from. Instead the default kernel boots usually with uboot mapping devices via device tree in order for hardware to be discovered. I am going to say that is about all I really know about the subject. There have been long threads in the Fedora ARM lists over the years on going over this versus ACPI, and I think it is better for an expert like Jon Masters or Peter Robinson to explain why APCI is preferred in Fedora Linux and Red Hat Enterprise Linux (RHEL) versus a novice like myself.
For the raspberry pi4, the current method to implement a UEFI interface is a port of the Tianocore by the Pi Firmware Task Force. TianoCore is an opensource implementation and extension of the Intel Extendable Firmware Interface which was written to replace the 16-bit BIOS used in personal computers since the 1980’s. A further extension was with the Open Virtual Machine Firmware which I believe was then used by the Pi Firmware Task Force for their version.
Easier than I thought
In reading various blogs and documentation on the status of the EFI support, I was prepared for a system that would only work via serial console or may not have networking or other utilities. Instead, I found the process to be fairly ‘painless’ and I only ended up with a non-booting system twice. The general steps were the following:
At this point, if you haven’t already, read the documentation. Our main tasks will be to setup the raspberry pi EFI boot partition to have the needed data in it. I had seen that this came with dtb files which I figured needed to be replaced. However as seen below, all of these are in a Fedora 34 and are of a similar timeframe. The only file we really need to work with is the
RPI_EFI.fd
andconfig.txt
files.At this point, it is time to replace the config.txt that came with the system with one which can be used for booting the UEFI program. This is where I caused my system to go into rainbow mode a couple of times. In the end, I put in the following file:
A couple of the commands in it are from a side trip on getting accelerated graphics working in CentOS Stream on Pi. I decided to document it for later as this is getting long. At this point I was able to get a booting system:
On the HDMI monitor, you get a nice raspberry pi and a timeout to choose if you want to go into setup, shell or boot. If you hit ESC, you will get what looks like a fairly standard BIOS/EFI screen asking if you want to change various settings. At this point it is a good idea to make a change to allow for full memory usage (or you will be limited to 3GB and a slower system). Following the upstream README,
Device Manager
→Raspberry Pi Configuration
→Advanced Configuration
. At this point you select ‘Limit RAM to 3GB’ and disable it. Save the settings and escape up to the top menu. Choose the boot manager and you should be given the choices of the following operating systems:Choose Fedora and after a short pause, you will get the grub config file. You shouldn’t need to change any defaults, but it is good in case you did. Once the kernel has been selected, the system will begin booting and the scary black screen occurs. This is a point where for some seconds nothing seems to be happening and a couple of times I was ready to power off and go back to other configs. Then you will see something similar to start scrolling:
[ 0.000000] Linux version 5.12.10-300.fc34.aarch64 (mockbuild@buildvm-a64-03.iad2.fedoraproject.org) (gcc (GCC) 11.1.1 20210531 (Red Hat 11.1.1-3), GNU ld version 2.35.1-41.fc34) #1 SMP T hu Jun 10 13:49:00 UTC 2021 [ 0.000000] efi: EFI v2.70 by https://github.com/pftf/RPi4 [ 0.000000] efi: ACPI 2.0=0x30720018 SMBIOS=0x33e00000 SMBIOS 3.0=0x33de0000 MEMATTR=0x321c2418 RNG=0x33fdb798 MEMRESERVE=0x30375118 [ 0.000000] efi: seeding entropy pool [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x0000000030720018 000024 (v02 RPIFDN) ...
My belief is that the pause is the kernel and initial ramdisk are getting gunzipped in memory for usage. Reading from the MMC is slow, and uncompressing the files is slow. A future project may be to see if there is a sizable speedup of just doing this on the filesystem beforehand. In any case, the system will boot and be usable as a workstation.
End of File
This post has gotten on in size, and there were several other side tasks I worked on while doing it. Those will need to be seperate posts in the near future.