Recent commit to firmware breaks mainline linux kernel booting with uboot on RPi4B (Arch ARM)

[graysky2]

Describe the bug
Using the current files on RPi4 8G causes the device to get stuck at the "rainbow" screen rather than booting

To reproduce
Booting linux-aarch64 (kernel) package with raspberrypi-bootloader-20240208

Expected behaviour
RPI4 should boot

Actual behaviour
RPi4 is stuck at "Rainbow" screen

System
Copy and paste the results of the raspinfo command in to this section. Alternatively, copy and paste a pastebin link, or add answers to the following questions:

• Which model of Raspberry Pi? RPi4B 8G
• Which OS and version (cat /etc/rpi-issue)? (No such file)
• Which firmware version (2023/12/14 16:43:25)
• Which kernel version (Linux rpi4 6.7.4-1-aarch64-ARCH #1 SMP PREEMPT_DYNAMIC Tue Feb 13 13:34:32 EST 2024 aarch64 GNU/Linux)?

Logs
If applicable, add the relevant output from dmesg or similar.
There is no dmesg output/stuck on "rainbow" screen

Additional context

Last known good commit = 5f795d1
First known bad commit = 2a1e5bf

The last know good is packaged in ArchARM's raspberrypi-bootloader version 20240122: archlinuxarm/PKGBUILDs@dedfb89

The first known bad is packaged in ArchARM's raspberrypi-bootloader version 20240208: archlinuxarm/PKGBUILDs@77ae14d

To fix the issue, users can simply downgrade to 20240122-1 and the system boots fine, so I believe that is pointing to a change between those two commits.

Files provided by ArchARM's raspberrypi-bootloader package:

% pacman -Ql raspberrypi-bootloader 
raspberrypi-bootloader /boot/
raspberrypi-bootloader /boot/bootcode.bin
raspberrypi-bootloader /boot/fixup.dat
raspberrypi-bootloader /boot/fixup4.dat
raspberrypi-bootloader /boot/fixup4cd.dat
raspberrypi-bootloader /boot/fixup4db.dat
raspberrypi-bootloader /boot/fixup4x.dat
raspberrypi-bootloader /boot/fixup_cd.dat
raspberrypi-bootloader /boot/fixup_db.dat
raspberrypi-bootloader /boot/fixup_x.dat
raspberrypi-bootloader /boot/start.elf
raspberrypi-bootloader /boot/start4.elf
raspberrypi-bootloader /boot/start4cd.elf
raspberrypi-bootloader /boot/start4db.elf
raspberrypi-bootloader /boot/start4x.elf
raspberrypi-bootloader /boot/start_cd.elf
raspberrypi-bootloader /boot/start_db.elf
raspberrypi-bootloader /boot/start_x.elf

[pelwell]

Can you confirm my understanding that, although the "bad" commit you reference includes kernel and firmware changes, you have seen a system go from good to bad by only changing the firmware?

[graysky2]

@pelwell - that is correct. One or more of the files I listed above seems to be breaking the boot of the non-RPi Foundation kernel, what I call scenario 2 in the following:

Users of RPi hardware on ArchARM can run one of two kernels:

1. linux-rpi which tracks your fork of the kernel
2. linux-aarch64 which tracks the vanilla linux kernel and requires uboot.

Both kernel packages have the dependency of raspberrypi-bootloader which is based on some of the files you provide in this repo (again, those which I have listed above).

Again, this bug affects users of scenario 2.

[pelwell]

Thanks - with hindsight it was a dumb question, "mainline" being the operative word.
The good news is that there are only 34 firmware changes between the two:

LKG: VC_BUILD_ID_VERSION: 30f0c5e4d076da3ab4f341d88e7d505760b93ad7 (clean)
BAD: VC_BUILD_ID_VERSION: e02d33b3122450accf9dea471a177d3b5623f5ad (clean)

most of them being Pi 5-specific.

I don't suppose you have any UART output, do you (enable_uart=1, uart_2ndstage=1)?

[graysky2]

I don't have any hardware to read from the UART... I believe there are only 2 commits between the last good and first bad that affect the /boot dir of this repo, see: https://github.com/raspberrypi/firmware/commits/master/boot

[pelwell]

I was referring to the source tree (the versions will make sense to other Pi engineers).

I don't have any hardware to read from the UART...

Email me ([email protected]) your address and I'll get a few Debug Probes sent out to you. In the meantime, if you have a second Pi and a few IDC patch wires you can cross-connect them:

    Pi 1          Pi 2
 6 (GND)  <->  6 (GND)
 8 (TXD0) <-> 10 (RXD0)
10 (RXD0) <->  8 (TXD0)

The numbers are pin numbers on the 40-pin header.

Run minicom etc. on the other Pi, with the baudrate set to 115200, 8 data bits, 1 stop bit, no flow control.

[graysky2]

• e7590ad built and booted fine.
• 7273369 built and booted fine.
• 2a1e5bf is the first bad commit.

[graysky2]

A number of active users have also reported this bug. I have invited them to help you debug as they may have both more time and the hardware to help. I am burning the candle at both ends.

Link to forum thread: https://archlinuxarm.org/forum/viewtopic.php?f=15&t=16753

[lategoodbye]

@graysky2 Thanks for reporting, could you please provide the affected U-Boot version?

[pelwell]

Out of the 34 commits, one stood out as being kernel related, seemingly harmless, and written by me.

This download (https://drive.google.com/file/d/1iwBPjviagitJEEE2feV1eAsCIYJIsN2j/view?usp=sharing) is a special build of start4.elf & fixup4.dat with that one commit reverted.

[pelwell]

(also posted to the Arch forum, with mangled links)

[graysky2]

Thanks for reporting, could you please provide the affected U-Boot version?

I just updated to 2024.01-1, prior to that, we were shipping a rather old version (2021.04).

This download (https://drive.google.com/file/d/1iwBPjviagitJEEE2feV1eAsCIYJIsN2j/view?usp=sharing) is a special build of start4.elf & fixup4.dat with that one commit reverted.

When I unzip it, I only find fixup4.dat not start4.elf

If I update my system to the bad version of raspberrypi-bootloader which is 20240208-1, and replace /boot/fixup4.dat with the file from your zip file, it does not boot. Perhaps I need the missing start4.elf as well?

[pelwell]

It should be OK now (the pitfalls of driving a double 4K PC desktop over VNC from a Macbook).

[graysky2]

OK, replacing both files from raspberrypi-bootloader-20240208-1 with those two files fixes the bug!

[popcornmix]

There is a potential fix in latest rpi-update firmware.
Can you test if that also fixes your issue.

[graysky2]

Disregard my last message, I accidentally omitted installing uboot-rasbperrypi. Building b208f8c does indeed fix the issue.

[popcornmix]

You had me worried for a minute...
I can confirm with RPiOS, rpi-update firmware does boot fine for me on a Pi 4.
Good to hear it also works with uboot.

[graysky2]

I also verified that b208f8c successfully boots on both RPi4B and RPi5B with our linux-rpi package (tracking your kernel fork) as well as with linux-aarch64 (tracking the vanilla linux kernel) on RPi4B. Many thanks to you guys for the rapid bug fix. This issue is closed from my perspective 👍

[pelwell]

Building b208f8c does indeed fix the issue.

Phew! That's a mean trick to play on my birthday!

[graysky2]

Ha! Happy Birthday!

[lategoodbye]

Thanks for fixing this so fast. AFAIU the root cause should be fixed in U-Boot v2024.04:
https://marc.info/?l=u-boot&m=170599732617675&w=2

[timg236]

btw: The same code is also present in the Pi5 bootloader. The fix from @pelwell will also be included in the next bump of the Pi5 firmware, tomorrow or early next week.

[tomty89]

@graysky2 I would like to note that uboot-raspberrypi 2024.01-1 breaks booting (with linux-aarch64) on RPI 3B+, even with raspberrypi-bootloader 20240214-1, if that matters. I tested again with the dtbs in 2021.04-1 (i.e., fb0bfa6), it doesn't work either.

Instead of getting stuck at the rainbow screen, apparently the issue is that somehow the new uboot prevents the early init (script) from being able to find / mount the root filesystem. (Or maybe it just froze or so, I can't tell for sure. ssh access via Ethernet failed as well.) I can see that the microSD and its partition table is detected by the kernel though.

initramfs was regenerated in the upgrade, but not in the downgrade (or the re-upgrade). boot.scr was always regenerated.

[graysky2]

Thank you for reporting. Two things to try:

1. Either build uboot-raspberrypi-2024.01-2 and test it or wait for it to hit a mirror and test it.
2. Only if that fails to boot, please try building 2024.04-rc2 from my personal fork and letting me know if it works?

[tomty89]

@graysky2 Okay so apparently the cause is that since some point, rpi_arm64_defconfig no longer works for RPI 3B+. I just built 2024.01-4 with rpi_3_b_plus_defconfig instead and it works fine then.

I'm not sure it was just some kind of regression or it is something expected, and if it's the latter case, I have no idea what should be done in terms of alarm packaging. The bottom line is, it doesn't seem to be just about CONFIG_DEFAULT_DEVICE_TREE, as it still has problem when I hardcode /dtbs/broadcom/bcm2837-rpi-3-b-plus.dtb in /boot/boot.{txt,scr} (yeah, FYI I was never sure if the "bcm271x / rpi" dtbs we inserted into the package are used during build time or boot time at all anyway. Or if they can even be used with upstream kernels.) (Not exactly the same problem though, apparently. Somehow ping works but ssh does not in that case.)

I suppose I'll just stick with the old version for now and build u-boot myself with rpi_3_b_plus_defconfig when I have to. (I don't think I'd bother taking it to upstream myself.) Feel free to let me know if you are going to work on / discuss the problem somewhere else though. (Probably we should stop discussing it here.)

Btw, there's a typo (extra 0x) in the patch update. I confirmed that it is not the root cause of the problem. (Well, it could be why the 2024.01-4 on repo didn't work with /dtbs/broadcom/bcm2837-rpi-3-b-plus.dtb hardcoded in /boot/boot.{txt,scr}. I didn't test the workaround with my own build. The bottom line is, fixing the patch is not going to fix everything.)

[graysky2]

Thanks for the report. Firstly, I fixed the typo: archlinuxarm/PKGBUILDs@7a526fb

To confirm, if you simply substitute rpi_arm64_config with rpi_3_b_plus_defconfig on line 65, the package works on RPi3B+?

[tomty89]

@graysky2 Well, when I used rpi_3_b_plus_defconfig, I also had the patch fixed, so I don't know if the typo could be problematic. (But I had some other builds with rpi_arm64_config and the patch fixed as well, and they didn't work. In other words, the typo was at least not the real culprit.)

Note: FWIW, https://github.com/u-boot/u-boot/blob/v2024.01/scripts/kconfig/Makefile#L99

[graysky2]

@tomty89 -

1. Can you try it with uboot-raspberrypi 2024.01-5 from the repos which has the patch fixed? Does it boot?
2. Can you build uboot-raspberrypi 2024.01-5 yourself modifying the make target on line 65 to rpi_3_b_plus_defconfig and try that. Does that work?

[tomty89]

@graysky2 Well as I said, I tested 2024.01-4 with the patch fixed (i.e., 2024.01-5), it doesn't work by itself, but works when rpi_arm64_config is replaced with rpi_3_b_plus_defconfig. What I don't know / didn't test is whether 2024.01-5 would work if I hardcode the/a correct dtb in boot.{txt,scr}, assuming "autodetection" is the only thing broken. (I don't think that matters much anyway since it would still mean it's practically broken upstream.)

If we want things to work "OOTB", we would either need to have different packages for different board (at least something like pre-PI4 and post-PI4), or take it to upstream and see if anyone knows what they have broken.

P.S. I do wonder if it's due to some conflict between rpi_arm64_config (on PI 3B+) and our patch, since the problem is a "late problem" (i.e., somehow the kernel / initramfs seem to partially work).