[v2023.1.x] Fritzbox 7520/7530 bricked after update to v2023.1.1

[maurerle]

Bug report

What is the problem?

The FB7520 and FB7530 do not boot properly since some commit between the tag v2023.1 and v2023.1.1

This bug exists only on this branch, a build from master is working properly.

I don not have serial logs for this bug - the broken firmware can be currently found here:
http://firmware.freifunk-aachen.de/firmware/gluon-ffac-v2023.1.1-1-avm-fritz-box-7530-sysupgrade.bin

The Power LED stays lit in green and the device does nothing.
No Wifi, no Gateway, no ip address..
Power cycle does not help either.

Someone reported that a Computer attached to the switch can talk to another computer on the switch, when the device is on - but the device itself does not boot properly.

We have 13 devices which have this problem.

What is the expected behaviour?
The device should work as normal - as bevor v2023.1.1 or on current master.

Gluon Version:
v2023.1 and v2023.1.1

Site Configuration:
https://github.com/ffac/site

Recovery

To recover, one has to use the respective AVM tool:
For 7520:
https://download.avm.de/fritzbox/fritzbox-7520/deutschland/recover/
For 7530:
https://download.avm.de/fritzbox/fritzbox-7530/deutschland/recover/
(works with playonlinux/WINE)

And can then run the (improved) fritz flash tool from here:
https://github.com/maurerle/fritz-tools/blob/master/fritzflash.py

Download an initramfs and a working firmware
And run:
sudo python3 fritzflash.py --dev eno2 --initramfs ./openwrt-22.03.2-ipq40xx-generic-avm_fritzbox-7530-initramfs-fit-uImage.itb --sysupgrade ./gluon-ffac-v2023.1.0-3-avm-fritz-box-7520-sysupgrade.bin
or execute this command without sudo in an admin prompt on windows.

[Djfe]

on windows there are definitely a few more differences, I'll update this post, once I have tested a working command.
One difference: you execute py instead of python3

[Djfe]

Finally got around to doing the upgrade on a box while being connected via serial
The box hangs at Starting kernel ...
https://pastebin.com/0NiNpjqr

Next step:
I'm building branch openwrt-22.03 to see whether it's affected or not.
Then I'm going to do a git bisect.

If openwrt upstream isn't affected then I might need further assistance to build Gluon with some kernel debugging enabled.
I might find some help already in some past github issues for similar booting issues

[maurerle]

Searching for the other two lines before it dies:

Read 1024 bytes from 02bd800
maca pointer invalid: a5472102
Using machid 0x8010001 from environment
 
Starting kernel ...

led me to this issue with an unbootable 7530: chunkeey/FritzBox-4040-UBOOT#9
So this might be related.
I currently do not have time to test various commits on the box

[grische]

@maurerle from what I understand in the ticket you linked, it's either this patch
chunkeey/FritzBox-4040-UBOOT@9d89013

Or some bad blocks on the NAND that can be fixed by having a smaller ubifs. I'm surprised that this affected all of the 7530 that were used in their community.

Or am I missing something here?

[blocktrron]

The other issue is not related, as ubifs inconsistency would fail the boot process when reading from NAND. In addition, the kernel image checksum is validated prior jumping to the image entry.

[Djfe]

Sorry, no progress from me since last week.
I wasn't at home last weekend and didn't get any tests on openwrt done this week. Next time I can do some tests (the bisect) is Sunday at the earliest or Wednesday at the latest.

[maurerle]

current master and 23.05 works

v2023.1.x (e521759) is broken
v2023.1.1 is broken
48582f2 is broken
v2023.1 works
So it is broken since one of:
v2023.1...48582f2
eventually the kernel bump

Openwrt: 057bf8fc5f390cf78dc440b3e72d540aeafe1e81 breaks - So its one of those 26 commits:
openwrt/openwrt@1ec274a...057bf8f

Edit: i was barking up the wrong tree as I thought to remember that v2023.1 was not working either. I always did build gluon by setting different openwrt commits in modules file.

I tested building from latest 22.05 branches in openwrt (to see if bumping helps). But this image fails too.
So we need to find the root of the issue and the issue is still present..
I used the following patch on top of v2023.1.x for my latest build (which fails):

diff --git a/modules b/modules
index 19a5a3ff..cfef9636 100644
--- a/modules
+++ b/modules
@@ -2,15 +2,15 @@ GLUON_FEEDS='packages routing gluon'
 
 OPENWRT_REPO=https://github.com/openwrt/openwrt.git
 OPENWRT_BRANCH=openwrt-22.03
-OPENWRT_COMMIT=2034387af45f11c933e36ebdde4ca198c45068f9
+OPENWRT_COMMIT=c2921044e75fb05d666d76850cb922f8f67c1784
 
 PACKAGES_PACKAGES_REPO=https://github.com/openwrt/packages.git
 PACKAGES_PACKAGES_BRANCH=openwrt-22.03
-PACKAGES_PACKAGES_COMMIT=e061716ae08e57e825cb50a07ba3e2afc833617d
+PACKAGES_PACKAGES_COMMIT=64d48d70a19884f4ae9a3e89f045bbecfaaa4abd
 
 PACKAGES_ROUTING_REPO=https://github.com/openwrt/routing.git
 PACKAGES_ROUTING_BRANCH=openwrt-22.03
-PACKAGES_ROUTING_COMMIT=f2b9e3536523b4e23d14dd7d21cfa17ceb622b87
+PACKAGES_ROUTING_COMMIT=777c115b0a6336d6582c5992d25b631b6d6d21fd
 
 PACKAGES_GLUON_REPO=https://github.com/freifunk-gluon/packages.git
 PACKAGES_GLUON_COMMIT=29912ec6308fd10b47763b4cf28a638d07f59973

[maurerle]

So the next steps are to check if these openwrt commits are working:

• 419218af13b4c036b3f27e450855727005b8d141 (broken)
• de29f15af173e9434d11a00ffcf437bd6bc97727 (broken)
• 188c49b32112aae146a3ac3f870ad486cf1db11f (must be broken too)

So now it must be one of these Kernel bumps:

openwrt/openwrt@1ec274a...419218a

Next steps should be to test plain openwrt images - as this issue persists in current gluon

EDIT:
As expected - I could reproduce this on OpenWRT:
works: openwrt/openwrt@1ec274a
Is stuck on boot: openwrt/openwrt@419218a

[maurerle]

This issue is fixed by the update to latest openwrt based on v22.03.6 in #3087
see openwrt/openwrt#13822 (comment) for more information