[c6 and likely c5 and c61 specific] esptool elf2image generates wrongly-positioned esp_app_desc_t (ESPTOOL-1004)

[ivmarkov]

Operating System

All

Esptool Version

4.8.1

Python Version

Whatever used in the "binary release" images (pyinstaller)

Full Esptool Command Line that Was Run

No response

Esptool Output

The conversion from an ELF to a `.bin` generates the `.bin` file segments wrongly. The `esp_app_desc_t` struct is:
- *Not* at the beginning of the first IROM (= DROM) segment
- The first segment is **not** an IROM (= DROM) one to begin with

What is the Expected Behaviour?

I would expect the .bin file to start with an "IROM" (or "DROM" - seems to be the same for c6+ chips) segment, which - furthermore - has the esp_app_desc_t structure right in the beginning of this segment, after the image header and the segment addr + len.

More Information

Background is this bug reported in esp-idf-sys (the Rust tooling around ESP-IDF) (please look from the linked comment onwards): esp-rs/esp-idf-sys#365 (comment)

I do understand that there is no distinction between DROM and IROM in newer chips, but still - esp_app_desc_t ends up at the wrong place in the binary image.

Might very well be (if you look at the ELF dump of an esp32c6-compiled firmware in the linked bug) because the .flash.appdesc section in the ELF starts at addr 0x421e8020, while the executable section .flash.text - is listed earlier and starts at an earlier addr - 0x42000020 but I'm speculating about the root cause a bit. To continue the speculation - this - in turn - might also be a problem with the ESP-IDF linker script, in that both .flash.text and .flash.appdesc are "lumped" together in arbitrary order in irom_seg = drom_seg which seem to be the same now (start at the same address) without any specific order...

Wondering how could it be, that this problem is not experienced with pure "C" ESP-IDF? Pure luck only? The Rust tooling is of course using the ESP-IDF linker scripts and in fact the ESP-IDF linker command line; it just appends extra .a files to the link line...

Other Steps to Reproduce

I can provide Rust-built executables for inspection. For one, building the esp-idf-matter examples for the c6 always reproduces the problem, but I can also upload the binaries somewhere, or assist you in any way with diagnosing the issue. For the Rust side of things, this problem is pretty severe...

[Dzarda7]

Hi @ivmarkov, thanks for noticing this. I will take a look at it. I just quickly tried to build a project using ESP-IDF v5.3 for ESP32-C6 and found that when I run esptool.py image_info -v 2 on the built binary file, it does not show esp_app_desc_t. It might not be esptool issue, but I will let you know.

[ivmarkov]

Hi @ivmarkov, thanks for noticing this. I will take a look at it. I just quickly tried to build a project using ESP-IDF v5.3 for ESP32-C6 and found that when I run esptool.py image_info -v 2 on the built binary file, it does not show esp_app_desc_t. It might not be esptool issue, but I will let you know.

Didf you use the latest esptool.py from master or the released version? The released version will not show anything due to this commit, which is not released yet: 3555fe1

With that said, with this commit or not, the breakage is much more severe in that the app image does not start with the correct segment (the one containing esp_app_desc_t) in the first place. You can check this by doing a hexdump -C on the .bin image from your ESP-IDF project. If you don't locate the esp_app_desc_t struct at the very beginning (at offset 32 or so), then yes you are experiencing the issue (you can visually recognize the struct because it contains stuff like your ESP IDF project name as a C-string)

[Dzarda7]

Yes, thanks, the new commit actually hides the issue a bit, because app description is visible even though it is not at the beginning of the image. Thank you for deep investigation. I had a while to look at and it definitely seems to be esptool issue. elf2image command does not work correctly for chips that have common IROM and DROM address which includes for example ESP32H2 too and other chips you mentioned. This is not a problem for ESP-IDF projects, because it does not generate .bin images using esptool's elf2image command. But when you use the elf2image command on the created elf file, it also messes up the order. I did a quick investigation and found that the bug is probably somewhere in the save function. There are many hacks and obfuscations on the way, so I would not be surprised 😄. I will try to find it and fix it, but if you will be faster, feel free to open the PR. Thanks.

[ivmarkov]

This is not a problem for ESP-IDF projects, because it does not generate .bin images using esptool's elf2image command.

Interesting! What command does it use then? I was hoping it is still esptool but then was also wondering how come nobody noticed the problem in production with esp32c6 and stock ESP-IDF C images yet. If a different "elf2image" codepath is used in ESP-IDF, that would explain it.

There are many hacks and obfuscations on the way, so I would not be surprised 😄. I will try to find it and fix it, but if you will be faster, feel free to open the PR. Thanks.

I prefer not to fix it myself, as my Python working knowledge is likely much much worse than yours. :)

[Dzarda7]

@ivmarkov sorry for confusion. ESP-IDF actually uses esptool elf2mage command. The difference is that it passes --flash-mmu-page-size as a parameter with it. This parameter is set to 64K in esptool for all chips when not passed as the parameter, however some newer chips like these mentioned have the mmu page size configurable. When there is a mismatch (like in case of ESP32C6 which has 32K by default), esptool tries to align it as seen here. It fills the non-alignment with RAM segments first and with 0x00 then, this is the reason why the segment is misplaced. The solution to this is to provide the --flash-mmu-page-size as the rust tooling should.

I will take a look if it is a good idea to calculate the mmu_page_size from the app_info segment address, but not sure about it right now, otherwise I will try to add a hint that this parameter should be passed if app info segment is not the first segments. ESP IDF specifies the mmu_page_size in the app_info segment since v5.4 I believe, so it can be obtain like, but any previous version or anything else does not provide it, so this is still a partial solution.

[ivmarkov]

@ivmarkov sorry for confusion. ESP-IDF actually uses esptool elf2mage command. The difference is that it passes --flash-mmu-page-size as a parameter with it. This parameter is set to 64K in esptool for all chips when not passed as the parameter, however some newer chips like these mentioned have the mmu page size configurable. When there is a mismatch (like in case of ESP32C6 which has 32K by default), esptool tries to align it as seen here. It fills the non-alignment with RAM segments first and with 0x00 then, this is the reason why the segment is misplaced. The solution to this is to provide the --flash-mmu-page-size as the rust tooling should.

Thank you for this information!
I confirm that passing --flash-mmu-page-size 32KB does fix the issue for me.

I will take a look if it is a good idea to calculate the mmu_page_size from the app_info segment address, but not sure about it right now, otherwise I will try to add a hint that this parameter should be passed if app info segment is not the first segments. ESP IDF specifies the mmu_page_size in the app_info segment since v5.4 I believe, so it can be obtain like, but any previous version or anything else does not provide it, so this is still a partial solution.

Hmmm, rather than messing up with whatever was written by the app in esp_app_desc_t (which is valid from 5.4 onwards only, as you say), wouldn't a simpler fix be to instead of always using a 64KB default, use a per-chip default. This way, if I pass --chip esp32c6 on the "elf2image" command line (as I always do anyway), you actually do know it is the c6 and you can switch to 32KB?

And then the user could still override this with --flash-mmu-page-size <whatever>?

[Dzarda7]

I have prepared the fix for this, it is under review now. I investigated this deeply and found that we cannot change the default MMU page size for these chips unfortunately because of ESP-IDF. It passes the --flash-mmu-page-size parameter to the esptool only when it is not set to 64 KB as seen here, that is why default needs to be 64 KB.

But I introduced the fix that checks if app info segment is present and if so, use page size from it (available from ESP-IDF v5.4) or from its address alignment, which should be placed immediately after image header and segment header as stated in the app image format section and should be correctly aligned to set MMU page size (ensured by linker script). It is under review.