Core Dump support via CrashCatcher and heap storage

[salkinium]

This uses the CrashCatcher to generate the core dump, store it in volatile memory and reboots so that the report is available for the application to use under normal operating conditions (ie. not in the HardFault handler).

This works as follows:

1. A hardfault is generated, CrashCatcher performs its job.
2. CrashCatcher calls into the :platform:fault module which stores the report in the memory sections designated for the heap as defined by the linkerscript's .table.heap section, effectively overwriting the heap.
3. CrashCatcher reboots the device.
4. modm's heap initialization does not overwrite the report stored in the heap, and initializes the remaining heap memory.
5. The application's boot process continues normally and may use the heap, even if it is smaller now.
6. The application can access the report and decide what to do with it (for example send it out via UART).
7. To clear the report and regain the full heap memory, the application must reboot the device again.

This gives significantly more freedom to the application for responding to a non-recoverable fault event, even allowing the use of dynamic memory for more complex communication protocols like XPCC over CAN.

The :platform:fault module may be configured to output just the core, or the core and stack or the core, stack and static data memories (.data and .bss sections), depending on how much heap memory is available.
The resulting report is to be used with the CrashDebug program that support post-mortem debugging.

TODO:

• Implement Core functionality
• Module documentation
• Implement heap truncation for all allocators
• Upgrade examples
• Better (stateless) reporting API: C++ Iterator design
• SCons tool for best-effort post-mortem debugging
• Extend SCons postmortem tool with artifact caching based on firmware hash
• Pre-Report hook to put hardware in safe mode
• Use Assembly instructions for accessing memory
• Move CrashCatcher stack to top of heap instead of static memory. 500B is a lot.

The current user facing reporting API is only FaultReporter::report(lambda) which is a synchronous API forcing sequential reading and a reboot afterwards. This is a convenience API for synchronous UART reporting, but doesn't support sending the report in smaller chunks over an asynchronous channel like XPCC over CAN.
Added an InputIterator for reading the data however you want it.

cc @rleh @dergraaf @chris-durand

[salkinium]

The module does not implement any GPIO blink behavior anymore, since it's out-of-scope.
The application can overwrite the extern "C" void HardFault_Handler itself to implement just LED blinking, or it can blink after the reboot when the report has been detected.

[salkinium]

I added a HeapTable input iterator, which completely hides the fault storage from the heap initializations as well as deduplicated some code and makes it readable.
I will add a FaultReporter input iterator as well.

[salkinium]

The F469 example correctly allows heap usage after hardfault reboot and before reporting:

Can I allocate 20kB? answer: 0x2000109C
Hold Button to cause a Hardfault!

Can I allocate 20kB? answer: 0x2000272C

=== CrashCatcher === HardFault === CoreDump ===

6343030001000000170000002B00000010000... lots of data

Can I allocate 20kB? answer: 0x2000109C
Hold Button to cause a Hardfault!

You can see the the first allocation, then a hardfault is triggered, the device reboots, allocates in a different spot, dumps the core, then reboots again, and can use the full heap again.

Works for all allocators transparently: newlib

Can I allocate 20kB? answer: 0xC0000008
Hold Button to cause a Hardfault!

Can I allocate 20kB? answer: 0xC0001658

=== CrashCatcher === HardFault === CoreDump ===

6343030001000000170000002B0000000000...

Can I allocate 20kB? answer: 0xC0000008
Hold Button to cause a Hardfault!

and block allocator:

Can I allocate 20kB? answer: 0xC0000004
Hold Button to cause a Hardfault!

Can I allocate 20kB? answer: 0xC0001654

=== CrashCatcher === HardFault === CoreDump ===

6343030001000000170000002B000000000000000...

Can I allocate 20kB? answer: 0xC0000004
Hold Button to cause a Hardfault!

[salkinium]

@dergraaf I need the binary to compute the firmware hash, but I couldn’t figure out how to depend on env.Bin(source) properly. Any ideas?

[dergraaf]

In general that's what env.Depends() is for. I can look into it in more detail tomorrow, at the moment I don't have my laptop.

[salkinium]

Yes, I got to env.Depends too, but I just didn't understand SCons enough to know when to call this? I tried something like env.Depends(action, "path/to/artifact.bin") in the store_artifact function but I didn't get that to work. SCons is super undocumented.

[salkinium]

I've added SCons support for caching the uploaded firmware ELF and binary files in the build directory and allow you to retrieve it using the firmware CRC32 sum: scons postmortem firmware={hash}.
This way you won't have to remember what commit the firmware is running on (if you even have a commit for that!!) and the whole thing is automated so you can't forget.

Of course you still have to manually copy the coredump data into the coredump.txt file, but then again this is post-mortem debugging, so you're not likely having your computer connected to the device at the time of the fault anyways.

I'm quite happy with this solution, it's also nicely documented I think.