Hugely increased memory usage after upgrade from 4.17.0 to 4.18.0?

[doctea]

Hi! I recently upgraded my platformio, which took the installed platform-teensy from 4.17.0 to 4.18.0.

This has resulted in a large apparent increase in code and data size:-

With 4.17.0:
teensy_size: Memory Usage on Teensy 4.1:
teensy_size: FLASH: code:430772, data:78424, headers:8944 free for files:7608324
teensy_size: RAM1: variables:123872, code:387000, padding:6216 free for local variables:7200
teensy_size: RAM2: variables:17664 free for malloc/new:506624

With 4.18.0:
teensy_size: Memory Usage on Teensy 4.1:
teensy_size: FLASH: code:447076, data:101980, headers:9020 free for files:7568388
teensy_size: RAM1: variables:151136, code:405336, padding:20648 free for local variables:-52832
teensy_size: RAM2: variables:17664 free for malloc/new:506624

Switching platform-teensy back to 4.17.0 (via platform = [email protected] in platformio.ini) gets things working as expect for me again, with the original compilation size.

Anything I can do to help solve this?

[valeros]

Hi @doctea, I'm not sure what's the reason but v4.18.0 contains an updated Teensy core and new toolchain which go hand-in-hand and cannot be tested separately. PlatformIO has a special tool called Memory Inspector that may help your find out what symbol has the most substantial memory footprint:

[doctea]

Hi @valeros, thanks for your response, sorry for the long delay in my reply.

Unfortunately, running 'Inspect' on the project when using platform = [email protected] doesn't finish compilation and give any stats, even though compiling with the usual 'Build' tasks does. So I can't see what the difference may be when using the two versions. ('Inspect' works OK usually, and with 4.17.0)

The errors I get are about functions causing section type conflicts with other functions, as I have several functions marked FLASHMEM (I frequently have to juggle which functions can be tagged FLASHMEM and which can't in order to avoid these errors -- as my uneducated guess is that something peculiar happens when memory usage gets very low and the linker has to start cramming functions into the last remaining segments.. I think this is probably a different issue, but does make it hard to figure out what's going on)!

[doctea]

Realised I could do a crude comparison using the .o files from each build - some obvious outliers from first glance are:-

4.17: 49264	./FrameworkArduino/CrashReport.cpp.o/CrashReport.cpp.o
4.18: 89676	./FrameworkArduino/CrashReport.cpp.o/CrashReport.cpp.o

4.17: 63432	./FrameworkArduino/HardwareSerial.cpp.o/HardwareSerial.cpp.o
4.18: 73856	./FrameworkArduino/HardwareSerial.cpp.o/HardwareSerial.cpp.o

4.17: 28508	./FrameworkArduino/HardwareSerial1.cpp.o/HardwareSerial1.cpp.o
4.18: 57256	./FrameworkArduino/HardwareSerial1.cpp.o/HardwareSerial1.cpp.o

4.17: 22484	./FrameworkArduino/HardwareSerial2.cpp.o/HardwareSerial2.cpp.o
4.18: 50704	./FrameworkArduino/HardwareSerial2.cpp.o/HardwareSerial2.cpp.o
(..same for the rest of HardwareSerial[2-8]..)

4.17:  6688	./FrameworkArduino/serialEvent.cpp.o/serialEvent.cpp.o
4.18 17364	./FrameworkArduino/serialEvent.cpp.o/serialEvent.cpp.o

4.17: 39596	./FrameworkArduino/Stream.cpp.o/Stream.cpp.o
4.18: 56804	./FrameworkArduino/Stream.cpp.o/Stream.cpp.o

4.17:  86812	./FrameworkArduino/WString.cpp.o/WString.cpp.o
4.18: 109544	./FrameworkArduino/WString.cpp.o/WString.cpp.o

4.17: 30120	./FrameworkArduino/yield.cpp.o/yield.cpp.o
4.18: 42068	./FrameworkArduino/yield.cpp.o/yield.cpp.o

4.17:  89738	./lib841/libSPI.a/libSPI.a
4.18: 101562	./libc76/libSPI.a/libSPI.a

Majority of the other object files seem to be around the same size, or a few k smaller in 4.18 compared to 4.17.

Am I barking up the right tree here, or are the size of the object files misleading compared to the post-link size..?

[valeros]

Am I barking up the right tree here, or are the size of the object files misleading compared to the post-link size..?

The linker is smart enough to get rid of unused code, so I wouldn't rely on comparing the size of binary objects. Instead, I'd rather recommend generating a special map file (just add the -Wl,-Map,output.map flag to the build_flags option) for two binaries compiled using the 4.18 and 4.17 versions accordingly. This map file contains all information about used/discarded symbols so you can analyze what unnecessary parts of the framework got pulled into the final binary.

[doctea]

@valeros thanks again for your help here. I've produced a .map file for each build. Opening the resulting files in 'amap' shows me a breakdown of the size of the .o files, or by module or by file etc, but I'm not sure how to interpret anything useful from it, other than being able to see again that many library's .o files are larger in the 4.18 version.

Wondering if the 'Memory Configuration' and 'DTCM' tab is what I should be looking at?

It seems to show that showing that the _sdata .data section of libFrameworkArduino.a is ~25k larger and the _sbss .bsss section is about 3k larger... and also seems to show some files that don't seem to appear at all in 4.17 (eg, libc_a-categories.o and libc_a-jp2uc.o)... and I think the combined sizes of those gets some way to totalling the extra size of the 4.18 version.

Are those clues as to what might be going on?

Any tips on what I should be looking for here, or what if anything I can do about it? :)

[valeros]

Mind sharing that two map files here? I'm also curious what is the memory usage if you compile your project using Arduino IDE.

[PaulStoffregen]

If you try with Arduino IDE 2.1.1, use Boards Manager to install Teensyduino 0.59.3 (beta) and select Tools > Optimize for Smallest Code or Smallest Code with LTO.

[doctea]

Sure thing -- here's a zip with the two output files, let me know if you have any trouble with that and thanks for taking a look!
doctea map files usb_teensy_clocker.zip

The project is quite large so I think it might be a sizeable undertaking to get it to compile in the Arduino IDE at this stage, but if its necessary then I'll see what I can do!

[valeros]

It seems that some additional amount of code is now pulled from the standard C library (the categories.o and jp2uc.o object files to be precise). Not sure what's the reason, so I'd be great to test your project in Arduino IDE, just to make sure the problem has something to do with PlatformIO.

I'm also wondering if #95 is related in any way.

[PaulStoffregen]

If the problem is upstream, of course I want to investigate and fix.

To begin any upstream investigation, a test case must be given which reproduces the problem with the latest Arduino IDE and most recent Teensyduino beta version. The beta versions appear in this Boards Manager drop-down list, with major version 0 to prevent Arduino IDE from suggesting them as updates. Please make sure you test with the latest beta.

The default optimization is "Faster" which means -O2 on the gcc command line. Larger code which runs faster is not considered a bug in this mode. For smaller code, you want -Os which is "Smallest Code" in the Tools > Optimize menu. Please make sure you have selected it from this menu before you run the test case.