Heap Memory - Performance monitoring for the last 5 hours #825
Replies: 9 comments 38 replies
-
|
Have you watched it over several days? I ask, as I’ve seen this over a few hours at startup historically, but it generally stabilizes after a bit. I always attributed it to people walking past with BLE devices that scans pick up. |
Beta Was this translation helpful? Give feedback.
-
|
Hello @thorrak many thanks for your contribution. I have not but I will, It will run for the next hours and results will be posted here. I just have not understood your last sentence "I always attributed it to people walking past with BLE devices that scans pick up.". By the way, it make senses what you have said about stabilization, I see it more stable in these few hours around 200-199KB memory heap. |
Beta Was this translation helpful? Give feedback.
-
|
Hello again, 23 hours since the first gateway (esp32) startup: (From previous report) (The performance now is) 2024-12-28 09:40:21 --> ---- Heap Statistics ---- 2024-12-28 10:06:08 am --> ---- Heap Statistics ---- Any concern ? Will it leak memory over time ? I am concerned that esp will restart with lack of memory available after days. |
Beta Was this translation helpful? Give feedback.
-
|
Although it's certainly possible that there is a memory leak somewhere, that memory usage change doesn't make it sound like it's in the bluetooth stack (to me). Keep in mind that you are constantly creating/destroying devices in memory as your controller scans, so this would have to be an edge case. My recommendation would still be to let it run longer and see what happens. Eventually your device is going to lose power/get disconnected/restart, so what is the minimum acceptable run-time between restarts that you can tolerate? Test that long, and see if you have any issues. |
Beta Was this translation helpful? Give feedback.
-
|
Some more logs from the last couple of minutes: 2024-12-28 10:36:37 --> ---- Heap Statistics ---- |
Beta Was this translation helpful? Give feedback.
-
|
This is all completely normal, there a lot of allocations that occur in the normal operation of the BLE stack and is eventually cleaned up by the RTOS during the Idle task. |
Beta Was this translation helpful? Give feedback.
-
|
For compatibility reasons I'm using 1.4.3 with a FireBeetle-ESP32 board (WROOM32) and observing something similar, it appears that the heap is decreasing by about 20 bytes +/- with every deinit/init cycle. Although I only intend to do this once an hour in order to send data to the cloud I wrote a test routine to accelerate the process with only a 10 second delay and this continued to decrease overnight for a total over 1000 cycles. I have to wonder if this short interval is preventing the Idle task or perhaps this is indeed a leak that would eventually reset the device. |
Beta Was this translation helpful? Give feedback.
-
Yes facing same issue here. Definitely there is something going on. My device is up for a day now, I will monitor for more days. |
Beta Was this translation helpful? Give feedback.
-
|
Memory heap used to be quite normal when using nimBLE, however when added MQTT functionality it seems that min_free_heap decreases over time: Yellow line is min_free_heap After some time, assert heap memory fail happens. Is that a reflect to use reconnections + MQTT ? I am using a Queue (buffer) and when it completes 4 sensors servers temperature data, MQTT task is responsible to send data to AWS IOT Core. |
Beta Was this translation helpful? Give feedback.
-
|
No problem, same memory drop happens with only nimBLE:
I believe I have reached a point that my e sensor server connections is the fault. My modified nimBLE_Client.ino is attached below. The only thing I am doing is that on top of the code I identifies the both 4 devices I want to be connected when advertising:
Then, I just verify if advertized sensor server is in my list:
My full code is below: |
Beta Was this translation helpful? Give feedback.
-
|
I would not concern yourself with the min_free_heap unless you start seeing allocations failures in backtraces. That only tells you the lowest it's ever been, what matters is what is still available, which looks fine. The min is still quite high event at 48k as it gets released quickly as shown with the blue line. |
Beta Was this translation helpful? Give feedback.
-
|
I did not make myself clear, sorry for that. I showed partially as a tendency. At the end it restarts the esp32 all the time after drops. Like I showed some posts while ago. |
Beta Was this translation helpful? Give feedback.
-
|
To find the source of the crash I need to see logs and a decoded backtrace. |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately it still dropping the min free heap at every cycling. I have not got any other way of diagnosing it. I am trying to use platform io with arduino and espidf as framework together to use heap tracing in order to check memory leakage in more details. |
Beta Was this translation helpful? Give feedback.
-
|
Hello @h2zero, I could manage to use heap tracing in my nimBLE arduino code. I removed all AWS and WIFI connections and have run for couple of minutes. Min free heap has not started to decrease yet but look memory leakage from heap tracing:
Look this part of the log: 23605 bytes 'leaked' in trace (88 allocations) It means that nimBLE is leaking memory during allocations and thats why I am facing min free heap drops over time., isn´t it? Every scan cycle this leakage memory is presented. I also removed 3 sensor servers from connection to see if it is caused by amount of sensors but I see same leaked in trace message every scan time.
Any guess? Look my code attached, it is the standard code with quite few modifications just to track 4 well kwon devices. |
Beta Was this translation helpful? Give feedback.
-
|
I can see that your testing is flawed, you were checking for leaks before freeing the resources. |
Beta Was this translation helpful? Give feedback.
-
Thank you for your promptness again. I have followed your suggestion and modified the code to:
So I see that this explains min free heap drops I have faced. |
Beta Was this translation helpful? Give feedback.
-
|
did you also delete the client instances? Try the modified file I gave you. |
Beta Was this translation helpful? Give feedback.
-
I did not really. Sorry. I am applying the changes now in order to check heap tracing again. |
Beta Was this translation helpful? Give feedback.
-
|
@andrevipid Are you familiar with decoding crash backtraces? You could try increasing Also note that the dump may contains false-positives. Also I believe you should be calling I'll try testing this with my application and see if I get similar results. I've had an esp32-s3 scanning/connecting to peripherals constantly for 3 days straight and never noticed memory leaks. |
Beta Was this translation helpful? Give feedback.
-
Thank you sir for your contribution. I am updating the code to stop right before I print the trace_dump following espressif example: |
Beta Was this translation helpful? Give feedback.
-
|
After implementing the delete client suggested:
Same behavior has been visualized:
|
Beta Was this translation helpful? Give feedback.
-
|
You missed one, at the end of connectToSever |
Beta Was this translation helpful? Give feedback.
-
Thats the last one I screenshot:
Updated comment with delete client and heap tracing after scan function |
Beta Was this translation helpful? Give feedback.
-
|
Hello all, now I have a more clear picture about the memory leakage after adjusting the heap trace stop in the code to avoid false positive.
Now, the memory leakage seems more real to what happens to my application, that drains the memory over time. If we see, every cycle some bytes are leaked by 200 bytes:
Therefore, these are the following elements reported in heap tracing: 2025-01-20 15:59:10 --> 6 allocations trace (500 entry buffer) I have used the xtensa do detect from which part of the code: Respective 0x40115945 and 0x400d2752 (Left side caller)PS C:\Users\Documents\PlatformIO\Projects\Teste2> ~/.platformio/packages/toolchain-xtensa-esp32/bin/xtensa-esp32-elf-addr2line -pfiaC -e .pio/build/esp32dev/firmware.elf 0x40115945 PS C:\Users\Documents\PlatformIO\Projects\Teste2> ~/.platformio/packages/toolchain-xtensa-esp32/bin/xtensa-esp32-elf-addr2line -pfiaC -e .pio/build/esp32dev/firmware.elf 0x400d2752 0x400d34d9PS C:\Users\Documents\PlatformIO\Projects\Teste2> ~/.platformio/packages/toolchain-xtensa-esp32/bin/xtensa-esp32-elf-addr2line -pfiaC -e .pio/build/esp32dev/firmware.elf 0x400d34d9 0x400d336cPS C:\Users\Documents\PlatformIO\Projects\Teste2> ~/.platformio/packages/toolchain-xtensa-esp32/bin/xtensa-esp32-elf-addr2line -pfiaC -e .pio/build/esp32dev/firmware.elf 0x400d336c 0x400d32a6PS C:\Users\Documents\PlatformIO\Projects\Teste2> ~/.platformio/packages/toolchain-xtensa-esp32/bin/xtensa-esp32-elf-addr2line -pfiaC -e .pio/build/esp32dev/firmware.elf 0x400d32a6 0x400d40eaPS C:\Users\Documents\PlatformIO\Projects\Teste2> ~/.platformio/packages/toolchain-xtensa-esp32/bin/xtensa-esp32-elf-addr2line -pfiaC -e .pio/build/esp32dev/firmware.elf 0x400d40ea 0x400d4022PS C:\Users\Documents\PlatformIO\Projects\Teste2> ~/.platformio/packages/toolchain-xtensa-esp32/bin/xtensa-esp32-elf-addr2line -pfiaC -e .pio/build/esp32dev/firmware.elf 0x400d4022 0x400d2795PS C:\Users\poaand29\Documents\PlatformIO\Projects\Teste2> ~/.platformio/packages/toolchain-xtensa-esp32/bin/xtensa-esp32-elf-addr2line -pfiaC -e .pio/build/esp32dev/firmware.elf 0x400d2795 |
Beta Was this translation helpful? Give feedback.
-
|
@andrevipid Could you use the code tags to make the logs and code a bit easier to read for future posts? Instead of only passing one address at a time, I think you'd be better off passing the whole trace. This way if you increase the tracing stack depth you can see it all in one go. Could you try removing all occurrences of this line of code and retest? |
Beta Was this translation helpful? Give feedback.
-
|
Hello, one update regarding to the heap results. I noticed based on my last assessments that there were some services or characteristics still persisted on memory. Therefore I added deleteServices method before deleting the Client to completely delete the client. Running my application again, I have faced now only 1 allocation remaining with only 60 bytes. I am not able so far to detect what is this address using xtensa because it has given something that the editor cannot display like "???? : ?? : ? ". But see the allocations now: With 60 bytes per scan cycle my gateway will work longer but still draining memory and overtime it will reset. I reset my application again and checked once more. I have noticed that now the allocation goes with different caller address 0x401158fd:0x400d349d consuming 60 bytes instead of previous 0x40115961:0x400d34ed: The xtensa result for caller caller 0x401158fd:0x400d349d is: The respective nimBLEDevice line 304 is: I made sure all created clients are deleted. So, it is probably something related to internal nimBLE allocations. |
Beta Was this translation helpful? Give feedback.
-
|
@andrevipid The If there was a persistent memory leak in this library I would be very surprised that you are the only one that has noticed after all these years. I have my devices running for months without resetting and logging the free heap to MQTT and it is very consistent. |
Beta Was this translation helpful? Give feedback.
-
I was wondering the same thing. But fortunately now It seems stable. It has run for over hours with stable min free heap. |
Beta Was this translation helpful? Give feedback.
-
Hello, first of all I would like to share that I am really happy with this library, its is awesome to handle BLE connections with stable performance. I was at the beggining strugglinh with arduino standard BLE library but failing all the time due to the memory leake. Now, things are much better. I can handle 4 BLE connections at time with no issues.
I do have a gateway (esp32 wroom32e) scanning 4 bluetooth devices sensors (servers, esp32 c3), servers advertises for 4 seconds and transmit data to gateway every 25 seconds. Code is pretty much the same presented for nimBLE Client.ino.
I have run a test and I have plotted the serial monitor every scan time, in addition to that adding memory information such as esp_get_free_heap_size() and esp_get_minimum_free_heap_size().
Below it is possible to see memory heap for the last 5 hours. Although heap memory variation is small and it cleary shows (when ploggin every second) a memory recovery when disconnecting and reconnecting devices, I still see it decreasing a litle bit over the hours. Is there any reason to concern about memory leake under this current scenario ? In case i run my device with this below´s performance, will it run out of memory over the weeks or months ?
2024-12-27 10:00 am --> ---- Heap Statistics ----
2024-12-27 10:00 am --> Free Heap: 206068 bytes bytes
2024-12-27 10:00 am --> Minimum Free Heap: 205656 bytes bytes
2024-12-27 1:00 pm --> ---- Heap Statistics ----
2024-12-27 1:00 pm --> Free Heap: 200016 bytes
2024-12-27 1:00 pm --> Minimum Free Heap: 199120 bytes
2024-12-27 2:00 pm --> Free Heap: 199776 bytes
2024-12-27 2:00 pm --> Minimum Free Heap: 199084 bytes
2024-12-27 2:00 pm --> -------------------------
2024-12-27 3:00 pm --> ---- Heap Statistics ----
2024-12-27 3:00 pm --> Free Heap: 200140
2024-12-27 3:00 pm --> Minimum Free Heap: 199084 bytes
2024-12-27 3:00 pm --> ------------------------
2024-12-27 4:00 pm --> ---- Heap Statistics ----
2024-12-27 4:00 pm--> Free Heap: 199672 bytes
2024-12-27 4:00 pm--> Minimum Free Heap: 199084 bytes
2024-12-27 4:00 pm --> -------------------------
Beta Was this translation helpful? Give feedback.
All reactions