Buffer acceleration data (while writing to SD)

[erichiggins]

Currently, the write-operation creates a 10ms delay during which time acceleration data is no longer being collected.

Ideally, data would still be collected during this time so as to prevent any loss.

[RipVW]

No problem. I'm not ignoring this one, but the words aren't coming to me as easily as with the other subjects. I think now that the others are written down it'll be easier to concentrate on this one.

I'm interested in what you thought about Bill Greiman's low latency data logging program. My buffering methods aren't nearly as spiffy as his, but they work for 1 kHz and I think they'll work much faster than that.

[erichiggins]

@RipVW Just to clarify, I've filed all of these issues just to track the improvements I'd like to eventually make with this particular project rather than assignments for you specifically. So, please don't feel obligated. Hopefully others will show up and make some contributions as well!

[RipVW]

@erichiggins That's understood. I was thinking about all those things already so figured I might as well write it down. I do feel like at some point I committed myself to sharing what I've learned so far that you might find helpful. When I share how I did the buffering I'll feel like I'm done with that commitment. After that, I still have some ideas to share and a willingness to answer questions if I can.

[RipVW]

My comments on buffering will likely have 3 parts, but I'm not sure I'll have time for all three this evening.

1. How I think the SD card works
2. Circular Buffer design considerations
3. Code for the buffer

How I think the SD card works

Inside each SD card is a processor, and 512 byte volatile buffer, and several gigabytes of 'permanent' memory. Our scripts write several bytes at a time into the 512 byte buffer, when it gets full the SD card is busy for a few mS transferring that data to the permanent memory. Sometime the SD card may be busy for as long as 200 mS doing 'wear leveling' or other housekeeping chores. When done writing to the SD card a file close or flush command tells the SD card to transfer the partially full buffer to permanent memory.

Circular Buffer design considerations

• The circular buffer must be big enough to hold 200+ ms of data. Generally this means an array(s) with 200+ rows. One possibility is having six separate arrays for timestamp, start, delta, x, y, and z.

How often will the SD buffer get filled up?

timestamp - 4 bytes
start - 4 bytes
delta - 4 bytes, although this could be reduced to 2 since it's never very big
x, y, z - 4 bytes each, there is a way to use 2 bytes each, but that requires to some math later on
5 commas - one byte each
CR/LF - 2 bytes (I think)
Total of 31 bytes for each reading.

(512/31) = 16.5.... indicates a SD busy spell every 16-17 readings. My experience has been that when I fill up the 512 byte buffer ther is generally a 3-4 ms busy spell, and that every few hundred readings there is a 13 ms busy spell. The most readings I've ever had in the circular buffer is 25.

Is there enough time between SD card busy times to get the circular buffer emptied out?

When reading at 1 kHz the buffer can send 25 readings to the SD card's buffer between each reading. Oddly enough, although that should just about refill the SD card's buffer it doesn't immediately trigger another busy spell.

Of course at sampling rates greater than 1 kHz, not only will the 512 byte buffer will fill up faster, there'll also be less time to empty it out. This is where shortening the data to be stored can make a big difference.

[RipVW]

It seemed like the best way to illustrate the functioning of the circular buffer was to take a copy of your code and show what changes would have to be made. I put that copy on the RipVW branch.

[erichiggins]

@RipVW Nice, thank you! I just merged my PR so the master branch should have the latest changes. I believe you can run git pull from your local branch to keep things synced.

[RipVW]

You are welcome.

[RipVW]

Have been meaning to mention that the circular buffer in the example code I sent does little in preventing or reporting the buffer being overwhelmed. In both of our applications it seems better to carry on with logging rather than to stop unexpectedly. When post processing the data file I check for missed readings, buffer overruns, etc.

[erichiggins]

It looks like the library from Sparkfun for this sensor is more feature-complete than Adafruit.
For example, they support the built-in FIFO feature of the sensor, along with example code.
https://github.com/sparkfun/SparkFun_LIS3DH_Arduino_Library/tree/master/examples/FifoExample

I don't know if the libraries are interchangeable for use with boards from either manufacturer. It's a useful reference at the very least.

[RipVW]

Thanks Eric, I'll take a look at that.

[RipVW]

@erichiggins
Last night I learned that reading just 5 bytes of data from the accelerometer instead of 6 made a significant improvement to the script's speed.

As I understand the on-board fifo, there is no way to read just the 3 bytes containing data. So, at 8-bit resolution, reading the fifo means getting 3 bytes of data, 3 empty data bytes, 5 commas, and a CR/LF. In other words it requires reading 13 bytes in order to get the 3 bytes of data. After seeing how much difference it made to not-read 1 byte, I think reading 10 useless bytes would really slow things down.

The fifo might be more useful when reading higher resolution data, especially if the processor was juggling a lot of other tasks. For your application the script is fast enough already to keep up at the lower data rates associated with higher resolution. I would guess you could occasionally read a temperature sensor and still keep up.

[RipVW]

It occurred to me later today that the accelerometer's fifo buffer might output a stream of bytes, and it's the library that adds the commas and CR/LF.