ZSVLIB makes efficient use of memory through the following techniques:
-
Memory copying is minimized. Bytes are read from the input (e.g. via
fread) and copied to a buffer. Usually, by the time the cell() and row() callbacks are called, no further copying is performed, andzsv_get_cell()will return a pointer back to the location in the buffer where the data was originally copied into from the initial read. Exceptions to this are:-
escaped double-quotes are removed using a
memmovecall (e.g."aaa""aaa"becomesaaa"aaa). Note: we are considering adding an option to skip this step; feel free to let us know if you have an opinion on that -
when the end of the buffer is reached, any partial row content if moved to the beginning of the row. This occurs on average once every N rows, where N is the average number of rows contained in each chunk of data read from the input. By default, the buffer size is 256k, which typically yields a high enough value of N that the impact of end-of-buffer copy is negligible. If your table rows are so large as to make this copy operation have a noticeable performance impact, use a higher buffer size
-
when a cell value is fetched, the parser returns the cell contents, the length, and a flag indicating whether the contents contain any delimiter or embedded dbl-quote that will require the value to be further processed if output as CSV (or TSV, JSON etc). This allows the caller to skip unnecessary scanning of cell contents in the common case where no quoting is required
-
-
Each row's entire contents are stored in a single contiguous block of memory. To operate on the entire block of row memory, you can simply operate on the memory that starts at the first cell's
.straddress and ends with the last cell's.str + .lenaddress. This can be advantageous for bulk operations, especially those that can be vectorized. -
The maximum row size is a function of the maximum size of the internal buffer, which is set (either to a default or a caller-specified value) when the parser is initialized.