What's Changed
- CSVField: new member function try_parse_decimal() to specify one or more decimal symbols by @wilfz in #226
- Replace the includes of Windows.h with windows.h (#204) by @ludovicdelfau in #235
- Use const CSVFormat& in calculate_score by @rajgoel in #236
- Fix memory issues in CSVFieldList by @vincentlaucsb in #237
Race Condition Notes
Background
The CSV Parser tries to perform as few allocations as possible. Instead of naively storing individual CSV fields as singular std::strings in a std::vector, the parser keeps references to the raw input and uses lightweight RawCSVField objects to mark where a specific field starts and ends in that field (as well as flag indicating if an escaped quote is present). This has the benefits of:
- Avoiding the cost of constructing many
std::stringinstances - Avoiding the cost of constant
std::vectorreallocations - Preserving locality of reference
Furthermore, the CSV Parser also uses separate threads for parsing CSV and for iterating over the data. As CSV rows are parsed, they are made available to the user who may utilize them without interrupting the parsing of new rows.
The Race Condition
The RawCSVField objects mentioned previously were stored as contiguous blocks, and an std::vector of pointers to these blocks were used to keep track of them.
However, as @ludovicdelfau accurately diagnosed, if the reading thread attempted to access a RawCSVField (e.g. through reading a CSVField ) at the same time that a parsing thread was pushing a new RawCSVField to an at-capacity std::vector, the parsing thread's push would cause the contents of the std::vector to be reallocated, thus causing the reading thread to access deallocated memory.
This issue was first reported in #217.
The Fix
The fix was simple. An std::deque was dropped in to replace std::vector to store RawCSVField pointers, as std::deque does not perform reallocations. This change appears to even improve the CSV Parser's performance as the cost of constant reallocations is avoided. The loss of memory locality typical in std::deque applications was avoided as, again, the CSV Parser is storing pointers to RawCSVField[] and not the RawCSVField objects themselves.
New Contributors
- @wilfz made their first contribution in #226
- @ludovicdelfau made their first contribution in #235
- @rajgoel made their first contribution in #236
Full Changelog: 2.2.3...2.3.0
