Parquet ByteArray representation

[jp0317]

The current representation for ByteArray is based on the Bytes crate which supports maintains atomic pointers for thread safety. However, it might be an overkill for use cases where thread safety is not needed, as it adds unnecessary overhead. Does it make sense to provide or is there any alternative of the Bytes for the ByteArray?

[tustvold]

The major overhead associated with ByteArray is the allocation, not the atomics that surround it.

For use-cases reading byte arrays and needing performance I would strongly encourage using the arrow interface, as that allows sharing a single allocation across an entire batch of values

[jp0317]

Thanks for your reply!

is the allocation

would you mind elaborate what this 'allocation' is for, Bytes objects or the actual bytes, e.g., when use the read_records to read a batch of ByteArray?

using the arrow interface, as that allows sharing a single allocation across an entire batch of values

does this refer to the OffsetBuffer for ByteArray ?

[tustvold]

Unless the non-arrow interface is smarter than I remember it will create a new allocation for each ByteArray to store the value data. Regardless the row-oriented nature of these APIs will make them always slower than the arrow APIs, even discounting the fact comparatively little effort has gone into optimising them. If you care about performance, you should use the arrow APIs. I recommend you run some benchmarks and make your own judgement

[jp0317]

create a new allocation for each ByteArray

iiuc the ByteArray is just constructed locally then cloned to the output Vec buffer, increasing the reference count on Bytes while not copying the actual bytes

should use the arrow APIs

thanks i'll look into this. I was wondering if an alternative makes sense in cases someone cannot use arrow API. Besides, it seems using arrow API will copy the actual bytes for ByteArray, unlike the parquet api which just increases reference count?

[tustvold]

I guess it's smarter than I remember, and yes the arrow API will do a memcpy (although theoretically the newer StringView could avoid needing to do this). Point still stands though