Preferred file format

[humcasma]

What is the preferred file format to work with Vaex: hdf5, arrow, feather,...?

[JovanVeljanoski]

Depending on what you are trying to achieve.

• For performance - HDF5. Arrow should be close second and I hope in the future there should be little to no difference between the two. For now HDF5.
• For interoperability - Arrow. If your goal is to pass data around to other eco-systems (say Java), or what to send/receive data from databases etc..
• For optimizing disk space - parquet. Sometimes people want to store data in the cloud, but want the data to be compressed etc.

Does this help a bit?

[humcasma]

Thanks.

Yes, it helps.

Right now I am more concerned with performance and with not having issues. I have been using HDF5, but wanted an alternative until the issues with saving NaNs are solved. I am now trying Arrow, but I am having another issue exporting a dataframe that contains an internal variable (var_isin_ordered_set |   | $\mathrm{}$ ). Error message is ArrowInvalid: Tried to write record batch with different schema. If saving first to Feather, then loading and then exporting to Arrow the problem disappears. I will try to open an issue if I manage to find a reproducible example.

[humcasma]

Arrow gives me also problems with dates: TypeError: data type 'timestamp[ns]' not understood

[xiaodaigh]

The speed gain from parquet to hdf5 was impressive.

The same query went from 10s to 0.1s. This would be faster than the same query in {disk.frame} (I'm the author), but {disk.frame} is still sub-second.

But

The hdf5 file was 30x bigger than the parquet file! And so, if we are talking about big datasets, parquet's compression still results in better performance. In fact the HDF5 file was almost 3 larger than the original CSV file.

Also, it took a while to convert the parquet to HDF5 for me.

[maartenbreddels]

Hmm, this tells me that we are not reading parquet ideally. Maybe a non ideal chunksize matters, re exporting to parquet using vaex might help here, still it should not matter that much.

[xiaodaigh]

BTW the data I used is just the Fannie Mae https://docs.rapids.ai/datasets/mortgage-data

I found the biggest CSV I can in those files, converted it to parquet then did this

import vaex
df = vaex.open("Performance_2004Q3.txt.parquet", convert="meh.hdf5")


df.head()

import time

t = time.time()
df.groupby(by="V11", agg = {'mean':vaex.agg.mean("V5")})
print(time.time()-t)

[maartenbreddels]

Thanks, we should compare it ourselves as well!

[JovanVeljanoski]

Hi @xiaodaigh

Thanks for this info!

Just so we can reproduce your example: the result your groupby operation is 2x2 dataframe right? Our data has column names, so just making sure we are using the same columns.

Also, is it possible to tell us the specs of the machine you are using? And finally the version of vaex please.

Thank you!

[xiaodaigh]

I got 128 (roughly) by 2.