Store variables in a hdf5 file.

[TomasRL]

Hi all,

I am working with a hdf5 table to which I want to add some variables so that other people can use the same set of variables. I could define a separate text file with those, but given that I can define variables within the hdf5 file, I think it is way more convenient this way (as I would have everything in one file, no need to have 2). However, after defining the variables, when I save that dataframe, it stores all the newly created columns but not the variables.

test = vaex.open('test.hdf5')
test.add_variable('Ntotal', Ntotal)
test.export('test2.hdf5', virtual=True)
test.info() --> (see test.png, variables are stored, first plot)

When I open the new dataframe I got test2.png (second plot):

test2 = vaex.open('/data/users/ruizlara/disc_files_bayestar_bab18/test2.hdf5')
test2.info() --> (see test2.png, variables are not stored)

I have tried working with the 'state' options (get, load, write):

test.state_write('test_state.json')
state_new = test2.state_load('test_state.json')

This works, but I need two files, if I try to export again this test2 into a test3 the same will happen, variables are not stored.

Is there a way to store the variables in vaex that I am missing? I think this is a cool feature of the hdf5 format and would be really useful to have it.

Thanks a lot for your help!!

[maartenbreddels]

Hi Tomas,

interesting question.
The way I see it is that vaex will export the 'data', in a vaex-independent way, so what we write in vaex, should be able to be interpreted by any other library, without needing vaex (say 20 years from now).

My idea always was, that if you want to 'save the dataframe', you should export the data without virtual columns into an hdf5 file, and the state to a separate file. This also gives you the flexibility to have multiple state files and 1 data file.
Also, the variables in vaex are meant to be used as input variables for the virtual columns, outside of that they don't have any use in vaex. I think you want to use them for storing some extra data, like model input parameters right?

But, if for some reason, you want to store extra data in the hdf5 file, you can totally do that yourself. You can use the h5py library, and modify/add entries to the hdf5 file, or even store arrays etc.

So, I'd be against putting the variables inside of the hdf5 file, or at least not near the data, and I think users can just use the h5py library to add any data.

Mayne @JovanVeljanoski can share his ideas as well.

cheers,

Maarten

[TomasRL]

Hi Maarten,

20 years from now? Still using VAEX, right? ;) Yes, I understand your point, I just thought it could be a good option to have the possibility of storing those variables (yes, I meant extra data) without the need of an auxiliary file. But it is true that using states you have way more flexibility and you do not need 5 files of 15Gb each, just 1 file of 15Gb and 5 state files. I finally used h5py to add attributes (https://docs.h5py.org/en/stable/high/attr.html#attributes).... just one detail.... Once I add attributes to an hdf5, when I open it with VAEX I cannot see those attributes, is there a way? Not crucial, although might be good ;)

Cheers,
Tomas

[maartenbreddels]

20 years from now? Still using VAEX, right? ;)

I was actually implying Vaex wouldn't be around but the data would :)

Once I add attributes to an hdf5, when I open it with VAEX I cannot see those attributes, is there a way

Yes, vaex simply ignores them. You probably have to reopen the file, since we close the H5File object (although we do have the file opened for the memory mapping).