[feature request] support different webdataset format

[ethanhe42]

to my understanding, the energon compatible format needs everything in the same folder like this

shard_000.tar
├── samples/sample_0000.jpg
├── samples/sample_0000.txt
├── samples/sample_0000.json
├── samples/sample_0001.jpg
├── samples/sample_0001.txt
├── samples/sample_0001.json
└── ...

however, users might have different dataset format, for example, images live in one folder while labels live in another.

We need to be able to customize dataset format definition. e.g., a user defined function for mapping data to dictionary.

[lvoegtle]

To clarify:

Using this feature, the following would be possible:

base/shard_000.tar
├── samples/sample_0000.jpg
├── samples/sample_0001.jpg
└── ...
ground_truth/shard_000.tar
├── samples/sample_0000.json
├── samples/sample_0001.json
└── ...
extracted_features/shard_000.tar
├── samples/sample_0000.features.pt
├── samples/sample_0001.features.pt
└── ...

i.e. in the respective folders, the same shards with the respective same sample key must be present. The features (here "jpg", "json", "features.pt") will then be available for the same samples.

This would allow e.g. precomputing features, or changing the ground truth without modifying the input data (in this case the image).

A difficulty might be the wizard for preparing datasets. That wizard must allow combining the shards.

Implementation thought:

• This should likely to be handled in the innermost WebdatasetSampleLoaderDataset, to combine the multiple tar loaders.
• Internally, the .nv-meta must somehow reflect these multiple files, which was not foreseen at the moment. Maybe one folder must be the "primary" folder, and the others are "secondary", such that the primary gives the sample keys, and the secondary are only additions and may not introduce additional keys.
• A slightly difficult case may be if a feature is optional and a sample does not have a feature. A solution may be that the .taridx points to the next sample, which will not match the key, thus not load the feature.

[ethanhe42]

hi @lvoegtle what you described is accurate! fwiw, in webdataset, your idea of "primary" folder is supported through https://webdataset.github.io/webdataset/column-store/#using-webdataset-as-a-column-store