feat(interleaved): add InterleavedParquetReader and InterleavedWebdatasetWriter - #1657

Merged

VibhuJawa merged 25 commits into

NVIDIA-NeMo:mainfrom

VibhuJawa:feat/interleaved-parquet-reader-wds-writer

Apr 3, 2026

Commits on Mar 24, 2026

feat(interleaved): foundation IO improvements — schema utilities, base-class fixes, and bug fixes
VibhuJawa
committed
34d2f34
simplify(interleaved): move resolve_schema to shared utils, remove dead code, pre-compute write_kwargs
VibhuJawa
committed
7702bfd
Merge remote-tracking branch 'origin/main' into feat/interleaved-io-foundation
VibhuJawa
committed
a1eeccc
chore: remove unrelated nixl-cu12 from override-dependencies
VibhuJawa
committed
33088b1
chore: restore nixl-cu12 override-dependency
VibhuJawa
committed
6f0e655
Remove source_id_field from WebDataset reader
VibhuJawa
committed
57263df
simplify: fix align_table, type hints, and add dict-type comment
VibhuJawa
committed
27cc69a
fix(tutorial): update quickstart notebook and set source_files in reader metadata
VibhuJawa
committed
987f5d4
Apply suggestion from @claude[bot]

VibhuJawa
and
claude[bot]
authored
575e505
ci: update secrets baseline for notebook image outputs (false positives)
VibhuJawa
committed
7df12f3
fix: make source_files unique per split in webdataset reader
VibhuJawa
committed
1a4dce4
fix: source_files per split lists only contributing tars in webdataset reader
VibhuJawa
committed
472b21c

Commits on Mar 30, 2026

fix: source_files per split lists only contributing tars in webdataset reader
VibhuJawa
committed
be9e7c5

Commits on Mar 31, 2026

test: add schema_utils tests and expand interleaved coverage
VibhuJawa
committed
86de41b

Commits on Apr 1, 2026