[WIP]Multimodal parquet reader wds writer - #1559

Closed

VibhuJawa wants to merge 72 commits into

NVIDIA-NeMo:mainfrom

VibhuJawa:multimodal_parquet_reader_wds_writer

Commits on Feb 24, 2026

Add multimodal MINT1T reader/writer pipeline and scoped benchmarks
VibhuJawa
committed
fcb7de3
Move parquet output metrics helper to benchmark utils and update 2026 headers
VibhuJawa
committed
9dbe24a
Clean up multimodal reader/writer config and cloud materialization path
VibhuJawa
committed
4b6f3e0
Clean up multimodal stage readability and consolidate test fixtures
VibhuJawa
committed
c42bacf
Refine multimodal JPEG filter abstractions and naming
VibhuJawa
committed
73a66d3
Require explicit source_id_field in multimodal webdataset reader
VibhuJawa
committed
5d3547f
Optimize parsed source column extraction for multimodal task
VibhuJawa
committed
21908bf
Remove redundant direct materialization branch and add regression test
VibhuJawa
committed
ed901be
Refactor multimodal materialization and shared IO defaults
VibhuJawa
committed
4e90268
Unify multimodal reader field projection and tighten validation
VibhuJawa
committed
47a147b
Extract source field validation util and reduce multimodal LOC
VibhuJawa
committed
fb29991
Require multimodal writer path and add Pillow image dependency
VibhuJawa
committed
ba3ccdc
Move multimodal validation helpers and harden metadata source parsing
VibhuJawa
committed
545f224
Avoid tar reopen in multimodal materialize-on-read path
VibhuJawa
committed
25e3bd8
Make PIL import lazy in multimodal aspect ratio filter
VibhuJawa
committed
279858c
Fix multimodal defaults for tar formats and empty task table
VibhuJawa
committed
0a99dce
Trim multimodal reader/task duplication and simplify parsing
VibhuJawa
committed
c5e39bc
Remove redundant multimodal source-id validation unit test
VibhuJawa
committed
c2a1cd3
Restore explicit multimodal parsing and reader field exclusions
VibhuJawa
committed
9d37539
Consolidate multimodal metadata source missing-value handling
VibhuJawa
committed
2d62af2
Update nemo_curator/stages/multimodal/io/writers/multimodal.py

VibhuJawa
and
Copilot
committed
89e7df2
Remove multimodal pyarrow writer backend toggle
VibhuJawa
committed
cc6e4b2
Update uv.lock for Pillow image extras
VibhuJawa
committed
7faee07
Simplify multimodal materialization and remove writer aliases
VibhuJawa
committed
7d2093c
Remove redundant multimodal JPEG column guard
VibhuJawa
committed
3c42153
Move multimodal row-validity filtering into base filter
VibhuJawa
committed
2a0e0f8
Fix ruff unused-noqa and local PIL import in multimodal stage
VibhuJawa
committed
0e6ccaa
Align multimodal PIL import with text-stage local import style
VibhuJawa
committed
fa50bea
Apply suggestions from code review

VibhuJawa
and
Copilot
committed
fdd3e22
Fix multimodal index alignment, content_type, schema, and parquet index
VibhuJawa
committed
5bf4e76
Fix multimodal alignment and materialization behavior
VibhuJawa
committed
fe57ab6
Remove stale BLE001 noqa directives in MINT1T benchmark
VibhuJawa
committed
85b9cf3

Commits on Feb 26, 2026

Commits on Feb 27, 2026

Commits on Mar 2, 2026

Fixed some reordering problems
VibhuJawa
committed
c54141e
Add per-modality passthrough fields (per_image_fields, per_text_fields)
VibhuJawa
committed
c0cffa6

Commits on Mar 3, 2026

Commits on Mar 4, 2026