[WIP]Multimodal parquet reader wds writer - #1559
Closed
VibhuJawa wants to merge 72 commits into
Closed
Conversation
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
… headers Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Vibhu Jawa <vibhujawa@gmail.com> Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Vibhu Jawa <vibhujawa@gmail.com> Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
…ilter position recompute - Change metadata_json category from Internal to Content (docs + README) - Add count(modality=) method for filtered row counts; num_items now returns unique sample count (distinct sample_id values) - Add add_rows() and delete_rows() stubs (NotImplementedError) - Recompute content positions in BaseMultimodalFilterStage after row drops to close gaps (metadata rows keep position=-1) - Add tests for position recomputation, count(), and pandas data path Signed-off-by: Vibhu Jawa <vjawa@nvidia.com> Made-with: Cursor
…mary Signed-off-by: Vibhu Jawa <vjawa@nvidia.com> Made-with: Cursor
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com> Made-with: Cursor
…quet_reader_wds_writer
…ation
- Add MultimodalLanceWriterStage extending BaseMultimodalWriter for Lance format
- Fix WebDataset writer to preserve interleaving via position-indexed parallel
texts/images arrays (previously compact arrays lost position information)
- Fix _sanitize_key to strip dots from WebDataset keys (dots are extension
separators per the WDS spec; sample_ids containing ".parquet" caused all
samples to merge into one)
- Use position-based image naming (e.g., "0.jpg", "3.jpg") so images[pos]
directly maps to sample[images[pos]] per WebDataset dict key convention
- Preserve all extra (non-schema) columns in WebDataset JSON via _row_extra
structured by modality: {text: [...], image: [...], metadata: {...}} with
1:1 alignment to texts/images arrays for zero data loss
- Fix BaseMultimodalWriter.write_data to not leak storage_options into
format-specific _write_dataframe calls
- Add test_merged_parquet_3format_write.py benchmark script for Parquet,
WebDataset, and Lance materialized writes from merged domain-bucket data
- Add explore_outputs.ipynb verification notebook with webdataset/lance/PIL
compliance checks and filtering UX demos
- Add unit tests for Lance writer and WDS extra column preservation
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Made-with: Cursor
The benchmark script and notebook are local test files that should not be tracked. They remain on disk for local use. Signed-off-by: Vibhu Jawa <vjawa@nvidia.com> Made-with: Cursor
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Previously all passthrough fields were broadcast to every row. This adds per_image_fields and per_text_fields which distribute list values 1:1 to their respective non-None content rows, with sample-level passthrough now only on the metadata row. Includes length-mismatch warnings, ValueError for non-list per-modality values, and centralized helper methods. Signed-off-by: Vibhu Jawa <vjawa@nvidia.com> Made-with: Cursor
…ory to interleaved Rename the task class, schema, module, stage directory, and all stage classes to use "Interleaved" naming, reflecting the interleaved nature of the data format: - MultiBatchTask -> InterleavedBatch - MULTIMODAL_SCHEMA -> INTERLEAVED_SCHEMA - tasks/multimodal.py -> tasks/interleaved.py - stages/multimodal/ -> stages/interleaved/ - BaseMultimodalAnnotatorStage -> BaseInterleavedAnnotatorStage - BaseMultimodalFilterStage -> BaseInterleavedFilterStage - MultimodalAspectRatioFilterStage -> InterleavedAspectRatioFilterStage - MultimodalParquetWriterStage -> InterleavedParquetWriterStage - BaseMultimodalReader -> BaseInterleavedReader - BaseMultimodalWriter -> BaseInterleavedWriter Signed-off-by: Vibhu Jawa <vjawa@nvidia.com> Made-with: Cursor
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com> Made-with: Cursor
Interactive notebook showing the end-to-end interleaved pipeline: read WebDataset tar → inspect schema → display interleaved document (text + inline images) → filter by aspect ratio → write to Parquet. Includes executed outputs with rendered MINT-1T sample data. Signed-off-by: Vibhu Jawa <vjawa@nvidia.com> Made-with: Cursor Signed-off-by: Vibhu Jawa <vjawa@nvidia.com> Made-with: Cursor
- Use IPython.display.Image (image/png output) instead of display(HTML) so images render on GitHub, JupyterLab, and VS Code - Add HuggingFace download cell (mlfoundations/MINT-1T-PDF-CC-2024-18) with MINT1T_TAR_PATH env var override for local data - Fix all ruff lint errors (type annotations, import order, magic numbers) Signed-off-by: Vibhu Jawa <vjawa@nvidia.com> Made-with: Cursor
The quickstart notebook embeds base64-encoded PNG thumbnails which trigger false positives in detect-secrets. Updated baseline to whitelist these along with existing pre-existing entries. Signed-off-by: Vibhu Jawa <vjawa@nvidia.com> Made-with: Cursor
Regenerated .secrets.baseline to include entries from the updated interleaved quickstart notebook (base64-encoded image outputs and hex strings from Parquet metadata) so the detect-secrets CI check passes. Signed-off-by: Varun Jawa <vjawa@nvidia.com> Signed-off-by: Vibhu Jawa <vjawa@nvidia.com> Made-with: Cursor
…ormat verification - Add multimodal_symmetric_benchmark.py: runs Reader -> [Filter] -> Writer for all 3 output formats (Parquet, WebDataset, Lance) with configurable reader type (WDS/Parquet), optional AspectRatioFilter, and materialization toggle - Add verify_cross_format_consistency.py: reads back Parquet, WDS, and Lance outputs into a common representation and verifies row counts, sample IDs, positions, modality distributions, and content tuples match across all formats - Fix multimodal_parquet_3format_benchmark.py: update imports from multimodal to interleaved API, add ObelicsReaderStage for HuggingFace OBELICS ingestion, add --reader-type, --use-filter/--no-filter, --input-path flags - Add collect_lance_output_metrics to benchmarking utils - Add generate_benchmark_comparison.py: produces COMPARISON.md with summary tables, per-stage timing, writer metrics, and cross-dataset analysis - Add run_all_format_benchmarks.sh with --clean, --materialization, --verify-lance - Add 6 symmetric benchmark entries + OBELICS dataset to nightly-benchmark.yaml with baseline requirements from 10GB MINT-1T runs - Add symmetric-smoke-test.yaml and symmetric-local.yaml for local runs Benchmark results (10GB MINT-1T, no materialization): Parquet writer: 1.35s write time, 132.9 MB output Lance writer: 1.10s write time, 249.3 MB output WDS writer: 14.68s write time, 144.1 MB output Signed-off-by: Vibhu Jawa <vjawa@nvidia.com> Made-with: Cursor
VibhuJawa
force-pushed
the
multimodal_parquet_reader_wds_writer
branch
from
March 3, 2026 08:50
099070a to
58e1594
Compare
Adds auto-generated analysis from symmetric benchmark results: - Writer ranking (Lance fastest in all scenarios) - Filter cost comparison (2.9-4.4x slowdown from AspectRatioFilter) - Reader cost comparison (WDS ~1.8x slower than Parquet reader) - Materialization cost (write time increases 7.8-19.3x depending on format) - Space efficiency (Parquet 1.81 KB/row, WDS 2.00 KB/row, Lance 3.44 KB/row) Signed-off-by: Vibhu Jawa <vjawa@nvidia.com> Made-with: Cursor
The --fields default hardcoded field names (language_id_whole_page_fasttext, previous_word_count) that ~5.3% of MINT-1T samples lack, causing a ValueError. Default to None (auto-discovery) to match benchmark behavior. Signed-off-by: Vibhu Jawa <vjawa@nvidia.com> Made-with: Cursor
- validate_and_project_source_fields now logs a warning and fills None when requested passthrough fields are absent from a source sample, instead of raising ValueError (heterogeneous data resilience). - _extract_per_modality_fields warns when a per-modality field is missing from the sample (previously silently ignored). - Changed type-mismatch raise from ValueError to TypeError (TRY004 fix). - Removed metadata_json from INTERLEAVED_SCHEMA -- passthrough fields already carry all source data; the redundant JSON blob is no longer needed. - Updated all tests, notebook, and README to match. Signed-off-by: Vibhu Jawa <vjawa@nvidia.com> Made-with: Cursor
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com> Made-with: Cursor
…, orphaned metadata, fragile cache - Track frame_index per content_key so distinct TIFF members each start at 0 - Preserve original TIFF bytes in binary_content when frame extraction fails - Drop metadata rows whose sample has no remaining content after filtering - Remove _resolve_frame id()-based cache; inline _extract_tiff_frame directly Signed-off-by: Vibhu Jawa <vjawa@nvidia.com> Made-with: Cursor
New parametrized and data-driven tests across 6 test files covering all previously untested branches in materialization, validation utils, InterleavedBatch task, filter/annotator stages, readers, and writers. Signed-off-by: Vibhu Jawa <vjawa@nvidia.com> Made-with: Cursor
…quet_reader_wds_writer Merge non-squashed history to share commit ancestry before merging main. Signed-off-by: Vibhu Jawa <vjawa@nvidia.com> Made-with: Cursor
Resolves conflicts from squash merge of NVIDIA-NeMo#1517 on main, keeping PR branch versions for interleaved module files (which include parquet reader, WDS writer, and Lance writer features). Signed-off-by: Vibhu Jawa <vjawa@nvidia.com> Made-with: Cursor
Remove references to the dropped metadata_json column. Metadata passthrough fields are now read directly from extra columns on the metadata row, matching the current INTERLEAVED_SCHEMA. Signed-off-by: Vibhu Jawa <vjawa@nvidia.com> Made-with: Cursor
14 tasks
Contributor
Author
|
Replaced by #1570 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Usage
# Add snippet demonstrating usageChecklist