Skip to content

[REVIEW] Feat/interleaved io readers writers #1570

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
VibhuJawa wants to merge 39 commits into from
Closed

[REVIEW] Feat/interleaved io readers writers #1570

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
8877955
Add interleaved parquet reader, webdataset writer, and lance fragment…
VibhuJawa Mar 4, 2026
827def1
Fix lance fragment schema alignment and enforce core column types
VibhuJawa Mar 4, 2026
d0c7f89
Centralize schema alignment into utils/schema.py
VibhuJawa Mar 4, 2026
13972b1
Batch S3 range reads into single cat_ranges call for 3-8x speedup
VibhuJawa Mar 4, 2026
91028bb
Fix dictionary column decode and preserve WDS image names without binary
VibhuJawa Mar 4, 2026
65ddac5
Reconstruct per-image/text metadata in WDS writer and fix None task h…
VibhuJawa Mar 4, 2026
58427e0
4000x faster WDS writer: replace iloc+iterrows with column list indexing
VibhuJawa Mar 4, 2026
5ad4359
Relax fsspec pin to >=2024.12.0 to fix CI dependency resolution
VibhuJawa Mar 4, 2026
0e47432
Address Greptile review: remove redundant lance alignment, fix read_s…
VibhuJawa Mar 4, 2026
be6d9f9
Remove dead code: _write_sample, _collect_rows, _extract_row_extras
VibhuJawa Mar 4, 2026
bd68db2
Fix Greptile review issues: storage_options, output_schema propagatio…
VibhuJawa Mar 4, 2026
86dbc42
Update nemo_curator/stages/interleaved/utils/schema.py
VibhuJawa Mar 4, 2026
a4d5606
Address Greptile review: lance commit mode validation, sanitize_key c…
VibhuJawa Mar 4, 2026
979f1b2
Fix align_table indentation from co-authored commit and add RUF021 pa…
VibhuJawa Mar 4, 2026
ca95788
Add multimodal_cpu optional dependency group with pylance, Pillow, s3fs
VibhuJawa Mar 4, 2026
02c993f
Add lance-encoding:blob metadata to binary_content in INTERLEAVED_SCHEMA
VibhuJawa Mar 4, 2026
005431a
Merge branch 'main' into feat/interleaved-io-readers-writers
VibhuJawa Mar 5, 2026
31fd370
Fix WebDataset writer/reader round-trip bugs
VibhuJawa Mar 5, 2026
8b869ed
Fix _escape_key to percent-encode unsafe chars, trim redundant _LARGE…
VibhuJawa Mar 5, 2026
6c3d11a
Merge branch 'main' into feat/interleaved-io-readers-writers
VibhuJawa Mar 9, 2026
cc5d386
Fix JPEG-in-TIFF pixel corruption in _extract_tiff_frame; parametrize…
VibhuJawa Mar 11, 2026
d8ad84a
Fix interleaved IO readers/writers: schema_overrides, passthrough fie…
VibhuJawa Mar 13, 2026
7c49a4e
Merge upstream main into feat/interleaved-io-readers-writers
VibhuJawa Mar 13, 2026
3bdeea1
Fix on_materialize_error policy to apply to upstream stage errors
VibhuJawa Mar 14, 2026
7a1830a
Fix interleaved IO review issues: remove lance writer, fix bugs, impr…
VibhuJawa Mar 21, 2026
bfcdd8f
Remove PLR0913 from global per-file-ignores (was causing RUF100 in pr…
VibhuJawa Mar 21, 2026
03b1af0
Update secrets baseline to mark PIL image thumbnails in notebook as f…
VibhuJawa Mar 21, 2026
473d7f9
Merge branch 'main' into feat/interleaved-io-readers-writers
VibhuJawa Mar 21, 2026
0a96654
Add tests: JPEG/PNG bytes preserved verbatim through materialize_on_r…
VibhuJawa Mar 21, 2026
f13be75
Revert unrelated RAY_ADDRESS pinning change from xenna executor
VibhuJawa Mar 21, 2026
9ab80fd
Revert out-of-scope conftest.py change
VibhuJawa Mar 21, 2026
eabc298
Revert write_benchmark_results to merge-on-exists behavior
VibhuJawa Mar 21, 2026
d6b0149
Fix Copilot review issues: _write_dataframe raises, schema_overrides …
VibhuJawa Mar 21, 2026
0b99dc3
Remove lance remnants from benchmarking utils and base writer docstring
VibhuJawa Mar 22, 2026
a725f8b
Rename WebdatasetReaderStage -> InterleavedWebdatasetReaderStage
VibhuJawa Mar 22, 2026
cdc8f66
tests(interleaved): add coverage for schema utils, base writer, and w…
VibhuJawa Mar 23, 2026
74e017e
tests(interleaved): add wds.WebDataset compatibility test with 4 samp…
VibhuJawa Mar 23, 2026
ec50284
Remove CLAUDE.md from repository (project-local file, not for upstream)
VibhuJawa Mar 23, 2026
177139f
Merge branch 'main' into feat/interleaved-io-readers-writers
VibhuJawa Mar 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 10 additions & 14 deletions .github/workflows/config/.secrets.baseline
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,3 +157,4 @@ data/
.DS_Store
AGENTS.md
alm_output/
benchmark_results/
Loading
Loading