perf(raw): StreamingRawDataset cloud download throughput - #863
Merged
Conversation
Dedicated LoopRunner/uvloop, look-ahead prefetch, size-gated hedging, atomic cache, fork-safe clients, and opt-in ranged GETs lift ImageNet-val raw throughput to ~7350 samples/s (w=24, prefetch=16); document knobs and sweep results. Co-authored-by: Cursor <cursoragent@cursor.com>
for more information, see https://pre-commit.ci
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #863 +/- ##
====================================
Coverage 81% 81%
====================================
Files 56 56
Lines 8374 9303 +929
====================================
+ Hits 6795 7566 +771
- Misses 1579 1737 +158 🚀 New features to boost your workflow:
|
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
for more information, see https://pre-commit.ci
Break compound exit-status asserts and use contextlib.suppress for expected OSError/BrokenBarrierError paths so pre-commit.ci passes. Co-authored-by: Cursor <cursoragent@cursor.com>
Publish A/B numbers for StreamingRawDataset (stock main vs LoopRunner/prefetch) in README, skill docs, and benchmark JSON for PR #863. Co-authored-by: Cursor <cursoragent@cursor.com>
for more information, see https://pre-commit.ci
Default hedge_delay=0 and skip hedge/timeout wrappers when both are off so prefetch=0 no longer pays for disabled safety features. Move optional uvloop to extras, lengthen the A/B harness warm/timed windows, and replace the short-window w=24 artifact (10556 / conflicting 5361 vs 7350) with a long-window remeasure (~6814 before, ~6635/6756 after). Co-authored-by: Cursor <cursoragent@cursor.com>
for more information, see https://pre-commit.ci
Capture practical chunk_bytes guidance and ordered-source shuffle caveats from expert discussion so README and agent skills stay aligned. Co-authored-by: Cursor <cursoragent@cursor.com>
for more information, see https://pre-commit.ci
Add agent-facing FUSE warning, shuffle-before-optimize tip, and order-of-magnitude ImageNet ballpark to litdata skills and the README FAQ. Co-authored-by: Cursor <cursoragent@cursor.com>
for more information, see https://pre-commit.ci
…atrix Enable look-ahead by default (~2× typical batch) and replace the w=24-only snippet with a full workers×prefetch Before vs After table from the long-window harness. Co-authored-by: Cursor <cursoragent@cursor.com>
for more information, see https://pre-commit.ci
Replace try/except/pass with contextlib.suppress for SIM105/S110. Clarify that max_prefetch is per DataLoader worker and aggregate look-ahead scales with num_workers × max_prefetch. Co-authored-by: Cursor <cursoragent@cursor.com>
Cap per-worker look-ahead to a ~64-item aggregate so default max_prefetch=16 stays strong at low workers without overscheduling at w≥8; document after-p0 vs main at high workers and reframe PR value around correctness. Co-authored-by: Cursor <cursoragent@cursor.com>
for more information, see https://pre-commit.ci
Per-item wait_for under default download_timeout=120 was the w=24 regression; hang protection now wraps the batch gather once so the fast path coexists with defaults. Index cache writes use tmp+replace. Co-authored-by: Cursor <cursoragent@cursor.com>
for more information, see https://pre-commit.ci
Cancelling _resolve_index wrappers alone left poisoned prefetch downloads in _inflight, so every retry paid the full budget. Also catch both TimeoutError aliases, avoid rewriting item timeouts when budget is None, and floor the batch budget by aggregate transfer time. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Lead README/PR presentation with regime summary and post-fix w=24 noise band; stop overwriting confirm/decisive/highw result JSON in place. Restore the 6697 confirm run under a SHA/ts-suffixed artifact. Co-authored-by: Cursor <cursoragent@cursor.com>
for more information, see https://pre-commit.ci
Co-authored-by: Cursor <cursoragent@cursor.com>
Keep lru/ranged/worker-sweep artifacts append-only so re-runs cannot overwrite prior dig results in place. Co-authored-by: Cursor <cursoragent@cursor.com>
for more information, see https://pre-commit.ci
Clamp per-worker download permits from a size-aware aggregate budget so high num_workers cannot open N×64 in-flight GETs. Fix A/B timing to max(batches, seconds), add repeats/interleave with median+spread, and document deferred adaptive stages. Co-authored-by: Cursor <cursoragent@cursor.com>
for more information, see https://pre-commit.ci
Reword "statics" to avoid codespell false positive, use a ternary for median, and noqa the trusted local subprocess interleave invocation. Co-authored-by: Cursor <cursoragent@cursor.com>
Budget is max(bandwidth, latency) clamped to [32, 512] so ImageNet-sized objects keep healthy mid-w aggregate. None defaults to adaptive; explicit int is exact permits. Cache permits per pid; document honest Stage 1 premise. Co-authored-by: Cursor <cursoragent@cursor.com>
Allow before trees that already have max_prefetch/LoopRunner as a fixed-64 baseline, pair same-(w,prefetch) in merge, and add --after-prefetch. Co-authored-by: Cursor <cursoragent@cursor.com>
High-throughput cells can exhaust ImageNet val (~782 batches) before the 30s floor; restart the iterator so max(batches, seconds) can finish. Co-authored-by: Cursor <cursoragent@cursor.com>
A/B showed w=8 p0 −23% under always-on clamp while w=16/24 gained +40–55%. Below the gate keep historical 64 permits; high-w keep the size-aware split. Track append-only A/B resume helper. Co-authored-by: Cursor <cursoragent@cursor.com>
Treat w8 −23% as likely A/A noise after budget widening. Latency model only for sub-MiB medians; single-process adaptive capped at 128. Record before_sha/after_sha from each PYTHONPATH tree; warn once on index PutObject. Co-authored-by: Cursor <cursoragent@cursor.com>
Provenance-verified before_sha=52dba61 after_sha=ba9da13; before ~3.8k confirms robustness framing, not wrong-tree. Co-authored-by: Cursor <cursoragent@cursor.com>
urlparse('C:\\...') yields scheme='c', which was rejected as an unsupported
cloud scheme and broke StreamingRawDataset on Windows CI.
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Sync the litdata skill with StreamingRawDataset adaptive concurrency, batch-level timeouts, fork/atomic invariants, and bench trust hierarchy so agents use and extend the raw path correctly. Co-authored-by: Cursor <cursoragent@cursor.com>
for more information, see https://pre-commit.ci
…esolution Exhaustive agent references for optimize/map I/O (FsProvider worker pools vs Downloader ABC, FUSE→cloud resolve) and multi-node Studio jobs (sharding, index merge, checkpoints, pitfalls). Co-authored-by: Cursor <cursoragent@cursor.com>
for more information, see https://pre-commit.ci
Skip broadcast_object for input/output dirs by default so multi-node ranks
keep locally resolved paths. Auto-enable when paths contain a `{%strftime}`
template so ranks still share one expanded timestamp; allow broadcast_paths=True
to force alignment.
Co-authored-by: Cursor <cursoragent@cursor.com>
for more information, see https://pre-commit.ci
Broadcast-path tests used /tmp strings that trip ruff S108; swap to non-tmp placeholders. Empty unused intersphinx_mapping so -W linkcheck no longer fails when docs.python.org resets inventory fetches. Co-authored-by: Cursor <cursoragent@cursor.com>
Align the skip with the existing reason: nested spawn workers under pytest-xdist can leave DataProcessor.join() stuck on darwin, burning CI. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR hardens
StreamingRawDatasetcloud downloads for correctness first, then improves default concurrency.LoopRunner(optional uvloop), fork/spawn-safe clients, atomic cache + index publishes, and batch-level hang protection.max_concurrent_downloads=Nonepicks a size-aware aggregate budget (bandwidth vs Little’s-law, clamped), instead of a fixed 64 permits/worker stampede.HEAD of this branch:
3b0bdcc.What's in this PR
litdata[extras]); fork/spawn-safe runtime clientstmp+os.replacedownload_timeout(default 120): oneasyncio.wait_foraround_download_batch; hang-recovery cancels poisoned_inflightso retries can proceedhedge_delay=0(default); size-gated request hedging stays off by defaultmax(≥300 batches, ≥30s), repeats/interleave, SHA/ts artifacts; harness recordsbefore_sha/after_shafrom each PYTHONPATH treemax_concurrent_downloads=None):max(bandwidth, Little’s-law)then clamp[32, 512], split across workers (floor 8)int→ exact permit count (no silent clamp)range_parallel_threshold=0)Benchmarks
Setup: ImageNet val raw (50k JPEGs) via S3 · batch_size=64 ·
spawn·persistent_workers=True· 48 vCPU · Python 3.12.11 ·range_parallel_threshold=0,hedge_delay=0.Protocol: warm
max(1, workers×prefetch_factor), then time until both ≥300 batches and ≥30s. Prefer medians + spreads.Confirm cell (authoritative for Stage 1 @ w=24)
Provenance-verified interleaved A/B, w=24 p=0, n=3.
52dba61(post batch-timeout, pre Stage 1; fixedmax_concurrent_downloads=64)ba9da13(Stage 1 adaptive)Δ median ≈ +59%. Verdict (a): before ≈3.7k matches the Stage 1 grid before band — robustness / session drift on fixed-64, not wrong-tree. Do not treat prior full-grid +53% as a headline without this SHA proof.
Artifacts:
benchmarks/results/raw_before_vs_after.ba9da13.1785268543.json,benchmarks/results/stage1_confirm_w24_ba9da13.log.Stage 1 full-grid medians (n=5)
Runner
46f2d45. Before = fixed 64; after = Stage 1 adaptive. Artifact:benchmarks/results/raw_before_vs_after.46f2d45.1785267013.json.Spreads are often large (tens of percent). The w=8 p=0 −23% cell looks like A/A noise (both sides ~60–64 permits) and was not used to gate the clamp at w≥16. Prefer the confirm cell for Stage 1 @ w=24 framing.
Historical dig (batch-timeout story, stale)
Single-run w=24 p=0 cells that motivated the timeout fix — not current HEAD claims:
5d8cfc1in merge) —raw_before_vs_after.before.jsonb991c7d—raw_before_vs_after.jsondownload_timeout=0fast path6ab527d—raw_decisive_timeout0.6ab527d.1785252483.jsondownload_timeout=120)6ab527d—raw_confirm_batch_timeout.6ab527d.1785252695.jsonTakeaway from the dig: post-timeout, w=24 with 64 permits was back near the old main band; the “must hard-clamp because of stampede” premise was weak. Stage 1 remains valuable as a size-aware default, verified at high-w by the confirm cell.
Design notes:
benchmarks/ADAPTIVE_CONCURRENCY.md.API notes
max_concurrent_downloads=None(default after this PR)max_concurrent_downloads=<int>download_timeout=1200disables)hedge_delay=0range_parallel_threshold=0User-facing README stays high-level; deep formula lives in the design note above.
Test plan
pytest tests/raw/ -v(113 passed @ba9da13)pytest tests/streaming/test_client.py tests/streaming/test_downloader.py -v(if touched)before_sha/after_shaLITDATA_RAW_DEBUG=1) → ~3%Follow-ups
Made with Cursor