feat: resume interrupted dataset generation runs (sync + async engine) - #526

Merged

nabinchha merged 34 commits into

NVIDIA-NeMo:mainfrom

przemekboruta:main

May 8, 2026

Commits on May 4, 2026

docs: add implementation plan for resume mechanism
przemekboruta
committed
5110d63
feat(storage): add resume flag and clear_partial_results()
przemekboruta
committed
6afb638
feat(batch-manager): add start_batch param to start()
przemekboruta
committed
699f510
feat(builder): implement resume logic in DatasetBuilder
przemekboruta
committed
866df6b
feat(interface): expose resume on DataDesigner.create()
przemekboruta
committed
e0b22d5
test: add tests for resume mechanism
przemekboruta
committed
4b514b2
feat(builder): extend resume to async engine (DATA_DESIGNER_ASYNC_ENGINE=1)
przemekboruta
committed
1db497a
fix(builder): skip after-generation processors when resume finds dataset already complete
przemekboruta
committed
812d7df
fix(builder): use filesystem count for initial_total_num_batches on async resume
przemekboruta
committed
4054610
feat(results): add export() method and --output-format CLI flag
przemekboruta
committed
0bdf24a
fix(builder): handle resume when metadata.json missing (interrupted before first batch)
przemekboruta
committed
4401e4b
docs(interface): fix resume docstring — async engine is supported
przemekboruta
committed
c2d0a77
fix(builder): derive initial_actual_num_records from filesystem in async resume
przemekboruta
committed
4ffd7f3
feat(resume): replace resume: bool with ResumeMode enum (NEVER/ALWAYS/IF_POSSIBLE)
przemekboruta
committed
b89f1a1
fix(resume): invalidate resolved_dataset_name cache when IF_POSSIBLE downgrades to NEVER
przemekboruta
committed
5a99f59
fix(builder): move partial-completion warning before return in _build_async
przemekboruta
committed
f69b3e7
fix(builder): IF_POSSIBLE now starts fresh when no dataset directory exists
przemekboruta
committed
4daf48b
fix(builder): use original target_num_records in async resume record count
przemekboruta
committed
69c3e55
fix(builder): IF_POSSIBLE starts fresh on empty dataset directory
przemekboruta
committed
487def2
fix(builder): ALWAYS raises DatasetGenerationError on config fingerprint mismatch
przemekboruta
committed
a08d9cc

Commits on May 5, 2026

Merge branch 'main' into main
nabinchha
authored
75fda14

Commits on May 6, 2026

Commits on May 7, 2026

Commits on May 8, 2026