cp: fix(cli): fail fast when gym eval run --split has no matching dataset (2911) into r0.6.0 - #2953
Merged
Merged
Conversation
…et (#2911) ## What Fixes #2685 (`[VDR][v0.6.0] ERR-91baf7d3`). `gym eval run --split train` against a config with no `train`-typed dataset (e.g. `example_single_tool_call`, which ships only an `example` dataset) walked the entire data-preparation banner sequence, printed "View your final data!" and "# Finished!", and only then exited 1 on a bare `assert input_jsonl_fpath.exists()` whose message was an internal path (and which disappears under `python -O`). ## Changes - **Fail fast before preparation**: validate that some loaded config declares a dataset of the requested split's type. The error lists the declared datasets and, when example datasets exist, prints the copy-pasteable documented recipe (`gym eval run --no-serve --input <server>/data/example.jsonl`). - **Replace the bare assert** with an explicit `ConfigError` backstop naming the split and the split files preparation actually produced (covers other missing-file causes: failed downloads, stale `reuse_existing_data_preparation`). - **Actionable `--split example` rejection**: caught before the pydantic `Literal` check with the documented `gym env start` + `--no-serve --input` flow, instead of a bare "Input should be 'train'". Per the discussion on #2685, example datasets stay non-runnable via `--split` (they are the PR data gate's committed smoke-test samples); the suggested "let `--input` satisfy the serving path" change is deliberately not included — it reverses an intentional guard and deserves its own issue. - **Latent bug fix in that guard**: the CLI passes an OmegaConf `DictConfig` (a `Mapping`, not a `dict`), so the existing `isinstance(data, dict)` check in `_reject_input_jsonl_fpath` never fired on the real path and `-i/--input` was silently dropped when serving — exactly the failure it was written to prevent. Now matches on `Mapping`, with a `DictConfig` regression test. ## New behavior ``` $ gym eval run --resources-server example_single_tool_call --model-type openai_model --split train -o out.jsonl Error: No dataset of type `train` is declared in this config, so `--split train` has nothing to run. Declared datasets: - example_single_tool_call_simple_agent: example (type: example) Example datasets are committed smoke-test samples and are not runnable via --split. To run one, start the servers (gym env start ...) and collect against the file directly: gym eval run --no-serve --input resources_servers/example_single_tool_call/data/example.jsonl --output <out>.jsonl ``` ## Testing - `pytest tests/unit_tests/`: 1264 passed (new: `test_cli_eval.py`, extended `TestE2EInputJsonlFpathRejected`, new `TestE2EExampleSplitRejected`). - Real CLI smoke test of both error paths against `example_single_tool_call` (output above; exit 1, no banners, no traceback). - `pre-commit run` on changed files: clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Adil Asif <adasif@nvidia.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Author
|
/ok to test b9ff40b |
Contributor
|
🌿 Preview your docs: https://nvidia-preview-cherry-pick-2911-r0-6-0.docs.buildwithfern.com/nemo/gym Here are the markdown pages you've updated: |
ananthsub
approved these changes
Sep 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
beep boop [🤖]: Hi @adil-a 👋,