fix(cli): fail fast when gym eval run --split has no matching dataset

[adil-a]

What

Fixes #2685 ([VDR][v0.6.0] ERR-91baf7d3).

gym eval run --split train against a config with no train-typed dataset (e.g. example_single_tool_call, which ships only an example dataset) walked the entire data-preparation banner sequence, printed "View your final data!" and "# Finished!", and only then exited 1 on a bare assert input_jsonl_fpath.exists() whose message was an internal path (and which disappears under python -O).

Changes

• Fail fast before preparation: validate that some loaded config declares a dataset of the requested split's type. The error lists the declared datasets and, when example datasets exist, prints the copy-pasteable documented recipe (gym eval run --no-serve --input <server>/data/example.jsonl).
• Replace the bare assert with an explicit ConfigError backstop naming the split and the split files preparation actually produced (covers other missing-file causes: failed downloads, stale reuse_existing_data_preparation).
• Actionable --split example rejection: caught before the pydantic Literal check with the documented gym env start + --no-serve --input flow, instead of a bare "Input should be 'train'". Per the discussion on [VDR][v0.6.0] ERR-91baf7d3 · gym eval run --split with no matching dataset prints "Finished!" then dies on a bare assert #2685, example datasets stay non-runnable via --split (they are the PR data gate's committed smoke-test samples); the suggested "let --input satisfy the serving path" change is deliberately not included — it reverses an intentional guard and deserves its own issue.
• Latent bug fix in that guard: the CLI passes an OmegaConf DictConfig (a Mapping, not a dict), so the existing isinstance(data, dict) check in _reject_input_jsonl_fpath never fired on the real path and -i/--input was silently dropped when serving — exactly the failure it was written to prevent. Now matches on Mapping, with a DictConfig regression test.

New behavior

$ gym eval run --resources-server example_single_tool_call --model-type openai_model --split train -o out.jsonl
Error: No dataset of type `train` is declared in this config, so `--split train` has nothing to run.
Declared datasets:
- example_single_tool_call_simple_agent: example (type: example)
Example datasets are committed smoke-test samples and are not runnable via --split. To run one, start the servers (gym env start ...) and collect against the file directly:
  gym eval run --no-serve --input resources_servers/example_single_tool_call/data/example.jsonl --output <out>.jsonl

Testing

• pytest tests/unit_tests/: 1264 passed (new: test_cli_eval.py, extended TestE2EInputJsonlFpathRejected, new TestE2EExampleSplitRejected).
• Real CLI smoke test of both error paths against example_single_tool_call (output above; exit 1, no banners, no traceback).
• pre-commit run on changed files: clean.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[adil-a]

/ok to test 8f7f1d9

[github-actions]

🌿 Preview your docs: https://nvidia-preview-fix-2685-eval-split-early-exit.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

• nemo/gym/reference/cli-commands

[adil-a]

/ok to test 0ebf761

[ffrujeri]

Thanks for improving this failure mode. Failing before data preparation and replacing the user-facing assert with ConfigError are clear improvements.

I think the reported behavior also exposes a deeper semantic mismatch that is worth acknowledging, even if it is outside the scope of this fix. A command such as:

gym eval run --resources-server example_single_tool_call --split train

reads as “run the train split of this benchmark,” but a resources server is not itself a benchmark. It provides environment-side behavior such as tools, state, and verification, and it does not inherently need to own a train, validation, or benchmark dataset. In practice, the CLI composes several loaded configs and treats --split as a global request, with datasets potentially declared on an agent or resources server.

The validator in this PR correctly checks the complete loaded configuration rather than requiring the selected resources server itself to have the split. However, that also highlights that the CLI currently blurs three concepts:

• the resources server (tools/state/verifier),
• the dataset or benchmark tasks,
• the complete environment configuration that composes resources, agent, model, and datasets.

I think this PR is a worthwhile backward-compatible guard for the current interface, but we should avoid wording that implies every resources server is a benchmark or must provide the requested split. The contract is instead: some dataset in the complete loaded configuration must match --split.

As a follow-up, it may be worth tracking a clearer environment/dataset-oriented CLI interface, where users select an environment/config and a named dataset directly rather than inferring dataset ownership through --resources-server plus --split.

[ffrujeri]

Could we run _validate_split_datasets_declared() before the should_skip_data_processing branch? As written, when reuse_existing_data_preparation=true and the expected split file already exists, the declaration check is skipped. That appears to allow a stale prepared file to run even when the current loaded configuration no longer declares that split. Is that intentional? If reuse is expected to bypass this validation, should the new documentation qualify its statement that a matching dataset “must be declared”?

[adil-a]

Thanks! I'll fix that

[ffrujeri]

A few comments worth taking a look. but otherwise I think we can proceed

[adil-a]

/ok to test a5309d7

[copy-pr-bot]

/ok to test a5309d7

@adil-a, there was an error processing your request: E2

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/2/

[adil-a]

/ok to test ee4fd8f