diff --git a/.claude/skills/add-benchmark/SKILL.md b/.claude/skills/add-benchmark/SKILL.md index 385666e384..d2b0cb65e4 100644 --- a/.claude/skills/add-benchmark/SKILL.md +++ b/.claude/skills/add-benchmark/SKILL.md @@ -72,7 +72,7 @@ Convert your source dataset to Gym JSONL format. Each line must have `responses_ **Data conversion**: Write conversion scripts in the **source repo** (e.g. your dataset repository), not in NeMo-Gym. Prompt files also belong in the source repo. Exception: when there is no external source repo. See `references/patterns.md` ยง "Data Conversion Script Pattern". -**`example.jsonl`**: Generate 5 entries for smoke testing. This file is committed directly to git in `data/example.jsonl`. +**`example.jsonl`**: Generate 5 entries for smoke testing, committed directly to git in `data/example.jsonl`. Each row must be in materialized Responses format (contain `responses_create_params`) โ€” example datasets are not templated via `prompt_config`, so raw rows won't validate. **`train`/`validation` datasets**: Upload to the GitLab dataset registry โ€” these must NOT be committed to git. diff --git a/.claude/skills/add-benchmark/references/patterns.md b/.claude/skills/add-benchmark/references/patterns.md index 68d3668b9d..5a56bf8156 100644 --- a/.claude/skills/add-benchmark/references/patterns.md +++ b/.claude/skills/add-benchmark/references/patterns.md @@ -247,8 +247,7 @@ my_benchmark_eval_agent: ### Key rules - The `verified: false` flag is auto-added by pre-commit hook. Set to `true` after baselining. -- `license` is required for `train` and `validation` datasets. -- Valid license values: `Apache 2.0`, `MIT`, `CC-BY-4.0`, etc. +- `license` is required for `train` and `validation` datasets. It must be one of the values accepted by the dataset config schema (validated enum in `config_types.py`). - `domain` should be one of: `coding`, `math`, `other`, or check `config_types.py` for current enum. - Dataset `type` must be one of: `train`, `validation`, `example`. - `gitlab_identifier` is required for `train`/`validation` datasets. `jsonl_fpath` is the local download path. Both fields coexist. diff --git a/fern/versions/latest/pages/data/prepare-validate.mdx b/fern/versions/latest/pages/data/prepare-validate.mdx index 3c024e6e95..528f1b4299 100644 --- a/fern/versions/latest/pages/data/prepare-validate.mdx +++ b/fern/versions/latest/pages/data/prepare-validate.mdx @@ -239,6 +239,14 @@ ng_prepare_data "+config_paths=[resources_servers/example_multi_step/configs/exa +mode=example_validation ``` + +**Resources server example-data contract (PR submission).** A `resources_servers//` passes data validation when its `data/` directory contains `example.jsonl`, the `example_metrics.json` produced by `example_validation`, and `example_rollouts.jsonl`. Commit all three. + + + +`example.jsonl` must already be in materialized Responses format (each row containing `responses_create_params`). Example-type datasets are not templated via `prompt_config`, so raw rows won't validate. + + ### Training Preparation ```bash @@ -378,6 +386,10 @@ datasets: | `train` | Training data | RL training | | `validation` | Evaluation during training | RL training | +### License values + +`license` is required for `train` and `validation` datasets. It is validated against an allowed set defined in code โ€” see the `license` enum in [`nemo_gym/config_types.py`](https://github.com/NVIDIA-NeMo/Gym/blob/main/nemo_gym/config_types.py). + --- ## Next Steps