feat: split run configs into launch TOML and resolved dir - #2429
Merged
Conversation
A run dir's configs/ now holds the launch TOML copied verbatim (configs/eval.toml) next to configs/resolved/eval.json, the resolved dump - the same layout prime-rl's launchers write. saved_config_path still finds legacy configs/<cli>.json runs. Also drop the startup log of the full eval config: it is persisted to disk now, so the log line was redundant. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The argv scan captured nested `--flag @ file` / `--flag @file` references and concatenated them into configs/eval.toml as if they were root launch files - re-running that copy would apply nested keys at root. Only space-form root `@ file` references are copied now, and only .toml sources (a resume from the resolved JSON writes no launch copy). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mikasenghaas
marked this pull request as ready for review
August 24, 2026 01:18
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 8307666. Configure here.
Contributor
ApprovabilityVerdict: Not approved Macroscope's review found this PR not approvable — This PR changes the run-output contract by adding launch TOML artifacts, moving resolved configs, and updating resume/replay behavior across multiple CLI workflows. The new parser-dependent persistence behavior, including an unresolved Medium-severity omission case, warrants human review. You can add or adjust custom eligibility rules. Learn more. |
The multi-file header zip paired the unfiltered path list against the toml-filtered texts, so a non-toml root reference shifted every header onto the wrong file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
samsja
approved these changes
Aug 24, 2026
hallerite
added a commit
that referenced
this pull request
Sep 2, 2026
Pure deletions; no behaviour change. - configs/eval, configs/gepa, configs/rl, configs/zero3.yaml, configs/endpoints.toml: v0 `vf-eval` / training formats, unreferenced since the v0 stack was removed; drop the test filter that skipped them - pyproject: inert nemo-gym dependency-metadata (not a dependency since #2476), pytest markers no test applies, cellpylib warning filter - Harness.run: superseded by HarnessSession.turn, zero callers - Agents.__iter__/__len__: no reader in verifiers, prime-rl, or the env repos - utils.git.snapshot_untracked: never called - cli.output.saved_config_path: drop the pre-#2429 configs/<cli>.json fallback; eval/gepa usage gate: drop the --taskset./--harness. prefixes of flags removed in #2157/#2237 - mcp: ServerBase.EXTRAS and RUNTIME_PYTHON hooks that no server declares - prime_agent: SUPPORTS_RESUME is never read on an ACP harness Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
xeophon
pushed a commit
that referenced
this pull request
Sep 2, 2026
Pure deletions. Every item has zero references in verifiers, prime-rl, prime-envs and community-environments (grep + GitHub code search); nothing here changes behaviour. **v0 config trees (−820 lines).** `configs/eval/`, `configs/gepa/`, `configs/rl/`, `configs/zero3.yaml`, `configs/endpoints.toml` are the `vf-eval` / DeepSpeed formats of the removed v0 stack. `tests/v1/test_configs.py` already excluded them; the filter goes with them. **pyproject.** The `[[tool.uv.dependency-metadata]] nemo-gym` block is inert (nemo-gym stopped being a dependency in #2476; the PEP 723 header in `tasksets/nemo_gym/server.py` is the one that is used), so `uv.lock` loses its manifest entry. Markers `integration`, `slow`, `unit`, `parsers`, `rubrics`, `environments` are applied by no test; the `cellpylib` warning filter targets a package that is not in the lock. **Dead code.** - `Harness.run()`: the pre-session launch/resume dispatch, superseded by `HarnessSession.turn`. - `Agents.__iter__`/`__len__`: no reader anywhere; every env addresses agents by attribute. - `saved_config_path`: the fallback to the pre-#2429 `configs/<cli>.json` location. Only `configs/resolved/` is written. - `eval`/`gepa` usage gate: the `--taskset.`/`--harness.` prefixes kept alive for flags removed in #2157/#2237. - `ServerBase.EXTRAS` and the `RUNTIME_PYTHON` hook in `mcp/launch.py`: declared by no server anywhere. - `PrimeAgentHarness.SUPPORTS_RESUME`: `ACPHarness` overrides `session()`, so both sites that read the flag short-circuit before reaching it. `snapshot_untracked` was removed in the first revision and restored in the second: prime-envs #792 calls it from four SWE tasksets (`capture_patch(ignore=...)`). Open PRs in the env repos are now part of the check. Deliberately not in this PR: anything on the `Trace`/`Episode` record model (`Episode` aggregates, `PolicySpan.drift`, `Trace.last_message`) — those have downstream consumers outside this repo; `agent_config_fields` (its `_declared_agent_configs` twin checks field defaults, not values, so swapping is not a pure deletion), `WireTrace`, `InterceptionError`, the `Branch` trainer-side properties, and every docs/docstring wording fix. Those are separate PRs. Checks run on the branch: `ruff check`, `ruff format --check`, `ty check verifiers`, and `pytest tests/v1 -m "not e2e"` (82 passed). 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Low Risk** > Mostly deletions of unreferenced configs and code; the only behavioral tweaks are stricter CLI axis detection, resolved-config-only replay paths, and always running full MCP sandbox installs. > > **Overview** > Removes **v0-era configuration** that is no longer part of the v1 stack: the entire `configs/eval/`, `configs/gepa/`, `configs/rl/` trees, `configs/endpoints.toml`, and `configs/zero3.yaml`. Config validation in `tests/v1/test_configs.py` no longer special-cases `endpoints.toml`. > > **Tooling cleanup** drops the unused `nemo-gym` `[[tool.uv.dependency-metadata]]` block from `pyproject.toml` (and the matching `uv.lock` manifest), unused pytest markers, and a `cellpylib` warning filter for a package not in the lock. > > **Dead v1 code paths** are deleted without replacing callers: `Harness.run()` (segment dispatch now lives on `HarnessSession.turn`), `Agents.__iter__`/`__len__`, legacy resolved-config lookup in `saved_config_path`, narrowed `eval`/`gepa` CLI usage gates (no `--taskset.` / `--harness.` shortcuts), MCP sandbox install no longer honors `ServerBase.EXTRAS` or `RUNTIME_PYTHON`, and `PrimeAgentHarness.SUPPORTS_RESUME` (unused because ACP harnesses use `session()`). > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 8389294. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> <!-- Macroscope's pull request summary starts here --> <!-- Macroscope will only edit the content between these invisible markers, and the markers themselves will not be visible in the GitHub rendered markdown. --> <!-- If you delete either of the start / end markers from your PR's description, Macroscope will append its summary at the bottom of the description. --> > [!NOTE] > ### Remove v0 config files, dead code, and fix CLI usage-gate bypass in v1 > - Deletes leftover v0 configs under `configs/` (endpoint registry, eval/gepa/rl TOMLs, `zero3.yaml`) and removes unused pytest markers, dependency metadata, and dead class members (`Agents.__iter__`, `Agents.__len__`, `ServerBase.EXTRAS`, `PrimeAgentHarness.SUPPORTS_RESUME`). > - Restricts the typed-axis usage-gate bypass in [main.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2496/files#diff-1335625d1379d7f1a4bf6a2f961d4b5fde5f3a9bfb8eb4cc736859f08698a8c8) and [gepa.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2496/files#diff-dc87040f272d5f7871025c1cd9da91e56670d1cf1deff808037cb5bc356376bc) to `--env.`/`--serve.` prefixes only; `--taskset.` and `--harness.` arguments no longer suppress the usage message. > - Removes the `configs/` fallback in [output.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2496/files#diff-07fce45dd0fa633f34d25e19962ab801b705a509774833de0d81bfe872787e8b): `saved_config_path` now searches only `configs/resolved/`. > - Changes non-subprocess MCP server launch in [launch.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2496/files#diff-989d87123d40d48511f9e1df74594d2e30776e9a99c11eca49d9cc20bf9e2230) to always install a sandbox venv and use its Python, dropping the `RUNTIME_PYTHON` override and server-class extras suffix. > - Risk: non-subprocess MCP servers no longer honor a prebuilt `RUNTIME_PYTHON` executable; `saved_config_path` returns `None` for runs with JSON only under `configs/` rather than `configs/resolved/`. > > <!-- Macroscope's review summary starts here --> > > <sup><a href="https://app.macroscope.com">Macroscope</a> summarized 8389294.</sup> > <!-- Macroscope's review summary ends here --> > <!-- Macroscope's pull request summary ends here --> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
@TOML file verbatim toconfigs/eval.toml, next to the resolved dump, which moves toconfigs/resolved/eval.json.@ filereferences are copied (nested--flag @ file/--flag @fileconfigs belong under their flag and are excluded), and only.tomlsources — a resume from the resolved JSON writes no launch copy.saved_config_path(resume/replay) checksconfigs/resolved/first and falls back to the legacyconfigs/<cli>.jsonlocation.Companion to the prime-rl run-dashboard PR (PrimeIntellect-ai/prime-rl#3351), which reads both layouts: the launch TOML as the default config view and the resolved JSONs concatenated into one searchable document.
Breaking
configs/eval.jsontoconfigs/resolved/eval.json. Reading (resume, replay) handles both layouts; anything that writes or globs the old path directly must switch toconfigs/resolved/.🤖 Generated with Claude Code
Note
Medium Risk
Moves the on-disk resolved config path used for resume/replay. Reading falls back to the legacy location, but anything that writes or hardcodes configs/*.json will break.
Overview
Runs now keep two config artifacts: a verbatim copy of root
@launch TOML atconfigs/<cli>.toml, and the resolved dump atconfigs/resolved/<cli>.jsoninstead ofconfigs/<cli>.json. Nested--flag @ filerefs and JSON resumes are not copied.saved_config_pathlooks inconfigs/resolved/first, then the oldconfigs/*.jsonlayout, so resume/replay still work on existing runs. Help text points at the new path. The full eval-config INFO dump at startup is removed because the file is already on disk.Reviewed by Cursor Bugbot for commit f7c83ad. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Split run configs into launch TOML at
configs/<cli>.tomland resolved JSON atconfigs/resolved/<cli>.jsonsave_confignow writes the resolved config JSON toconfigs/resolved/<cli>.jsonand copies the launch TOML (from@-referenced files insys.argv) toconfigs/<cli>.toml, creating nested directories as needed.saved_config_pathlooks underconfigs/resolved/first and falls back to the legacyconfigs/path so older runs still resume.write_launch_tomlconcatenates multiple TOML inputs with a header comment naming each source path.configs/resolved/eval.json, and the startup config JSON log line inrun_evalis removed.write_configin output.py writes to a newconfigs/resolved/subdirectory; any code or scripts expecting<run-dir>/configs/<cli>.jsonwill need to look underconfigs/resolved/instead.Macroscope summarized f7c83ad.