Skip to content

Simplify v1 CLI plumbing - #2427

Merged
mikasenghaas merged 3 commits into
mainfrom
v1-cli-cleanup
Aug 23, 2026
Merged

mikasenghaas merged 3 commits into
mainfrom
v1-cli-cleanup

Conversation

@xeophon

@xeophon xeophon commented Aug 23, 2026

Copy link
Copy Markdown
Member

Overview

This makes the v1 command-line tools smaller and more consistent. It removes code that was doing the same work twice, uses information that is already saved with each run, and fixes debug runs being recorded as failures when their command succeeded.

What changed

Evaluation and resume

  • Resume now uses the task hash already stored in each saved episode instead of recalculating it every time.
  • Runs created before task hashes were stored still resume correctly by calculating the hash from their saved task data.
  • Missing hashes use that fallback, while malformed list or object hashes are treated as invalid rows and rerun safely.
  • Saved and current configurations are compared directly instead of hashing both configurations first.
  • Saved JSON configurations are read through Pydantic directly, and the already calculated run path is reused throughout command startup.
  • Removed an unused evaluation environment-id shortcut and a one-use resume-message helper.

Debug and validation

  • A successful debug command now produces a successful trace instead of being saved as failed.
  • Debug runs save their resolved configuration as configs/debug.json, matching the other commands.
  • Gold-answer and setup-only validation now share one execution path instead of maintaining two nearly identical implementations.
  • Resume loaders use one JSON parser and safely skip malformed result rows.

Environment scaffolding and command configuration

  • The environment name is now required by the configuration schema instead of being checked manually later.
  • Removed unreachable per-file skip logic, ignored return values, and temporary aliases from environment scaffolding. The existing top-level overwrite protection and --force behavior remain unchanged.
  • Removed unused eager command-configuration exports and an unused default argument from command id resolution.

Dashboards and saved output

  • The evaluation dashboard builds its static header and runtime description once instead of rebuilding them four times per second.
  • Progress display work is reused within each refresh, and terminal arrow-key handling is disabled when the dashboard is showing logs because paging is inactive there.
  • Dashboard sizing now uses the same Rich console as the live display.
  • Replay dashboard state is typed directly as replay progress, removing a fallback lookup that could never be needed.
  • Saved-output descriptions now consistently refer to configs/<command>.json, traces.jsonl, and run.dir rather than the old TOML paths or display name.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9b15b4ff21

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread verifiers/v1/cli/eval/resume.py Outdated
@macroscopeapp

macroscopeapp Bot commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This PR combines a mostly mechanical CLI refactor with changes to persisted debug status, resume rollout selection, and the exported configuration API. Those effects cross durable-run and compatibility boundaries, so the changes merit human review.

You can add or adjust custom eligibility rules. Learn more.

Comment thread verifiers/v1/cli/eval/resume.py Outdated
@mikasenghaas
mikasenghaas merged commit 807aaf3 into main Aug 23, 2026
13 checks passed
@mikasenghaas
mikasenghaas deleted the v1-cli-cleanup branch August 23, 2026 17:03
mikasenghaas added a commit to PrimeIntellect-ai/prime-rl that referenced this pull request Aug 24, 2026
## Summary
- Bump the `deps/verifiers` submodule from `d4f3d5a8c` to `807aaf389`
(latest `main`).
  - Pulls in:
- [vf#2425](PrimeIntellect-ai/verifiers#2425):
forward sampling parameters across dialects.
- [vf#2428](PrimeIntellect-ai/verifiers#2428):
centralize v1 egress policy decisions.
- [vf#2427](PrimeIntellect-ai/verifiers#2427):
simplify v1 CLI plumbing.
- Bump the `deps/prime-envs` submodule from `b30aad36` to `26dafdc9`
(latest `main`).
  - Pulls in:
-
[prime-envs#761](PrimeIntellect-ai/prime-envs#761):
environment-design rules in AGENTS.md.
-
[prime-envs#762](PrimeIntellect-ai/prime-envs#762):
CI fix for swebench 5 + prime-sandboxes 0.2.37.
-
[prime-envs#684](PrimeIntellect-ai/prime-envs#684):
deshuffle-papers environment.
-
[prime-envs#675](PrimeIntellect-ai/prime-envs#675):
DeepSWE environment.
- Relock `uv.lock`: new workspace members `deep-swe` and
`deshuffle-papers`; `mcp-atlas` 0.3.0 → 0.3.1, `swesmith-env` 0.1.2 →
0.1.3.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Dependency/submodule bump that pulls new training environments and
tighter swebench/prime-sandboxes pins; not auth-critical, but can change
eval/training behavior.
> 
> **Overview**
> Bumps `deps/verifiers` and `deps/prime-envs` to latest `main` and
relocks `uv.lock`.
> 
> **verifiers** picks up sampling-parameter forwarding across dialects,
centralized v1 egress policy, and simpler v1 CLI plumbing.
> 
> **prime-envs** adds the `deep-swe` and `deshuffle-papers` workspace
environments, plus env-design AGENTS.md rules. `mcp-atlas` goes 0.3.0 →
0.3.1 (`prime-sandboxes>=0.2.37`); `swesmith-env` goes 0.1.2 → 0.1.3
(`swebench>=4.1.0,<5`).
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
b5b07f4. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants