feat: vf v1 <> nano bridge - #1576
Closed
mikasenghaas wants to merge 189 commits into
Closed
Conversation
…orts First step of replacing v1 with vf-nano. Deletes verifiers/v1/ wholesale and strips its surface from verifiers/__init__.py (lazy imports, __all__, TYPE_CHECKING) and utils/env_utils.py (load_taskset/load_harness + the typed-config/component machinery). load_environment is now v0-only. Example v1 envs, v1 tests, eval.py v1 path, and docs are removed in follow-up commits. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…envs Removes the 20 v1-native example envs (tau2_bench_v1, hello_*_v1, bfcl_v3, dspy_*, openenv_*, rlm_swe_v1, sft_replay, mcp_search_env, nemo_gym_env, openai_agents_env, opencode_harbor, langchain_*, wordle_v1, nested_harness_v1) and their *_v1 siblings; removes the v1 test suite (test_v1_*, test_eval_cli, test_wordle_v1_env, test_wiki_search_v1, test_mcp_search_env); strips the v1 flag/branch from the kept v0 envs (reverse_text, alphabet_sort, math_python, wiki_search). Follow-ups: eval.py/init.py v1 paths, remaining v1 test refs, docs, State v1-contract. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Vendor vf-nano as a submodule under deps/vf-nano and extend the verifiers package __path__ so verifiers.nano imports from it; alias verifiers.v1 -> verifiers.nano 1:1 (verifiers.v1.Trace, .serve.EnvServer, .EnvConfig are the nano objects). Add a v1 extra with nano's runtime + serve deps. One verifiers package now carries both the v0 API and v1 (=nano). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…verse-text-v1) Strip the v1 taskset/harness CLI-override path from scripts/eval.py so vf-eval is v0-only; expose nano's eval as vf-eval-v1 so both run side by side. Bump deps/vf-nano to the reverse-text-v1 rename. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mikasenghaas
changed the title
Remove the v1-only machinery the deleted v1 framework grafted onto State: the _vf_state_contract contract (+ its guards in every dict method), the runtime/endpoint/tools/runtime-handle method cluster (get_model/get_client/get_endpoint_config/get_tools/add_tool/_runtime*/strip_runtime_handles), the for_task borrow/group-state params, and the module-level group-state/borrow helpers. State is now plain v0: dict semantics + _set_* + stop + timing + finalize + _legacy_for_task. Verified: State.for_task/stop/finalize and v0 env load work. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…doc) Remove the vf-init --v1/--openenv/--with-harness scaffolding (templates + flags) now that v1 is vf-nano; vf-init is v0-only. Delete the v1-specific test functions (test_imports, test_init_script, test_trajectory_processing) and the v1 harness-authoring doc. Remaining: a docs prose pass (overview/environments/evaluation/reference/training still mention the old v1 API). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
De-submodule vf-nano and vendor it 1:1 into the repo as the verifiers.v1
subpackage, then drop the legacy v1 packages it replaces.
- Copy vf-nano (latest main) in: package -> verifiers/v1/, plus examples/,
configs/, packages/{tasksets/harbor, harnesses/{default,rlm}}. Remove the
deps/vf-nano submodule and the verifiers/__init__ __path__ shim.
- verifiers.v1 is now a real subpackage (drop the verifiers/v1.py alias); the
v0 -> vf.Trace bridge lives at verifiers.v1.legacy.
- Rename nano -> v1 throughout (code, comments, configs); model names like
gpt-*-nano / Nemotron-Nano are untouched.
- Delete the old-v1 tasksets/harnesses packages and their tests + publish
workflows; rework pyproject to source/group the v1 plugins (default-installed),
drop the old extras/conflicts, and relax the plugins to >=3.10.
- Exclude vendored verifiers/v1 from verifiers' ty gate; restore textarena/nltk
in dev so the v0 textarena env type-checks.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mikasenghaas
changed the title
…gins - Scripts: the v1 CLIs are now `eval` / `serve` (was `vf-eval-v1`), matching the CLI's own usage strings and the example config headers. - Move the v1 runtime deps (loguru, tomli-w, renderers) into base `dependencies` and drop the `v1` extra, so `import verifiers.v1` always works. - Shipped plugins are vendored by default (no extras): `tasksets` bundles harbor, `harnesses` bundles default + rlm. Each plugin is a top-level package resolved by id (`import <id>`); example plugins stay standalone under examples/. - Flatten core: verifiers/v1/harnesses/base.py -> verifiers/v1/harness.py; drop the one-module harnesses/ subpackage. - Bump prime-tunnel>=0.1.8, prime-sandboxes>=0.2.27 (latest). - Drop the <3.14 cap from the shipped/example plugin pyprojects. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mikasenghaas
changed the title
- Drop the "Run Prime sandbox tests" CI step: its tests lived in the removed test_v1_runtime_lifecycle.py, so `pytest -m prime_sandbox` collected nothing and exited 5. - Semgrep job: `uv sync --no-default-groups --group policy` (the plugin groups are default + declared incompatible with policy, so the old `--no-dev` still pulled them and the resolve conflicted). - Drop Python 3.10: requires-python >=3.11 (+ classifier, CI matrix). With renderers/v1 deps in base and example plugins pulling chromadb -> onnxruntime (no 3.10 wheel), 3.10 is no longer supported. - tests/test_envs.py: remove the obsolete v1 tests (alphabet_sort_v1 / test_v1_wrapper_*) and the stale prime-pydantic-config exclude-newer cap that conflicted with renderers' required version. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The .semgrep/verifiers.yml policy enforced the old hand-authored v1's conventions: env-authoring rules targeting load_environment(config) shims (the v1 env API is gone), package rules pointing at the old packages/<x>/<x> layout, State methods that were removed, and a canonical-shim exclude list of deleted files — plus typing rules (no Any/Mapping/__future__ annotations) that contradict the vendored vf-nano code (already excluded from the ty gate). Remove the policy wholesale: .semgrep/verifiers.yml, the Semgrep CI job, the `policy` dependency group + its uv conflicts, the pre-commit hook, the now-empty [tool.ruff] exclude, and the dead nosemgrep waivers. A lint policy for the new architecture can be written against vf-nano separately. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Task gains `system_prompt: str | None`. Harness adds the `APPENDS_SYSTEM_PROMPT` class var + `resolve_prompt`: harnesses that support it emit the system prompt as a real system message (default via program.py; rlm via RLM_APPEND_TO_SYSTEM_PROMPT, which rlm appends to its generated prompt); others fold it into the user instruction with a warning. - default harness adds a one-line bash system prompt (before the task's) only when `enable_bash`. - reverse_text_v1 sets `system_prompt` separately so its prompt is byte-identical to the v0 env ([system, user]) — the model answers directly instead of leaking <think>. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mikasenghaas
force-pushed
the
feat/nano-as-v1
branch
from
June 9, 2026 08:13
6ef7ace to
00c7b77
Compare
The renderer client built its tokenizer/renderer pool from the per-request `model`, which becomes the LoRA adapter name (e.g. `r32-a64.0`) after a weight update — there is no HF tokenizer published under that name, so rollouts 404'd. Add `renderer_model_name` to `RendererClientConfig` (pin it to the base model). The v1 `RendererClient` and the v0 legacy bridge use it for the tokenizer pool while the per-request `model` still selects the sampling target, so LoRA sampling keeps routing by the adapter name. Restores parity with the v0 `ClientConfig.renderer_model_name` wiring used on prime-rl main. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The openai_chat_completions client now best-effort parses the prompt and completion token ids and sampling logprobs that vLLM returns (return_token_ids + logprobs) into Response.tokens, so MITO training (no renderer) can train on real on-policy tokens instead of re-tokenizing the messages downstream. Sampling args still pass straight through; tokens stay None when the provider returns neither token ids nor logprobs (e.g. eval, or non-vLLM providers). Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The bridge only kept token ids: it dropped the prompt messages, the response message (content / reasoning / tool calls), finish_reason, usage, and the task's system prompt / answer — so a v0-bridged Trace was a near-empty skeleton next to a native v1 Trace. The cause: v0 RolloutOutput nests these as pydantic objects (messages, Response) and records finish_reason on response.message, but the mapping only handled plain dicts and read finish_reason off the response. Coerce v0 objects to dicts before mapping (_as_dict), read finish_reason/usage from their v0 locations, mirror tokens onto the response (as the native client does), and carry the prompt's system_prompt / instruction / answer onto the task. A v0-bridged Trace now matches the native v1 schema (verified by diffing reverse-text rollouts). Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
) Rename every taskset under examples/tasksets/ to a `-v1` id (package name, module, and directory) so they no longer collide with the v0 environments of the same name (gsm8k, wiki-search, math-env, ...) when both are installed in one env. reverse-text-v1 was already suffixed; harbor (a bundled taskset with no v0 counterpart) is left as-is. - examples/tasksets/<x> -> <x>_v1, module <x>.py -> <x>_v1.py; verify.py / server.py / facts.json keep their names (read via __file__, never imported) - package tasksets: inner package wiki_search/wikispeedia -> *_v1, with their self-imports and `-m <pkg>.server` launch paths updated to match - root pyproject [tool.uv.sources] + examples group, and configs/*.toml taskset ids - refresh uv.lock Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add RetryConfig (attempts / include / exclude) on EnvConfig.retry and retry a whole rollout with tenacity when it ends with a captured error — parity with v0's rollout-level retries. Matching is by exception type name; include/exclude name exception classes (e.g. ModelError, ProgramError). Flags: --retry.attempts / --retry.include / --retry.exclude. EvalConfig inherits EnvConfig and the env server runs through Environment.episode, so both eval and training get retries. Retries are first-class on the Trace: `errors` is the list of per-attempt errors (oldest first), and `error` is now a computed field returning the most recent — so a retried-then-failed trace shows every error that led to a retry. Retry utilities live in verifiers/v1/retries.py. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
* feat(v1): per-rollout token limits (EnvConfig.max_{input,output,total}_tokens)
Add framework-enforced token budgets alongside max_turns: max_input_tokens,
max_output_tokens, max_total_tokens on EnvConfig. The interception server checks
them before each turn via a new RolloutLimits bundle (which also subsumes
max_turns), capping the trace's prompt_len / completion_len / total_tokens
computed properties. Reaching any limit refuses the turn and records it as the
stop condition, and is_truncated now treats the token-limit conditions as
truncation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(v1): drop 'like max_turns' from token-limit field docstrings
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* style(v1): trim limit-check comment in interception
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* style(v1): ruff format interception
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(v1): reclaim orphaned subprocess workspaces
A rollout's /tmp workspace is removed in `stop()`, but a process killed mid-rollout
(SIGKILL, OOM, hard crash, interrupted teardown) never reaches it, so the workspace
leaks with no way to reclaim it — repeated runs eventually fill /tmp ("No space left
on device" at mkdtemp).
Name each workspace `/tmp/v1-<pid>-*` and, once per process on the first `start()`,
sweep `/tmp/v1-<pid>-*` whose pid is no longer alive. PID-keyed, so a concurrent live
process's workspaces are never touched; graceful per-rollout cleanup (`stop()`) is
unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(v1): atexit-based runtime teardown; drop the SIGKILL reaper
Make resource cleanup a backend-agnostic property of `Runtime`:
- a sync `cleanup()` is the teardown source of truth; the public async `stop()` runs it
off the event loop on the happy path.
- `make_runtime` registers each runtime in a WeakSet and arms one sync `atexit` hook that
calls `cleanup()` on anything still live — so a Ctrl-C / SIGTERM that cancels the
rollout's `finally` mid-teardown still frees the workspace / container / sandbox, reusing
each backend's own cleanup. The hook must be sync: at interpreter shutdown the event loop
and its thread-pool are gone, so async teardown raises "cannot schedule new futures".
Drop the PID-tagged `reap_orphans` startup sweep. A SIGKILL/OOM runs no in-process code at
all, so reclaiming it needs an external mechanism; prime sandboxes already self-terminate
via their server-side max-lifetime, and the local subprocess/docker cases are out of scope.
Prefix workspaces/containers/scripts with `vf-` (was `v1-`).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(v1): delete the prime sandbox in the sync atexit cleanup too
`cleanup()` (the atexit backstop) only stopped the tunnels and left the sandbox — the
costly resource — to its server-side max-lifetime. prime_sandboxes ships a sync
`SandboxClient`, so delete the sandbox synchronously there as well (the async client can't
run once the loop is gone). Idempotent with the async `stop` on the normal path: a second
delete just 404s.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* style: move teardown comments off the statement line (ruff format)
The inline comments pushed two lines past the 88-col limit; moving them above the
statement keeps `ruff format` happy without ruff's awkward auto-wrap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(v1): public register/cleanup_at_exit, trim runtime-teardown comments
- rename the module-level helpers to public `register` / `cleanup_at_exit`
- trim the `_LIVE` block comment and drop the inline "no event loop" why-comments
(the `cleanup` docstring already covers why teardown is sync)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Use native JSON bytes for upstream eval traffic * Validate non-finite JSON structurally
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
* Await scoring hooks directly * Preserve concurrent multi-hook scoring
* Persist large traces off the event loop * Preserve queued trace writes on cancellation * Use public Pydantic JSON serialization
…'s verifiers dep (#1824) * fix(v1): declare verifiers dep in compact harness package The compact harness package imports verifiers.v1 but declared dependencies = [], so installing/publishing the wheel on its own could fail at import time. Match the other v1 environment packages. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(v1): carry v0 truncation flag through the legacy bridge Trace.is_truncated is derived from the v1 stop-condition vocabulary and the final turn's finish_reason. v0 stop names (e.g. max_turns_reached, prompt_too_long) don't map onto that vocabulary, so legacy traces reported is_truncated=False even when the v0 rollout was truncated. Add an explicit, serialized `truncated` override on Trace (None for native v1, so the derivation is unchanged) that is_truncated honors first, and set it from the v0 rollout's own is_truncated flag in the bridge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(v1): derive legacy truncation from stop condition, not a Trace field Keep Trace.is_truncated purely derived (no stored field). Instead, the v0->v1 bridge translates a truncated v0 rollout's stop name into v1's truncation vocabulary (max_turns_reached -> max_turns, prompt_too_long -> context_length, ...; unmapped truncated stops fall back to max_output_tokens) so the property derives True. Untruncated rollouts keep their v0 stop condition unchanged. This reverts the `truncated` field added to Trace in the previous commit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 3 total unresolved issues (including 2 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 6e8b3cc. Configure here.
Member
Author
|
superseded by #1825 |
pull Bot
pushed a commit
to Stars1233/verifiers
that referenced
this pull request
Jun 23, 2026
…ison doc (PrimeIntellect-ai#1619) * chore: add v1 PrimeIntellect-ai#1559-vs-PrimeIntellect-ai#1576 comparison doc Feature parity, branch-unique features, and validation done for the two open v1 refactor PRs (PrimeIntellect-ai#1559 codex/v1-nano-refactor-draft, PrimeIntellect-ai#1576 feat/nano-as-v1). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: trim comparison (drop names, some PrimeIntellect-ai#1559 items, validation section) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: scope the size row to the verifiers/ module diff Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Revert "docs: scope the size row to the verifiers/ module diff" This reverts commit 500bd30. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
pull Bot
pushed a commit
to Stars1233/verifiers
that referenced
this pull request
Jun 23, 2026
* chore: final touches - harness configs no longer pin `id`; the caller supplies it (--harness.id / toml / a taskset's bundled harness), mirroring tasksets. Update the base HarnessConfig docstring and the `init` scaffold to match. - refine the v1 user guide (GUIDE.md, README.md) - register the reverse-text v0 env as an editable dep for the `eval --id reverse-text` legacy-bridge example Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: don't use auto-tracked num_turns as the @vf.metric example trace.num_turns is recorded by the framework; show a custom signal instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: move the load_tasks example into the Loading tasks section Keep the config code block focused on the config class. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: show full GSM8KTask construction in the load_tasks example Build typed task instances from the rows instead of stopping at the dataset load. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: allow @reward to return dict[str, float] A reward can now report a family of named contributions, like @Metric already can; each entry is recorded under its own key and scaled by the function's weight before being summed into trace.reward. This aligns the reward/metric return contracts — the only remaining (and intended) differences are that rewards are summed and carry a weight. Documented in the v1 guide. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: drop the score/score_group override note Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: livelier scoring examples + reframe the notes heading Use an enthusiasm metric and a brevity (length-penalty) group reward instead of stubs, and reword "Notes that bite if missed" to "Good to know". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: restructure the trace-reading table and reframe in-runtime scoring Group the trace members by area and fix the table: drop the nonexistent trace.messages (use trace.branches[-1].messages), add usage / has_response / id, and note trace.nodes. Reframe in-runtime scoring around when to use the runtime object (host-unsafe/heavy computation, or runtime-only information). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: separate scoring outputs from read inputs; say "host" not "eval process" The trace-reading table now lists only what a reward reads; reward/rewards/ metrics are called out as outputs you shouldn't read mid-scoring. Rename that group to "Carried state" (info/state). Use "host" instead of "eval process". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: drop validate from the lifecycle-hooks table validate isn't part of the setup -> harness -> finalize -> scoring rollout loop (it's run only by `uv run validate`, documented in the CLI reference). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: give info its own section, contrasted with state Lift the trace.info paragraph into a "Persisted info" section beside "Per-rollout state", with a table contrasting the two per-rollout stores (persisted vs transient). Cross-link both ways. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: note that a Toolset wraps an MCP server @vf.tool methods are served as MCP tools the harness connects to. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: show the self-launching __main__ line in the tool/user examples Each tool/user server is its own self-launching module under servers/. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: merge own-host/own-sandbox into a single "own runtime" placement row Both are the same placement (a per-rollout runtime); subprocess on the host by default, a docker/prime sandbox when runtime.type is set. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: add the missing reference examples general-agent-v1 (per-task dynamic tools), tau2-bench-v1 (tools + user sim + bundled harness), and swebench-verified-v1 (SWE-bench on prebuilt images) were the only *_v1 envs absent from the example table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: reframe the custom-harness intro and list mini-swe-agent/kimi-code Frame authoring a harness as "when you need rollout logic the built-ins can't express" rather than "you rarely need this", and add the mini-swe-agent and kimi-code built-ins to the table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: surface the NEEDS_CONTAINER taskset capability flag Mention it in the taskset authoring intro alongside the generic params. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: rename harness capability flag SUPPORTS_TASK_TOOLS -> SUPPORTS_MCP Rename the flag on the base Harness and every harness that sets it (built-ins + tau2/compact env harnesses), the Environment check, and align the user-facing error message and the README/GUIDE docs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: harness example configures via CLI args, drops max_steps Pass endpoint/secret/model as CLI args (as the built-in default harness does) instead of OPENAI_* env vars, which have footguns; note why. Drop the max_steps knob (turn limits are framework-enforced, not a harness concern) and the id pin (harness configs don't pin id), leaving a placeholder config. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: split the harness "Writing one" section into subsections Break the wall of bold-lead paragraphs into ### subsections (contract, launch, resolve_prompt, program styles, harness metrics), matching the taskset docs' ##/### structure. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: correct the tau2-bench-v1 example description tau2's taskset defines no vf.User/vf.Toolset; it bundles its own harness that runs the whole tau2 simulation in a subprocess and stores the result in trace.info. It's the bundled-harness example, not a tools+user-sim one. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: drop internal bench/ scripts and COMPARE.md from v1 These are work-in-progress benchmarking scripts and a transient PrimeIntellect-ai#1559-vs-PrimeIntellect-ai#1576 comparison doc — not part of the public v1 surface. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: restore taskset/harness PyPI publish workflows; bump to 0.2.0 Bring back publish-tasksets.yml / publish-harnesses.yml (dropped on the v1 branch). v1 versions packages statically in pyproject.toml rather than via __init__.py __version__, so the version-detection reads [project].version from pyproject. Bump both packages 0.1.0 -> 0.2.0: the v1 branch had reset them to 0.1.0, below what's already on PyPI (tasksets 0.1.5, harnesses 0.1.2), so a progression is required to publish. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: drop the experimental flag from the renderer client The renderer client is no longer marked experimental; keep the substantive caveat (per-model renderers cover a subset of models). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: drop stale v1-runtime dependency comment Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: tidy verifiers pyproject - drop the v0 reverse-text entry from the examples group - drop stale comments in [dependency-groups]/[tool.uv.sources] - drop the redundant [tool.ruff] line-length = 88 (88 is ruff's default) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: exclude v1 envs from the auto env-publish for now publish-envs.yml matrixes over environments/*; skip the *_v1 packages and the compact harness example — the v1 envs aren't ready for the Environments Hub yet. Classic v0 envs still publish. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: fix version-mismatch message to reference pyproject, not __init__.py The version is read from [project].version in pyproject.toml; the mismatch error still pointed at the old __init__.py source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
High Risk
Large surface-area change: drops automated Semgrep v1 policy enforcement, narrows type-check scope, changes publish matrices and public docs/API expectations, and restructures environment packaging—easy to miss regressions in CI, releases, or hub publishes.
Overview
This PR reshapes how v1 is exercised and documented alongside the nano bridge: CI and pre-commit drop the Semgrep policy job and
policydependency group, and ty now checks onlyverifiers(nottasksets/harnesses). Tests drop Python 3.10, limit default branches tomain, replace the dedicated Prime sandbox job with a paralleltests/v1pytest pass (-m "not prime and not modal"), and adjust the main suite markers.Publishing reads tasksets/harnesses versions from
pyproject.toml(tomllib) instead of__init__.py, and hub env auto-publish skips*_v1dirs andcompactuntil v1 packages are hub-ready.User-facing docs remove the BYO Taskset/Harness guide, v1 reference sections, Mintlify nav entry, and lifecycle SVGs; eval/training docs no longer describe
--taskset/--harnessoverrides. Sample eval configs land underconfigs/*.tomlfor v1 taskset/harness combos (GSM8K, harbor, wiki search + compact, etc.).Environments move to installable v1 taskset/harness packages (e.g.
aime24_v1,alphabet_sort_v1with colocated users,code_golf_v1group rewards,color_codeword_v1,deepwiki_v1,compactcontext-rewrite harness). Legacy monolithic v1 modules (bfcl_v3, inlinealphabet_sort_v1.py, v0→v1 shims) are removed or trimmed.Reviewed by Cursor Bugbot for commit 6e8b3cc. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Add verifiers v1 framework with taskset/harness bridge, runtimes, MCP tooling, and built-in environments
verifiers.v1public API:Taskset,Harness,Environment,Episode,Rollout,Trace,State, and supporting decorators (@vf.reward,@vf.metric,@vf.stop,@vf.tool,@vf.group_reward) for authoring evaluation environments.BashHarness,DefaultHarness,CodexHarness,KimiCodeHarness,MiniSWEAgentHarness,RLMHarness,Terminus2Harness) that launch external agent CLIs against an OpenAI-compatible interception proxy.SubprocessRuntime,DockerRuntime,ModalRuntime,PrimeRuntime) with unified config, atexit cleanup, and per-task resource resolution.Toolset,User) with colocated, shared, and forked placement modes and aInterceptionPoolfor routing model requests through the framework.GSM8K,AIME24,Math,ReverseText,CodeGolf,AlphabetSort,ColorCodeword,Wikispeedia,WikiSearch,DeepWiki,Glossary,Scratchpad,TextArena/Wordle,Tau2Bench,Harbor/SWEBenchVerified/TerminalBench2,R2EGym,ScaleSWE,SWELego, andGeneralAgent.eval,serve,validate, andinitCLI entrypoints with rich live dashboards, resumable runs, dry-run mode, and TOML-based config.verifiersandverifiers.v1public surfaces;load_harness/load_tasksetare no longer lazily resolved via__getattr__, v0load_environmentnow raises if the module lacks that function, andverifiers.typesnow requiresrenderersat import time.verifiersandverifiers.v1public APIs; code importing previously available v0-era symbols (ArtifactConfig,BindingsConfig,load_taskset,Stateruntime handles, etc.) will fail at import time.Macroscope summarized 6e8b3cc.