V1 runtime/taskset/harness refactor by willccbb · Pull Request #1559 · PrimeIntellect-ai/verifiers

willccbb · 2026-06-06T18:49:43Z

Summary

This PR rewrites the v1 runtime/taskset/harness boundary around a strict, transcript-native contract while keeping v0 separate.

Adds EnvRun, rollout Context, and group lifecycle ownership for v1, including env-scope reuse, explicit close/cleanup, and group scoring.
Makes Toolset and User MCP-backed authoring surfaces with hidden bound args plus sets / extends updates for state and extras.
Adds runtime placement/config support across subprocess, Docker, Prime, and stubbed Modal/Daytona runtimes; subprocess-backed MCP servers get graceful interrupt cleanup before termination.
Migrates representative v1 envs and packages to the component layout, including single-turn, tool, user-simulator, group-reward, replay, BFCL, Harbor, OpenEnv, Tau2, Wiki Search, and command/program-style cases.
Updates v1 docs and the prime-rl integration sketch around transcript-only state, token-level advantages, teacher config flow, dynamic tools, and taskset-owned setup.

Final Validation

GitHub CI is green on head 5d1ef09e1.

Ruff: pass
Ty: pass
Semgrep: pass
CodeQL / analyzers: pass
Cursor Bugbot: pass
Macroscope - Correctness Check: pass
Verifiers (3.10), (3.11), (3.12), (3.13): pass
Environments: pass
Macroscope - Approvability Check: skipped by the app

Local validation:

env PYTHONDONTWRITEBYTECODE=1 uv run pytest tests/test_v1_core.py -m 'not prime_sandbox' -q
env PYTHONDONTWRITEBYTECODE=1 uv run pytest tests/test_v1_core.py::test_env_user_startup_failure_closes_started_env_toolsets tests/test_v1_core.py::test_env_scope_startup_failure_closes_entered_env_user tests/test_v1_core.py::test_nemo_gym_task_row_preserves_explicit_task_fields -q
env PYTHONDONTWRITEBYTECODE=1 CHANGED_ENVS='wiki_search_v1,bfcl_v3_v1' uv run pytest tests/test_envs.py -vv -k 'wiki_search_v1 or bfcl_v3_v1'
env PYTHONDONTWRITEBYTECODE=1 uv run pytest tests/test_wiki_search_v1.py -q
env PYTHONDONTWRITEBYTECODE=1 uv run --python 3.10 pytest tests/test_v1_core.py::test_v1_prime_runtime_run_honors_timeout -q
env PYTHONDONTWRITEBYTECODE=1 uv run --python 3.13 ty check verifiers/v1 packages/tasksets/tasksets packages/harnesses/harnesses
env PYTHONDONTWRITEBYTECODE=1 uv run --no-dev --group policy semgrep --metrics=off --disable-version-check --config .semgrep/verifiers.yml --error --quiet
env PYTHONDONTWRITEBYTECODE=1 uv run python scripts/sync.py --check
git diff --check

Live prime eval smoke matrix, default settings unless the package default had fewer eval examples. Commands were run with PYTHONPATH pinned to this worktree so prime eval loaded this PR branch:

reverse-text-v1: 5 examples x 3 rollouts, average reward 0.948.
alphabet-sort-v1: 5 examples x 3 rollouts, average reward 0.801.
mcp-search-env-v1: 5 examples x 3 rollouts, average reward 1.000.
math-python-v1: 5 examples x 3 rollouts, average reward 1.000.
hello-group-reward-v1: 5 examples x 3 rollouts, average reward 0.645.
sft-replay-v1: 1 example x 3 rollouts, structural replay completed with stop replayed_messages.
openenv-echo-v1: 5 examples x 3 rollouts, average reward 1.800; verified OpenEnv sandbox cleanup after run.
openenv-textarena-v1: 5 examples x 3 rollouts, average reward 0.500; verified OpenEnv sandbox cleanup after run.
tau2-bench-v1: 5 examples x 3 rollouts, average reward 0.533, tau2_num_errors average 0; no Tau2 environment/tool errors.

Review State

PR is marked ready for review.
No unresolved review threads remain.
Merge is blocked only by required human review.

macroscopeapp · 2026-06-08T12:20:56Z

Approvability

Verdict: Needs human review

2 blocking correctness issues found. Diff is too large for automated approval analysis. A human reviewer should evaluate this PR.

^{You can customize Macroscope's approvability policy. Learn more.}

aniravich · 2026-06-09T21:48:49Z

+            EnvRun.normalize_model(teacher) if teacher is not None else None
+        )
+        self.states = list(
+            await asyncio.gather(


gather here has no return_exceptions=True. Most rollout errors are absorbed by run_lifecycle (via capture_error) and come back as states, which is great — but a failure raised before that try-block, e.g. the runtime failing to start or a toolset/user failing to enter in run_context, will propagate through this gather and fail the entire group with no partial credit. Is that intended? Two options worth discussing: (a) capture setup/runtime-acquisition failures into the state too, so one bad member can't sink its siblings, or (b) return_exceptions=True here with per-member handling. Also — since this waits for the slowest member before scoring, do we have a straggler policy at training scale, or is max_turns + runtime timeout the only backstop?

Also: first failing rollout propagates while siblings keep running, then stop_env_scope() tears down servers under live rollouts. Use TaskGroup.

aniravich · 2026-06-09T21:49:43Z

+                state.capture_error(exc)
+            except Error as exc:
+                state.capture_error(exc)
+            except BaseException as exc:


Catching BaseException turns a failed rollout into recorded data, which I think is the intent. But it also swallows asyncio.CancelledError (and KeyboardInterrupt/SystemExit). On a timeout or shutdown this could prevent cancellation from propagating and leave work running. Should we re-raise CancelledError (catch Exception for the "error becomes data" path, and let cancellation through)? Same question for the except BaseException in interception.py's handler.

aniravich · 2026-06-09T21:51:05Z

+            "VF_PROMPT": prompt_text,
+        }
+
+    async def run_with_context(self, context: vf.Context) -> None:


The command harness wires the agent to the interception endpoint + task env, but not to any MCP tool servers — so external agents use their own built-in tools, and verifiers' @vf.tool(args/sets/extends) binding effectively does nothing on this path. That asymmetry vs the in-process base harness is subtle. Could we either document it loudly, or warn/error if a task declares bound toolsets but runs under a command harness, so an author doesn't think their sets/extends are taking effect?

aniravich · 2026-06-09T21:52:04Z

+                return
+        max_turns = self.max_turns(task)
+        turns = 0
+        while max_turns <= 0 or turns < max_turns:


max_turns <= 0 means unlimited (bounded only by timeout). Worth a one-line doc on Task.max_turns / HarnessConfig.max_turns so 0 isn't mistaken for "no turns."

aniravich · 2026-06-09T21:52:15Z

+    def has_model_prompt(messages: Messages) -> bool:
+        return any(getattr(message, "role", None) != "system" for message in messages)
+
+    def max_turns(self, task: Task) -> int:


max_turns <= 0 means unlimited (bounded only by timeout). Worth a one-line doc on Task.max_turns / HarnessConfig.max_turns so 0 isn't mistaken for "no turns."

aniravich · 2026-06-09T22:09:44Z


-    def _validate_owner_bindings(self, owner: RuntimeOwnerMixin | None) -> None:
-        if owner is None:
+    async def stop(self) -> None:


Cleanup not cancellation-safe — contextlib.suppress(Exception) around await client.delete(sandbox_id) doesn't survive cancellation → leaked billed Prime sandboxes / orphaned Docker containers exactly when users interrupt. Shield cleanup awaits. Also SubprocessRuntime kills only the direct child (no process groups), and Docker/Prime timeouts abandon the remote command without killing it.

aniravich · 2026-06-09T22:10:38Z

+        result = await docker(
+            "run",
+            "--detach",
+            "--network",


Docker uses --network host — containers see every host-local service; also makes free_port()'s TOCTOU collisions real across concurrent rollouts.

aniravich · 2026-06-09T22:11:37Z

+    server: ServerConfig, name: str, *, in_runtime: bool = False
+) -> list[str]:
+    python = "python" if in_runtime else sys.executable
+    payload = {


Secrets in argv - full ServerConfig (incl. env/headers, the natural home for API keys) is serialized onto the command line, visible via ps. Pass over stdin.

aniravich · 2026-06-09T22:12:05Z

+    @property
+    def has_group_signals(self) -> bool:
+        return any(signal["stage"] == "group" for signal in self.signals)
+
    def to_task(self, task: Task | JsonData) -> Task:


to_task bypasses task_type — taskset.py:218: the Task-instance path calls prepare_task(task) with default task_type=Task, so a taskset declaring a subclass silently accepts base Tasks. Should be prepare_task(task, self.task_type).

aniravich · 2026-06-09T22:12:38Z

+
+
+def parse_anthropic_messages(body: JsonData) -> Messages:
+    messages: Messages = []


Anthropic interception bugs — protocols.py:421 drops list-form system prompts (what Claude SDKs emit with cache_control) silently; protocols.py:484–509 reorders text before tool results, producing invalid assistant→user→tool sequences.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: add v1 #1559-vs-#1576 comparison doc Feature parity, branch-unique features, and validation done for the two open v1 refactor PRs (#1559 codex/v1-nano-refactor-draft, #1576 feat/nano-as-v1). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: trim comparison (drop names, some #1559 items, validation section) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: scope the size row to the verifiers/ module diff Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Revert "docs: scope the size row to the verifiers/ module diff" This reverts commit 500bd30. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

These are work-in-progress benchmarking scripts and a transient #1559-vs-#1576 comparison doc — not part of the public v1 surface. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@reward

* chore: final touches - harness configs no longer pin `id`; the caller supplies it (--harness.id / toml / a taskset's bundled harness), mirroring tasksets. Update the base HarnessConfig docstring and the `init` scaffold to match. - refine the v1 user guide (GUIDE.md, README.md) - register the reverse-text v0 env as an editable dep for the `eval --id reverse-text` legacy-bridge example Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: don't use auto-tracked num_turns as the @vf.metric example trace.num_turns is recorded by the framework; show a custom signal instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: move the load_tasks example into the Loading tasks section Keep the config code block focused on the config class. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: show full GSM8KTask construction in the load_tasks example Build typed task instances from the rows instead of stopping at the dataset load. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: allow @reward to return dict[str, float] A reward can now report a family of named contributions, like @Metric already can; each entry is recorded under its own key and scaled by the function's weight before being summed into trace.reward. This aligns the reward/metric return contracts — the only remaining (and intended) differences are that rewards are summed and carry a weight. Documented in the v1 guide. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: drop the score/score_group override note Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: livelier scoring examples + reframe the notes heading Use an enthusiasm metric and a brevity (length-penalty) group reward instead of stubs, and reword "Notes that bite if missed" to "Good to know". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: restructure the trace-reading table and reframe in-runtime scoring Group the trace members by area and fix the table: drop the nonexistent trace.messages (use trace.branches[-1].messages), add usage / has_response / id, and note trace.nodes. Reframe in-runtime scoring around when to use the runtime object (host-unsafe/heavy computation, or runtime-only information). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: separate scoring outputs from read inputs; say "host" not "eval process" The trace-reading table now lists only what a reward reads; reward/rewards/ metrics are called out as outputs you shouldn't read mid-scoring. Rename that group to "Carried state" (info/state). Use "host" instead of "eval process". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: drop validate from the lifecycle-hooks table validate isn't part of the setup -> harness -> finalize -> scoring rollout loop (it's run only by `uv run validate`, documented in the CLI reference). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: give info its own section, contrasted with state Lift the trace.info paragraph into a "Persisted info" section beside "Per-rollout state", with a table contrasting the two per-rollout stores (persisted vs transient). Cross-link both ways. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: note that a Toolset wraps an MCP server @vf.tool methods are served as MCP tools the harness connects to. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: show the self-launching __main__ line in the tool/user examples Each tool/user server is its own self-launching module under servers/. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: merge own-host/own-sandbox into a single "own runtime" placement row Both are the same placement (a per-rollout runtime); subprocess on the host by default, a docker/prime sandbox when runtime.type is set. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: add the missing reference examples general-agent-v1 (per-task dynamic tools), tau2-bench-v1 (tools + user sim + bundled harness), and swebench-verified-v1 (SWE-bench on prebuilt images) were the only *_v1 envs absent from the example table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: reframe the custom-harness intro and list mini-swe-agent/kimi-code Frame authoring a harness as "when you need rollout logic the built-ins can't express" rather than "you rarely need this", and add the mini-swe-agent and kimi-code built-ins to the table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: surface the NEEDS_CONTAINER taskset capability flag Mention it in the taskset authoring intro alongside the generic params. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: rename harness capability flag SUPPORTS_TASK_TOOLS -> SUPPORTS_MCP Rename the flag on the base Harness and every harness that sets it (built-ins + tau2/compact env harnesses), the Environment check, and align the user-facing error message and the README/GUIDE docs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: harness example configures via CLI args, drops max_steps Pass endpoint/secret/model as CLI args (as the built-in default harness does) instead of OPENAI_* env vars, which have footguns; note why. Drop the max_steps knob (turn limits are framework-enforced, not a harness concern) and the id pin (harness configs don't pin id), leaving a placeholder config. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: split the harness "Writing one" section into subsections Break the wall of bold-lead paragraphs into ### subsections (contract, launch, resolve_prompt, program styles, harness metrics), matching the taskset docs' ##/### structure. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: correct the tau2-bench-v1 example description tau2's taskset defines no vf.User/vf.Toolset; it bundles its own harness that runs the whole tau2 simulation in a subprocess and stores the result in trace.info. It's the bundled-harness example, not a tools+user-sim one. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: drop internal bench/ scripts and COMPARE.md from v1 These are work-in-progress benchmarking scripts and a transient #1559-vs-#1576 comparison doc — not part of the public v1 surface. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: restore taskset/harness PyPI publish workflows; bump to 0.2.0 Bring back publish-tasksets.yml / publish-harnesses.yml (dropped on the v1 branch). v1 versions packages statically in pyproject.toml rather than via __init__.py __version__, so the version-detection reads [project].version from pyproject. Bump both packages 0.1.0 -> 0.2.0: the v1 branch had reset them to 0.1.0, below what's already on PyPI (tasksets 0.1.5, harnesses 0.1.2), so a progression is required to publish. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: drop the experimental flag from the renderer client The renderer client is no longer marked experimental; keep the substantive caveat (per-model renderers cover a subset of models). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: drop stale v1-runtime dependency comment Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: tidy verifiers pyproject - drop the v0 reverse-text entry from the examples group - drop stale comments in [dependency-groups]/[tool.uv.sources] - drop the redundant [tool.ruff] line-length = 88 (88 is ruff's default) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: exclude v1 envs from the auto env-publish for now publish-envs.yml matrixes over environments/*; skip the *_v1 packages and the compact harness example — the v1 envs aren't ready for the Environments Hub yet. Classic v0 envs still publish. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: fix version-mismatch message to reference pyproject, not __init__.py The version is read from [project].version in pyproject.toml; the mismatch error still pointed at the old __init__.py source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ison doc (PrimeIntellect-ai#1619) * chore: add v1 PrimeIntellect-ai#1559-vs-PrimeIntellect-ai#1576 comparison doc Feature parity, branch-unique features, and validation done for the two open v1 refactor PRs (PrimeIntellect-ai#1559 codex/v1-nano-refactor-draft, PrimeIntellect-ai#1576 feat/nano-as-v1). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: trim comparison (drop names, some PrimeIntellect-ai#1559 items, validation section) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: scope the size row to the verifiers/ module diff Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Revert "docs: scope the size row to the verifiers/ module diff" This reverts commit 500bd30. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@reward

* chore: final touches - harness configs no longer pin `id`; the caller supplies it (--harness.id / toml / a taskset's bundled harness), mirroring tasksets. Update the base HarnessConfig docstring and the `init` scaffold to match. - refine the v1 user guide (GUIDE.md, README.md) - register the reverse-text v0 env as an editable dep for the `eval --id reverse-text` legacy-bridge example Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: don't use auto-tracked num_turns as the @vf.metric example trace.num_turns is recorded by the framework; show a custom signal instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: move the load_tasks example into the Loading tasks section Keep the config code block focused on the config class. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: show full GSM8KTask construction in the load_tasks example Build typed task instances from the rows instead of stopping at the dataset load. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat: allow @reward to return dict[str, float] A reward can now report a family of named contributions, like @Metric already can; each entry is recorded under its own key and scaled by the function's weight before being summed into trace.reward. This aligns the reward/metric return contracts — the only remaining (and intended) differences are that rewards are summed and carry a weight. Documented in the v1 guide. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: drop the score/score_group override note Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: livelier scoring examples + reframe the notes heading Use an enthusiasm metric and a brevity (length-penalty) group reward instead of stubs, and reword "Notes that bite if missed" to "Good to know". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: restructure the trace-reading table and reframe in-runtime scoring Group the trace members by area and fix the table: drop the nonexistent trace.messages (use trace.branches[-1].messages), add usage / has_response / id, and note trace.nodes. Reframe in-runtime scoring around when to use the runtime object (host-unsafe/heavy computation, or runtime-only information). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: separate scoring outputs from read inputs; say "host" not "eval process" The trace-reading table now lists only what a reward reads; reward/rewards/ metrics are called out as outputs you shouldn't read mid-scoring. Rename that group to "Carried state" (info/state). Use "host" instead of "eval process". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: drop validate from the lifecycle-hooks table validate isn't part of the setup -> harness -> finalize -> scoring rollout loop (it's run only by `uv run validate`, documented in the CLI reference). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: give info its own section, contrasted with state Lift the trace.info paragraph into a "Persisted info" section beside "Per-rollout state", with a table contrasting the two per-rollout stores (persisted vs transient). Cross-link both ways. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: note that a Toolset wraps an MCP server @vf.tool methods are served as MCP tools the harness connects to. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: show the self-launching __main__ line in the tool/user examples Each tool/user server is its own self-launching module under servers/. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: merge own-host/own-sandbox into a single "own runtime" placement row Both are the same placement (a per-rollout runtime); subprocess on the host by default, a docker/prime sandbox when runtime.type is set. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: add the missing reference examples general-agent-v1 (per-task dynamic tools), tau2-bench-v1 (tools + user sim + bundled harness), and swebench-verified-v1 (SWE-bench on prebuilt images) were the only *_v1 envs absent from the example table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: reframe the custom-harness intro and list mini-swe-agent/kimi-code Frame authoring a harness as "when you need rollout logic the built-ins can't express" rather than "you rarely need this", and add the mini-swe-agent and kimi-code built-ins to the table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: surface the NEEDS_CONTAINER taskset capability flag Mention it in the taskset authoring intro alongside the generic params. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: rename harness capability flag SUPPORTS_TASK_TOOLS -> SUPPORTS_MCP Rename the flag on the base Harness and every harness that sets it (built-ins + tau2/compact env harnesses), the Environment check, and align the user-facing error message and the README/GUIDE docs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: harness example configures via CLI args, drops max_steps Pass endpoint/secret/model as CLI args (as the built-in default harness does) instead of OPENAI_* env vars, which have footguns; note why. Drop the max_steps knob (turn limits are framework-enforced, not a harness concern) and the id pin (harness configs don't pin id), leaving a placeholder config. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: split the harness "Writing one" section into subsections Break the wall of bold-lead paragraphs into ### subsections (contract, launch, resolve_prompt, program styles, harness metrics), matching the taskset docs' ##/### structure. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: correct the tau2-bench-v1 example description tau2's taskset defines no vf.User/vf.Toolset; it bundles its own harness that runs the whole tau2 simulation in a subprocess and stores the result in trace.info. It's the bundled-harness example, not a tools+user-sim one. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: drop internal bench/ scripts and COMPARE.md from v1 These are work-in-progress benchmarking scripts and a transient PrimeIntellect-ai#1559-vs-PrimeIntellect-ai#1576 comparison doc — not part of the public v1 surface. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: restore taskset/harness PyPI publish workflows; bump to 0.2.0 Bring back publish-tasksets.yml / publish-harnesses.yml (dropped on the v1 branch). v1 versions packages statically in pyproject.toml rather than via __init__.py __version__, so the version-detection reads [project].version from pyproject. Bump both packages 0.1.0 -> 0.2.0: the v1 branch had reset them to 0.1.0, below what's already on PyPI (tasksets 0.1.5, harnesses 0.1.2), so a progression is required to publish. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: drop the experimental flag from the renderer client The renderer client is no longer marked experimental; keep the substantive caveat (per-model renderers cover a subset of models). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: drop stale v1-runtime dependency comment Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: tidy verifiers pyproject - drop the v0 reverse-text entry from the examples group - drop stale comments in [dependency-groups]/[tool.uv.sources] - drop the redundant [tool.ruff] line-length = 88 (88 is ruff's default) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: exclude v1 envs from the auto env-publish for now publish-envs.yml matrixes over environments/*; skip the *_v1 packages and the compact harness example — the v1 envs aren't ready for the Environments Hub yet. Classic v0 envs still publish. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: fix version-mismatch message to reference pyproject, not __init__.py The version is read from [project].version in pyproject.toml; the mismatch error still pointed at the old __init__.py source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

willccbb added 2 commits June 6, 2026 11:42

Draft v1 nano refactor

e0bdde4

Remove v1 Schema helper

122cc60

cursor Bot reviewed Jun 6, 2026

View reviewed changes

Comment thread environments/bfcl_v3_v1/bfcl_v3_v1/taskset.py

Comment thread ...onments/langchain_deep_agents_wikispeedia_v1/langchain_deep_agents_wikispeedia_v1/taskset.py Outdated

Add v1 server stubs to init template

be7f1d8

cursor Bot reviewed Jun 6, 2026

View reviewed changes

Comment thread environments/hello_subagent_v1/hello_subagent_v1/taskset.py Outdated

Comment thread environments/hello_subagent_v1/hello_subagent_v1.py Outdated

Comment thread environments/dspy_flights_v1/dspy_flights_v1/taskset.py Outdated

Migrate v1 environments to component packages

8f2f9f0

cursor Bot reviewed Jun 6, 2026

View reviewed changes

Comment thread environments/hello_rlm_v1/hello_rlm_v1/taskset.py Outdated

macroscopeapp Bot reviewed Jun 6, 2026

View reviewed changes

willccbb added 3 commits June 6, 2026 18:19

Finalize v1 nano toolset contract

061a155

checkpoint

6ceb3c7

checkpoint

a7b2dce

cursor Bot reviewed Jun 7, 2026

View reviewed changes

Comment thread environments/bfcl_v3_v1/pyproject.toml

checkpoint

d392bd9

cursor Bot reviewed Jun 7, 2026

View reviewed changes

Comment thread environments/bfcl_v3_v1/bfcl_v3_v1/taskset.py

Comment thread environments/alphabet_sort_v1/README.md Outdated

macroscopeapp Bot reviewed Jun 7, 2026

View reviewed changes

willccbb added 3 commits June 7, 2026 03:20

checkpoint

bae370c

checkpoint

8548a9b

Fix dynamic tool payload typing

d7fda73

macroscopeapp Bot reviewed Jun 7, 2026

View reviewed changes

Comment thread environments/tau2_bench_v1/tau2_bench_v1/servers/user/user.py

Comment thread verifiers/v1/env.py Outdated

Tighten v1 state runtime contracts

ff16cd0

cursor Bot reviewed Jun 7, 2026

View reviewed changes

Comment thread environments/bfcl_v3_v1/bfcl_v3_v1/taskset.py

macroscopeapp Bot reviewed Jun 7, 2026

View reviewed changes

Comment thread verifiers/v1/state.py Outdated

Comment thread docs/byo-harness.md Outdated

Comment thread verifiers/v1/env.py

Tighten v1 runtime and advantage defaults

b23ece4

macroscopeapp Bot reviewed Jun 7, 2026

View reviewed changes

Comment thread verifiers/v1/runtime.py

Comment thread docs/byo-harness.md

willccbb added 2 commits June 7, 2026 04:36

Fix v1 lifecycle review issues

b392de7

Fix Docker runtime binary reads

06277b8

cursor Bot reviewed Jun 7, 2026

View reviewed changes

Comment thread environments/alphabet_sort_v1/README.md Outdated

Fix v1 README env args

b4d896d

macroscopeapp Bot reviewed Jun 7, 2026

View reviewed changes

Comment thread verifiers/v1/state.py

Comment thread verifiers/v1/env.py Outdated

macroscopeapp Bot reviewed Jun 7, 2026

View reviewed changes

Comment thread environments/math_python_v1/README.md Outdated

Restore Qwen3 VL endpoint aliases

ccbc20a

Retarget v1 prime runtime CI smoke

b27ee8b

macroscopeapp Bot reviewed Jun 8, 2026

View reviewed changes

Comment thread verifiers/v1/runtime.py Outdated

Comment thread configs/endpoints.toml

Comment thread verifiers/v1/harness.py Outdated

Fix v1 environment smoke contracts

24e4093

macroscopeapp Bot reviewed Jun 8, 2026

View reviewed changes

Comment thread packages/tasksets/tasksets/nemo_gym.py

willccbb added 2 commits June 8, 2026 05:56

Fix v1 runtime timeout and env startup cleanup

510e09c

Normalize v1 runtime timeouts

930b843

macroscopeapp Bot reviewed Jun 8, 2026

View reviewed changes

Comment thread verifiers/v1/harness.py

willccbb added 3 commits June 8, 2026 06:27

Fix v1 env package smokes

3222e7b

Fix BFCL v1 task serialization

5c6a92f

Fix v1 env-scope cleanup contracts

5d1ef09

willccbb requested review from aniravich, cdreetz, eligotts, hallerite, hubert-marek, mikasenghaas, samsja and xeophon June 8, 2026 16:17

aniravich requested changes Jun 9, 2026

View reviewed changes

mikasenghaas mentioned this pull request Jun 11, 2026

chore: add v1 #1559-vs-#1576 comparison doc #1619

Merged

mikasenghaas added a commit that referenced this pull request Jun 11, 2026

docs: trim comparison (drop names, some #1559 items, validation section)

61ef454

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

jiviny mentioned this pull request Jun 12, 2026

feat: add daytona runtime #1663

Open

xeophon closed this Jul 13, 2026



		def parse_anthropic_messages(body: JsonData) -> Messages:
		messages: Messages = []

Uh oh!

Conversation

willccbb commented Jun 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Final Validation

Review State

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

macroscopeapp Bot commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Approvability

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

willccbb commented Jun 6, 2026 •

edited

Loading

macroscopeapp Bot commented Jun 8, 2026 •

edited

Loading