chore: final touches

[mikasenghaas]

Summary

Final-touches pass over the v1 docs, plus a small scoring feature, CI changes, and cleanup.

Docs (verifiers/v1/GUIDE.md, README.md)

• Major GUIDE pass: livelier scoring examples (enthusiasm metric, brevity length-penalty group reward); restructured the trace-reading reference into grouped tables and fixed it against the real Trace (dropped the nonexistent trace.messages, added usage/has_response/id); a dedicated Persisted info section contrasted with Per-rollout state; split the harness "Writing one" section into subsections; reframed in-runtime scoring around when to use the runtime object; "host" instead of "eval process".
• Reference-examples table completed and corrected (added general-agent-v1, swebench-verified-v1, tau2-bench-v1; fixed the tau2 description; merged the own-host/own-sandbox placement rows).
• Surfaced the NEEDS_CONTAINER taskset capability flag; noted a Toolset wraps an MCP server; documented the self-launching __main__ server pattern; listed the mini-swe-agent / kimi-code built-in harnesses; harness example configures via CLI args (not OPENAI_* env vars) and drops max_steps.

Scoring feature

• @vf.reward may now return float | dict[str, float] (a family of named contributions, each scaled by weight), mirroring @vf.metric. Existing float rewards are unaffected.

Harness configs

• Harness configs no longer pin id (the caller supplies it), mirroring tasksets; updated the base HarnessConfig docstring and the init scaffold.

CI

• Restored publish-tasksets.yml / publish-harnesses.yml (dropped on the v1 branch). v1 versions packages statically in pyproject.toml, so the version-detection reads [project].version from pyproject. Bumped both packages 0.1.0 → 0.2.0 (the branch had reset them below what's on PyPI — tasksets 0.1.5, harnesses 0.1.2).
• publish-envs.yml: excluded the v1 envs (the *_v1 packages + the compact harness) from the auto env-publish for now; classic v0 envs still publish.
• Renderer client docs (docs/training.md, docs/faqs.md): dropped the (experimental) flag.

Cleanup

• Removed the internal bench/ scripts and the transient COMPARE.md.
• Pyproject tidy: dropped the redundant [tool.ruff] line-length = 88 (88 is ruff's default) and stale dependency comments.
• Added a [tool.uv.sources] entry for the reverse-text v0 env (for the eval --id reverse-text legacy-bridge example).

Breaking

• Harness capability flag SUPPORTS_TASK_TOOLS → SUPPORTS_MCP. Renamed on the base Harness and every built-in/bundled harness, the Environment check, and the error message. Any external harness setting SUPPORTS_TASK_TOOLS must rename it to SUPPORTS_MCP.

---

Note

Medium Risk
The SUPPORTS_TASK_TOOLS→SUPPORTS_MCP rename breaks external harness subclasses; PyPI publish workflows on main can release packages when versions change.

Overview
This pass tightens v1 docs and release plumbing, extends scoring, and makes a breaking harness rename.

@vf.reward dict returns — Rewards can return dict[str, float] like @vf.metric; each entry is weighted and merged into trace.reward via Taskset.score.

Harness API — SUPPORTS_TASK_TOOLS is renamed to SUPPORTS_MCP everywhere (base Harness, built-ins, Environment load check). Harness config subclasses no longer pin id (caller supplies --harness.id); the init scaffold matches.

CI / publishing — New publish-harnesses.yml and publish-tasksets.yml auto-tag from pyproject.toml version on main, build with uv, and publish to PyPI (harnesses / tasksets bumped to 0.2.0). publish-envs.yml now skips *_v1 dirs and compact so only classic v0 envs hub-publish.

Docs — Large verifiers/v1/GUIDE.md refresh (trace tables, persisted info vs state, harness authoring, examples). renderer client is no longer labeled experimental in docs/training.md / docs/faqs.md. Minor README legacy-bridge flag tweak.

Cleanup — Removes internal bench/ scripts and COMPARE.md. Adds reverse-text to [tool.uv.sources] for the v0 bridge example; drops redundant ruff line-length and stale pyproject comments.

^{Reviewed by Cursor Bugbot for commit 2f977db. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add PyPI publish workflows for harnesses and tasksets packages and rename SUPPORTS_TASK_TOOLS to SUPPORTS_MCP

• Adds CI workflows for the harnesses and tasksets packages that auto-tag on pushes to main, build with uv, and publish to PyPI via pypa/gh-action-pypi-publish. Both packages are bumped to 0.2.0.
• Renames the harness capability flag SUPPORTS_TASK_TOOLS to SUPPORTS_MCP across the base class, all harness implementations, environment code, and documentation.
• Filters the publish-envs workflow matrix to exclude *_v1 and compact environments, limiting it to classic v0 environments.
• Removes the bench/ directory and COMPARE.md, removes the (experimental) qualifier from renderer client type in docs, and adds reverse-text as an editable uv source.

^{Macroscope summarized 2f977db.}

[macroscopeapp]

🟢 Low workflows/publish-harnesses.yml:12

The push event includes a tags filter for harnesses-v* (lines 13-14), so when auto-tag-on-main pushes a new tag (line 65), the tag push triggers a second workflow run. That second run executes build-tag → publish-tag and attempts to upload the same version to PyPI again, which fails with a 409 Conflict because PyPI rejects duplicate uploads. Consider removing the tags filter from the push event so tag pushes do not start a redundant run.

      - main
-    tags:
-      - "harnesses-v*"

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @.github/workflows/publish-harnesses.yml around lines 12-14:

The `push` event includes a `tags` filter for `harnesses-v*` (lines 13-14), so when `auto-tag-on-main` pushes a new tag (line 65), the tag push triggers a second workflow run. That second run executes `build-tag` → `publish-tag` and attempts to upload the same version to PyPI again, which fails with a 409 Conflict because PyPI rejects duplicate uploads. Consider removing the `tags` filter from the `push` event so tag pushes do not start a redundant run.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 2f977db. Configure here.}

[cursor]

Duplicate PyPI publish on release

Medium Severity

When auto-tag-on-main creates and pushes a release tag, the same workflow run publishes to PyPI via publish-from-auto-tag, and the tag push starts a second run whose publish-tag job uploads the same version again. PyPI typically rejects the duplicate, so the tag-triggered workflow fails after a successful release.

Additional Locations (2)

• .github/workflows/publish-tasksets.yml#L64-L115
• .github/workflows/publish-harnesses.yml#L195-L210

 

^{Triggered by project rule: BugBot Instructions}

^{Reviewed by Cursor Bugbot for commit 2f977db. Configure here.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR introduces new PyPI publishing workflows with unresolved review comments identifying a bug that will cause duplicate publish failures. New deployment infrastructure combined with an unfixed issue in that infrastructure warrants human attention.

^{You can customize Macroscope's approvability policy. Learn more.}