feat: add scaleswe v1 taskset + per-task setup/workdir hooks

[mikasenghaas]

Summary

Adds the scaleswe-v1 taskset (a v1 port of the v0 ComposableEnv Scale-SWE taskset) plus three small, general framework hooks that agentic/SWE tasksets need.

scaleswe-v1 (examples/tasksets/scaleswe_v1)

Scale-SWE (AweAI-Team/Scale-SWE) rows ship a per-task Docker image with the repo, pre_commands that reset it to the base commit on a clean scaleswe branch, F2P/P2P pytest ids, and an optional f2p_patch / f2p_script carrying the failing test.

• Each row → a ScaleSWETask carrying its image + workdir and the SWE metadata.
• setup(task, runtime) runs the row's pre_commands in the live runtime before the agent.
• The single solved reward restores the test files to base (the agent only fixes the source), applies the f2p test, and runs the merged F2P+P2P ids through a self-contained score.py (run by the testbed python) — 1.0 iff every expected id passes. Host-side reward stays tiny; the pytest run + JUnit matching live in score.py.
• Dataset hardcoded (AweAI-Team/Scale-SWE, train); the row's image_url is used as-is (the prime sandbox pulls the raw Docker Hub image — no registry prefix needed).

Framework hooks (general)

• Task.workdir — injected into the runtime config's workdir (symmetric with Task.image), so the agent and scoring run in the row's repo dir, not the default /app.
• Taskset.setup(task, runtime) — no-op by default; run by the rollout after runtime.start() and before the harness, for per-task runtime prep.
• Taskset.NEEDS_CONTAINER — ClassVar; the Environment refuses the subprocess runtime for a taskset that sets it (scaleswe-v1 sets it True).

Validation

100-rollout eval on glm-5.1 (rlm harness, prime runtime, uncapped):

• 49/100 solved (49%), avg_reward=0.490; 51.6% among cleanly-completed (5 rollouts hit rlm/sandbox infra failures — harness exited 1 / prime exec failed — none in the taskset or scorer).
• Confirms per-task image+workdir injection, setup() pre_commands, and the f2p/P2P pytest scorer all produce correct 0/1 rewards at scale.
• ruff check --isolated / ruff format --isolated --check clean.

---

Note

^{Cursor Bugbot is generating a summary for commit 46da6d1. Configure here.}

Note

Add ScaleSWE v1 taskset with per-task setup and workdir hooks

• Adds a new scaleswe_v1 taskset that loads tasks from the AweAI-Team/Scale-SWE dataset, runs pre-commands to prepare the repo, applies optional fail-to-pass patches/scripts, and scores results via pytest JUnit XML output.
• Adds a score.py CLI scorer that normalizes test IDs, parses JUnit XML, and emits 1.0 or 0.0 based on whether all expected tests passed.
• Extends Taskset with a NEEDS_CONTAINER class variable and an async setup hook; Rollout.run now calls setup before starting tools.
• Extends Task with an optional workdir field; Environment.__init__ propagates it to the runtime config when supported.
• Risk: Environment now raises ValueError at construction time if a container-required taskset is paired with a SubprocessConfig runtime.

^{Macroscope summarized 46da6d1.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR introduces new v1 framework features (per-task setup hooks, workdir injection, container validation) and a substantial new taskset. These are new capabilities affecting runtime behavior, not minor changes. Additionally, two unresolved review comments identify potential logic issues in the taskset implementation.

^{You can customize Macroscope's approvability policy. Learn more.}

[cursor]

Cursor Bugbot has reviewed your changes and found 3 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 6818779. Configure here.}

[cursor]

Missing pre_commands validation

Medium Severity

setup runs pre_commands without checking they are non-empty after normalization, so a row with missing or whitespace-only pre_commands still passes setup. The v0 Scale-SWE taskset raises in that case; here the repo may never be reset before the agent runs.

 

^{Reviewed by Cursor Bugbot for commit 6818779. Configure here.}

[cursor]

Missing base commit check

Medium Severity

The solved reward runs the RESTORE script with an empty base when base_commit is missing, instead of failing fast. The v0 taskset rejects rows without parent_commit/base_commit before restoring tests.

 

^{Reviewed by Cursor Bugbot for commit 6818779. Configure here.}

[cursor]

Undocumented v1 framework hooks

Medium Severity

This PR adds user-facing v1 APIs (Task.workdir, Taskset.setup, Taskset.NEEDS_CONTAINER, and related env/rollout wiring) but does not update the project docs that describe v1 authoring and evaluation. Authors need that behavior documented to use SWE-style tasksets correctly.

Additional Locations (2)

• verifiers/v1/taskset.py#L92-L121
• verifiers/v1/env.py#L144-L183

 

^{Triggered by project rule: BugBot Instructions}

^{Reviewed by Cursor Bugbot for commit 6818779. Configure here.}