fix(tbench2_env): bring Docker mode up to the canonical scoring contract

[nblintao]

Summary

#965 / #972 landed the canonical Terminal-Bench-2 contract in local mode (Tbench2Environment): scoring via the task's own tests/test.sh (pinned toolchain, /logs/verifier/reward.txt verdict), the task image's real WORKDIR, and verifier assets kept out of the agent's reach. Docker mode (Tbench2DockerEnvironment) only received the verifier-asset withholding half of #972 — its evaluate still ran bare pytest tests/ from /task (the task-source copy), skipping the canonical harness and the image's real WORKDIR. This PR closes that gap so both execution modes speak one scoring contract, and removes two silent-degradation paths found along the way.

Changes

• Canonical evaluate in Docker mode: stage tests/ at the official fixed path (/tests) for exactly the verify window (pre-wipe fail-closed, post-wipe of both /tests and /logs/verifier, as before), run tests/test.sh from the agent's workdir bounded by the task's own verifier budget (task.toml [verifier].timeout_sec, via coreutils timeout when present in the image), read the verdict from /logs/verifier/reward.txt. Task dirs without test.sh fall back to pytest, mirroring local mode's _evaluate_fallback. Command construction and marker parsing are factored into module helpers shared with local mode.
• Real task workdir: agent exec commands and scoring run in the task image's own WORKDIR, resolved from image metadata (Config.WorkingDir, which sees base-image WORKDIRs) with the task Dockerfile parse as fallback, then /task.
• exec quoting fix: commands now ride as a bash -c argv element instead of being spliced into a single-quoted shell string, which mangled any agent command containing a single quote.
• No host-execution fallback: a task dir that declares no [environment] docker_image is rejected at reset. The old silent fallback ran agent commands directly on the env-server host — a containment hole (arbitrary agent shell outside any container) and a scoring-fidelity hole at once. The containerless code paths (host subprocess exec, docker-class _evaluate_local) are removed with it.
• TB2_MODE=auto removed: it claimed to auto-detect Docker but unconditionally served local mode; unknown modes now fail at startup instead of silently serving local.

Who is affected?

Local mode — the default, and what HF Spaces run — is completely unchanged. The refactor generates byte-identical commands there, and the existing local-mode tests pass without modification. If you never set TB2_MODE=docker, you can stop reading.

For TB2_MODE=docker operators, the intended effect is simply: official TB2 tasks now score the official way (previously: bare pytest in the wrong directory, without the task's pinned toolchain — i.e. the old rewards were wrong). Scores will move; that's the fix working.

The only setups that need a change on their side:

• Custom task dirs whose tests hard-code the old layout (cwd /task, tests at /task/tests) and whose image declares a WORKDIR. Images without a WORKDIR keep /task as cwd, so most custom setups are untouched.
• Tasks with no docker_image in task.toml: these used to silently run agent commands directly on the server host — outside any container. That's a containment hole, so reset now rejects them with an error pointing at TB2_MODE=local. (If someone has a real use case for the old behavior, happy to gate it behind an explicit opt-in env var instead.)
• TB2_MODE=auto: never actually auto-detected anything (it silently ran local mode), so it now errors at startup; set local explicitly to keep the exact same behavior.

Nothing here can fail silently: every changed path either scores the official way or refuses loudly with an actionable message. One cosmetic note for downstream parsers: docker-mode evaluate now reports info["harness"] on the canonical path (same shape as local mode) instead of info["exit_code"]; the pytest-fallback path keeps exit_code.

Validation

• Unit tests (tests/envs/test_tbench2_env.py): 23 passed. Existing docker-mode tests updated to the new contract; new coverage for the pytest fallback without test.sh, workdir resolution precedence (image metadata → Dockerfile → /task), imageless reset rejection, and argv-form exec (single quotes survive byte-identical).
• Real-container golden run on an x86 docker host, driving a live TB2_MODE=docker server from this branch with a plain OpenEnv client (reset → stage the task's official solution/solve.sh → exec it → standard evaluate):
  • fix-git: reward 1.0, info.harness=tests/test.sh, and the in-container pwd probe returned /app/personal-site — the task's non-trivial image WORKDIR, resolved correctly (the old code would have run everything in /task).
  • chess-best-move: reward 1.0.
  • Negative control (no solution staged, straight to evaluate): reward 0.0 — the canonical harness genuinely ran and failed the tests.
  • Imageless task dir: reset raised RuntimeError: task imageless-task declares no [environment] docker_image … as intended.

🤖 Generated with Claude Code

---

Note

Medium Risk
Docker-mode rewards and cwd change for official tasks (intended fix); breaking for imageless tasks, TB2_MODE=auto, and any setup that relied on host fallback or old pytest-in-/task scoring.

Overview
Docker mode now matches local mode for scoring and isolation: evaluate stages tests at /tests, runs the task's tests/test.sh (verifier timeout via timeout when available), and reads /logs/verifier/reward.txt instead of bare pytest under /task. Agent exec and scoring use the image's resolved WORKDIR (image config → Dockerfile → /task).

Shared helpers (_canonical_eval_cmd, reward/exit-code parsing) dedupe local and Docker paths; local behavior is intended to stay byte-identical.

Fail-closed behavior: tasks without docker_image error on reset (host subprocess fallback removed). TB2_MODE=auto and unknown modes error at startup. Container exec uses argv-form bash -c so agent commands with quotes are not mangled. Failed resets close() the prior container so stale sessions cannot mix tasks.

Docs note the stricter Docker contract. Unit tests cover canonical evaluate, workdir, imageless reset, and exec quoting.

^{Reviewed by Cursor Bugbot for commit 4180b7c. Bugbot is set up for automated code reviews on this repo. Configure here.}

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[cursor]

Docker fallback scoring lacks timeout

Medium Severity

Docker mode reads verifier_timeout_s for every evaluate, but only passes it into _canonical_eval_cmd. The pytest fallback via _fallback_eval_cmd has no in-shell timeout, and Docker exec has no server-side budget either, so a hung custom-task pytest can block the session indefinitely. Local mode still bounds that path through the terminal toolkit.

Additional Locations (1)

• envs/tbench2_env/server/tbench2_env_environment.py#L214-L228

 

^{Reviewed by Cursor Bugbot for commit 31d0fa6. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 4180b7c. Configure here.}

[cursor]

Rejected reset leaves stale state

Low Severity

Imageless reset calls close() then raises, but close() clears the container and _task_dir without resetting _state or _task_image. After a rejected reset, /state can still report the previous episode as ready even though step will fail because no task is initialized.

Additional Locations (1)

• envs/tbench2_env/server/tbench2_env_environment.py#L1067-L1078

 

^{Reviewed by Cursor Bugbot for commit 4180b7c. Configure here.}

[burtenshaw]

LGTM. I handled the left over container issue.