feat: add textarena taskset with a framework-driven user simulator

[mikasenghaas]

Summary

• Add vf.User — a first-class user simulator that is structurally a tool server (vf.Tools: an MCP server with a runtime), registered on a taskset via the new Taskset.user(task) hook (parallel to Taskset.tools).
• Drive the user simulator from the framework. The per-rollout interception server, after each model turn with no tool call, calls the simulator's respond tool, injects its reply as a user turn, and re-prompts the model — so a whole multi-turn game plays out within a single program request, transparently to the harness and its program.py (both unchanged; they never see the simulator). With no user simulator the interception loop runs exactly once, as before.
• Ship textarena-v1 in the tasksets package: single-player TextArena games via a generic, seed-based ta.Env → tasks conversion. game is a Literal of the verified games: Wordle-v0, Wordle-v0-long, Hangman-v0, WordLadder-v0, WordSearch-v0.
• Game-authoritative scoring: when the episode ends, the simulator writes the game's own outcome (env.state.rewards) to a file in the runtime, and one generic @reward reads it back — no per-game guess parsing. (Win = 1.0; many games also report partial credit.)
• Generic conversion: each task is an RNG seed (carried in the task's info dict). load_tasks seeds the game to build the instruction (per-seed for games whose prompt embeds the setup — WordLadder's start/target, WordSearch's grid) and the simulator re-seeds to the same episode. No per-game word-list or state-key knowledge, so any single-player TextArena game fits. A num_tasks config sets how many seeds to generate.
• Add two pinned example tasksets (each a ~10-line subclass that fixes one field and reuses everything else, showing how to pin a built-in taskset): wordle-v1 (textarena_v1 with game pinned to Wordle-v0) and terminal-bench-2-v1 (harbor with dataset pinned to terminal-bench/terminal-bench-2).
• Rename the tool surface (see Breaking). textarena + nltk are an optional textarena extra on the tasksets package.

Breaking

• verifiers.v1.ToolServer → verifiers.v1.Tools (vf.ToolServer(...) → vf.Tools(...)); shipped example tasksets migrated.
• verifiers.v1.tools.serve_mcp → serve_tools.
• Taskset.tool_servers(task) → Taskset.tools(task); new sibling Taskset.user(task) returns the optional user simulator.

How the user simulator works

vf.User is consumed by verifiers/v1/interception.py, not the harness:

• Rollout serves the user server (colocated, like a tool server) and hands the interception server an async respond(message) -> (messages, done) (verifiers/v1/user.py).
• InterceptionServer.handle_chat loops: model turn → record a Turn → if the model made a tool call or there is no simulator, return to the program; otherwise call respond, append the user message(s) to the prompt, and re-prompt. The loop respects the framework's RolloutLimits (turns / token budget) and @stops; done ends it.
• The game's final outcome travels out-of-band via the runtime file, so it survives even though the terminal user turn isn't a model call.

Verification

uv run eval @ configs/textarena.toml (subprocess runtime, deepseek/deepseek-v4-flash), on this branch rebased onto feat/nano-as-v1:

• Wordle-v0, Wordle-v0-long, Hangman-v0, WordLadder-v0 reward 1.0; WordSearch-v0 solves within the turn budget (harder; longer episodes). Hangman shows both a win (1.0) and a loss (0.0), confirming game-authoritative scoring reads wins and losses. err 0.00, stop=user_completed.
• Non-regression: gsm8k-v1 (no user simulator) runs unchanged.
• The Tools/serve_tools rename is smoke-verified across every tool-server placement (each reward 1.0): glossary-v1 (colocated), wikispeedia-v1 (own runtime), wiki-search-v1 (shared), deepwiki-v1 (remote URL).
• ruff check + ruff format --check clean repo-wide; uv run pytest -k "v1 or interception" passes.

Notes

• The user simulator runs colocated (host-reachable for the subprocess/docker runtimes), so the example pins the subprocess runtime; a prime colocated simulator would need a public URL.
• game is gated to the tested set. Out of scope: WordChains (needs 2 players), games with no word_list (GuessTheNumber, Crosswords), or LLM-gamemaster games (TwentyQuestions, Taboo, …).
• The Wordle/Hangman -hardcore variants are intentionally excluded: their lists include capitalized proper nouns that are unwinnable under random seeding (TextArena lowercases the guess, not the stored secret).

---

Note

Medium Risk
Breaking public v1 API renames affect all tasksets using tools; interception’s new multi-turn loop changes rollout behavior for user-simulator tasksets but leaves non-simulator paths as a single iteration.

Overview
Introduces vf.User and Taskset.user(task) so tasksets can register an MCP-backed user simulator the framework drives (not the model). InterceptionServer.handle_chat now loops: after each assistant turn without tool calls, it calls the simulator’s respond, injects user messages, and re-prompts until done or limits/@stop fire—so multi-turn games run inside one harness chat completion while the trace still records full assistant/user turns.

Breaking renames: ToolServer → Tools, serve_mcp → serve_tools, Taskset.tool_servers → Taskset.tools; example tasksets updated.

Ships textarena-v1: seed-based TextArena episodes via a colocated user server (textarena_v1/server.py), with rewards read from a runtime outcome file written when the game ends. Optional tasksets[textarena] extra pins textarena/nltk. Adds thin wordle-v1 and terminal-bench-2-v1 example plugins plus eval configs (configs/textarena.toml, wordle.toml, terminal-bench-2.toml).

^{Reviewed by Cursor Bugbot for commit ff76d31. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add TextArena taskset with a framework-driven user simulator for multi-turn game rollouts

• Adds textarena_v1 taskset that runs TextArena games (e.g. Wordle-v0) as seeded episodes, scoring via game outcomes written by a colocated user simulator.
• Adds textarena_v1/server.py, a FastMCP server exposing a respond tool that steps the game and signals completion via an outcome file.
• Extends InterceptionServer to support multi-turn conversations: it loops model calls and invokes the user simulator until done or the model requests tools.
• Adds verifiers/v1/user.py with User, Respond, connect_user, and serve_user — first-class MCP-based user simulator support in the rollout framework.
• Renames ToolServer → Tools, tool_servers() → tools(), and serve_mcp() → serve_tools() across the public API; adds optional user() hook to the Taskset base class.
• Risk: ToolServer is no longer exported from verifiers.v1; any external taskset importing it will break.

^{Macroscope summarized ff76d31.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR introduces a new user simulator framework with significant changes to the core interception server loop, new abstractions (User class), and TextArena game integration. The scope and runtime behavior changes warrant human review.

^{You can customize Macroscope's approvability policy. Learn more.}

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit ff76d31. Configure here.}

[cursor]

Outcome read errors mishandled

Medium Severity

game_reward treats a missing textarena_outcome.json as score 0.0 only when runtime.read raises FileNotFoundError or OSError. Docker and Prime runtimes raise ProgramError on a failed read, so an unfinished episode can fail scoring instead of returning 0.0.

 

^{Reviewed by Cursor Bugbot for commit ff76d31. Configure here.}

[cursor]

Missing v1 API documentation

Low Severity

This PR adds and exports vf.User, Taskset.user, and the Tools / serve_tools rename as core v1 APIs, but no updates appear in the user-facing docs under docs/ (for example docs/environments.md, docs/reference.md, or docs/faqs.md) describing the user simulator or the breaking tool-server rename.

Additional Locations (1)

• verifiers/v1/taskset.py#L84-L91

 

^{Triggered by project rule: BugBot Instructions}

^{Reviewed by Cursor Bugbot for commit ff76d31. Configure here.}