feat: add alphabet-sort-v1 taskset

[mikasenghaas]

Summary

• Port the v0 alphabet-sort environment to a v1 taskset under examples/tasksets/alphabet_sort_v1/ (a package: __init__.py + user.py, like the other example tasksets).
• Task generation, prompts, and scoring are kept identical to v0 — the two environments differ only at the harness boundary.
• The multi-turn follow-ups (v0's MultiTurnEnv.env_response) are colocated with the agent as a vf.User (user.py): after each assistant turn the interception server drives the simulator, which replays the next pre-generated follow-up as a user turn and signals done after the last turn.
• The reward grades each turn's <alphabetical_sorted> / <combined_alphabetical_sorted> content against the per-turn ground truth with the same difflib-similarity, power-scaling, and strictly-improving multi-attempt logic as v0.
• AlphabetSortConfig exposes min_turns / max_turns / min_names_per_turn / max_names_per_turn / similarity_power / power_per_turn / split (the source dataset and seed are fixed module constants).
• Register alphabet-sort-v1 in pyproject.toml (examples group + [tool.uv.sources]) and add configs/alphabet_sort.toml.
• Drive-by: drop the redundant enable_bash = false from the user-simulator eval configs (alphabet_sort, wordle, textarena) — it's already the harness default.

Verification

Identity vs the v0 env (environments/alphabet_sort):

• Task generation + prompts: instruction, follow-ups, ground truths, and turn counts are byte-identical across all 37,853 generated episodes (0 mismatches).
• Scoring: the v1 reward (called directly) vs v0's actual rubric — 0 diffs across thousands of checks (perfect / partial / no-tag / multi-attempt improving & non-improving, both power_per_turn modes).

Smoke test:

uv run eval @ configs/alphabet_sort.toml -m qwen/qwen3-30b-a3b-instruct-2507

All rollouts user_completed, 2 turns each, no errors, rewards 0.15–1.0. Trace confirms user → assistant <alphabetical_sorted> → [user-sim follow-up] → assistant <combined_alphabetical_sorted> → user_completed.

🤖 Generated with Claude Code

---

Note

Low Risk
New example taskset and config-only harness tweaks; no changes to core auth, harness defaults, or existing task behavior beyond removing redundant config keys.

Overview
Adds alphabet-sort-v1, a v1 port of the multi-turn alphabetical name-sorting eval: episodes are built from a Hugging Face author-names dataset, graded with the same tagged-output difflib reward as v0, and follow-up user turns are replayed by a colocated vf.User MCP subprocess instead of harness-side env responses.

Wires the plugin into pyproject.toml (examples group + uv source) and adds configs/alphabet_sort.toml for subprocess evals with fixed 2-turn episodes.

Removes redundant enable_bash = false from the wordle, textarena, and alphabet_sort harness configs because the default harness already disables bash.

^{Reviewed by Cursor Bugbot for commit 308da10. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add alphabet-sort-v1 multi-turn sorting taskset

• Adds a new taskset in examples/tasksets/alphabet_sort_v1/ that generates multi-turn episodes where a model must alphabetically sort lists of names, with follow-up turns adding new names to a cumulative list.
• Episode generation is seeded and configurable via AlphabetSortConfig (turn counts, names per turn, similarity power scaling); ground truths include ' // new name!' annotations after the first turn.
• Reward scoring uses difflib.SequenceMatcher ratio against ground truth per turn, with strict improvement required across multiple attempts and optional per-turn power scaling.
• A FastMCP-based user-simulator subprocess (alphabet_sort_v1.user) drives follow-up turns by reading pre-generated episode data from the ALPHABET_SORT_INFO environment variable.
• Registers the package as the alphabet-sort-v1 extra in pyproject.toml and adds a run config at configs/alphabet_sort.toml (5 tasks, 2 rollouts, max 4 turns, bounded to 2 turns per episode).

^{Macroscope summarized 308da10.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 308da10. Configure here.}

[cursor]

Lockfile missing new package

Medium Severity

The root pyproject.toml registers alphabet-sort-v1 in the examples dependency group and [tool.uv.sources], but uv.lock is not updated to include that workspace package. A releasable merge should keep the lockfile aligned with dependency changes so uv sync and CI use a consistent resolved graph.

 

^{Triggered by project rule: BugBot Instructions}

^{Reviewed by Cursor Bugbot for commit 308da10. Configure here.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR introduces a complete new taskset with substantial new logic (~230 lines) including custom reward functions, multi-turn episode generation, and an MCP-based user simulator. New features introducing new components warrant human review even when well-scoped.

^{You can customize Macroscope's approvability policy. Learn more.}