feat(evals): shared evals package and an eval for T3 tool-use mistakes

.github/scripts/ci-changes.cjs

@@ -32,6 +32,8 @@ const RUST = /^native\/(resource-monitor|kde-snap-shot|hyprland-snap-shot)\//;
 
 // These tests read other packages without declaring dependencies.
 const TEST_ONLY_DEPS = {
+  // Evals import the server's source to test its real model-facing text.
+  "@t3tools/evals": ["t3"],
   "@t3tools/provider-pi": ["@t3tools/client-runtime"],
   "@t3tools/scripts": [
     "@t3tools/desktop",

AGENTS.md

@@ -136,6 +136,10 @@ An empty database is a bad test. Seed your worktree's `.t3` with a copy of real
 
 For authorized mobile verification, a missing or outdated native client is an install step, not a blocker. Run `vp run mobile:install <ios|android> [device-id]` on the device host before starting Metro; it targets the only booted simulator or connected Android device when the id is omitted. It installs the dev client for this checkout's native fingerprint from the shared cache in seconds and builds only when no machine has built it yet, so prefer it over `expo run:*`, `xcodebuild`, or `gradlew`, which rebuild from scratch. Its stdout is one JSON line; build output goes to the log it names. See `test-t3-mobile` for the full workflow.
 
+## Evals
+
+`evals/` checks how real models behave with T3 Code's model-facing text: agent instructions, MCP tool descriptions and schemas, and generated titles. Before you change that text, read [evals/README.md](evals/README.md) and run the evals it names. When you fix a model behavior bug, add a case that fails before the fix.
+
 ## Pull requests
 
 - Never make a PR unless the developer explicitly asks you to do so.

apps/server/src/mcp/OrchestratorMcpService.ts

@@ -809,7 +809,7 @@ function threadTitle(input: {
   return detail.length > 80 ? `${detail.slice(0, 77)}...` : detail;
 }
 
-function taskPrompt(input: OrchestratorMcpDelegateTaskInput): string {
+export function taskPrompt(input: OrchestratorMcpDelegateTaskInput): string {
   return input.role === undefined || input.role === "general"
     ? input.task
     : `Act as the ${input.role} sub-agent for this task.\n\n${input.task}`;

apps/server/src/orchestration-v2/PullRequestSyncReactor.ts

@@ -139,7 +139,7 @@ type CommandExecution = Extract<OrchestrationV2TurnItem, { readonly type: "comma
  * does, existed before it ran. More URLs than create commands means something else printed
  * some, such as `gh pr list`, so none are taken.
  */
-function changeRequestsCreatedBy(input: string, output: string) {
+export function changeRequestsCreatedBy(input: string, output: string) {
   if (!CREATE_COMMANDS.some(({ command }) => command.test(input))) return [];
   const script = shellScript(input);
   const code = shellCode(script);

evals/.gitignore

@@ -0,0 +1 @@
+runs/

evals/README.md

@@ -0,0 +1,52 @@
+# Evals
+
+An eval runs a real model against T3 Code's real text or code and scores what the model does. Use one where a unit test cannot help: model output changes from run to run, and a small copy change can move it.
+
+Write or run an eval when a change affects what a model sees or does: agent instructions, MCP tool descriptions and schemas, tool results, generated titles, or the guidance an agent gets for a feature such as Device Manager.
+
+| Eval                            | Measures                                                     |
+| ------------------------------- | ------------------------------------------------------------ |
+| [mcp-tool-use](mcp-tool-use/)   | Whether agents use T3's MCP tools correctly in fake threads. |
+| [thread-titles](thread-titles/) | Whether generated thread titles name the right subject.      |
+
+## Run an eval
+
+1. Run `vp i` at the repo root.
+2. Put the API keys the eval needs in the repo root `.env`. It is gitignored, and the eval scripts load it. Never print or commit a key.
+3. From `evals/`, run `pnpm <eval> --out runs/<name>`. `runs/` is gitignored.
+
+Each eval's README lists its models, flags, and cost. Start with one case and one sample to check the setup.
+
+Evals call paid models, so CI does not run them. CI typechecks this package and runs the graders' unit tests.
+
+## The shape of an eval
+
+Every eval follows the same shape, so a new one is easy to write and an old one is easy to run.
+
+- `evals/<eval>/run.ts` is the entry point. `package.json` has a script with the eval's name.
+- `evals/<eval>/README.md` says what the eval measures, how to run it, how to add a case, and what a run costs.
+- Cases are data, in `cases/` or `cases.ts`. Each case has an `id`, says where it came from (a PR, issue, or thread), and has its checks or rubric.
+- The eval imports the code under test from source. Never copy prompts, tool descriptions, or schemas into an eval. Then the eval always tests the current text.
+- Flags: `--out <dir>` (required), `--model <id>`, and `--baseline <earlier results>` to compare. Add `--case <ids>` when a full run is slow or costly.
+- A run writes results rows to `--out`, plus a summary or review file for a person.
+- When the eval grades itself, `run.test.ts` tests the grader, which is plain logic. It runs in CI. When a person grades, the run hides which output is new.
+
+## Rules for cases
+
+- A case reproduces a real bad experience, or guards a real good one. Name its source.
+- Add the case before the fix. It must fail on the old text and pass on the new text.
+- If you change a case, the changed case must still fail on the text it was written against. Prefer to fix T3 Code over loosening a case.
+- Add controls: cases where the correct action is ordinary work. They catch a fix that overcorrects.
+- Model output varies. Run each case several times. A case fails when half or more of its samples fail. Compare against a baseline on the same model, not against an absolute score.
+- No user data in cases, fixtures, or templates. Use `/home/user` and `acme/app`. Read every captured fixture before you commit it.
+- No unreleased model details in committed files, except model ids in an eval's model table.
+
+## Cost
+
+Run only the cases your change touches until they pass. Then do one full run per model before the PR merges. Each eval's summary records what the run cost.
+
+## Add an eval
+
+1. Copy the shape above from the closest eval.
+2. Add a script to `package.json`, a README, and a row to the table above.
+3. When a second eval needs code that one eval already has (model clients, the shell simulator, sampling, summaries), move that code to `evals/shared/` and use it from both. Do not build shared code before a second eval needs it.

evals/mcp-tool-use/README.md

@@ -0,0 +1,47 @@
+# MCP tool use
+
+Replays small fake threads on Claude and Codex models and checks how they use T3's MCP tools: delegation, PR linking, previews, scheduling, and more. Each case reproduces a real mistake an agent made in T3 Code.
+
+The model sees what it sees in T3 Code. A trimmed capture of a real harness turn supplies the harness prompt and native tools. T3's instructions, tool descriptions, and schemas come from this checkout. A cheap model plays the shell, so nothing runs on your machine.
+
+## Run
+
+```sh
+cd evals
+pnpm mcp-tool-use --case review-round-uses-thread-send --samples 1 --out runs/check
+pnpm mcp-tool-use --model claude-opus-5-5 --out runs/opus
+```
+
+- `ANTHROPIC_API_KEY` in the repo root `.env` pays for Claude models and the Haiku shell simulator.
+- Codex models (`gpt-*` in `MODELS` in `run.ts`) go through a CLIProxyAPI server, set up as the `vibeproxy` model provider in `~/.codex/config.toml`. Keep the proxy under about 12 samples in flight across all runs.
+- Flags: `--case <ids>`, `--cases <file.jsonl>`, `--samples <n>` (default 2, and a 3rd when the first 2 disagree), `--concurrency <n>` (samples in flight, default 8), `--max-steps <n>`, `--effort <level>`, `--dry-run` (prints request bodies, no model calls), and `--context` (prints the size of T3's text and schemas, no key needed).
+- `--baseline <results.jsonl> --stop-on-regression` stops as soon as a case that passed every baseline sample fails. Use it to test one copy change against a baseline run.
+- Cost: a full Sonnet run is about $9, and a full Opus run about $17. Codex models through the proxy cost only the shell simulator.
+
+## Add a case
+
+Cases are JSON lines in `cases/`, one file per area. The case schema in `run.ts` (`McpToolUseCase`) documents every field.
+
+- `kind`: `bad` reproduces a mistake, `guard` protects correct behavior, `control` is ordinary work.
+- `experience`: the real mistake, in a sentence. `source`: where it happened.
+- `setup`: the fake thread, its `world` (repo, branch, files, PRs, pages), `priorTurns`, and `mocks` for tool results.
+- `expect`: the checks, such as `calls`, `callsMustNot`, `maxCalls`, `order`, `textMust`, and `linked`.
+
+Check that the case builds with `--dry-run --case <id>`. Then run it on the text before your fix, where it must fail, and on the text after, where it must pass.
+
+## Where T3's text lives
+
+- `packages/provider-core/src/server/runtimeInstructions.ts`
+- `packages/provider-core/src/server/orchestrationInstructions.ts`
+- `apps/server/src/provider/CodexDeveloperInstructions.ts`
+- `apps/server/src/mcp/toolkits/*/tools.ts` (tool descriptions)
+- `packages/contracts/src/*Mcp.ts` (parameter descriptions and result text)
+
+## Templates
+
+`templates/` holds one trimmed capture per harness. Refresh them when a harness changes its prompt or native tools:
+
+1. Capture one T3 Code turn through CLIProxyAPI, with T3's MCP server connected.
+2. From `evals/`, run `pnpm mcp-tool-use:trim-capture <claude|codex> <capture.json> mcp-tool-use/templates/<claude|codex>.json.gz`. It keeps only what `run.ts` reads, and removes user instructions, the user's own skills and agents, account ids, and home paths.
+3. Read the output for user data before you commit it: `zcat <file> | less`.
+4. Run the full eval on the old and new templates, and compare.