[teamleaderleo]

Leo-ask: ask-c344b7792d

Leo asked for agents to read things earlier and concurrently while multitasking, and to investigate whether the prompting could be improved. This adds a fixed paired prompt-behavior evaluator, plus call/turn and parallel metrics in the existing tool evaluator.

The standalone evaluator uses a captured Claude Code system prompt with its native Read/Bash schemas and deterministic repository fixtures. Baseline and candidate use the same three tasks and sample count with alternating arm order. It records prompt/fixture hashes, full transcripts, success, wall time, first-read latency, tokens, calls per tool turn, parallel share and background starts. The candidate is the compact asking rule plus the canonical early/concurrent-work guidance. No generated shell commands execute. This is synthetic prompt-behavior coverage, not DEV app or Codex verification.

Validation: Bun bundles all three changed modules; git diff --check and cmux-next prepush gate pass.

Paired exact-head evaluator receipt (local lane host, gateway-reachable): commit fd751732688cba73b423d31c6db7fe81c5350b88; ANTHROPIC_BASE_URL=http://sr2-gateway.tail137216.ts.net:4016 ANTHROPIC_AUTH_TOKEN=subrouter node tests/tool-evals/prompt-behavior.mjs --samples 3; baseline 9/9, guidance 9/9. Baseline medians: wall 7564 ms, first read 2734 ms, tool calls 4, tool turns 2, parallel share 0.5, tokens 13689. Guidance medians: wall 3247 ms, first read 1839 ms, tool calls 3, tool turns 1, parallel share 1, tokens 9084. The paired run is recorded at /home/cmux/lane-scratch/next-system-prompt-2/eval-local/log.

The fleet fast-check run 41794283465 could not reach sr2-gateway because its Linux workers are outside the tailnet (fetch failed); this is an infrastructure route limitation, not an evaluator or code failure. The same exact-head evaluator completed locally through the lane host's gateway route.

Metric	Before	After	Delta	Harness
Always-loaded asking/guidance text, estimated tokens	198	174	-24 (-12%)	Source chars / 4; paired manifest records exact text and method

Risk: evaluator-only. Remove prompt-behavior.mjs or omit --prompt-arm to revert the added evaluation path; application behavior is unchanged.

🤖 Generated with Codex

Agent session: agent session

---

^{Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.}

---

Summary by cubic

Evaluates whether the compact asking rule plus early/concurrent-work guidance changes model behavior on paired repository tasks, using A/B prompt-guidance instrumentation.

• The new standalone prompt-behavior.mjs runs three fixed tasks against deterministic fixtures with a captured Claude Code system prompt, alternating arm order per sample. It appends the guidance block to the first user turn to match product delivery and routes through the gateway via ANTHROPIC_BASE_URL and ANTHROPIC_AUTH_TOKEN. It records prompt hashes, transcripts, success, wall time, first-read latency, tokens, tool-turn counts, parallel share, and background starts. The baseline asking rule is frozen at a recorded source revision, and a guard rejects templates that already contain cmux guidance.
• The tool evaluator gains --prompt-arm none|baseline|guidance and the same tool-turn, parallel-share, and first-read metrics, delivering the candidate through the Claude preset's append field or the Codex developer-instruction field. The paired-run workflow is documented in the tool-evals README.
• Changes are evaluator-only: application behavior is unchanged, and the new path is reverted by deleting prompt-behavior.mjs or omitting --prompt-arm.

^{Written for commit 8df2591. Summary will update on new commits.}

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review fd75173: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr34637 -f ref=lane/prompt-eval -f platform=macos -f backend_mode=local

web and size checks: green

Shipped in:

• cmux NEXT nightly 1.0.0-nightly.4185979991201 (8a134ee405)
• cmux NEXT nightly 1.0.0-nightly.4186744784501 (890be4f45a)
• cmux NEXT nightly 1.0.0-nightly.4186855704101 (53421cdccd)
• cmux NEXT nightly 1.0.0-nightly.4187086980301 (df4454e051)
• cmux NEXT nightly 1.0.0-nightly.4188039004101 (3e464b08ff)
• cmux NEXT nightly 1.0.0-nightly.4190094575701 (036f8aa538)
• cmux NEXT nightly 1.0.0-nightly.4191131679101 (e01bbae5c9)

[teamleaderleo]

Review (codex, router-review): SHIP at 9dd7ace

Reviewer: codex gpt-6-astra high, fast tier.

Follow-ups: 1. Each lands as an issue in this repo that links this receipt; the lane that opened the PR fixes it forward.

SHIP at 9dd7ace

Visual verdict: no images; Leo tries the build

1. [P2][follow-up] first_read_ms is recorded for any Bash, cmux_exec, Codex MCP call, or file change, including mutations and tests, so it does not measure first-read latency and can skew A/B conclusions. Restrict this metric to actual read operations (agents-sdk.mjs:92, run.mjs:521).

Follow-ups:

• [P2] Correct first-read event classification before relying on that metric.

[cmux-bot]

cmux-bot approves 9dd7ace: an independent review passed this exact head (router-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

[teamleaderleo]

Review (subagent opus medium): SHIP at c285386

Checked: the full diff (4 files under tests/tool-evals/, +325/-11), plus run.mjs, agents-sdk.mjs and the captured template at the head. No workflow under .github or scripts runs tests/tool-evals, so the harness is opt-in. scripts/check-test-determinism.py is clean on the new file. With the default --prompt-arm none, run.mjs behaves as it does on base apart from a summary header suffix. prompt-behavior.mjs runs no shell commands, writes only under --out, and never writes the auth token to the manifest or transcripts.

Visual verdict: unchanged by this diff (no UI).

Follow-ups:

• F1 [follow-up] correctness tests/tool-evals/agents-sdk.mjs:88-93: the Claude SDK path counts one tool turn per assistant message. Claude Code's stream output usually sends each content block as its own assistant message, all sharing message.id, so parallel calls may count as separate one-call turns and parallel_share stays at about 0. Group by message.message.id. outcome:pending
• F2 [follow-up] test-quality tests/tool-evals/agents-sdk.mjs:203-216: the Codex path groups calls into a turn only while their item/started events overlap, so its parallel figures are approximate. outcome:pending
• F3 [follow-up] test-quality tests/tool-evals/prompt-behavior.mjs:236-238: the compare-contracts grader only checks that the reply contains "server", "retries" and "retry_limit", which a wrong answer (one that blames the client) can also contain. Grade the actual conclusion. outcome:pending
• F4 [follow-up] correctness tests/tool-evals/prompt-behavior.mjs:364: the first sample error (a 429 or gateway failure) aborts the run before summary.json is written. Retry the way run.mjs does, or record the error and continue. outcome:pending
• F5 [follow-up] correctness tests/tool-evals/prompt-behavior.mjs:371-374: with --samples 0, or an arm with no tool turns, rs[0].usage throws and the ratios come out as NaN. outcome:pending
• F6 [follow-up] security tests/tool-evals/prompt-behavior.mjs:208, README.md:42: passing the token as --auth puts it in process listings. Document only the ANTHROPIC_AUTH_TOKEN env path. outcome:pending
• F7 [follow-up] correctness tests/tool-evals/agents-sdk.mjs:172: guidance replaces CMUX_EVAL_CODEX_DEVELOPER instead of combining with it. outcome:pending
• F8 [follow-up] docs tests/tool-evals/run.mjs:390: the error message omits the accepted none value. outcome:pending

[teamleaderleo]

Owned by an agent session · session

[teamleaderleo]

Review (subagent claude-opus-5-5 medium): SHIP at fd75173
Visual verdict: unchanged by this diff (evaluator-only, no UI code).

Checked prompt-behavior.mjs (fixtures answer Read/Bash deterministically, no model command executes, arms alternate per sample, manifest hashes the texts), the --prompt-arm path in run.mjs (defaults to none, so existing runs are unchanged; trailing-rule replacement only when no later section follows) and the tool-turn / first-read metrics in both SDK and Codex app-server paths. No failing checks at this head; the fleet eval's gateway reach failure is a route limit, and the paired receipt is in the body.

Follow-ups:

• F1 [follow-up] prompt-behavior.mjs throws on the first sample error, so one gateway blip discards the run; record the error row and continue (or retry like runCliAgent). outcome:pending
• F2 [follow-up] Codex first_read_ms counts any fileChange as a read; only reads/commands/MCP calls should count. outcome:pending

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[teamleaderleo]

Review (subagent claude-opus-5-5 medium): SHIP at 1a0a418
Visual verdict: unchanged by this diff (evaluator-only, no UI code).

Merge-resolution check since the SHIP at fd75173:

• The three commits after it (52c768a, f003a39, 1a0a418) are all Merge branch 'feat-cmux-next'. Their second parents (21302c6, d96836b, dbc381e) are ancestors of origin/feat-cmux-next, and no non-merge commit in fd75173..1a0a418 is missing from base.
• git diff <merge-base> fd751732 and git diff <merge-base> 1a0a4184 match byte for byte, index lines included. Both touch the same 4 files under tests/tool-evals/ (+345/-11). git range-diff shows all 5 PR commits as =. So no conflict resolution was needed.
• None of the 423 base commits merged in since then touch tests/tool-evals/. prompt-behavior.mjs freezes both arm texts inline (baseline pinned to agent_tools.rs@6dedb9cc) and reads only its own templates/claude.json.gz, so base changes can't shift the comparison. No semantic conflict.

Tests at this head: node --check passes for prompt-behavior.mjs, run.mjs and agents-sdk.mjs. bun test live-guard.test.mjs gives 0 fail / 10 skip (the macOS sandbox cases skip on Linux). I did not rerun the paired evaluator. It needs a live model gateway, and its code and fixtures are byte-identical to the reviewed head, so the 9/9 receipt still applies.

Findings: none new. F1/F2 from the fd75173 receipt stay as follow-ups.

Reds at this head: the cmux-next checks (god files/concurrency/l10n) and the nightly-guards release-notary/Guard status are outside this PR's files, which are all under tests/tool-evals/ and wired to no workflow. Before the pinned merge they still need base_red_excuse.py. I did not run it.

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[teamleaderleo]

Review (subagent claude-opus-5-5 medium): SHIP at 8df2591

Carried from the SHIP at 1a0a418 (comment 6632404879) through pure base merges: 077f5b1 (1a0a418 + 45aeecc), then 8df2591 (077f5b1 + feat-cmux-next 6d5aaed, from update-branch).

Proof: the net diff against the merge-base (6d5aaed) is unchanged. git diff <merge-base> 8df2591c | sha1sum = 9a3132a8b792f582cd381e1decea330f198fafeb, the same hash as at 1a0a418 and 077f5b1. It is 4 files, all under tests/tool-evals/ (README.md, agents-sdk.mjs, prompt-behavior.mjs, run.mjs), +345/-11.