ci: refuse to re-dispatch a focused test that already failed at that commit

[teamleaderleo]

test-e2e.yml took 959 dispatches in 7 days, 531 of them red, for 6,730 runner-minutes. The expensive failures are not flaky tests. Sampled logs show Swift compile errors on the dispatched ref:

SplitPaneBackgroundUITests.swift:89:13: error: invalid redeclaration of 'terminal'
DeviceSurfaceProviderRegistry.swift:56:46: error: call to main actor-isolated initializer ...
Cannot find 'pullRequestWorkspacePathBelongsToRepository' in scope

A focused run compiles the whole tree before it runs anything, so a 10-20 minute macOS run existed only to discover that the branch does not build. scripts/ci/require_selected_test_execution.sh then correctly reports that no selected test executed — the guard works, it is being fed refs that do not compile.

The single worst line item: cmuxUITests/SplitPaneBackgroundUITests was dispatched 24 times in 12 hours for 397 runner-minutes and passed zero times. Two of those failures were the same assertion, two were the same compile error. test-depot.yml shows the same pattern (348 dispatches, 213 red, 5,335 runner-minutes) against branches that do not compile.

Two changes

1. The launcher refuses a repeat that cannot say anything new.

A focused run's result is a property of the commit. Before dispatching, the launcher now looks for a completed run of the same selector at the same commit — the run name already carries both halves, <selector> on <runner> @ <commit> [<dispatch id>], so no local state is needed — and refuses when one failed and none succeeded, naming the earlier run:

cmuxTests/Foo already failed at <sha> (3 time(s)); the newest is <url>.
A focused run compiles the tree first, so the most common red result is a
compile error in the branch, not a flaky test -- and re-running the same
selector at the same commit returns the same answer. Read that run, fix the
branch, push, and dispatch the new commit. Pass --force to dispatch anyway.

--force keeps the escape hatch for a failure you know was infrastructure. The lookup is an economy measure and never a gate: if the run history cannot be read for any reason, it dispatches exactly as before.

2. The instruction that produced most of this could not work.

skills/cmux-testing/references/local-vs-ci-validation.md said:

Run through GitHub Actions or the VM: gh workflow run test-e2e.yml.

test_filter is required: true, so that command is rejected by the API. Anyone following it literally gets an error and improvises, which is where filters matching zero tests come from. It also omits --ref, so a dispatch that does succeed lands on whatever ref is current rather than the commit under test — while docs/contributor-verification.md has the correct form via scripts/run-e2e.sh, which the skill never mentions.

The section now points at the wrapper with --ref, says to compile the test target locally first (the same file already gives that exact cmux-unit command eight lines earlier), and says not to repeat a selector at a commit that already failed.

Validation

Two commits, per the repository's regression policy:

1. d2039ef adds five cases to tests/test_run_e2e.py only — two fail against the current launcher.
2. ce45716 adds the guard and the skill fix — all 19 pass.

The new cases cover: a prior failure at the same commit refuses and does not dispatch; --force dispatches anyway; a prior success does not block; a failure of a different selector does not block; a failure at a different commit does not block. The fake gh harness distinguishes the pre-dispatch lookup from the post-dispatch correlation by whether the caller asked for conclusion, rather than by call ordering.

The 14 pre-existing cases still pass unchanged, including with the guard's extra lookup in place.

Scope and what I did not do

This targets the repeat-dispatch class specifically. It does not stop the first dispatch against a non-compiling ref — that is what the local cmux-unit build in the skill is for, and enforcing it mechanically would mean a cheap pre-gate on the workflow, which is a real cost decision and a separate change.

I also have not touched test-depot.yml, which shows the same pattern from the same source. It has no in-repo dispatcher to add a guard to, and its workflow_call trigger is dead (no callers anywhere) while its workflow_dispatch half is actively used ~50 times a day — so it needs its own change, not deletion.

Numbers are from the 7-day totals in docs/ci/workflow-inventory.md; the root-cause sampling is from a 17-hour window of runs, so the per-class proportions are a sample rather than a weekly figure.

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

---

Summary by cubic

Stops the focused-test launcher from re-dispatching a selector at a commit where it already failed, cutting wasted macOS runner minutes that mostly reprint Swift compile errors rather than flaky failures. The launcher now looks up completed runs by selector and commit (the run name already encodes both) and refuses with a link to the earlier run unless --force is passed; if history can't be read, it dispatches as before. The testing skill now points to scripts/run-e2e.sh --ref <sha>, which is the only working form (test_filter is a required input), and tells readers to compile the test target locally first and never repeat a failed selector at the same commit.

^{Written for commit ce45716. Summary will update on new commits.}

[coderabbitai]

Warning

Review limit reached

Next included review available in 16 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 0a9647c2-291f-4f90-87f4-f9f83f66aaf0

📥 Commits

Reviewing files that changed from the base of the PR and between 6b2ba6d and ce45716.

📒 Files selected for processing (3)

• scripts/ci/dispatch-focused-test.py
• skills/cmux-testing/references/local-vs-ci-validation.md
• tests/test_run_e2e.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}