Skip to content

ci(cmux-next): path-based risk tiers with a required evidence check - #21691

Closed
teamleaderleo wants to merge 3 commits into
feat-cmux-nextfrom
claude/risk-tiers
Closed

teamleaderleo wants to merge 3 commits into
feat-cmux-nextfrom
claude/risk-tiers

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

Pull requests into feat-cmux-next now get a risk:low, risk:medium or risk:high label from the paths they change, and a risk evidence check that stays red until the description has the evidence that tier needs. The policy is in docs/ci/risk-tiers.md, and CLAUDE.md tells every agent to follow it.

Tier Paths Evidence before merge
low docs, tests, skills, plans, CI and tooling green CI
medium app UI, SDK bindings, anything unclassified plus a screenshot or clip under ## Proof, or Not UI: <why>
high daemon, acpmux, protocol and spec, persistence, migrations, schemas/, typing path, remote, cloud and browser import, signing, release and publish, vendored pins, and the merge gates themselves plus Review: naming the reviewed commit and Runtime test: naming the run that executed the change

Why now: 2,049 pull requests merged into feat-cmux-next from 2026-10-04 to 2026-10-18. 932 had a failing check on their merged head, not counting review bots and report-only lanes. cmux-next swift test runs on a pull request only with full-ci, never on a push to the branch (.github/workflows/cmux-next.yml:1760), and in the last 25 runs that executed it, 24 failed. So a new Swift test usually compiles and never runs before merge. On that history the classifier grades 380 pull requests low, 992 medium and 677 high.

How it works:

  • scripts/ci/risk_tier.py holds the ordered path rules. A pull request takes the highest tier of any file. An unknown path is medium, never low. A medium change over 1,000 lines outside tests and generated files becomes high.
  • .github/workflows/risk-tier.yml runs on pull_request_target, so fork pull requests get labels too. It checks out only the base's classifier and reads the file list, commit list and description through the API. It never runs the pull request's code. It re-runs when the description is edited, so adding the evidence turns the check green.
  • Review: must name a commit on the pull request. Runtime test: must name a run URL, or a commit on the pull request that a tagged build or Testbox ran. Proof must be media, not a link to a CI run.
  • A maintainer can add risk:waived. It counts only when the timeline shows an admin or maintainer added it. Agents are told never to.
  • The unit tests run in their own workflow, and the grading job has no if:, so no skipped risk evidence run can become the newest check and read as a pass. A newer event cancels an older grading run, and gh-merge-green refuses a cancelled check.
  • gh-merge-green: never waive a red risk evidence check #21692 (on main, where the installed helper lives) makes gh-merge-green refuse a red risk evidence check even with --override, the same way it treats god-file and l10n lint.

The policy also states two practices that already have tooling: ship user-visible high-tier behavior behind a setting that defaults off, and treat nightly-next as the canary, reverting with the existing gh-merge-green --revert when a high-tier change breaks it and the fix misses the next cut.

Because this workflow runs from the base, it starts labeling only after it merges. Open pull requests pick up a label on their next push or description edit.

Testing

This pull request grades itself high, because it changes a merge gate.

Review: BLOCK at af6ab7a (a skipped run could hide a red check, plus classifier and parser gaps), then SHIP with nits at 9132ed0. That was a cmux-review subagent, not the author's agent. 2397770 only adds the waiver-lookup nit from that review.
Runtime test: python3 tests/test_risk_tier.py and risk_tier.py pr against live #21334, #21645, #21646 and #21690, run at 2397770.

Follow-ups from review, not in this pull request: grading any workflow with pull_request_target or a write token as high needs the file's contents, not only its path; and Runtime test: accepts any run URL without tying it to this pull request.

Changelog

none

Proof

Not UI: adds a CI label, a check and docs. Nothing in the app changes.

🤖 Generated with Claude Code

teamleaderleo and others added 3 commits October 18, 2026 14:32
Label each feat-cmux-next pull request risk:low, risk:medium or risk:high
from the paths it changes, and keep a 'risk evidence' check red until the
description has that tier's evidence: a screenshot for medium, plus a
reviewed SHA and an executed runtime test for high. The policy is in
docs/ci/risk-tiers.md and CLAUDE.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Grade on every pull_request_target event with job-level concurrency, and
  move the unit tests to risk-tier-tests.yml, so no skipped 'risk evidence'
  run can become the newest check and read as a pass.
- Check out the classifier at github.workflow_sha.
- Grade merge gates, CODEOWNERS, publish/deploy workflows, schemas/, and the
  BrowserImport, BrowserAutomation and Cloud targets high; stop treating
  every .txt and src/ fixture as low.
- Proof needs media, not any URL; field values must be on the same line;
  a runtime SHA must be a commit on the pull request.
- risk:waived counts only when an admin or maintainer added it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A failed permission lookup no longer turns a check with full evidence red,
and reads as no waiver.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cursor

cursor Bot commented Oct 18, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@coderabbitai

coderabbitai Bot commented Oct 18, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: Repository: manaflow-ai/cmux/.coderabbit.yaml
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 6d48e02d-fa14-40c6-93eb-b7de328f0ef2

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

Receipt for 2397770 (this PR's head; a new push replaces it)

  • Visual review: HQ feed card · gallery before/after runs
  • Checks: this head
  • App of this head (app.zip, kept 1 day): gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr21691 -f ref=claude/risk-tiers -f platform=macos -f backend_mode=local

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Bead cx-1xk2a created for this PR by the cmux coordinator.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Superseded by #21718, which now targets feat-cmux-next and carries this PR's commits unchanged plus the reworked designations and review fixes, so it lands as one PR.

teamleaderleo added a commit that referenced this pull request Oct 19, 2026
…sk evidence check (#21718)

* ci(cmux-next): path-based risk tiers with an evidence check

Label each feat-cmux-next pull request risk:low, risk:medium or risk:high
from the paths it changes, and keep a 'risk evidence' check red until the
description has that tier's evidence: a screenshot for medium, plus a
reviewed SHA and an executed runtime test for high. The policy is in
docs/ci/risk-tiers.md and CLAUDE.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci(cmux-next): close the risk evidence bypasses review found

- Grade on every pull_request_target event with job-level concurrency, and
  move the unit tests to risk-tier-tests.yml, so no skipped 'risk evidence'
  run can become the newest check and read as a pass.
- Check out the classifier at github.workflow_sha.
- Grade merge gates, CODEOWNERS, publish/deploy workflows, schemas/, and the
  BrowserImport, BrowserAutomation and Cloud targets high; stop treating
  every .txt and src/ fixture as low.
- Proof needs media, not any URL; field values must be on the same line;
  a runtime SHA must be a commit on the pull request.
- risk:waived counts only when an admin or maintainer added it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci(cmux-next): read risk:waived only when evidence is missing

A failed permission lookup no longer turns a check with full evidence red,
and reads as no waiver.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci(cmux-next): grade workflows high and keep file names out of workflow commands

From coord-2's review of #21691: every .github/workflows and .github/actions
file runs with secrets or a write token and decides what other pull requests
skip, so it is high tier. File names in the report have control characters
and backticks replaced, so a newline in a name can't start a :: command.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci(cmux-next): sharper risk designations

- Always high, at any size: release and signing, migrations, schemas/defaults,
  the protocol spec, pins, the merge gates and the workflows that route
  checks (cmux-next.yml, ci.yml, ci-guards.yml, ...).
- Runtime paths (daemon, acpmux, backend, Remote, Terminal, ...) are high
  only when the change is over 40 lines outside tests and generated files;
  a smaller one is medium, which a reviewer can read whole.
- Other workflows and actions are medium.
- New: a pull request that takes back another pull request's change from
  the last 7 days (deletes what it added, restores what it replaced) is
  high and names it. On #21194's squash this names exactly the five PRs it
  undid (#21199, #21234, #21369, #21384, #21446); moves and generated files
  don't count. The workflow fetches the base's last 9 days blobless for it.

On the 10-04..10-18 history by path: 225 low, 1207 medium, 617 high (30%),
down from 856 high (42%).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci(cmux-next): printable() also replaces C1 controls and line separators

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci(cmux-next): a line moved to another file isn't taken back

#21242 moved #20398's mouseDownCanMoveWindow override from
SidebarContainerView into SidebarResizeHandle; the revert check read the
move as a revert.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci(cmux-next): full-blob base fetch, visible blame gaps, advisory until 10-21

From coord-2's re-review:
- Fetch the base's 9 days with blobs. Blobless, blame fetched blobs one by
  one and passed its 30 s timeout on busy files, so #21724's takeback of
  #21320 and #21536 went unflagged; with blobs it is flagged in seconds.
- A file the revert check couldn't read is listed in the report for a
  manual check, never skipped silently.
- Generated settings exports (*.sha256, settings-export/, the settings
  schema) don't count toward the 40-line runtime threshold.
- The evidence check is advisory (a warning, check passes) until
  2026-10-21T00:00Z, so pull requests already in flight can add evidence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: register tests/test_risk_tier.py in the execution registry

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci(cmux-next): base history in a fresh repo, a time budget, and the merge base

From review of fa3750e:
- actions/checkout turns a sparse checkout into a blobless partial clone,
  and a later fetch into it stays blobless, so blame still fetched blobs one
  at a time. Fetch the base's 9 days into a fresh repository instead (about
  60 s, not a partial clone; blame then takes about 2 s on #21724).
- The revert check stops after 120 s and lists what it didn't reach as
  'check by hand', so the job always ends inside its 5-minute timeout (a
  timed-out risk evidence check is red and can't be waived).
- Blame at the merge base the patches are relative to, not the base tip.
- Docs: re-run risk evidence on open PRs at the advisory cutover.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci(cmux-next): cap the base history fetch at 150 s

From review of 8a61191: an uncapped fetch could still run the job past its
5-minute timeout, and a timed-out risk evidence check is red. A slow fetch
now skips the revert check with a warning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(risk tiers): cover the audit misgrades and advisory-era verdicts

The designation audit found paths graded too high (Rust *_tests.rs, spec
prose), too low (nested deploy and signing scripts, merge-gate helpers,
OptChat backend, pins), and that a risk evidence check that passed while
advisory stays green after the cutover. These tests fail until the fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci(risk tiers): apply the designation audit and refuse advisory-era verdicts

Agents audited the designations against 40 merged and 30 open PRs.
- Rust *_tests.rs, tests.rs and tests_v2/ grade as tests, and spec Markdown
  as docs. The cmux-tui/spec JSON stays a contract.
- Nested deploy, signing, nightly, promote and rollback scripts and workflows,
  scripts/ci/lib, the gh-merge-green helpers, merge-main.sh and the god-file
  check now grade high. The merge-gate rule moves first, so
  select_package_tests.py can never match the test rule.
- The OptChat agent backend counts as runtime (its inspector stays UI).
  ghostty.h, homebrew-cmux and .xcode-version count as pins.
- A Demo, Video or Recording heading holds proof.
- ENFORCE_FROM moves to 2026-10-24 00:00 UTC. After that, gh-merge-green
  reads ENFORCE_FROM from the base's risk-tier.yml and refuses a
  risk evidence verdict that finished before the cutover, or no verdict at all.
- The module docstring now matches the rules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(risk tiers): judge a verdict by its start and refuse an unreadable cutover

Review nits on 83f662d: a job that started before the cutover can grade
as advisory and still finish after it, and a risk-tier.yml whose ENFORCE_FROM
no longer parses silently skipped the gate. Also test_*.py scripts and
fixtures should grade as tests. These fail until the fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci(risk tiers): judge a verdict by its job start; refuse an unreadable cutover

gh-merge-green now compares the risk evidence job start time with
ENFORCE_FROM, since the job decides advisory-or-not while it runs, and
refuses when risk-tier.yml is readable but has no ENFORCE_FROM in the
expected form instead of skipping the gate. test_*.py scripts and fixtures
under scripts/ grade as tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant