Repository navigation
ci(cmux-next): path-based risk tiers with a required evidence check - #21691
teamleaderleo wants to merge 3 commits into
Conversation
Label each feat-cmux-next pull request risk:low, risk:medium or risk:high from the paths it changes, and keep a 'risk evidence' check red until the description has that tier's evidence: a screenshot for medium, plus a reviewed SHA and an executed runtime test for high. The policy is in docs/ci/risk-tiers.md and CLAUDE.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Grade on every pull_request_target event with job-level concurrency, and move the unit tests to risk-tier-tests.yml, so no skipped 'risk evidence' run can become the newest check and read as a pass. - Check out the classifier at github.workflow_sha. - Grade merge gates, CODEOWNERS, publish/deploy workflows, schemas/, and the BrowserImport, BrowserAutomation and Cloud targets high; stop treating every .txt and src/ fixture as low. - Proof needs media, not any URL; field values must be on the same line; a runtime SHA must be a commit on the pull request. - risk:waived counts only when an admin or maintainer added it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A failed permission lookup no longer turns a check with full evidence red, and reads as no waiver. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bugbot needs on-demand usage enabledBugbot uses usage-based billing for this team and requires on-demand usage to be enabled. A team admin can enable on-demand usage in the Cursor dashboard. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Receipt for
|
|
Bead |
|
Superseded by #21718, which now targets feat-cmux-next and carries this PR's commits unchanged plus the reworked designations and review fixes, so it lands as one PR. |
…sk evidence check (#21718) * ci(cmux-next): path-based risk tiers with an evidence check Label each feat-cmux-next pull request risk:low, risk:medium or risk:high from the paths it changes, and keep a 'risk evidence' check red until the description has that tier's evidence: a screenshot for medium, plus a reviewed SHA and an executed runtime test for high. The policy is in docs/ci/risk-tiers.md and CLAUDE.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(cmux-next): close the risk evidence bypasses review found - Grade on every pull_request_target event with job-level concurrency, and move the unit tests to risk-tier-tests.yml, so no skipped 'risk evidence' run can become the newest check and read as a pass. - Check out the classifier at github.workflow_sha. - Grade merge gates, CODEOWNERS, publish/deploy workflows, schemas/, and the BrowserImport, BrowserAutomation and Cloud targets high; stop treating every .txt and src/ fixture as low. - Proof needs media, not any URL; field values must be on the same line; a runtime SHA must be a commit on the pull request. - risk:waived counts only when an admin or maintainer added it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(cmux-next): read risk:waived only when evidence is missing A failed permission lookup no longer turns a check with full evidence red, and reads as no waiver. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(cmux-next): grade workflows high and keep file names out of workflow commands From coord-2's review of #21691: every .github/workflows and .github/actions file runs with secrets or a write token and decides what other pull requests skip, so it is high tier. File names in the report have control characters and backticks replaced, so a newline in a name can't start a :: command. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(cmux-next): sharper risk designations - Always high, at any size: release and signing, migrations, schemas/defaults, the protocol spec, pins, the merge gates and the workflows that route checks (cmux-next.yml, ci.yml, ci-guards.yml, ...). - Runtime paths (daemon, acpmux, backend, Remote, Terminal, ...) are high only when the change is over 40 lines outside tests and generated files; a smaller one is medium, which a reviewer can read whole. - Other workflows and actions are medium. - New: a pull request that takes back another pull request's change from the last 7 days (deletes what it added, restores what it replaced) is high and names it. On #21194's squash this names exactly the five PRs it undid (#21199, #21234, #21369, #21384, #21446); moves and generated files don't count. The workflow fetches the base's last 9 days blobless for it. On the 10-04..10-18 history by path: 225 low, 1207 medium, 617 high (30%), down from 856 high (42%). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(cmux-next): printable() also replaces C1 controls and line separators Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(cmux-next): a line moved to another file isn't taken back #21242 moved #20398's mouseDownCanMoveWindow override from SidebarContainerView into SidebarResizeHandle; the revert check read the move as a revert. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(cmux-next): full-blob base fetch, visible blame gaps, advisory until 10-21 From coord-2's re-review: - Fetch the base's 9 days with blobs. Blobless, blame fetched blobs one by one and passed its 30 s timeout on busy files, so #21724's takeback of #21320 and #21536 went unflagged; with blobs it is flagged in seconds. - A file the revert check couldn't read is listed in the report for a manual check, never skipped silently. - Generated settings exports (*.sha256, settings-export/, the settings schema) don't count toward the 40-line runtime threshold. - The evidence check is advisory (a warning, check passes) until 2026-10-21T00:00Z, so pull requests already in flight can add evidence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: register tests/test_risk_tier.py in the execution registry Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(cmux-next): base history in a fresh repo, a time budget, and the merge base From review of fa3750e: - actions/checkout turns a sparse checkout into a blobless partial clone, and a later fetch into it stays blobless, so blame still fetched blobs one at a time. Fetch the base's 9 days into a fresh repository instead (about 60 s, not a partial clone; blame then takes about 2 s on #21724). - The revert check stops after 120 s and lists what it didn't reach as 'check by hand', so the job always ends inside its 5-minute timeout (a timed-out risk evidence check is red and can't be waived). - Blame at the merge base the patches are relative to, not the base tip. - Docs: re-run risk evidence on open PRs at the advisory cutover. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(cmux-next): cap the base history fetch at 150 s From review of 8a61191: an uncapped fetch could still run the job past its 5-minute timeout, and a timed-out risk evidence check is red. A slow fetch now skips the revert check with a warning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(risk tiers): cover the audit misgrades and advisory-era verdicts The designation audit found paths graded too high (Rust *_tests.rs, spec prose), too low (nested deploy and signing scripts, merge-gate helpers, OptChat backend, pins), and that a risk evidence check that passed while advisory stays green after the cutover. These tests fail until the fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(risk tiers): apply the designation audit and refuse advisory-era verdicts Agents audited the designations against 40 merged and 30 open PRs. - Rust *_tests.rs, tests.rs and tests_v2/ grade as tests, and spec Markdown as docs. The cmux-tui/spec JSON stays a contract. - Nested deploy, signing, nightly, promote and rollback scripts and workflows, scripts/ci/lib, the gh-merge-green helpers, merge-main.sh and the god-file check now grade high. The merge-gate rule moves first, so select_package_tests.py can never match the test rule. - The OptChat agent backend counts as runtime (its inspector stays UI). ghostty.h, homebrew-cmux and .xcode-version count as pins. - A Demo, Video or Recording heading holds proof. - ENFORCE_FROM moves to 2026-10-24 00:00 UTC. After that, gh-merge-green reads ENFORCE_FROM from the base's risk-tier.yml and refuses a risk evidence verdict that finished before the cutover, or no verdict at all. - The module docstring now matches the rules. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(risk tiers): judge a verdict by its start and refuse an unreadable cutover Review nits on 83f662d: a job that started before the cutover can grade as advisory and still finish after it, and a risk-tier.yml whose ENFORCE_FROM no longer parses silently skipped the gate. Also test_*.py scripts and fixtures should grade as tests. These fail until the fix. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci(risk tiers): judge a verdict by its job start; refuse an unreadable cutover gh-merge-green now compares the risk evidence job start time with ENFORCE_FROM, since the job decides advisory-or-not while it runs, and refuses when risk-tier.yml is readable but has no ENFORCE_FROM in the expected form instead of skipping the gate. test_*.py scripts and fixtures under scripts/ grade as tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Pull requests into
feat-cmux-nextnow get arisk:low,risk:mediumorrisk:highlabel from the paths they change, and arisk evidencecheck that stays red until the description has the evidence that tier needs. The policy is indocs/ci/risk-tiers.md, andCLAUDE.mdtells every agent to follow it.## Proof, orNot UI: <why>schemas/, typing path, remote, cloud and browser import, signing, release and publish, vendored pins, and the merge gates themselvesReview:naming the reviewed commit andRuntime test:naming the run that executed the changeWhy now: 2,049 pull requests merged into
feat-cmux-nextfrom 2026-10-04 to 2026-10-18. 932 had a failing check on their merged head, not counting review bots and report-only lanes.cmux-next swift testruns on a pull request only withfull-ci, never on a push to the branch (.github/workflows/cmux-next.yml:1760), and in the last 25 runs that executed it, 24 failed. So a new Swift test usually compiles and never runs before merge. On that history the classifier grades 380 pull requests low, 992 medium and 677 high.How it works:
scripts/ci/risk_tier.pyholds the ordered path rules. A pull request takes the highest tier of any file. An unknown path is medium, never low. A medium change over 1,000 lines outside tests and generated files becomes high..github/workflows/risk-tier.ymlruns onpull_request_target, so fork pull requests get labels too. It checks out only the base's classifier and reads the file list, commit list and description through the API. It never runs the pull request's code. It re-runs when the description is edited, so adding the evidence turns the check green.Review:must name a commit on the pull request.Runtime test:must name a run URL, or a commit on the pull request that a tagged build or Testbox ran. Proof must be media, not a link to a CI run.risk:waived. It counts only when the timeline shows an admin or maintainer added it. Agents are told never to.if:, so no skippedrisk evidencerun can become the newest check and read as a pass. A newer event cancels an older grading run, and gh-merge-green refuses a cancelled check.gh-merge-greenrefuse a redrisk evidencecheck even with--override, the same way it treats god-file and l10n lint.The policy also states two practices that already have tooling: ship user-visible high-tier behavior behind a setting that defaults off, and treat nightly-next as the canary, reverting with the existing
gh-merge-green --revertwhen a high-tier change breaks it and the fix misses the next cut.Because this workflow runs from the base, it starts labeling only after it merges. Open pull requests pick up a label on their next push or description edit.
Testing
python3 tests/test_risk_tier.py: 17 tests pass. They cover path rules, the size bump, the evidence parser, template comments and bare links not counting as proof, values on the next line, stale review and runtime SHAs, and the waiver labeler.python3 scripts/ci/risk_tier.py pr --repo manaflow-ai/cmux --number Nagainst live cmux-next: harden sidebar motion and workspace journeys #21334 (medium), fix(cmux-next): restore composer row rhythm and open-chip wash (base red since #21574) #21645 (medium), ci(cmux-next): run the split guard's classifier from the base #21646 (low) and fix(cmux-next): restore #21369 and #21384, undone by the Chief Dock squash #21690 (medium) printed the expected tiers and missing evidence. That run read the API and wrote nothing.bash tests/test_ci_self_hosted_guard.shaccepts both jobs' runners.python3 scripts/verify-local.py --affected origin/feat-cmux-next: everything passes except localization parity, which fails the same way (34,264 errors) at the tip without this change.bash scripts/ci/guards-local.sh: 100 steps pass and 19 fail, the same 19 as at the tip 4b1b023 without this change.pull_request_targetworkflows run from the base, so the first real label comes after merge.actionlintisn't installed on this host.This pull request grades itself high, because it changes a merge gate.
Review: BLOCK at af6ab7a (a skipped run could hide a red check, plus classifier and parser gaps), then SHIP with nits at 9132ed0. That was a cmux-review subagent, not the author's agent. 2397770 only adds the waiver-lookup nit from that review.
Runtime test:
python3 tests/test_risk_tier.pyandrisk_tier.py pragainst live #21334, #21645, #21646 and #21690, run at 2397770.Follow-ups from review, not in this pull request: grading any workflow with
pull_request_targetor a write token as high needs the file's contents, not only its path; andRuntime test:accepts any run URL without tying it to this pull request.Changelog
none
Proof
Not UI: adds a CI label, a check and docs. Nothing in the app changes.
🤖 Generated with Claude Code