Skip to content

ci: a label CI does not read no longer starts a full CI run - #29747

Merged
teamleaderleo merged 5 commits into
feat-cmux-nextfrom
ci-risk-label-no-rerun
Oct 31, 2026
Merged

teamleaderleo merged 5 commits into
feat-cmux-nextfrom
ci-risk-label-no-rerun

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 31, 2026 •

Copy link
Copy Markdown
Collaborator

Leo (2026-10-31): "JUST MERGE" "IT IS BINARY DONE OR NOT DONE".

Problem

ci.yml triggers on pull_request labeled/unlabeled, so every label change, risk:* swaps included, started a second full CI run at the same head. On #29618 the duplicate run 41241282080 sat queued for over an hour behind the first one (label runs share its concurrency group and don't cancel), and while it was pending the webview gate kept waiting. cmux-vm.yml had the same trigger with cancel-in-progress: true, so any label change cancelled and restarted its check.

What changed

  • ci.yml, for pull requests into feat-cmux-next: every job but ci-status gets one guard that is false only for a label event whose label is not full-ci or dev-build (the two labels it reads). ci-status still runs on those events and passes at once into feat-cmux-next, because a skipped ci-status newer than a red or pending one would read as passing to a newest-by-name gate like gh-merge-green (review finding at 336d0db). PRs into other bases (main) keep re-running as before, so main's required checks are never superseded by a skip. The concurrency group gets a -label-<run_id> suffix for those events, so the skipped run doesn't wait behind the real run, replace a pending one, or cancel anything. It finishes in seconds with every job skipped. full-ci and dev-build work as before.
  • scripts/ci/next_batch.py (risk:high, since it touches a merge enforcer): newest_checks now treats a skipped newest run like a cancelled one and falls back to the newest run that has a verdict. Without this, a label run's skipped jobs (VM check, web) would hide an earlier red from the batch queue (review finding at 9d793d0). merge_guard's newest already works this way. Red without the fix, green with it: tests/test_next_batch_groups.py::GroupCycle::test_newest_run_edge_cases.
  • cmux-vm.yml: same pattern (feat-cmux-next only), keyed on cmux-vm-breaking-change-ok.
  • Left alone because they're cheap and read labels on purpose: risk-tier, merge-freeze, pr-capture-check, pr-perf-section-check, backend-migrations, cmux-next-batch (debounce), cmux-next-media and cmux-next-motion-golden (already gated on their label).
  • The skipped duplicate doesn't hide an earlier verdict: the status rollup still lists the earlier suite's checks (checked on Agent chat: no background band behind the header tools in the tab row #29618), and merge-guard reads the newest check that has a verdict.

Test

tests/test_ci_delta_since_green.py drops the guard before its small evaluator, which has no !, runs. The new test also runs as a step in ci-guards.yml's ci group.

New tests/test_ci_label_events_skip_full_ci.py (linux-guard lane). Every label-triggered workflow must be either GUARDED (its labels listed; every job a label event could start ends with the guard; its concurrency group isolates other label events) or in REACTS with a reason. It also evaluates the guard: synchronize, opened, push, its own labels and a risk:high label into main give true; risk:high, risk:normal and ready-to-land give false. It fails on the base workflows (4+ job failures) and passes on the head. I updated one existing assertion (linux-preflight) that pinned the old if: text. test_ci_pr_runs_cancel_superseded.py, test_ci_linux_compile_runners.py and test_ci_linux_single_size.py pass, and actionlint output is unchanged.

Observed / Not run

  • Observed: the tests above, run locally. test_ci_workflow_run_sources::test_ci_failfast... and test_ci_change_areas::test_repository_helpers_route_by_where_they_run fail the same way on base (no tests job; owned_pool_rescue route).
  • Not run: a live label event on a PR. That's the first label swap after merge.

Changelog

none

Agent session: agent session

🤖 Generated with Claude Code

ci.yml and cmux-vm.yml trigger on labeled/unlabeled for the labels they read
(full-ci, dev-build; cmux-vm-breaking-change-ok). Any other label event, such
as a risk:* swap, now skips every job and runs in a concurrency group of its
own, so it neither queues behind, replaces nor cancels the real run at that
head. tests/test_ci_label_events_skip_full_ci.py fails a label-triggered
workflow that neither guards its jobs nor says why it reacts to every label.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-93043e213c5bbc5a6a17
@teamleaderleo teamleaderleo added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 31, 2026
@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 59f2529: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr29747 -f ref=ci-risk-label-no-rerun -f platform=macos -f backend_mode=local

Shipped in:

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by an agent session · session

@blacksmith-sh

blacksmith-sh Bot commented Oct 31, 2026 •

Copy link
Copy Markdown

Found 21 test failures on Blacksmith runners:

Failures

Test View Logs
1 runner guard check(s) failed:/1 runner guard check(s) failed: View Logs
1 runner guard check(s) failed:/1 runner guard check(s) failed: View Logs
test_big_red (main.FleetRuleLints.test_big_red)/
test_big_red (main.FleetRuleLints.test_big_red)
View Logs
test_dev_build_label_builds_a_head_that_only_merged_main (main.WorkflowTests.test_d
ev_build_label_builds_a_head_that_only_merged_main) (label=False, macos='false', delta_
base='aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa')/
test_dev_build_label_builds_a_head_that_only_merged_main (main.WorkflowTests.test_d
ev_build_label_builds_a_head_that_only_merged_main) (label=False, macos='false', delta_
base='aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa')
View Logs
test_dev_build_label_builds_a_head_that_only_merged_main (main.WorkflowTests.test_d
ev_build_label_builds_a_head_that_only_merged_main) (label=False, macos='true', delta_b
ase='')/
test_dev_build_label_builds_a_head_that_only_merged_main (main.WorkflowTests.test_d
ev_build_label_builds_a_head_that_only_merged_main) (label=False, macos='true', delta_b
ase='')
View Logs
test_dev_build_label_builds_a_head_that_only_merged_main (main.WorkflowTests.test_d
ev_build_label_builds_a_head_that_only_merged_main) (label=True, macos='false', delta_b
ase='')/
test_dev_build_label_builds_a_head_that_only_merged_main (main.WorkflowTests.test_d
ev_build_label_builds_a_head_that_only_merged_main) (label=True, macos='false', delta_b
ase='')
View Logs
test_dev_build_label_builds_a_head_that_only_merged_main (main.WorkflowTests.test_d
ev_build_label_builds_a_head_that_only_merged_main) (label=True, macos='false', delta_b
ase='aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa')/
test_dev_build_label_builds_a_head_that_only_merged_main (main.WorkflowTests.test_d
ev_build_label_builds_a_head_that_only_merged_main) (label=True, macos='false', delta_b
ase='aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa')
View Logs
test_dev_build_label_builds_a_head_that_only_merged_main (main.WorkflowTests.test_d
ev_build_label_builds_a_head_that_only_merged_main) (label=True, macos='true', delta_ba
se='')/
test_dev_build_label_builds_a_head_that_only_merged_main (main.WorkflowTests.test_d
ev_build_label_builds_a_head_that_only_merged_main) (label=True, macos='true', delta_ba
se='')
View Logs
test_every_blacksmith_linux_label_fails_over (main.BlacksmithLinuxFailoverTest.test
_every_blacksmith_linux_label_fails_over)/
test_every_blacksmith_linux_label_fails_over (main.BlacksmithLinuxFailoverTest.test
_every_blacksmith_linux_label_fails_over)
View Logs
test_every_upload_is_short_lived_and_small_or_allowlisted (main.WorkflowTests.test_
every_upload_is_short_lived_and_small_or_allowlisted)/
test_every_upload_is_short_lived_and_small_or_allowlisted (main.WorkflowTests.test_
every_upload_is_short_lived_and_small_or_allowlisted)
View Logs

...and 11 more test failures. View all on Blacksmith

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

Review at 336d0db: a skipped ci-status newer than a red or pending one reads
as passing to gh-merge-green, which takes the newest check by name. ci-status
now runs on every label event (into feat-cmux-next it passes at once), and the
skip applies only to pull requests into feat-cmux-next, so main's required
checks are never superseded by a skipped run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-93043e213c5bbc5a6a17
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high, router-review range 7f26639..5ca065b): SHIP at 5ca065b

No correctness findings. Focused CI tests passed.

Follow-ups: none

The earlier BLOCK at 336d0db (a skipped ci-status superseding a red one under a newest-by-name gate) is fixed at this head: ci-status runs on every label event, and the skip covers only PRs into feat-cmux-next.

The test execution registry needs a job that pull requests into
feat-cmux-next run to execute it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-93043e213c5bbc5a6a17
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high, router-review range 7f26639..e850b06): SHIP at e850b06

Follow-ups: none

Since 5ca065b, this head only adds the new test as a step in ci-guards.yml's ci group, which the test execution registry required.

…guard

Its small evaluator has no '!'; the guard has its own test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-93043e213c5bbc5a6a17
Review at 9d793d0: a label event the workflow skips leaves a newer skipped
run of each job, and newest_checks took it over a failed one, so the batch
admitted a PR whose VM check was red. A skipped newest run now gives way to
the newest run with a verdict, as a cancelled one already did (merge_guard's
newest does the same). Red without the fix, green with it:
tests/test_next_batch_groups.py::GroupCycle::test_newest_run_edge_cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-93043e213c5bbc5a6a17
@teamleaderleo teamleaderleo added risk:high Risk tier (docs/ci/risk-tiers.md) and removed risk:normal Risk tier (docs/ci/risk-tiers.md) labels Oct 31, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high, router-review range 7f26639..59f2529): SHIP at 59f2529

Follow-ups: none

The BLOCK at 9d793d0 (a skipped label run hiding a red VM check from next_batch) is fixed at this head: newest_checks treats a skipped newest run like a cancelled one, with a red/green test.

@teamleaderleo
teamleaderleo merged commit b8ee44e into feat-cmux-next Oct 31, 2026
227 of 241 checks passed
@teamleaderleo
teamleaderleo deleted the ci-risk-label-no-rerun branch October 31, 2026 23:02

This branch was previously deployed

1 inactive deployment
ci-routing-agent — 59f2529d Deployed Oct 31, 2026 by teamleaderleo via cmux-tui tree preflight #91904
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant