Skip to content

fix(ci): base_red_excuse skips base tier first runs that deferred their tree jobs - #29384

Merged
teamleaderleo merged 1 commit into
feat-cmux-nextfrom
dep-fleet-3/base-red-deferred-base-tier
Oct 31, 2026
Merged

teamleaderleo merged 1 commit into
feat-cmux-nextfrom
dep-fleet-3/base-red-deferred-base-tier

Conversation

@teamleaderleo

Copy link
Copy Markdown
Collaborator

Reported: every recent feat-cmux-next push run of cmux-next was cancelled, so the base had no daemon-tests verdict and PRs could not show a daemon-test red is a base red.

What actually happens

  • The push runs' cancelled conclusion comes from the fast tier's job group (cmux-next-push-refs/heads/feat-cmux-next-fast-tier, cancel-in-progress: false): a newer push replaces the pending fast tier job ("Canceling since a higher priority waiting request ... exists"). That is the intended coalesce-to-newest, and nothing that started gets cancelled. Pushes never run daemon tests anyway: since ci(cmux-next): run the base Mac tier at most once every 15 minutes, on the tip #28553 the base Mac tier (daemon tests, generated files, scheme compile) runs from cmux-next base tier · <tip> dispatches.
  • A base tier (or tip daemon suite) dispatch whose cmux-tui tree hasn't published yet skips daemon tests and the scheme compile. It hands them to the same-tree run that the tree publish dispatches, and that run has the same name. The same-tree run then waits in the Mac queue, often for an hour or more.
  • base_red_excuse.tip_suite_verdict reads at most TIP_SUITE_JOB_READS (6) job lists from the newest completed dispatches. Those were mostly the first runs, each with daemon tests skipped, so the reads ran out before the same-tree run that had the verdict. Result: "no completed feat-cmux-next run of its workflow has a verdict for it".

Fix

  • The dispatch fallback now lists dispatches of every status. A suite run that has a newer run of the same name (queued, running or done) is the first run, so it is skipped unread for the tree jobs (TREE_JOBS: daemon tests, scheme compile, crash ends the app). It spends no job-list read.
  • Generated files still read the first run, because path routing's generated_host checks them there and never in the same-tree run.
  • Adds no runs and no Mac time, and changes no workflow or concurrency.

Live check against the current dispatch list: before the change, the first five of the six reads went to runs with daemon tests skipped or failed before the suite. After it, the second read is the base tier same-tree run 41193817088, which has a daemon-tests verdict.

Tests (tests/test_base_red_excuse.py, all 109 pass; the two new behaviour tests fail without the change)

  • First runs that left their tree jobs to a same-tree run are skipped unread, for both daemon tests and the scheme compile.
  • A first run still counts for generated files.
  • TREE_JOBS stays within the suites that defer them.

Not in this PR: same-tree base tier runs sit in the Mac queue for 1 to 2 hours, so the newest base daemon verdict is about that old. That belongs to the Mac queue and admission area.

Agent session: agent session

🤖 Generated with Claude Code

…ir tree jobs

A tip daemon suite or base tier dispatch whose cmux-tui tree has not published
skips daemon tests and the scheme compile and leaves them to the same-tree run
its publish dispatches, under the same run name. The dispatch fallback read
those first runs' job lists and ran out of its six reads before the run with
the daemon-test verdict, so no PR's daemon-test red could be compared with the
base. A suite run with a newer run of the same name is now skipped unread for
the tree jobs; generated files, which the first run checks itself, still read
it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-cabecb079fe356666905
@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review e7b34c2: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr29384 -f ref=dep-fleet-3/base-red-deferred-base-tier -f platform=macos -f backend_mode=local

Shipped in:

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by an agent session · session

@github-actions github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 31, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6.1-sol xhigh, router-review): BLOCK at e7b34c2

BLOCK at e7b34c2

Visual verdict: no images; Leo tries the build

  1. [P1] [blocking] scripts/ci/base_red_excuse.py:158 — Duplicate names do not prove the older run deferred its tree jobs. Publication between marker upload and the workflow’s second probe can make the initial run execute those jobs while also dispatching a same-tree run. If the initial run finishes green while its namesake remains queued, this filter discards that green verdict and can excuse a PR regression using an older red. A targeted reproduction confirms the base version refuses on green run 201, while this head excuses using red run 40. Only discard runs proven to lack the requested verdict, and cover this case in tests.

All 109 existing focused tests pass.

@teamleaderleo teamleaderleo removed the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 31, 2026
@github-actions github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 31, 2026
@blacksmith-sh

blacksmith-sh Bot commented Oct 31, 2026

Copy link
Copy Markdown

Found 1 test failure on Blacksmith runners:

Failure

Test View Logs
test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt
epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)/
test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt
epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex, router-review): SHIP at e7b34c2

Reviewer: codex gpt-6.1-sol xhigh, fast tier.

SHIP at e7b34c2

Visual verdict: no images; Leo tries the build

No correctness findings. Focused tests: 109 passed.

Follow-ups: none

@teamleaderleo teamleaderleo removed the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 31, 2026
@github-actions github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 31, 2026
@teamleaderleo
teamleaderleo merged commit 9f10f35 into feat-cmux-next Oct 31, 2026
226 of 228 checks passed
@teamleaderleo
teamleaderleo deleted the dep-fleet-3/base-red-deferred-base-tier branch October 31, 2026 13:18

This branch was previously deployed

1 inactive deployment
ci-routing-agent — e7b34c2a Deployed Oct 31, 2026 by teamleaderleo via wait for same-key tree publisher #89913
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant