Skip to content

ci(next): a tip daemon suite gives the base a daemon-test verdict - #28283

Merged
teamleaderleo merged 5 commits into
feat-cmux-nextfrom
lane/same-tree-daemon-base-log
Oct 30, 2026
Merged

teamleaderleo merged 5 commits into
feat-cmux-nextfrom
lane/same-tree-daemon-base-log

Conversation

@teamleaderleo

Copy link
Copy Markdown
Collaborator

Problem

cmux-next daemon tests never runs for a feat-cmux-next push. Its if: skips same_tree_origin == 'push', and a push's own run is superseded before its tree publishes. So scripts/ci/test_reds.py / base_red_excuse.py have had no base log for that job since 04:34Z, and no pull request's daemon red can be compared with the base (#27692 is stuck on 17 red daemon tests, #27868 likewise). Base logs existed only from hand dispatches (41001517417, 41024114885).

Even a dispatch did not reliably help. Its tree wait marker (origin dispatch) was dropped as superseded whenever the tip moved before the tree published, which in a landing burst is every time: 41023525520 and 41022857098 ended with daemon tests skipped.

Change

This follows how tip_swift_suite gives swift-test a base verdict. The dispatcher side is #28261 on main.

  • cmux-next.yml takes a tip_daemon_suite input.
    • Path routing routes only the daemon tier, and only on refs/heads/feat-cmux-next.
    • The run is named cmux-next tip daemon suite · <tested sha>, using same_tree_sha in same-tree mode.
    • daemon-test's own if: is unchanged: a dispatch, and its same-tree run with origin dispatch, already run it.
  • Its tree wait marker carries tip_suite: daemon.
    • cmux_next_tree_notify.py treats such a marker as current after the tip moves, like the DEV daily's: it tests its recorded sha.
    • It passes tip_daemon_suite on, so the same-tree run keeps the name.
    • Only a daemon-only dispatch marker on feat-cmux-next heads may carry it. Anything else is rejected as malformed.
  • base_red_excuse.py looks up tip suite runs by name. If its window of the 100 newest base runs (up to 40 job reads) holds no verdict for cmux-next swift test or cmux-next daemon tests, it lists the base's workflow_dispatch runs and reads the newest tip suite runs of that job by name, at most 6 job lists. Base pushes crowd that window out within a couple of hours. test_reds.classify then judges per test as before. A crash or a cut-short run is still never a base red.

No workflow permission or secret changes.

Performance

Runner minutes added per base push (estimate). The daemon-test job took 9 to 13 minutes in 41024114885 and 41001517417. The base gets about 30 pushes an hour, and the dispatcher (#28261) starts at most 2 daemon suite runs an hour: when the daemon tier changed, or every 20 landings.

metric before after delta harness or run link
Mac runner-minutes per base push (daemon-test) 0 ≤ 0.9 (≤ 2 runs/h × ~13 min ÷ ~30 pushes/h) +≤ 0.9 runs 41024114885, 41001517417
REST reads per daemon/swift excuse lookup with no verdict in the window 1 + 40 1 + 40 + 1 + ≤ 6 +≤ 7 (only on that miss) tests/test_base_red_excuse.py

Tests

These pass on the branch:

  • tests/test_base_red_excuse.py:
    • a daemon red is excused by the tip daemon suite run past 45 base pushes with no verdict
    • a tip suite run green on the base still blocks
    • a DEV daily's jobs are never read
    • a job with no tip suite makes no extra lookup
  • tests/test_cmux_next_tree_notify.py:
    • a tip daemon suite marker dispatches its exact sha with tip_daemon_suite after the tip moved
    • malformed tip_suite markers are ignored
  • tests/test_cmux_next_tip_swift_suite.py:
    • the input, the run name, a daemon-only route and the marker field
  • tests/test_test_reds.py and tests/test_cmux_next_generated_fold.py also pass.

Stop condition: after this and #28261 land, the next base daemon-test result appears without a hand dispatch, and #27692's daemon reds are compared by name against it.

Agent session: agent session

🤖 Generated with Claude Code

A feat-cmux-next push never runs daemon-test: its same-tree run has origin
push, and a dispatch's tree wait marker is superseded the moment the tip moves,
which in a landing burst is always before the tree publishes. So the base had
no "cmux-next daemon tests" verdict for hours, and base_red_excuse.py could
not compare any pull request's daemon red with it (#27692, #27868).

- cmux-next.yml takes tip_daemon_suite (main's tip suite dispatcher asks for
  it): path routing routes only the daemon tier, and the run is named
  `cmux-next tip daemon suite · <tested sha>` (same_tree_sha in same-tree mode).
- Its tree wait marker carries tip_suite=daemon. cmux_next_tree_notify.py
  starts it on the recorded sha after the tip moves (like the DEV daily) and
  passes tip_daemon_suite on, so the same-tree run keeps the name. Only a
  daemon-only dispatch on feat-cmux-next heads may carry it.
- base_red_excuse.py: when its window of the newest 100 base runs holds no
  verdict for swift test or daemon tests, it reads the base's newest tip suite
  runs of that job by name (at most 6 job lists).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by an agent session · session

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 026fafd: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr28283 -f ref=lane/same-tree-daemon-base-log -f platform=macos -f backend_mode=local

web and size checks: green

@github-actions github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 30, 2026
@blacksmith-sh

blacksmith-sh Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Found 2 test failures on Blacksmith runners:

Failures

Test View Logs
test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt
epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)/
test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt
epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)
View Logs
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl
ights_validate)/
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl
ights_validate)
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high, next-review): BLOCK at b0b1b0e

  • [P1] scripts/ci/base_red_excuse.py:228 — The fallback can excuse a regression using an obsolete failure. It filters out ordinary dispatches before checking their verdicts. After 40 newer runs exhaust the initial lookup, a passing ordinary base dispatch is ignored while an older failing tip suite is accepted. I reproduced this with the supplied fake GitHub harness: judge() excused the PR despite the newer base daemon result being green. Preserve the newest-verdict rule across dispatch types, or refuse when newer verdicts remain unchecked.

…y lane names

#27840 registered tests/test_cmux_tui_owned_mini_caches.py as lane
"linux-guard" but no ci-guards.yml step ran it, so the execution registry
validator fails every pull request into feat-cmux-next ("linux-guard lane is
not run by any workflow"). The release-notary group of ci-guards now runs it,
beside the other cmux-tui artifact guards.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 43312b4)
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Head is now c0c10bc0e8ac788138a35a39d25b02d52f5b03a3.

tooling-tests (job 124018292337): the registry error was tests/test_cmux_tui_owned_mini_caches.py: linux-guard lane is not run by any workflow. The base fails the same way: #27840 registered the file as linux-guard, but no ci-guards.yml step ever ran it. This head adds that step in the release-notary group, and validate_test_execution_registry.py now passes. This PR adds no new test files; its tests are edits to files that already have registry entries. The same commit also stands alone for other PRs as branch lane/owned-mini-caches-guard (43312b47).

daemon tests (job 124022634651): one failing test, theFirstLaunchBringsEverythingOnce(_:) in the multiWindow case (ClassicUpgradeSimulationTests.swift:45 and :52). Two of the three classic workspaces come over in swapped order. This PR changes no Swift and no daemon code, only CI workflow and scripts, so the failure is not caused by this diff. I could not compare it by name, though: the only base daemon logs since 04:34Z did not run the suites at all.

  • 41024114885 at d62d4d75: compile error in CloudAppLinkOwnerAnswerTests.swift:22 (main-actor isolation).
  • 41033244125 at 7050ae30: no published cmux-tui tree.

That gap, no base daemon-test log that ran the suites, is exactly what this PR and #28261 fix. Once a tip daemon suite run completes, base_red_excuse.py can classify this test by name.

test_a_dispatch_whose_tree_was_building_gets_its_daemon_tests_later runs the
marker step's script with its own env, which has no TIP_DAEMON_SUITE, and the
KeyError failed "Checks job structure" on #28283. The script reads it with
.get(), and the test now covers the tip daemon suite marker: tip_suite=daemon
only for a tip_daemon_suite dispatch on feat-cmux-next.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review: this branch's base predates #27840, so tests/test_cmux_tui_owned_mini_caches.py
is not in the branch and the step only passed on the merge commit with the
base. The registry error it fixed is a base red (feat-cmux-next fails the same
way); its fix goes in its own PR from a branch that has the file.

This reverts commit c0c10bc.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ncluded

Review of #28283: the fallback read only tip suite runs, so a newer ordinary
dispatch that ran daemon-test (or swift-test) green was skipped and an older
red tip suite run excused the head. It now reads the base's completed
dispatches newest first and the first with a verdict for the job decides, a
green one included. DEV daily runs and the other job's tip suite never run the
job and are skipped unread; at most 6 job lists. Test: a newer green ordinary
dispatch over an older red tip suite refuses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@teamleaderleo
teamleaderleo merged commit 3981014 into feat-cmux-next Oct 30, 2026
143 of 147 checks passed
@teamleaderleo
teamleaderleo deleted the lane/same-tree-daemon-base-log branch October 30, 2026 18:59
@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

This branch was previously deployed

1 inactive deployment
ci-routing-agent — 026fafd6 Deployed Oct 30, 2026 by teamleaderleo via cmux-tui tree preflight #84278
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant