Skip to content

ci(nightly-next): the Ghostty helper takes an idle owned Mac while the app build holds the dedicated builder - #27894

Merged
teamleaderleo merged 4 commits into
feat-cmux-nextfrom
lane/nightly-helper-owned-route
Oct 31, 2026
Merged

teamleaderleo merged 4 commits into
feat-cmux-nextfrom
lane/nightly-helper-owned-route

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

NEXT nightly fell 3h25m behind feat-cmux-next on 2026-10-30 (the rule is at most 30 minutes). Two causes, the second fixed here:

  1. A Release compile red from about 23:40Z to 01:23Z: JoinConsent+App.swift:222:29: error: 'async' call in a function that does not support concurrency failed build-nightly-app for every promoted tip from 43303f7 to 00c873b. cmux NEXT: await FolderChanges.events() in MachineConsentChanges (build fix) #27758 (cec0a36) fixed it.
  2. The first green tip after that, 373ec0c (run 40990112293), built its app on the dedicated builder by 02:30Z. Its Ghostty CLI helper then sat queued for 39 minutes on blacksmith-6vcpu-macos-26, and the run was cancelled unpublished at 02:51Z. The helper reuse lookup had missed (the repo GITHUB_TOKEN got a 403 from the contents API), and the helper never takes the dedicated builder, whose hook admits only build-nightly-app. So the helper's runs-on fell to blacksmith, even though owned side Macs were idle.

The change: pick-nightly-build-runner now also outputs helper.

  • helper is owned when owned is not the dedicated builder.
  • When the app build has the dedicated builder, helper comes from a fresh owned_route() read: an idle side or nightly-slot Mac. The verify jobs already pick their Mac this way.
  • With no idle Mac, helper is empty and the helper takes the moving Blacksmith label, as before.

build-nightly-ghostty-cli-helper now reads helper instead of owned, on the same rule: the pick must be for this attempt, and it is never the dedicated builder. The job output stands on the same terms as owned (marker uploaded outside attempt 1 of a push).

Risk: low. The change only affects routing of the helper job on nightly-next. The fallback is still Blacksmith macOS 26, and the dedicated builder is still excluded.

Tests (run on a Linux lane VM):

  • tests/test_nightly_build_runner.py: 33 tests OK, including a new test, the helper takes an idle owned Mac while the app build holds the builder.
  • tests/test_nightly_trusted_routing.py: 22 tests OK.
  • tests/test_ci_nightly_off_blacksmith.py: 15 tests OK.
  • tests/test_nightly_next_train.py: 38 tests OK.
  • tests/test_nightly_sign_runner.py: OK.
  • tests/test_nightly_universal_build.sh: PASS.
  • test_nightly_next_guard: PASS.
  • test_nightly_workflow_size: PASS.
  • test_ci_workflow_guards_are_wired: PASS.

tests/test_ci_signing_owned_mac.py fails one test (test_every_listed_hook_is_one_glaeda_released_with_the_check) on base too.

Agent session: agent session

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Fixes a nightly-next CI routing gap where the Ghostty CLI helper queued on Blacksmith while idle owned Macs were available, leaving nightly-next 3h25m behind feat-cmux-next after a Release compile red.

Because the dedicated builder admits only build-nightly-app, the helper previously fell back to the Blacksmith label whenever the app build held the builder, even with idle side Macs. pick-nightly-build-runner now also outputs helper: owned when that isn't the dedicated builder, otherwise a fresh owned-route read for an idle side or nightly-slot Mac, or empty so the helper keeps taking Blacksmith when no Mac is idle. build-nightly-ghostty-cli-helper reads the new output on the same marker terms as owned.

Written for commit 5e5b42c. Summary will update on new commits.

View guided diff Turn on auto-fix

…e app build holds the dedicated builder

pick-nightly-build-runner now prints a helper route: owned when that is not
the dedicated builder, else a fresh owned_route() read. Until now
build-nightly-ghostty-cli-helper took blacksmith whenever the app build took
the dedicated builder; in run 40990112293 (373ec0c, the first green tip
after the JoinConsent Release compile red) the helper queued 39 minutes on
blacksmith-6vcpu-macos-26 while owned side Macs were idle, and the run was
cancelled unpublished.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@teamleaderleo

teamleaderleo commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator Author

Owned by an agent session · took over from an earlier session · session

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 5e5b42c: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr27894 -f ref=lane/nightly-helper-owned-route -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); NEW: red check: Prebuilt web bundles install and stamp (+2 more) is not red on the base 625035e; do not merge until green; see lane-rules

Shipped in:

@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Oct 30, 2026
@blacksmith-sh

blacksmith-sh Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Found 21 test failures on Blacksmith runners:

Failures

Test View Logs
test_every_linux_nightly_job_takes_a_runner_set_member (main.NightlyTrustedRouting.
test_every_linux_nightly_job_takes_a_runner_set_member) (job='attest-nightly-remote-dae
mon', runner='', primary='failure')/
test_every_linux_nightly_job_takes_a_runner_set_member (main.NightlyTrustedRouting.
test_every_linux_nightly_job_takes_a_runner_set_member) (job='attest-nightly-remote-dae
mon', runner='', primary='failure')
View Logs
test_every_linux_nightly_job_takes_a_runner_set_member (main.NightlyTrustedRouting.
test_every_linux_nightly_job_takes_a_runner_set_member) (job='attest-nightly-remote-dae
mon', runner='', primary='success')/
test_every_linux_nightly_job_takes_a_runner_set_member (main.NightlyTrustedRouting.
test_every_linux_nightly_job_takes_a_runner_set_member) (job='attest-nightly-remote-dae
mon', runner='', primary='success')
View Logs
test_every_linux_nightly_job_takes_a_runner_set_member (main.NightlyTrustedRouting.
test_every_linux_nightly_job_takes_a_runner_set_member) (job='attest-nightly-remote-dae
mon', runner='blacksmith-16vcpu-ubuntu-2404', primary='failure')/
test_every_linux_nightly_job_takes_a_runner_set_member (main.NightlyTrustedRouting.
test_every_linux_nightly_job_takes_a_runner_set_member) (job='attest-nightly-remote-dae
mon', runner='blacksmith-16vcpu-ubuntu-2404', primary='failure')
View Logs
test_every_linux_nightly_job_takes_a_runner_set_member (main.NightlyTrustedRouting.
test_every_linux_nightly_job_takes_a_runner_set_member) (job='attest-nightly-remote-dae
mon', runner='blacksmith-16vcpu-ubuntu-2404', primary='success')/
test_every_linux_nightly_job_takes_a_runner_set_member (main.NightlyTrustedRouting.
test_every_linux_nightly_job_takes_a_runner_set_member) (job='attest-nightly-remote-dae
mon', runner='blacksmith-16vcpu-ubuntu-2404', primary='success')
View Logs
test_every_linux_nightly_job_takes_a_runner_set_member (main.NightlyTrustedRouting.
test_every_linux_nightly_job_takes_a_runner_set_member) (job='attest-nightly-remote-dae
mon', runner='blacksmith-32vcpu-ubuntu-2404', primary='failure')/
test_every_linux_nightly_job_takes_a_runner_set_member (main.NightlyTrustedRouting.
test_every_linux_nightly_job_takes_a_runner_set_member) (job='attest-nightly-remote-dae
mon', runner='blacksmith-32vcpu-ubuntu-2404', primary='failure')
View Logs
test_every_linux_nightly_job_takes_a_runner_set_member (main.NightlyTrustedRouting.
test_every_linux_nightly_job_takes_a_runner_set_member) (job='attest-nightly-remote-dae
mon', runner='blacksmith-32vcpu-ubuntu-2404', primary='success')/
test_every_linux_nightly_job_takes_a_runner_set_member (main.NightlyTrustedRouting.
test_every_linux_nightly_job_takes_a_runner_set_member) (job='attest-nightly-remote-dae
mon', runner='blacksmith-32vcpu-ubuntu-2404', primary='success')
View Logs
test_every_linux_nightly_job_takes_a_runner_set_member (main.NightlyTrustedRouting.
test_every_linux_nightly_job_takes_a_runner_set_member) (job='attest-nightly-remote-dae
mon', runner='blacksmith-4vcpu-ubuntu-2404', primary='failure')/
test_every_linux_nightly_job_takes_a_runner_set_member (main.NightlyTrustedRouting.
test_every_linux_nightly_job_takes_a_runner_set_member) (job='attest-nightly-remote-dae
mon', runner='blacksmith-4vcpu-ubuntu-2404', primary='failure')
View Logs
test_every_linux_nightly_job_takes_a_runner_set_member (main.NightlyTrustedRouting.
test_every_linux_nightly_job_takes_a_runner_set_member) (job='attest-nightly-remote-dae
mon', runner='blacksmith-4vcpu-ubuntu-2404', primary='success')/
test_every_linux_nightly_job_takes_a_runner_set_member (main.NightlyTrustedRouting.
test_every_linux_nightly_job_takes_a_runner_set_member) (job='attest-nightly-remote-dae
mon', runner='blacksmith-4vcpu-ubuntu-2404', primary='success')
View Logs
test_every_linux_nightly_job_takes_a_runner_set_member (main.NightlyTrustedRouting.
test_every_linux_nightly_job_takes_a_runner_set_member) (job='attest-nightly-remote-dae
mon', runner='ubuntu-24.04', primary='failure')/
test_every_linux_nightly_job_takes_a_runner_set_member (main.NightlyTrustedRouting.
test_every_linux_nightly_job_takes_a_runner_set_member) (job='attest-nightly-remote-dae
mon', runner='ubuntu-24.04', primary='failure')
View Logs
test_every_linux_nightly_job_takes_a_runner_set_member (main.NightlyTrustedRouting.
test_every_linux_nightly_job_takes_a_runner_set_member) (job='attest-nightly-remote-dae
mon', runner='ubuntu-24.04', primary='success')/
test_every_linux_nightly_job_takes_a_runner_set_member (main.NightlyTrustedRouting.
test_every_linux_nightly_job_takes_a_runner_set_member) (job='attest-nightly-remote-dae
mon', runner='ubuntu-24.04', primary='success')
View Logs

...and 11 more test failures. View all on Blacksmith

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high, router-review): SHIP at fafe629

SHIP

No blocking findings.

teamleaderleo and others added 2 commits October 30, 2026 04:25
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high, next-review): SHIP at ae374c8

No blocking findings.

cmux-bot[bot]
cmux-bot Bot previously approved these changes Oct 30, 2026

@cmux-bot cmux-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cmux-bot approves ae374c8: an independent review passed this exact head (next-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

For this PR's owner: the drain lander carried this PR as #28245 (current feat-cmux-next merged, to refresh its old-base checks). I didn't close this PR, because pr-owner shows you alive. #28245 is a draft. Either base-merge this PR yourself or say here that #28245 should carry it.

@cmux-bot
cmux-bot Bot dismissed their stale review October 30, 2026 10:15

A newer verdict blocks 5e5b42c (no independent SHIP at the head).

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high, next-review): SHIP at 5e5b42c

No blocking findings.

@cmux-bot cmux-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cmux-bot approves 5e5b42c: an independent review passed this exact head (next-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

@teamleaderleo
teamleaderleo merged commit 0a67ab0 into feat-cmux-next Oct 31, 2026
168 of 178 checks passed
@teamleaderleo
teamleaderleo deleted the lane/nightly-helper-owned-route branch October 31, 2026 01:29

This branch was previously deployed

1 inactive deployment
ci-routing-agent — 5e5b42ce Deployed Oct 30, 2026 by teamleaderleo via cmux-tui tree preflight #82247
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant