ci(nightly-next): run verify-signed's checks on the DMG Mac, beside the DMG by teamleaderleo · Pull Request #27508 · manaflow-ai/cmux

teamleaderleo · 2026-10-29T19:03:41Z

Summary

A pool-signed cmux NEXT nightly sent its stapled app to two office Macs: signer-dmg-image (to build the DMG) and verify-signed-nightly (a gzip that signer-dmg made on Linux, downloaded again over the office link). That second download took 2-5 min in recent runs and set the length of the verify stage whenever it outran verify-final-dmg-nightly.

signer-dmg-image now runs verify-signed-nightly's checks itself (scripts/ci/nightly-steps/signer-verify-signed.sh: version surfaces, the signed diff sidecar contract, check-signed-app.sh with the Computer Use helper ticket required, the bundled Ghostty theme picker regression, the SSH daemon contract), on a copy of the app it already holds.
Those checks and the existing signed-app checks start in the background after Apple's checks of the stapled app, run while the DMG is built and uploaded, and the job fails after the upload when any of them failed. signer-dmg needs signer-dmg-image, so it never signs a DMG whose app failed a check.
verify-signed-nightly runs for the macOS signer's app only. publish-nightly takes signer-dmg's success in its place for a pool-signed app (its if and its OWN_VERIFIED).
signer-dmg no longer packages or uploads the verify handoff; signer-handoff.sh is removed.

No check is dropped: verify-signed-nightly's codesign --verify --deep --strict is the first of signer-apple-checks.sh's checks, which the job runs before it starts the others. No secret reaches the Mac job. Nothing changes in build-nightly-app, prepare, or the macOS fallback signer path.

Performance

Chain = build-nightly-app end to publish-nightly end. The before rows are the last five arm64 pool-signed runs with a publish, from the jobs API. The after column is filled from real nightly-next runs once this lands.

metric	before	after	delta	harness or run link
verify stage, max(verify-signed, verify-final)	3.2, 5.8, 2.8, 2.6, 2.5 min	pending	expected -0.0 to -3.9 min (verify-final alone: 3.0, 1.9, 2.8, 2.6, 1.9)	runs 40893555177, 40905688542, 40909458084, 40913618519, 40918131595
verify-signed download of the signed app	120-321 s	0 (no second download)	-120 to -321 s of office-link traffic per run	same runs
signer-dmg-image, checks then DMG	3.0-5.5 min (checks 38-73 s run before the DMG)	pending	expected about -0.6 min (checks overlap the DMG build and upload)	same runs
chain (build end to publish end)	16.5, 17.9, 17.7, 15.4, 15.9 min	pending	expected about -1 to -4 min	same runs
Mac jobs per pool-signed run	3 (dmg-image, verify-signed, verify-final)	2	-1	nightly.yml job graph

Testing

python3 tests/test_nightly_signer_jobs.py: 20 tests pass, including new SignerVerifySigned tests that run the script with stub checks (every check runs on a copy; a failure fails finish after the rest ran; bundled-assets and CLI flags match verify-signed-nightly; nothing a check writes reaches the app the DMG is built from) and the wiring test for the new step order.
python3 tests/test_nightly_step_coverage.py and python3 tests/test_nightly_next_recovery.py pass.
Every other tests/test_*nightly* / workflow test that reads nightly.yml passes here, except tests/test_nightly_next_arm64_only.py and tests/test_check_ghostty_zig_workflows.py, which fail the same way on base in this Linux environment (a git rev-parse outside a repo; no bashlex).

Observed / Not run

Observed: the tests above on Linux. Not run: a real nightly-next run with this change; the first runs after it lands fill the Performance table.

Changelog

none

🤖 Generated with Claude Code

Agent session: agent session

…he DMG A pool-signed nightly-next build sent its stapled app to two office Macs: signer-dmg-image (to build the DMG) and verify-signed-nightly (gzipped on Linux by signer-dmg, then downloaded again). That second download took 2-5 min in runs 40905688542, 40913618519 and 40918131595 and set the chain's verify stage whenever it outran verify-final-dmg-nightly. signer-dmg-image now runs verify-signed-nightly's checks (signer-verify-signed.sh: version surfaces, the signed diff sidecar contract, check-signed-app.sh with the helper ticket required, the bundled Ghostty theme picker regression, the SSH daemon contract) and the signed-app checks on copies of the app in the background while the DMG is built and uploaded, and fails after the upload when one fails, so signer-dmg never signs that DMG. verify-signed-nightly runs for the macOS signer's app only, publish-nightly takes signer-dmg's success for a pool-signed app, and signer-dmg no longer packages or uploads the verify handoff (signer-handoff.sh is gone). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-a7306ef145c005ad24ab

github-actions · 2026-10-29T19:03:56Z

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review a71f97c: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr27508 -f ref=dep-fleet--nightly-chain-verify-on-dmg-mac -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); NEW: green on the base 37b93da; do not merge until green; see lane-rules

blacksmith-sh · 2026-10-29T19:09:20Z

Found 14 test failures on Blacksmith runners:

Failures

Test	View Logs
`/tmp/tmp1zdrsopv/summary-fail.json (live):/summary-fail.json (live):`	View Logs
`/tmp/tmp68l1fr9y/summary-missing.json (live):/summary-missing.json (live):`	View Logs
`1 runner guard check(s) failed:/1 runner guard check(s) failed:`	View Logs
`1 runner guard check(s) failed:/1 runner guard check(s) failed:`	View Logs
`[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/` `[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows`	View Logs
`[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames`	View Logs
`[tabs] debug.tab_switch timed 0 of 40 switches/` `[tabs] debug.tab_switch timed 0 of 40 switches`	View Logs
`[tabs] the window draws no frames/[tabs] the window draws no frames`	View Logs
`test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt` `epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)/` `test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt` `epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)`	View Logs
`test_every_upload_is_short_lived_and_small_or_allowlisted (main.WorkflowTests.test_` `every_upload_is_short_lived_and_small_or_allowlisted)/` `test_every_upload_is_short_lived_and_small_or_allowlisted (main.WorkflowTests.test_` `every_upload_is_short_lived_and_small_or_allowlisted)`	View Logs

...and 4 more test failures. View all on Blacksmith

^{Need help on this PR? Tag @codesmith-bot with what you need.}

…erify handoff upload Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-a7306ef145c005ad24ab

teamleaderleo · 2026-10-29T19:15:21Z

Bead cx-x6mnhd created for this PR by the cmux coordinator.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-a7306ef145c005ad24ab

teamleaderleo · 2026-10-29T21:40:14Z

Review (codex gpt-6-astra high, next-review): SHIP at a71f97c

No blocking findings.

cmux-bot

cmux-bot approves a71f97c: an independent review passed this exact head (next-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

teamleaderleo · 2026-10-29T22:18:31Z

Owned by an agent session · session

…own, resumed or recovered Review findings on 18c51bd: - publish-nightly's source selection takes this run's own build only when await-nightly-tree-marker passed (or the build never waited for its tree), so a recovered build can no longer stand in while the own build is used; - the marker check also runs for an unsigned resume, against the commit the resumed app installed; - recovery refuses a run whose marker check failed, was cancelled or has not finished (recover-tree-marker-ok.sh); - await-tree-client.sh reads each candidate's own artifacts runs before the newest-50 listing, so a crowded listing never reads as no publisher. publish-nightly's own if clause is left as it is (#27508 changes it). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-a7306ef145c005ad24ab

github-actions · 2026-10-30T00:40:45Z

Shipped in:

cmux NEXT nightly 1.0.0-nightly.4097664234401 (167472006a)

…own, resumed or recovered Review findings on 18c51bd: - publish-nightly's source selection takes this run's own build only when await-nightly-tree-marker passed (or the build never waited for its tree), so a recovered build can no longer stand in while the own build is used; - the marker check also runs for an unsigned resume, against the commit the resumed app installed; - recovery refuses a run whose marker check failed, was cancelled or has not finished (recover-tree-marker-ok.sh); - await-tree-client.sh reads each candidate's own artifacts runs before the newest-50 listing, so a crowded listing never reads as no publisher. publish-nightly's own if clause is left as it is (#27508 changes it). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-a7306ef145c005ad24ab

* ci(nightly-next): build the tip at once and install its tree's client when the tree publishes A tree-changing landing waited about 27 minutes for its cmux-tui tree (build, R2 upload, tree publish, marker) before its NEXT nightly could start. Main's promote guard now accepts such a commit when its nightly.yml carries NIGHTLY_NEXT_TREE_AT_INSTALL: v1 (#27498). This is that nightly: - resolve-nightly-cmux-tui-client: a tip whose tree is a clean 404 is built as it is (tree_pending, with its same-key first-parent candidates); - build-nightly-app compiles at once. Bundle acpmux gets a stand-in input (the phase only aliases bin/acpmux to bin/cmux), and "Bundle the cmux-tui client" waits for the tree's attested manifest (await-tree-client.sh), then installs it with the same install-cmux-tui-client.sh command; - await-nightly-tree-marker gates publish-nightly on the build commit's tree-commit marker naming the installed client's commit (G5, in the run); - request-nightly-next.sh requests such a tip while an artifacts run builds its tree. nightly.yml stays under its size limit: two steps live in scripts/ci/nightly-steps/ and some comments in the touched jobs are shorter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-a7306ef145c005ad24ab * ci(nightly-next): the tree marker gates the build the run publishes, own, resumed or recovered Review findings on 18c51bd: - publish-nightly's source selection takes this run's own build only when await-nightly-tree-marker passed (or the build never waited for its tree), so a recovered build can no longer stand in while the own build is used; - the marker check also runs for an unsigned resume, against the commit the resumed app installed; - recovery refuses a run whose marker check failed, was cancelled or has not finished (recover-tree-marker-ok.sh); - await-tree-client.sh reads each candidate's own artifacts runs before the newest-50 listing, so a crowded listing never reads as no publisher. publish-nightly's own if clause is left as it is (#27508 changes it). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-a7306ef145c005ad24ab * ci(nightly): shorter warm-checkout comment keeps nightly.yml under its size limit Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-a7306ef145c005ad24ab * ci(nightly-next): the recovery marker check reads one bounded job list (no gh api paginate) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-a7306ef145c005ad24ab * test(nightly): sign variants require the client build-nightly-app bundled The sign variant now reads build-nightly-app.outputs.cmux_tui_commit first (the commit await-tree-client.sh found while the tip tree publishes), so the universal-build guard accepts that expression instead of the resolver output alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-d35d25d288d130b0c84c * fix(nightly-next): gate resume with bundled tree client Agent-Session: https://cmux.com/deeplink/agent?id=aid-d35d25d288d130b0c84c * fix(nightly-next): gate resume with bundled tree client Agent-Session: https://cmux.com/deeplink/agent?id=aid-d35d25d288d130b0c84c * fix(nightly-next): carry bundled client through resume receipt Agent-Session: https://cmux.com/deeplink/agent?id=aid-d35d25d288d130b0c84c * fix(nightly-next): publish when marker wait is skipped Agent-Session: https://cmux.com/deeplink/agent?id=aid-d35d25d288d130b0c84c * fix(nightly-next): use synthetic tree key for resumed build Agent-Session: https://cmux.com/deeplink/agent?id=aid-d35d25d288d130b0c84c * fix(nightly-next): read resumed client from app archive Agent-Session: https://cmux.com/deeplink/agent?id=aid-d35d25d288d130b0c84c --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

teamleaderleo requested a review from lawrencecchen as a code owner October 29, 2026 19:03

teamleaderleo added the risk:high Risk tier (docs/ci/risk-tiers.md) label Oct 29, 2026

ci(nightly-next): drop the retention allowlist entry of the removed v…

b7941ba

…erify handoff upload Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-a7306ef145c005ad24ab

teamleaderleo temporarily deployed to ci-routing-agent October 29, 2026 19:13 — with GitHub Actions Inactive

teamleaderleo mentioned this pull request Oct 29, 2026

ci(nightly-next): notarize the app beside the DMG Mac's download, staple it there #27518

Closed

ci(nightly-next): the universal-build test reads the new publish gate

a71f97c

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-a7306ef145c005ad24ab

teamleaderleo temporarily deployed to ci-routing-agent October 29, 2026 19:28 — with GitHub Actions Inactive

cmux-bot Bot approved these changes Oct 29, 2026

View reviewed changes

cmux-bot Bot removed the risk:high Risk tier (docs/ci/risk-tiers.md) label Oct 29, 2026

github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Oct 29, 2026

teamleaderleo merged commit f5082ea into feat-cmux-next Oct 29, 2026
161 of 168 checks passed

teamleaderleo deleted the dep-fleet--nightly-chain-verify-on-dmg-mac branch October 29, 2026 23:13

teamleaderleo mentioned this pull request Oct 30, 2026

carry of #27518: notarize app beside DMG download #28183

Merged

Navigation Menu

Repository navigation

ci(nightly-next): run verify-signed's checks on the DMG Mac, beside the DMG - #27508

ci(nightly-next): run verify-signed's checks on the DMG Mac, beside the DMG#27508
teamleaderleo merged 3 commits into
feat-cmux-nextfrom
dep-fleet--nightly-chain-verify-on-dmg-mac

teamleaderleo commented Oct 29, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 29, 2026 •

edited

Loading

Uh oh!

blacksmith-sh Bot commented Oct 29, 2026 •

edited

Loading

Uh oh!

teamleaderleo commented Oct 29, 2026

Uh oh!

teamleaderleo commented Oct 29, 2026

Uh oh!

cmux-bot Bot left a comment

Uh oh!

teamleaderleo commented Oct 29, 2026

Uh oh!

Uh oh!

github-actions Bot commented Oct 30, 2026

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

teamleaderleo commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Performance

Testing

Observed / Not run

Changelog

Uh oh!

github-actions Bot commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

blacksmith-sh Bot commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Failures

Uh oh!

teamleaderleo commented Oct 29, 2026

Uh oh!

teamleaderleo commented Oct 29, 2026

Uh oh!

cmux-bot Bot left a comment

Choose a reason for hiding this comment

Uh oh!

teamleaderleo commented Oct 29, 2026

Uh oh!

Uh oh!

github-actions Bot commented Oct 30, 2026

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

teamleaderleo commented Oct 29, 2026 •

edited

Loading

github-actions Bot commented Oct 29, 2026 •

edited

Loading

blacksmith-sh Bot commented Oct 29, 2026 •

edited

Loading