Skip to content

ci(nightly): shorten the release-pool signing chain (about 18 to 15 min to publish) - #33554

Merged
teamleaderleo merged 3 commits into
feat-cmux-nextfrom
lane/nightly-sign-chain
Nov 3, 2026
Merged

teamleaderleo merged 3 commits into
feat-cmux-nextfrom
lane/nightly-sign-chain

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Leo, 2026-11-03: "the signing/publishing don't require 30 cores". The NEXT nightly's release-pool signing chain is mostly waiting, not work. In run 41570134320 it took 18.2 min from the end of build-nightly-app (03:25:28) to publish-nightly's end (03:43:38). The step times show where the waiting is:

  • signer-build needed pick-nightly-sign-runner, so its hosted pickup and cache restore (43 s) sat between the pick and signer-app. Nothing in it reads the pick's outputs since ci(nightly): route credentialed signers through trusted Linux set #30081 put it back on the signer route. It now needs only decide and builds while the app does.
  • signer-app spent 37 s on a full checkout of the build tree, though it reads only scripts/ and the top-level entitlements. It now checks out sparse (sparse-checkout: scripts; cone mode keeps the root files).
  • Apple's clock started only once signer-app had uploaded its artifact and signer-app-notary had been picked up, checked out, and had downloaded and unpacked the app (≈50 s after signing). signer-app now starts the upload to Apple itself (release-signer/submit-app.sh start) right after packing. It runs in the background beside the artifact uploads, and submit-app.sh finish hands the submission id on as an output. signer-app-notary is gone.
  • signer-dmg downloaded the whole 355 MB signed app (50 s) only to read its Info.plist for the appcast. signer-app now hands that Info.plist on as its own small artifact, with its digest as an output, and signer-dmg checks it against the digest.
  • launch-smoke-nightly read the unsigned DMG in one download-artifact stream (205 s). With signer-dmg shorter, that would have put it on publish's path. It now reads in parallel ranges first (fetch_run_artifact.py), as signer-dmg-image and the final-DMG verify already do, with download-artifact as the fallback.

The DMG image step stays on a Mac (hdiutil). Verify DMG and the launch smoke already run side by side (#32378).

What stays

  • Both notarizations, in order. The Mac still staples the app's ticket before it builds the DMG (signer-staple-app.sh polls Apple for the ticket, which exists only for an accepted submission). signer-dmg adds an explicit Accepted check, release-signer/wait-app-notary.sh, before it signs the DMG: notary-wait, then the notary log's status must be Accepted. The DMG's own sign, notarize and staple is unchanged.
  • The Computer Use helper's notarization (sign-app.sh, helper-first) is untouched.
  • The cmux NIGHTLY-identity (legacy) chain. legacy-signer-* jobs are unchanged. They still need signer-build, which now finishes earlier, so the change can only bring them forward.
  • notary_submit (release signer: wait out Apple notary rate limits instead of failing the nightly #33028). submit-app.sh submits through it, so a 429 rate limit is still waited out and retried.
  • The overtaken gate (nightly: spend no notarization on a build a newer published build is above #33109). signer-app still needs pick-nightly-sign-runner with overtaken != 'true'.
  • Keys still live only in a per-job mktemp -d (umask 077) that the script's EXIT trap removes. The background upload owns its own key directory and removes it when the upload ends. No secret, environment or permission changes. Runners are unchanged: GitHub-hosted Ubuntu first, Blacksmith 4 vCPU on a re-run, so tests/test_ci_self_hosted_guard.sh needs no new allowlist line.
  • If Apple rejects the app, the staple still fails at its deadline and signer-dmg never signs, as before. signer-dmg's wait now fails on any answer other than Accepted.

nightly.yml goes from 487,267 to 489,623 bytes (cap 490,000).

Performance

Before: run 41570134320's job and step times. After: projected from those same step times; the first NEXT nightly after merge gives the measured number (follow-up comment).

metric before after (projected) delta harness or run link
end of build-nightly-app → publish-nightly done 18.2 min ≈15.3 min ≈−2.9 min run 41570134320 job/step times
pick done → signer-app start (signer-build hop) 79 s ≈36 s −43 s same
signer-app build checkout 37 s ≈5 s (sparse, as the tools checkout) −32 s same
app signed → Apple submission starts ≈50 s ≈2 s −48 s same
signer-dmg-image end 03:34:41 ≈03:32:40 ≈−2.0 min same
signer-dmg signed-app download 50 s ≈2 s (Info.plist only) −48 s same
launch-smoke unsigned-DMG download 205 s ranged read (signer-dmg-image's took 64 s on the same mini for 355 MB) off publish's path same
signer jobs per run 5 4 −1 hosted job —

Testing

Run locally with system python3, all passing:

  • tests/test_nightly_signer_jobs.py (updated: signer-app-notary gone; submit and wait order; Info.plist digest; signer-build needs only decide; launch smoke's ranged read)
  • tests/test_release_signer_*.py (new AppNotarizationTests: background submit hands on the id; failed upload or missing id fails finish; wait passes only Accepted; key never on argv or in the signer's env; key dirs removed)
  • tests/test_nightly_sign_runner.py, tests/test_nightly_step_coverage.py, tests/test_workflow_file_size.py, tests/test_ci_signing_keychain_is_deleted_before_bundle_code.py, tests/test_ci_nightly_off_blacksmith.py, tests/test_supplychain_security.py, tests/test_ci_self_hosted_guard.sh, plus tests/test_nightly_trusted_routing.py, tests/test_fetch_run_artifact.py and the other tests that read the signer jobs.
  • tests/test_nightly_universal_build.sh fails ("nightly preparation must run on Blacksmith Linux…") identically on base origin/feat-cmux-next; it is not from this change.

Observed / Not run

  • Observed: the step timings above (run 41570134320 via the jobs API), and Apple's ≈35 s upload and 113 s processing in its signer-app-notary log.
  • Not run: a real nightly with these jobs. Only nightly-next with the release environment can sign and notarize. The first nightly after merge is the end-to-end check. If the pool path fails, build-sign-notarize-nightly (the macOS signer) still signs that build, as today.

Changelog

none

risk: high (release signing, CI)

🤖 Generated with Claude Code

Agent session: agent session


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Shortens the release-pool signing chain in the nightly workflow, cutting the time from the end of build-nightly-app to publish from about 18 minutes to about 15 minutes.

  • signer-build now needs only decide; signer-app checks out the build tree sparse and starts Apple's notarization upload in the background, so the signer-app-notary job is gone.
  • signer-dmg waits for Apple's Accepted answer before signing the DMG and reads only the app's Info.plist for the appcast.
  • launch-smoke-nightly downloads the unsigned DMG in parallel ranges first.

Both notarizations stay in order: the Mac still staples the app's ticket before the DMG, and signer-dmg checks the Accepted answer before signing. The legacy NIGHTLY-identity chain, the Computer Use helper notarization, notary_submit retries, and the overtaken gate are unchanged.

Risk: high (release signing, CI).

Written for commit 1ccc9a8. Summary will update on new commits.

View guided diff Turn on auto-fix

signer-build no longer waits for the runner pick; signer-app checks out the
build tree sparse and starts the app's notarization upload itself, beside its
artifact upload, so signer-app-notary goes away; signer-dmg waits for Apple's
Accepted answer before signing the DMG and reads only the app's Info.plist for
the appcast instead of the whole signed app; launch-smoke-nightly reads the
unsigned DMG in parallel ranges first. Both notarizations, the helper's, the
legacy-identity chain, notary_submit and the overtaken gate stay.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-bf341308e0feeab87f14
@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 3ffd4a7: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr33554 -f ref=lane/nightly-sign-chain -f platform=macos -f backend_mode=local

Shipped in:

@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 3, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6.1-sol high): SHIP at 3ffd4a7

No blocking findings. The helper, app and DMG notarization order, the overtaken gate, the legacy jobs, the sparse-checkout dependencies and the digest checks remain intact. Release environments, permissions and runner routes are unchanged. 140 scoped tests passed, plus YAML parsing and shell syntax checks.

Follow-ups:

  • Confirm the new commands against the real rcodesign 0.29 behaviour. Done by the author against the published apple-codesign 0.29.0 crate source: in cli/mod.rs, NotarySubmit without --wait/--staple passes wait_limit = None, and notarize_path returns NotarizationUpload::UploadId after the upload, logging created submission ID: (notarization.rs:277). NotaryWait takes the submission id positionally and calls wait_on_notarization_and_fetch_log. The real behaviour is confirmed on the first nightly after merge (the macOS signer stays the fallback).

@blacksmith-sh

blacksmith-sh Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Found 2 test failures on Blacksmith runners:

Failures

Test View Logs
test_every_compile_job_names_its_warm_sources (main.ThisTree.test_every_compile_job
_names_its_warm_sources)/
test_every_compile_job_names_its_warm_sources (main.ThisTree.test_every_compile_job
_names_its_warm_sources)
View Logs
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl
ights_validate)/
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl
ights_validate)
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@cmux-bot
cmux-bot Bot temporarily deployed to ci-routing-agent November 3, 2026 10:28 Inactive
@cmux-bot
cmux-bot Bot temporarily deployed to ci-routing-agent November 3, 2026 10:28 Inactive
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high, sweep, via the review router; full diff from merge base e50975b to this head): SHIP at 5b32961

No P1/P2 findings.

@cursor

cursor Bot commented Nov 3, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@cmux-bot
cmux-bot Bot temporarily deployed to ci-routing-agent November 3, 2026 12:36 Inactive
@cmux-bot
cmux-bot Bot temporarily deployed to ci-routing-agent November 3, 2026 12:37 Inactive
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6-astra high, carried): SHIP at 1ccc9a8

The head moved only by a base merge (5b32961 to 1ccc9a8). The PR's own diff against its base parent has the same stable patch-id (aee11828e9a2) at both heads, so the SHIP at 5b32961 carries.

@teamleaderleo
teamleaderleo merged commit 6f99cb6 into feat-cmux-next Nov 3, 2026
153 of 161 checks passed
@teamleaderleo
teamleaderleo deleted the lane/nightly-sign-chain branch November 3, 2026 14:12

This branch was previously deployed

1 inactive deployment
ci-routing-agent — 1ccc9a85 Deployed Nov 3, 2026 by cmux-bot[bot] via wait for same-key tree publisher #106080
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant