Repository navigation
nightly-next: every build ships both Apple silicon and Intel DMGs - #23811
austinywang wants to merge 27 commits into
Conversation
nightly-next has built arm64 only since #18424, so Intel Macs got no new cmux NEXT after 2026-10-07 and the release kept a stale x86_64 DMG. decide now builds thin arm64 and x86_64 on nightly-next (two notarizations, no universal) and marks x86_64 optional. Each matrix job lets an optional leg fail alone; only a required leg sets notary_pending and the stapled-app fallback id, and an optional leg never hands a pending notarization to the arm64 recovery. publish-nightly publishes x86_64 only when this run's own build passed every leg for it (tested final DMG plus a per-variant signed-app record); otherwise it drops those files and the Intel DMG and feed keep their last build. Deltas run only for published variants. The release title carries the build number, since GitHub keeps showing the first publish date of the release that is edited in place. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…arm64's stapled fallback Review follow-up: a failed optional x86_64 leg leaves every job green, so report-nightly-next-intel keeps one nightly-next-intel-failure incident open while a published build skips Intel and closes it when Intel publishes again. The x86_64 verify legs no longer read arm64's stapled-app fallback id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
cmux status (one comment, edited in place; each line is updated by the job that owns it) Review App of this head (app.zip, kept 1 day)
web and size checks: green |
|
Correctness review (independent subagent) at 366edc0: FIX. Findings and how each was resolved:
The reviewer also confirmed that the "Choose the variants to publish" bash, the digest gate and the delta gating are correct, and that main and RC are unchanged. |
|
Found 6 test failures on Blacksmith runners: Failures
|
|
Bead |
…hips with its Intel leg Per Austin: every nightly-next build ships the Apple silicon and the Intel app together, never one without the other. This drops the optional x86_64 machinery (job-level continue-on-error, the published-variant choice and the separate Intel incident): a failed x86_64 leg fails the run and the normal nightly-next failure incident reports it. Only the arm64 leg still hands a slow notarization to a later run; the x86_64 leg waits 40m like main's legs, and its final DMG is smoked even while arm64 is pending. The recovery continues a pending arm64 build only when that build's own run has a verified x86_64 leg (variant, tested digest of the exact bytes, signed-app record for the same build), checked before any notarization step, and uploads it as the recovered Intel variant. publish-nightly refuses a build missing any decided variant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review of 2e43e6d: a build whose arm64 notarize step failed still leaves a recovery artifact, but both verify jobs were gated on the whole sign job's success, so its x86_64 was never verified and every later recovery skipped it. On nightly-next the verify jobs now also run after a failed sign job (each leg fails on its own without its artifact); the own publish still needs the sign job to pass. The recovery also requires the candidate's arm64 signed-app record, and the tests cover a wrong Intel digest and a missing arm64 record. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Correctness review (independent subagent) at 2e43e6d: FIX. Requirement (a), no nightly-next publish with only one architecture, was found to hold on every path, including recovered builds. Findings and resolution:
Test gaps: recovery tests now cover a wrong Intel digest and a missing arm64 record. |
|
Review (independent correctness subagent): SHIP at b5324c0 Verified: this run's own build still cannot publish after a failed sign job (publish-nightly |
nightly-next now ships an x86_64 app with every build, so the same Release compile (nightly Xcode, -O, whole-module) can check the Intel slice ahead of the nightly. The default stays arm64. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… own PR The fleet runs an allowlisted script only as reviewed on feat-cmux-next, so the option must land first in a separate PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bugbot needs on-demand usage enabledBugbot uses usage-based billing for this team and requires on-demand usage to be enabled. A team admin can enable on-demand usage in the Cursor dashboard. |
With NIGHTLY_NEXT_DEV_RING on (#23835), every new build is published as cmux-next-dev, and the verdict moved only appcast-arm64.xml to Nightly. Now that each nightly-next build also ships x86_64, its appcast-x86_64.xml item would have stayed on Dev, so a Mac on the Nightly ring would never be offered an Intel update. The promotion step retags both cmux NEXT feeds from the same verdicts. The old cmux NIGHTLY-identity feed stays arm64 only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Intel compile: green. Fleet step The earlier build_only dispatch (run 39688351887) failed only because a branch dispatch runs as the Head 46d3262 merges feat-cmux-next and adds one fix. With The new visual smoke (#23780) runs on the x86_64 leg through |
Review of 46d3262: the rollback the canary dispatches pulled a bad build off appcast-arm64.xml only, so appcast-x86_64.xml kept offering it to Intel Macs. New job rollback-intel drops every Intel item newer than the build the arm64 feed now serves (nightly_next_feed_rollback.py plan --above, which need not find that build on the Intel feed), checks the fallback DMG, writes with If-Match and reads it back. It is its own job so a failure is a red run that never holds the fix-forward, and it runs when arm64 was already pulled, so a re-run converges. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Review (independent correctness subagent): SHIP at d9b6253 The review of 46d3262 (FIX) had one blocker: a rollback pulled a bad build off Verified by the reviewer:
Tests pass: feed rollback, variants, recovery (52), ring (28), universal build, bash 3.2 parse and actionlint. Low follow-ups (not blocking):
Earlier receipts: SHIP at b5324c0 (both-or-neither, recovery and verify), and the Intel compile on the fleet (step |
…t-intel # Conflicts: # .github/workflows/nightly.yml
|
Review (independent correctness subagent): SHIP at 092fb27 Merge of feat-cmux-next
This PR's own red, fixed in the merge commit. Tests run locally (all pass):
Review findings and resolution
Checked by the reviewer:
Not verified: the x86_64 nightly-next Release compile on this exact head. The earlier Intel compile receipt (fleet step No visual change: CI workflows and tests only. |
…64 and x86_64 feeds The base's feed test (2026-10-25) pinned nightly-next to arm64 only; this PR ships thin arm64 and x86_64. Ran tests/test_nightly_next_identity.py: 7/8 before (this assertion), 8/8 after. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bugbot needs on-demand usage enabledBugbot uses usage-based billing for this team and requires on-demand usage to be enabled. A team admin can enable on-demand usage in the Cursor dashboard. |
|
Review (independent correctness subagent): SHIP at 94f12fe The full review at 092fb27 (SHIP, comment above) still holds. This review covers only the delta, commit 94f12fe:
Non-PR reds at this head
Both are being rerun on the same SHA. |
# Conflicts: # .github/workflows/nightly.yml # tests/upload_artifact_retention_allowlist.json
|
Review (independent correctness subagent): SHIP at 8de8b12 This head merges feat-cmux-next Conflicts. #26491 removed
What the reviewer checked:
Tests on this head:
|
…ner, not a GitHub-hosted label check_no_github_hosted_runners (tests/test_ci_self_hosted_guard.sh) refused the bare ubuntu-24.04 on the new job; it now matches the rollback job's runner expression. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The base's Build Ghostty CLI helper step now tees to $GITHUB_STEP_SUMMARY under set -u, so compile_commands failed before recording any architecture. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Review (independent correctness subagent): SHIP at f338d9d Base: feat-cmux-next Changes since the last SHIP
Signer path. The owned-pool signer path is arm64-only: it is gated on Tests run on this head (all PASS):
Base reds, not this PR's. These fail identically at feat-cmux-next
Both shell tests stop at their first failure, so the reviewer also ran temporary copies past it. On this head, every later assertion passes. These base reds are what make nightly-guards release-notary and Guard status red. The fast-tier red is a runner refused at setup (infrastructure, held for a re-run). The earlier fleet Release compile red was controller 429/524 throttling. This PR changes no Swift. Not run: a tagged dev build. This PR changes only the nightly workflows, their step scripts and tests, so a tagged app build would be the same as the base's app. What this change does can only be seen in a real two-leg nightly-next publish, which runs after merge. |
…t-intel # Conflicts: # tests/test-execution.toml # tests/upload_artifact_retention_allowlist.json
…t-intel # Conflicts: # .github/workflows/nightly.yml
Resolves the nightly-next Intel branch against base: - nightly-next-rollback.yml: base's pulled-json job and this branch's rollback-intel job both kept. - nightly_next_feed_rollback.py: plan takes --above and --if-newest. - recover-continue-pending-notarization.sh: the Intel leg check, then base's visual-gate admit. - nightly.yml: base's step scripts (publish-tag-outcome, github-publish-require-tested-digest, github-publish-release-assets, github-names-publish-rolling-name) now loop over every architecture's feed; this branch's duplicate scripts are removed. verify-signed keeps base's signer-dmg split plus the nightly-next failed-notarize leg. - test_nightly_next_variants.py takes base's rust-targets.sh fixture. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Sweep fix. Blocker: conflicts in 5 files (1331 commits behind). Merged
Local checks: |
|
For awareness: this PR conflicts with feat-cmux-next, so it can't land until it's rebased or base is merged in. Its current review verdict stays as is; a hand-resolved merge gets a fresh review at the new head. |
|
Awareness note from an agent lane that does not own this PR; it is not a review or a request to take the PR over. Heads-up: #36348 split |
![Fix with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/fix-with-codesmith-light-3x.png)
Summary
cmux NEXT nightly has been built for Apple silicon only since #18424 (2026-10-07). Since then Intel Macs have gotten no new cmux NEXT, and the
nightly-nextrelease still carries the 2026-10-07x86_64anduniversalDMGs.With this PR, every nightly-next build ships a thin arm64 and a thin x86_64 DMG together. A build publishes both or neither, including a build a later run publishes through notarization recovery. There is no universal DMG, so each build waits on two notarizations instead of the three that caused #18424.
nightly.yml
['arm64', 'x86_64'], and its feeds areappcast-arm64.xmlandappcast-x86_64.xml. Main and RC are unchanged.notary_pendingand the stapled-app fallback id. The x86_64 leg waits 40m, as main's legs do; if it fails, the run fails and the normalnightly-next-failureincident opens.verify-final-dmg-nightlysmokes x86_64 even while arm64 is pending.verify-signed-nightlyrecords each verified variant.Nightly (cmux-next) <build>. The release is edited in place, so GitHub keeps showing its first publish date (Oct 4), and the newest build looked stale.Release rings
nightly-next-verdict.yml: withNIGHTLY_NEXT_DEV_RING=on(nightly-next: new builds go to the Dev ring first and move to Nightly on a canary pass #23835), a new build starts on Dev, and a canary pass now retags both cmux NEXT feeds to Nightly from the same verdicts. Before, onlyappcast-arm64.xmlwas promoted, which would have left Intel on Dev forever. The NIGHTLY-identity legacy feed stays arm64.nightly-next-rollback.yml: a newrollback-inteljob pulls the same bad builds offappcast-x86_64.xml.nightly_next_feed_rollback.py plan --abovedrops every Intel item newer than the build the arm64 feed now serves. The job is separate, so a failure there never holds the fix-forward, and a re-run converges.Design notes
smoke-signed-app-cli.sh's existing policy (Rosetta when the runner has it, skipped when it doesn't). These have not yet run on an Intel app in CI, so the first two-leg nightly may expose Rosetta timing. Under the both-or-neither rule, that would hold the publish and open the failure incident.signer-build(ci(nightly-next): sign on the release runner pool too, behind an opt-in input #26299, ci(nightly-next): one Mac job on the release-pool path; the final DMG is checked in verify-final #26491) runs only whenvariants == '["arm64"]'. With two variants it is skipped, and both legs sign inbuild-sign-notarize-nightly, the designed fallback (needs.signer-dmg.result != 'success').CI_NIGHTLY_NEXT_SIGN_POOL=1therefore has no effect on nightly-next until the pool path learns to sign x86_64 (follow-up). The owned verify Mac also gets four verify legs per build instead of two.Size: one behaviour change (nightly-next builds, verifies, recovers and publishes an Intel leg beside arm64) that touches every stage of the one pipeline in nightly.yml and its step scripts; about half the lines are the tests that run those steps on fixture DMGs, which belong with the code. Splitting by stage would ship a nightly-next that builds Intel but cannot publish it, which fails every publish.
Changelog
Testing
python3 tests/test_nightly_next_variants.py(renamed fromtest_nightly_next_arm64_only.py): runsdecideper track, runs "Require every variant" against fixtures, checks verdict promotion and rollback-intelpython3 tests/test_nightly_next_recovery.py:ContinueIntelTestsrun the real continue step against verified, missing, untested, wrong-digest, other-build and missing-arm64-record fixturespython3 tests/test_nightly_next_feed_rollback.py: new--abovecasetests/test_nightly_universal_build.shscripts/ci/check-bash32-parse.shon nightly.yml and nightly-next-rollback.ymlIntel compile: fleet step
e6e876698874rancheck-release-compile.sh --archs x86_64(#23919) at feat-cmux-next55f9d7ec3ddc: Release, Xcode 26.6,ARCHS=x86_64,** BUILD SUCCEEDED **, exit 0. This PR changes no Swift.Observed / Not run
nightly-nextbranch after merge.tests/test_r2_upload_requests.pylocally: its self-signed TLS server is rejected by this Mac's Python, and this PR doesn't touchupload-r2-object.py.🤖 Generated with Claude Code