Skip to content

nightly-next: every build ships both Apple silicon and Intel DMGs - #23811

Open
austinywang wants to merge 27 commits into
feat-cmux-nextfrom
nightly-next-intel
Open

austinywang wants to merge 27 commits into
feat-cmux-nextfrom
nightly-next-intel

Conversation

@austinywang

@austinywang austinywang commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

cmux NEXT nightly has been built for Apple silicon only since #18424 (2026-10-07). Since then Intel Macs have gotten no new cmux NEXT, and the nightly-next release still carries the 2026-10-07 x86_64 and universal DMGs.

With this PR, every nightly-next build ships a thin arm64 and a thin x86_64 DMG together. A build publishes both or neither, including a build a later run publishes through notarization recovery. There is no universal DMG, so each build waits on two notarizations instead of the three that caused #18424.

nightly.yml

  • decide: nightly-next variants are ['arm64', 'x86_64'], and its feeds are appcast-arm64.xml and appcast-x86_64.xml. Main and RC are unchanged.
  • Notarization hand-off: only the arm64 leg hands a slow notarization to a later run, and only it sets notary_pending and the stapled-app fallback id. The x86_64 leg waits 40m, as main's legs do; if it fails, the run fails and the normal nightly-next-failure incident opens.
  • Verification:
    • verify-final-dmg-nightly smokes x86_64 even while arm64 is pending.
    • verify-signed-nightly records each verified variant.
    • Both verify jobs run after a failed nightly-next sign job, so a failed arm64 notarization stays recoverable.
  • Recovery: a pending arm64 build is continued only when that build's own run has a verified x86_64 leg (variant, tested digest of the exact bytes, signed-app record for the same build) and a verified arm64 signed app. This is checked before any notarization step. The recovered publish carries both DMGs.
  • publish-nightly:
    • "Require every variant" refuses a build missing any decided variant.
    • Both aliases and both feeds are published, and the release notes list both downloads.
    • The release title becomes Nightly (cmux-next) <build>. The release is edited in place, so GitHub keeps showing its first publish date (Oct 4), and the newest build looked stale.

Release rings

  • nightly-next-verdict.yml: with NIGHTLY_NEXT_DEV_RING=on (nightly-next: new builds go to the Dev ring first and move to Nightly on a canary pass #23835), a new build starts on Dev, and a canary pass now retags both cmux NEXT feeds to Nightly from the same verdicts. Before, only appcast-arm64.xml was promoted, which would have left Intel on Dev forever. The NIGHTLY-identity legacy feed stays arm64.
  • nightly-next-rollback.yml: a new rollback-intel job pulls the same bad builds off appcast-x86_64.xml. nightly_next_feed_rollback.py plan --above drops every Intel item newer than the build the arm64 feed now serves. The job is separate, so a failure there never holds the fix-forward, and a re-run converges.

Design notes

  • One verdict promotes both architectures. The canary, feed-watch and the deltas republish's freshness check read the arm64 feed. An Intel item reaches Nightly on that build's arm64 canary pass. Before publication the Intel app has passed its own launch, CLI and final-DMG smokes and the new visual smoke and canary.
  • The x86_64 leg runs the base's new smokes under Rosetta (ci(nightly-next): a visual smoke gate before publication #23780): the visual smoke and the in-job canary, via smoke-signed-app-cli.sh's existing policy (Rosetta when the runner has it, skipped when it doesn't). These have not yet run on an Intel app in CI, so the first two-leg nightly may expose Rosetta timing. Under the both-or-neither rule, that would hold the publish and open the failure incident.
  • An x86_64 compile, notarization or smoke failure holds the whole nightly-next publish. That is the intent. The x86_64 Swift compile also adds the second-architecture compile time that ci: compile only the architectures shipped by each nightly #21154 removed.
  • The owned-pool signer path is arm64-only, so it stops running for nightly-next. signer-build (ci(nightly-next): sign on the release runner pool too, behind an opt-in input #26299, ci(nightly-next): one Mac job on the release-pool path; the final DMG is checked in verify-final #26491) runs only when variants == '["arm64"]'. With two variants it is skipped, and both legs sign in build-sign-notarize-nightly, the designed fallback (needs.signer-dmg.result != 'success'). CI_NIGHTLY_NEXT_SIGN_POOL=1 therefore has no effect on nightly-next until the pool path learns to sign x86_64 (follow-up). The owned verify Mac also gets four verify legs per build instead of two.
  • cmux NEXT's Chromium engine is arm64-only, so the Intel app ships without it, as before 2026-10-07.

Size: one behaviour change (nightly-next builds, verifies, recovers and publishes an Intel leg beside arm64) that touches every stage of the one pipeline in nightly.yml and its step scripts; about half the lines are the tests that run those steps on fixture DMGs, which belong with the code. Splitting by stage would ship a nightly-next that builds Intel but cannot publish it, which fails every publish.

Changelog

  • Fixed: cmux NEXT nightly has an Intel (x86_64) download again, published with every Apple silicon build.

Testing

Command Base Head
python3 tests/test_nightly_next_variants.py (renamed from test_nightly_next_arm64_only.py): runs decide per track, runs "Require every variant" against fixtures, checks verdict promotion and rollback-intel fails PASS
python3 tests/test_nightly_next_recovery.py: ContinueIntelTests run the real continue step against verified, missing, untested, wrong-digest, other-build and missing-arm64-record fixtures 4 failures 52 OK
python3 tests/test_nightly_next_feed_rollback.py: new --above case error OK
ring, guard, notary handoff, release notes, post-publish, fix-forward, canary mode, identity, background steps, protected-env, delta guard, tests/test_nightly_universal_build.sh PASS
scripts/ci/check-bash32-parse.sh on nightly.yml and nightly-next-rollback.yml 190 / 173 scripts parse

Intel compile: fleet step e6e876698874 ran check-release-compile.sh --archs x86_64 (#23919) at feat-cmux-next 55f9d7ec3ddc: Release, Xcode 26.6, ARCHS=x86_64, ** BUILD SUCCEEDED **, exit 0. This PR changes no Swift.

Observed / Not run

  • Observed: the tests above, the bash 3.2 parse, actionlint (nothing new against the base) and the x86_64 Release compile.
  • Not run:
    • A real two-leg nightly-next publish, promotion and rollback. These run only on the nightly-next branch after merge.
    • tests/test_r2_upload_requests.py locally: its self-signed TLS server is rejected by this Mac's Python, and this PR doesn't touch upload-r2-object.py.

🤖 Generated with Claude Code

austinywang and others added 2 commits October 21, 2026 15:34
nightly-next has built arm64 only since #18424, so Intel Macs got no new
cmux NEXT after 2026-10-07 and the release kept a stale x86_64 DMG.

decide now builds thin arm64 and x86_64 on nightly-next (two notarizations,
no universal) and marks x86_64 optional. Each matrix job lets an optional
leg fail alone; only a required leg sets notary_pending and the stapled-app
fallback id, and an optional leg never hands a pending notarization to the
arm64 recovery. publish-nightly publishes x86_64 only when this run's own
build passed every leg for it (tested final DMG plus a per-variant
signed-app record); otherwise it drops those files and the Intel DMG and
feed keep their last build. Deltas run only for published variants. The
release title carries the build number, since GitHub keeps showing the
first publish date of the release that is edited in place.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…arm64's stapled fallback

Review follow-up: a failed optional x86_64 leg leaves every job green, so
report-nightly-next-intel keeps one nightly-next-intel-failure incident open
while a published build skips Intel and closes it when Intel publishes again.
The x86_64 verify legs no longer read arm64's stapled-app fallback id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@austinywang austinywang added the risk:high Risk tier (docs/ci/risk-tiers.md) label Oct 21, 2026
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review f14690a: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr23811 -f ref=nightly-next-intel -f platform=macos -f backend_mode=local

web and size checks: green

@austinywang

Copy link
Copy Markdown
Contributor Author

Correctness review (independent subagent) at 366edc0: FIX. Findings and how each was resolved:

  1. High: the x86_64 Swift/helper compile is shared and not optional. An x86-only compile error would fail build-nightly-app and block arm64. The second-architecture compile also lengthens every build. Status: checking with a build_only run on this branch: https://github.com/manaflow-ai/cmux/actions/runs/39688351887. A separate optional x86 compile job is a possible follow-up.
  2. Medium: a failed optional leg left every job green, so nothing reported it, and the nightly incident could even close. Fixed in d91a86c: report-nightly-next-intel opens or updates a nightly-next-intel-failure incident, and closes it when Intel publishes again.
  3. Medium: nightly-next-rollback.yml rolls back only appcast-arm64.xml. Follow-up, documented in the PR body.
  4. Medium: operational costs. Two concurrent notarizations, arm64 publish waits for the x86 leg, and an arm64 recovery publish ships no Intel. Accepted and documented.
  5. Low: relies on empty matrix outputs not overwriting another leg's value. This is GitHub's documented pattern, and it fails safe through the tested-digest gate. No change.
  6. Low: arm64's stapled-app fallback id reached the x86 verify legs. Fixed in d91a86c: blanked for optional legs.
  7. Low, suspected: a cancelled optional leg may not be masked by continue-on-error. Accepted: a manual cancel already stops the publish today.
  8. Low, existing behavior: pruning after 100 Intel-less builds, and Rosetta-less smokes being skipped. No change.

The reviewer also confirmed that the "Choose the variants to publish" bash, the digest gate and the delta gating are correct, and that main and RC are unchanged.

@blacksmith-sh

blacksmith-sh Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Found 6 test failures on Blacksmith runners:

Failures

Test View Logs
main.DeveloperIdAppTests.test_download_prints_the_active_profile_only/
test_download_prints_the_active_profile_only
View Logs
stubs: /System/Library/Frameworks/Security.framework/Versions/A/Security has no stub/
Security has no stub
View Logs
stubs: CoreFoundation.tbd: install-name /System/Library/Frameworks/
CoreFoundation.framework/Versions/A/CoreFoundation, missing 1: ['_CFRetain']/
CoreFoundation, missing 1: ['_CFRetain']
View Logs
stubs: CoreFoundation.tbd: install-name /System/Library/Frameworks/
CoreFoundation.framework/Versions/A/CoreFoundation, missing 1: ['_log2']/
CoreFoundation, missing 1: ['_log2']
View Logs
test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt
epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)/
test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt
epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)
View Logs
test_the_macos_signer_is_the_fallback_and_one_path_hands_on (main.Wiring.test_the_m
acos_signer_is_the_fallback_and_one_path_hands_on)/
test_the_macos_signer_is_the_fallback_and_one_path_hands_on (main.Wiring.test_the_m
acos_signer_is_the_fallback_and_one_path_hands_on)
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@teamleaderleo

Copy link
Copy Markdown
Collaborator

Bead cx-8zkrd created for this PR by the cmux coordinator.

…hips with its Intel leg

Per Austin: every nightly-next build ships the Apple silicon and the Intel
app together, never one without the other. This drops the optional x86_64
machinery (job-level continue-on-error, the published-variant choice and
the separate Intel incident): a failed x86_64 leg fails the run and the
normal nightly-next failure incident reports it.

Only the arm64 leg still hands a slow notarization to a later run; the
x86_64 leg waits 40m like main's legs, and its final DMG is smoked even
while arm64 is pending. The recovery continues a pending arm64 build only
when that build's own run has a verified x86_64 leg (variant, tested
digest of the exact bytes, signed-app record for the same build), checked
before any notarization step, and uploads it as the recovered Intel
variant. publish-nightly refuses a build missing any decided variant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@austinywang austinywang changed the title nightly-next: ship an Intel (x86_64) DMG again, without holding arm64 nightly-next: every build ships both Apple silicon and Intel DMGs Oct 21, 2026
Review of 2e43e6d: a build whose arm64 notarize step failed still
leaves a recovery artifact, but both verify jobs were gated on the whole
sign job's success, so its x86_64 was never verified and every later
recovery skipped it. On nightly-next the verify jobs now also run after a
failed sign job (each leg fails on its own without its artifact); the own
publish still needs the sign job to pass. The recovery also requires the
candidate's arm64 signed-app record, and the tests cover a wrong Intel
digest and a missing arm64 record.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@austinywang

Copy link
Copy Markdown
Contributor Author

Correctness review (independent subagent) at 2e43e6d: FIX. Requirement (a), no nightly-next publish with only one architecture, was found to hold on every path, including recovered builds. Findings and resolution:

  1. Medium: a build whose arm64 notarize step failed could never be recovered. Both verify jobs were gated on the whole sign job's success, so that run's x86_64 was never verified. Fixed in b5324c0: on nightly-next the verify jobs also run after a failed sign job. Each leg fails on its own without its artifact, and the own-run publish still requires the sign job to pass.
  2. Medium: the x86_64 leg has no hand-off. If Apple takes more than 40m for Intel, the build is lost even when arm64 would have recovered. Accepted: the notarize step's 130m budget covers two waits plus the helper. An x86_64 hand-off in the recovery is a follow-up.
  3. Low: x86 artifacts are kept 3 days, but recovery artifacts 14. Accepted: after 3 days a pending candidate is below newer published builds and is skipped anyway. Keeping 225 MB DMGs for 14 days per build is costly.
  4. Low: recovery did not require the candidate's arm64 signed-app record. Fixed in b5324c0.
  5. Low, suspected: nightly-next-only checks (version surfaces, canary) now run against x86_64 under Rosetta. Watching: an enforce-mode canary failure would hold the publish and open the failure incident.
  6. Nit: stale "optional x86_64" step name in ci-guards.yml. Fixed.

Test gaps: recovery tests now cover a wrong Intel digest and a missing arm64 record. test_nightly_next_variants.py checks that both verify jobs run after a failed nightly-next sign job.

@austinywang

Copy link
Copy Markdown
Contributor Author

Review (independent correctness subagent): SHIP at b5324c0

Verified: this run's own build still cannot publish after a failed sign job (publish-nightly if and "Select the build to publish" both require it). Main and RC are unchanged. A failed x86_64 notarization can never yield the recovery's x86 records: the variant artifact is uploaded only after notarization succeeds, and the tested digest needs it. The new arm64 signed-app record is produced for both pending and failed arm64 legs. Low/nits, not blocking: extra verify runner time after failed builds; the new verify if is checked by string only; parentheses around the nightly-next clause for readability.

austinywang and others added 2 commits October 21, 2026 17:12
nightly-next now ships an x86_64 app with every build, so the same Release
compile (nightly Xcode, -O, whole-module) can check the Intel slice ahead of
the nightly. The default stays arm64.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… own PR

The fleet runs an allowlisted script only as reviewed on feat-cmux-next, so
the option must land first in a separate PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cursor

cursor Bot commented Oct 22, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

austinywang and others added 2 commits October 22, 2026 06:06
With NIGHTLY_NEXT_DEV_RING on (#23835), every new build is published as
cmux-next-dev, and the verdict moved only appcast-arm64.xml to Nightly. Now
that each nightly-next build also ships x86_64, its appcast-x86_64.xml item
would have stayed on Dev, so a Mac on the Nightly ring would never be
offered an Intel update. The promotion step retags both cmux NEXT feeds from
the same verdicts. The old cmux NIGHTLY-identity feed stays arm64 only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@austinywang

Copy link
Copy Markdown
Contributor Author

Intel compile: green. Fleet step e6e876698874be2dc61b7329 ran scripts/cmux-next/check-release-compile.sh --archs x86_64 (option added in #23919) at feat-cmux-next 55f9d7ec3ddc19f2f654efef8fb6dd602a3b6b6b on cmuxs-Mac-mini-3. It used Xcode 26.6 (17F113) to build CmuxNextApp in Release with ARCHS=x86_64: ** BUILD SUCCEEDED **, exit 0, 700 x86_64 build steps, no errors. This PR changes no Swift, so that is the code the Intel nightly compiles.

The earlier build_only dispatch (run 39688351887) failed only because a branch dispatch runs as the nightly track, which skips the cmux-next web bundles. It is not Intel evidence either way.

Head 46d3262 merges feat-cmux-next and adds one fix. With NIGHTLY_NEXT_DEV_RING=on (#23835), new items start on Dev, and nightly-next-verdict.yml promoted only appcast-arm64.xml. Intel items would have stayed on Dev forever, so a Mac on the Nightly ring would never be offered an Intel update. A canary pass now retags both cmux NEXT feeds from the same verdicts. The NIGHTLY-identity legacy feed stays arm64.

The new visual smoke (#23780) runs on the x86_64 leg through smoke-signed-app-cli.sh's existing policy: Rosetta when the runner has it, skipped when it doesn't. Tests on this head pass: variants (including a new verdict check that is red on the base), recovery (52), ring, guard, notary handoff, release notes, universal build, bash 3.2 parse (190 scripts). A fresh independent review of the exact head is running.

Review of 46d3262: the rollback the canary dispatches pulled a bad
build off appcast-arm64.xml only, so appcast-x86_64.xml kept offering it to
Intel Macs. New job rollback-intel drops every Intel item newer than the
build the arm64 feed now serves (nightly_next_feed_rollback.py plan
--above, which need not find that build on the Intel feed), checks the
fallback DMG, writes with If-Match and reads it back. It is its own job so
a failure is a red run that never holds the fix-forward, and it runs when
arm64 was already pulled, so a re-run converges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@austinywang

Copy link
Copy Markdown
Contributor Author

Review (independent correctness subagent): SHIP at d9b6253

The review of 46d3262 (FIX) had one blocker: a rollback pulled a bad build off appcast-arm64.xml only. It is fixed in d9b6253: rollback-intel runs plan --above <arm64 serving build> on appcast-x86_64.xml in its own job, so it never holds the fix-forward.

Verified by the reviewer:

  • set -euo pipefail and bash 3.2 behaviour.
  • serving is set on dry run and on a converging re-run. An arm64 refusal skips the Intel job and the run goes red.
  • Dry run writes nothing.
  • Every --above edge case, including a serving build older than every Intel item: that refuses with exit 1, loudly.
  • The workflow concurrency group and the If-Match writes.

Tests pass: feed rollback, variants, recovery (52), ring (28), universal build, bash 3.2 parse and actionlint.

Low follow-ups (not blocking):

  1. Early on, an Intel pull may fall back to the 2026-10-07 build, because that is the only older Intel item on the feed.
  2. pulled.json is still built from the arm64 pull only.
  3. The race where republish-nightly-deltas can undo a concurrent rollback is older than this PR; the Intel feed now inherits it.
  4. docs/ci/nightly-next-canary.md still describes only appcast-arm64.xml.

Earlier receipts: SHIP at b5324c0 (both-or-neither, recovery and verify), and the Intel compile on the fleet (step e6e876698874, ** BUILD SUCCEEDED **).

@austinywang austinywang added the ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds label Oct 22, 2026
…t-intel

# Conflicts:
#	.github/workflows/nightly.yml
@austinywang austinywang removed the ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds label Oct 26, 2026
@austinywang

Copy link
Copy Markdown
Contributor Author

Review (independent correctness subagent): SHIP at 092fb27

Merge of feat-cmux-next 6f065929a157 (plain merge, no rebase). Six conflicts in nightly.yml, resolved to keep both sides' intent:

  • The base removed the launch-smoke gates (e8652c2). They stay removed. The verify jobs keep the base's static checks plus this PR's "Record the verified variant" step and its upload.
  • verify-signed-nightly and verify-final-dmg-nightly keep the base's pick-nightly-sign-runner in needs/runs-on, together with this PR's nightly-next recovery if:.
  • "Verify final DMG artifact" (the base's name for the old smoke step) takes this PR's matrix.variant != 'arm64' || notary_pending != 'true' condition.
  • The recovery's Intel upload steps are kept. The base's comment changes ("verifies", not "launch-smokes") are kept.

This PR's own red, fixed in the merge commit. nightly-guards / workflow-guard-tests / release-notary failed tests/test_nightly_step_coverage.py because two new steps had no registry entry. .github/nightly-step-coverage.json now lists "Record the verified variant" (plumbing) and "Require every variant" (nightly_only; tests/test_nightly_next_variants.py runs that step's run block on fixtures).

Tests run locally (all pass):

  • tests/test_nightly_step_coverage.py
  • tests/test_nightly_next_variants.py
  • tests/test_nightly_next_recovery.py (52 tests)
  • tests/test_nightly_next_feed_rollback.py
  • tests/test_nightly_universal_build.sh

Review findings and resolution

  1. Medium, latent, follow-up. "Upload the Chromium runtime pack" uploads a fixed artifact name, cef-runtime-pack, from every build-sign-notarize matrix leg. With two nightly-next legs, the second upload would collide. Also, publish-nightly publishes only one pack. Both happen only when vars.CMUX_NEXT_CEF_ON_DEMAND == '1', which is not set (checked at repo level). Before turning that on, the pack needs a per-variant name and both packs need publishing, or on-demand Chromium stays arm64-only. Not changed here.
  2. Low, fails safe. The Intel artifacts that recovery needs are kept 3 days, while the recovery artifact is kept 14. An arm64 build pending longer than 3 days is skipped with "no verified Intel" instead of publishing arm64 alone. Accepted.
  3. Low. rollback-intel refuses, and goes red, if the x86_64 feed has nothing at or below the arm64 serving build. It runs as a separate job, so fix-forward is not held. Accepted.
  4. Low, scope. The nightly-next release title now ends with the build number (NAME_BUILD_SUFFIX). It is correct and nothing reads the release name. Noted here.

Checked by the reviewer:

  • No smoke step came back.
  • The if: precedence is correct.
  • The empty matrix outputs on the x86_64 leg cannot clear arm64's notary_pending.
  • Main and RC behaviour is unchanged.
  • Neither architecture can publish alone, in a normal run or a recovered one.

Not verified: the x86_64 nightly-next Release compile on this exact head. The earlier Intel compile receipt (fleet step e6e876698874be2dc61b7329, at feat-cmux-next 55f9d7ec3ddc) is in a comment above. CI's cmux-next Release compile (Xcode 26) runs on this head.

No visual change: CI workflows and tests only.

…64 and x86_64 feeds

The base's feed test (2026-10-25) pinned nightly-next to arm64 only; this
PR ships thin arm64 and x86_64. Ran tests/test_nightly_next_identity.py:
7/8 before (this assertion), 8/8 after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cursor

cursor Bot commented Oct 26, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@austinywang

Copy link
Copy Markdown
Contributor Author

Review (independent correctness subagent): SHIP at 94f12fe

The full review at 092fb27 (SHIP, comment above) still holds. This review covers only the delta, commit 94f12fe:

  • This PR's own red, fixed. nightly-guards / workflow-guard-tests / release-notary failed test_each_app_reads_the_feed_its_publisher_writes in tests/test_nightly_next_identity.py. A base test (2026-10-25) pinned nightly-next to ['arm64'] and ['appcast-arm64.xml']; this PR intentionally changes that. The assertions now pin ['arm64', 'x86_64'] and ['appcast-arm64.xml', 'appcast-x86_64.xml'], exactly matching nightly.yml. Result: 7/8 before, 8/8 after.
  • Hook addition, kept. The pre-commit hook registered the base's unregistered tests/test_nightly_delivery_concurrency.py in tests/test-execution.toml (lane linux-guard). The format is valid and it passes, along with tests/test_ci_test_execution_registry.py.
  • Remaining tests. The reviewer grepped the rest of tests/ for arm64-only nightly-next assertions and found none stale. Every tests/test_nightly*.py passes, apart from base or local-environment failures that also fail on the merge base.

Non-PR reds at this head

  • post cmux-next/fast-gates: the gates themselves passed; the GitHub App installation hit its API rate limit while posting the status.
  • high-signal review link: failed on the same rate limit.

Both are being rerun on the same SHA.

# Conflicts:
#	.github/workflows/nightly.yml
#	tests/upload_artifact_retention_allowlist.json
@austinywang austinywang removed the ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds label Oct 28, 2026
@austinywang

Copy link
Copy Markdown
Contributor Author

Review (independent correctness subagent): SHIP at 8de8b12

This head merges feat-cmux-next 0c8336391b32d into the reviewed 6a9f4b1. It is a plain merge: no rebase, no force push.

Conflicts. #26491 removed signer-verify; signer-dmg now hands off from the release pool.

  • verify-signed-nightly and verify-final-dmg-nightly: each takes the base's needs: [..., signer-dmg] and keeps this PR's if: with signer-dmg in place of signer-verify: (build-sign-notarize-nightly success || signer-dmg success) || nightly-next && build-sign-notarize-nightly failure. On a failed notarize every leg still verifies, so a recovery can take the x86_64 leg.
  • tests/upload_artifact_retention_allowlist.json: the entries for the removed job are dropped, and the PR's cef-runtime-pack-${{ matrix.variant }} entry is kept.

What the reviewer checked:

  • Nothing in .github/, scripts/ci/ or tests/ still refers to signer-verify.
  • The PR's per-variant publish, recovery and STAPLED_APP_FALLBACK_ID code does not depend on any signer job.
  • The pool-signed check the base moved into verify-final-dmg-nightly is gated on signer-dmg success. That cannot happen on nightly-next with both variants: signer-build requires variants == ["arm64"].
  • Latent note for whoever widens the pool to two variants: that step has no per-variant gate.

Tests on this head:

  • Pass: test_nightly_next_variants, test_nightly_next_recovery, test_nightly_next_identity, test_workflow_file_size, test_publish_release_assets_callers, test_nightly_unsigned_resume.
  • Base reds: test_nightly_signer_jobs, test_nightly_step_coverage and test_ci_upload_artifact_retention fail with output identical to the base tip 0c83363.
  • Size: nightly.yml is 486222 bytes.

@austinywang austinywang added the ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds label Oct 28, 2026
austinywang and others added 3 commits October 27, 2026 21:03
…ner, not a GitHub-hosted label

check_no_github_hosted_runners (tests/test_ci_self_hosted_guard.sh) refused the bare
ubuntu-24.04 on the new job; it now matches the rollback job's runner expression.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The base's Build Ghostty CLI helper step now tees to $GITHUB_STEP_SUMMARY
under set -u, so compile_commands failed before recording any architecture.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@austinywang austinywang removed the ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds label Oct 28, 2026
@austinywang

Copy link
Copy Markdown
Contributor Author

Review (independent correctness subagent): SHIP at f338d9d

Base: feat-cmux-next d4c78be875a8 merged into the reviewed 8de8b12 with a plain git merge (no rebase, no force push). Agent: Claude Code (Opus 5.5) for @austinywang.

Changes since the last SHIP

  1. Merge (6d4b569f403c): git show --remerge-diff is empty. ci(nightly-next): dispatched and re-run nightlies verify on an idle owned Mac too #26489 changed only how the owned Mac is picked for the verify jobs, and the pick is the same for every matrix leg. Both x86_64 verify legs, the notary_pending gate and the fallback-id gate are intact.
  2. 4f0b455129b6: this PR's own red. rollback-intel ran on the GitHub-hosted ubuntu-24.04 label, and check_no_github_hosted_runners refused it. It now uses the same runner expression as rollback and fix-forward. The CI guard group is green on this head; it was 2 red on this change at 8de8b12.
  3. f338d9d278f8: the base's Build Ghostty CLI helper step now writes to $GITHUB_STEP_SUMMARY under set -u, so tests/test_nightly_next_variants.py failed before it recorded any architecture. The harness now provides a step-summary file. The pre-commit hook also registered tests/test_ci_runs_on_fallback.py and tests/test_workflow_file_size.py (linux-guard); both pass.

Signer path. The owned-pool signer path is arm64-only: it is gated on variants == '["arm64"]' (nightly.yml:5429). Under this PR's two variants it is skipped, and both legs sign in build-sign-notarize-nightly, the designed fallback. No x86_64 DMG can be packed under arm64 names. This is advisory, not a blocker, and is now noted in the description.

Tests run on this head (all PASS):

  • tests/test_nightly_next_variants.py, test_nightly_next_recovery.py (54), test_nightly_next_feed_rollback.py, test_nightly_next_identity.py
  • test_publish_release_assets_callers.py, test_nightly_sign_runner.py, test_nightly_signer_jobs.py
  • test_ci_self_hosted_guard.sh, test_ci_runs_on_fallback.py, test_workflow_file_size.py, the test-execution registry validators
  • scripts/cmux-next/prepush-gate.py: no new violations

Base reds, not this PR's. These fail identically at feat-cmux-next d4c78be875a8:

Both shell tests stop at their first failure, so the reviewer also ran temporary copies past it. On this head, every later assertion passes. These base reds are what make nightly-guards release-notary and Guard status red.

The fast-tier red is a runner refused at setup (infrastructure, held for a re-run). The earlier fleet Release compile red was controller 429/524 throttling. This PR changes no Swift.

Not run: a tagged dev build. This PR changes only the nightly workflows, their step scripts and tests, so a tagged app build would be the same as the base's app. What this change does can only be seen in a real two-leg nightly-next publish, which runs after merge.

@austinywang austinywang added the ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds label Oct 28, 2026
@cmux-bot
cmux-bot Bot temporarily deployed to ci-routing-agent October 28, 2026 04:46 Inactive
@cmux-bot
cmux-bot Bot temporarily deployed to ci-routing-agent October 28, 2026 04:47 Inactive
…t-intel

# Conflicts:
#	tests/test-execution.toml
#	tests/upload_artifact_retention_allowlist.json
…t-intel

# Conflicts:
#	.github/workflows/nightly.yml
@austinywang austinywang removed the ready-to-land Reviewed at this head; next-autoland merges it into feat-cmux-next when its only reds are base reds label Oct 29, 2026
Resolves the nightly-next Intel branch against base:
- nightly-next-rollback.yml: base's pulled-json job and this branch's
  rollback-intel job both kept.
- nightly_next_feed_rollback.py: plan takes --above and --if-newest.
- recover-continue-pending-notarization.sh: the Intel leg check, then
  base's visual-gate admit.
- nightly.yml: base's step scripts (publish-tag-outcome,
  github-publish-require-tested-digest, github-publish-release-assets,
  github-names-publish-rolling-name) now loop over every architecture's
  feed; this branch's duplicate scripts are removed. verify-signed keeps
  base's signer-dmg split plus the nightly-next failed-notarize leg.
- test_nightly_next_variants.py takes base's rust-targets.sh fixture.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@austinywang

Copy link
Copy Markdown
Contributor Author

Sweep fix. Blocker: conflicts in 5 files (1331 commits behind). Merged origin/feat-cmux-next (new head db476d42e877e177034a7f1853f0d79c025bbbbc):

  • nightly-next-rollback.yml: base's new pulled-json job and this PR's rollback-intel job are both kept.
  • nightly_next_feed_rollback.py: plan takes both --above (this PR) and --if-newest (base).
  • recover-continue-pending-notarization.sh: this PR's Intel-leg check runs first, then base's visual-gate admit.
  • nightly.yml: base had moved the same steps into differently named step scripts. I ported this PR's per-architecture loops into base's github-publish-require-tested-digest.sh, github-publish-release-assets.sh and github-names-publish-rolling-name.sh, kept base's publish-tag-outcome.sh, and removed this PR's now-duplicate github-require-tested-dmg.sh, github-publish-assets.sh and publish-marker-outcome.sh. verify-signed keeps base's signer-dmg split plus this PR's failed-notarize leg on nightly-next.
  • tests/test_nightly_next_arm64_only.py stays renamed to test_nightly_next_variants.py and picks up base's rust-targets.sh fixture.

Local checks: tests/test_nightly_next_feed_rollback.py passes (32 tests), bash -n and YAML parse OK. The head needs CI (nightly-guards) and a review receipt before relabel.

@teamleaderleo

Copy link
Copy Markdown
Collaborator

For awareness: this PR conflicts with feat-cmux-next, so it can't land until it's rebased or base is merged in. Its current review verdict stays as is; a hand-resolved merge gets a fresh review at the new head.

@teamleaderleo

Copy link
Copy Markdown
Collaborator

Awareness note from an agent lane that does not own this PR; it is not a review or a request to take the PR over.


Heads-up: #36348 split tests/test-execution.toml into tests/test-execution.d/<area>.toml (one file per area, sorted by path), so this branch will hit a modify/delete conflict on that file when it next merges or rebases feat-cmux-next. To resolve it, run python3 scripts/ci/test_execution_registry.py absorb in the conflicted checkout. It moves this branch's registry edits into the right area files, deletes the old file and stages the result. Then continue the merge or rebase. Nothing else in the branch changes.

This branch was previously deployed

1 inactive deployment
ci-routing-agent — db476d42 Deployed Oct 30, 2026 by austinywang via cmux-tui tree preflight #85619
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants