[teamleaderleo]

HOLD: paused pending Leo's Apple SDK/tools-on-Linux decision

Leo, 2026-10-27, about the nightly compiling on a GitHub-hosted Mac: "WHY / THESE ARE SLOW AS STUFF / WHERE THE F*** IS LINUX".

What

The Linux Darling whole-app build plans from the stored build rules of one real Mac build of the same shape (its REF). #25740 made the DEV compile record its own REF. This PR does the same for the nightly-next Release compile (build-nightly-app, "Build nightly app (Release)").

Which file: nightly-next builds from the nightly.yml on the nightly-next branch, which is a feat-cmux-next commit. main's nightly.yml builds classic cmux NIGHTLY (a different app), so this PR changes feat-cmux-next's copy.

• Off by default. It runs only when the new repo variable NIGHTLY_HYBRID_REF_CAPTURE is 1, and only on the nightly-next track.
• Owned minis with cmux-ci only. On attempt 1, build-nightly-app runs on vars.CI_SIDE_LANE_RUNNER (glaeda-side-std-xcode-26.6, the owned side pool: run 40380339203 ran on cmuxs-mac-mini-5-glaeda-3, run 40378565252 on cmux8s-mac-mini-glaeda-2). Rescue attempts and every fallback run on vars.MACOS_RUNNER_26 (blacksmith-6vcpu-macos-26, e.g. run 40389706045). A Blacksmith or GitHub-hosted Mac has no cmux-ci and can't reach the fleet's private store, so it never captures. The decide step requires runner.environment == 'self-hosted', a *-glaeda[-N] runner name and an executable cmux-ci (the job's $HOME first, then the build user's home).
• Decides like the DEV compile (ci(cmux-next): the DEV REF capture retries a rate-limited lookup and compiles instead of reusing #25866). The decide step calls hybrid_ref_capture.py have --budget 30 under the nightly-next profile. That counts complete captures of the key (13 days) and incomplete ones (1 day), and waits out a rate limit for at most 30 s: it runs before the nightly compile. A capture is needed when there's no complete one. Then compile=true unless one came out incomplete in the last day. A lookup that still fails compiles and captures anyway and fails the step red, so a capture can't be lost silently. If the job's start time can't be read, the deadline counts from 30 minutes earlier, which only brings it sooner.
• Never from a reused product. nightly-next now reuses a compiled app when its inputs didn't change (ci(nightly-next): reuse the compiled nightly app when its inputs did not change #25731). The decide step runs before "Look up the nightly app compiled on the same inputs". On compile=true it hands that lookup SKIP_REASON "a REF capture needs a real compile", and the capture starts only when the lookup missed.
• Keyed on the nightly's own build. hybrid_ref_capture.py gets a nightly-next profile (HYBRID_REF_PROFILE). Its key hashes the project inputs that ci(cmux-next): the DEV compile can record the fleet whole-app build's REF (off by default) #25740 also hashes (Package.resolved, cmux.xcodeproj, the macOS xcconfigs and the local package manifests), the toolchain, and the nightly's own recipe in place of the DEV one: nightly-swift-defines.sh, the text of the "Build nightly app (Release)" step (its xcodebuild flags are inline), and the env that step reads (BUILD_TRACK, SWIFT_IMPORT_FLAGS, EMPTY_BROWSER_SOURCES, NIGHTLY_COMPILE_ARCHS, CHANNEL_ICON_NAME). The DEV key doesn't change (a test checks this). A capture runs only when no complete, readable capture exists for the key: artifact hybrid-ref-capture-nightly-next-<key>, from the last 13 days.
• The compile: on a capture run, the only change is EMIT_FRONTEND_COMMAND_LINES=YES plus a named xcodebuild log (CMUX_XCODEBUILD_OUTPUT_LOG). A non-capture run is unchanged.
• Under the nightly.yml size cap (ci(nightly): move seven run blocks into scripts so nightly.yml keeps room under GitHub's size limit #25879). The decide, start and finish run blocks live in scripts/ci/nightly-steps/hybrid-ref-{decide,start,finish}.sh, so this adds about 6 KB to nightly.yml (475 KB; the cap is 490 KB). The tests read the scripts back in place through tests/nightly_workflow_text.py.
• Bounded, so signing barely waits:
  • Right after the compile and the helper bundling (which changes only the app), one clonefile(2) snapshots the workspace. That takes about a second, and build-universal, the DerivedData, is inside it.
  • The capture then packs and uploads in the background at nice 10 (background_build.py), alongside the rest of the job (notices, the CUA helper, daemon assets, the unsigned-app upload, Sentry).
  • At the end, "Finish the Linux build's REF capture" waits at most 4 minutes, and never past minute 130 of the job's 150. A capture that hasn't finished by then is stopped and uploads no manifest; the next build with the same key takes it.
  • The capture also stops itself at minute 130 (capture --deadline) and on SIGTERM from the job's cleanup. Each upload runs in its own process group, which is killed however the upload ends, so nothing an upload started outlives the job.
  • It doesn't start once the job is more than 100 minutes old.
  • The decide step's job-start gh call never waits out a rate limit (GH_BUDGET_WAIT_MAX_SECONDS: "0"), so it spends none of the job's one-hour gh budget; the have lookup waits 30 s at most. That lookup joins the DEV one in tests/test_ci_no_github_token_polling.py's allowlist.
  • Every new step is continue-on-error with a timeout, and the always() cleanup stops a capture that is still running.
• Storage: packs go only to the build fleet's private store (cmux-ci artifact-put --lane hybrid-ref-nightly-next). The public manifest artifact (14 days) lists only names, sizes and sha256 values.
• No new permission. The job keeps contents: write, actions: read, attestations: write, id-token: write. Listing artifacts uses the existing actions: read.

Manifest differences from #25740 (for the HQ assembler)

• The kept xcodebuild log now starts with the wrapper's xcodebuild command: line (run-xcodebuild-with-diagnostics.sh writes it into a caller-kept log), so xcodebuild_args is filled in.
• A runner that links .spm-cache to its kept SourcePackages gets a real clone of the packages in the snapshot; the capture refuses a linked .spm-cache.
• Driver modules come from the builtin-SwiftDriver line's -module-name (ci(cmux-next): the REF capture names a Swift driver by its module, not its target #25903): the app target isn't its module.
• New field "profile": "nightly-next" here, "dev" on DEV manifests. This is additive and schema stays 1.
• settings holds the five nightly names above instead of CMUX_NEXT_*.
• workspace is the runner's $GITHUB_WORKSPACE, and derived_data is $GITHUB_WORKSPACE/build-universal. ws-src.tgz leaves out build-universal, which ships once, in build-noobj.tgz/xcbuilddata.tgz. Swift packages are cloned to .spm-cache in the workspace (-clonedSourcePackagesDirPath .spm-cache), so their checkouts are in ws-src.tgz; their git mirrors are not.
• xcodebuild_args starts with env CMUX_SKIP_ZIG_BUILD=1 xcodebuild ... (the nightly's wrapper line), and the build is universal (ARCHS=arm64 x86_64) unless compile_archs is arm64.
• Artifact names: hybrid-ref-capture-nightly-next-<key> for a complete capture and hybrid-ref-capture-nightly-next-incomplete-<key> for an incomplete one, the same names have reads. DEV keeps hybrid-ref-capture-[incomplete-]<key>.

Release surface

.github/workflows/nightly.yml and .github/workflows/cmux-next.yml are both on .github/release-surface.txt. So the nightly-next promotion whose range contains this commit waits for an approval in release-surface-ack, as #25740's range (cmux-next.yml) already does. hybrid_ref_capture.py isn't added to the surface: build-nightly-app holds no signing, notarization or publishing secret, and the other scripts/cmux-next/ files it runs aren't listed either.

Risk

Low for the shipped app. With the variable unset, no new step runs, and the compile step only gains two empty env values. On a capture run the shipped compile gains EMIT_FRONTEND_COMMAND_LINES=YES, which only prints the frontend lines. That makes the Actions log larger, the same as on #25740's DEV capture runs. The worst case adds about 10 minutes to the job: decide 1, start 2, finish 5, manifest upload 2. The typical case adds 0 to 4 minutes, once per REF key.

Tests

• tests/test_cmux_next_hybrid_ref_capture.py (34 tests):
  • the nightly key moves with the nightly step, its settings and defines, and not with the DEV recipe or another step's comment; the DEV key and DEV artifact names are unchanged;
  • have reads the nightly profile's own names, and the upload step writes the same ones;
  • a failed lookup compiles, captures and fails red; a capture run makes the product lookup miss;
  • a DerivedData inside the workspace is packed once, and a linked .spm-cache is refused;
  • the deadline, a failed upload and the cleanup's SIGTERM each leave no uploader or helper running (the test is its own subreaper and checks the helper died by SIGKILL);
  • the real xcodebuild wrapper writes the command line into the kept log;
  • the steps are off by default, owned-mini only, continue-on-error and bounded, and add no permission.
• tests/test_ci_no_github_token_polling.py: allowlist entry for the nightly have lookup.
• tests/test_gh_budget_wait.py: a step that never waits (GH_BUDGET_WAIT_MAX_SECONDS 0) no longer needs the 65-minute step timeout.
• .github/nightly-step-coverage.json: entries for the three new steps.
• Run locally on the merged tree: the capture suite (also 180 runs of its three cleanup tests, no failure), test_workflow_file_size.py, the polling, gh-budget, helper-ordering, product-reuse and xcodebuild-wrapper tests all pass. Nine other tests that read nightly.yml fail here exactly as on a clean base checkout. actionlint reports the same findings as base.

Stop condition: merged, then NIGHTLY_HYBRID_REF_CAPTURE=1, and the first nightly-next capture's manifest artifact exists.

Agent session: agent session

🤖 Generated with Claude Code

[teamleaderleo]

Owned by an agent session · session

[teamleaderleo]

Bead cx-42xbi created for this PR by the cmux coordinator.

[blacksmith-sh]

Found 1 test failure on Blacksmith runners:

Failure

Test	View Logs
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl ights_validate)/ test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl ights_validate)	View Logs

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review cea669f: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr25812 -f ref=nightly-hybrid-ref-capture -f platform=macos -f backend_mode=local

web and size checks: green

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[teamleaderleo]

Review (codex gpt-6-astra xhigh, next-review): SHIP at bec4e03

No blocking findings.

[teamleaderleo]

cmux-next guards / nightly-guards / workflow-guard-tests / release-notary is red on 13 of 17 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

[teamleaderleo]

Orphaned: no activity for 13 h, and no lane holds a lease on it. The orphan sweep could not merge feat-cmux-next in: conflicts in .github/nightly-step-coverage.json, tests/test_ci_no_github_token_polling.py. Listed for a lane to resolve.

[teamleaderleo]

Merging feat-cmux-next into this branch doesn't resolve cleanly. Base has changed the parts this PR builds on, so it needs a rework rather than a conflict fix:

• ci(nightly): headroom under the size budget (14 more run blocks, step notes into scripts, expressions through env) #31013 (and ci(nightly): both nightly-next build slots on a Mac build in one root, with a 10 GiB compile cache #32514 after it) moved the "Build nightly app (Release)" run block into scripts/ci/nightly-steps/app-build-release.sh. The nightly-next profile key in hybrid_ref_capture.py hashes that step's inline text, and the EMIT_FRONTEND_COMMAND_LINES hook edits that inline block. Both now need to target the script, along with the tests that pin them.
• nightly.yml on base is 486,950 bytes, and tests/test_workflow_file_size.py caps it at 490,000. The roughly 6 KB of new steps won't fit unless their env and comments move into the step scripts too.
• Base renamed the DEV steps in cmux-next.yml to "Decide the fleet build's REF capture" and "Capture the fleet build's REF". The new nightly step names, the nightly-step-coverage.json entries and the test_ci_no_github_token_polling.py allowlist key should use the same names.

A rework means re-applying the three steps on top of current base, with the key, the hook and the tests pointed at app-build-release.sh, and keeping nightly.yml under its cap. The HOLD in the body stays in place.