Skip to content

ci(cmux-next): the DEV compile can record the fleet whole-app build's REF (off by default) - #25740

Merged
teamleaderleo merged 6 commits into
feat-cmux-nextfrom
p0/hybrid-ref-capture
Oct 27, 2026
Merged

teamleaderleo merged 6 commits into
feat-cmux-nextfrom
p0/hybrid-ref-capture

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Lets the per-landing Mac DEV Release compile record the fleet whole-app build's REF. The REF is the stored rules of one real Mac build of the same shape, which the fleet build plans its Swift compiles and links from. This is off by default, behind the repository variable CMUX_NEXT_HYBRID_REF_CAPTURE=1.

Leo, 2026-10-26: "but we have the fleet... why are we STILL TALKING ABOUT BLACKSMITH MACOS"

The only REF today (hybrid-11) was assembled by hand from a 2026-10-18 nightly build. The fleet path refuses the current tip because Package.resolved has since gained swift-atomics. The DEV app also has its own build shape, so it needs its own REF:

  • singlefile for target cmux;
  • DEBUG_INFORMATION_FORMAT=dwarf;
  • an arm64 destination;
  • the DEV bundle settings.

This change records one from the DEV compile itself, at the paths every mini shares in host mode. The private HQ side assembles the pack from it.

Change

  • Decide step (before the compile; continue-on-error, 2 min timeout). It runs on a mini, with the variable on, and only when the compile isn't reused. It computes scripts/cmux-next/hybrid_ref_capture.py key, a sha256 over:

    • Package.resolved;
    • every file of cmux.xcodeproj except xcuserdata;
    • the macOS xcconfigs;
    • the local macOS and shared package manifests;
    • the compile recipe (check-cmux-scheme-compile.sh, compile-dev-release.sh, lib/compilation-cache.sh);
    • the compile step's build-line settings;
    • xcodebuild -version and the SDK build.

    It then looks for an unexpired artifact named hybrid-ref-capture-<key> from the last 13 days, using the job's existing actions: read. The store keeps packs 14 days and the manifest keeps the same 14. If there is none, it sets capture=true. In the last 48 h, 18 of 887 feat-cmux-next landings changed these inputs, so almost every landing skips the capture.

  • The compile. When capture=true, check-cmux-scheme-compile.sh adds EMIT_FRONTEND_COMMAND_LINES=YES, the only added xcodebuild setting. Measured on the nightly shape (HQ HYBRID.md section 9), it changed no output: 5767 of 5767 .o and 176 of 176 .swiftmodule were identical with and without it. compile-dev-release.sh also writes build_mode, build_root and derived_data to the step outputs.

  • Capture step (the last step of the job; continue-on-error, 10 min timeout). It runs only after a successful host-mode compile, from a snapshot compile-dev-release.sh cloned while it still held the host lock (one clonefile(2) each for the tree and DerivedData), and only while the job is under 30 minutes, so it cannot reach the job's 60-minute limit.

    • Completeness check. Every SwiftDriver module must have a frontend compile line in the log. If any is missing, the capture is marked incomplete (missing: [...]) and nothing is uploaded.
    • Packs. A complete capture writes these, deterministic (sorted, owner 0, mtime 0):
      • the log;
      • XCBuildData;
      • DerivedData without objects, the compilation cache, logs, the app or SwiftPM's repositories;
      • the workspace without .git, target/ or node_modules, with the compiled-out browser sources the compiler read;
      • the committed Package.resolved.
    • Upload. Each pack is split under 480 MiB and every part goes to the build fleet's private store with cmux-ci artifact-put --lane hybrid-ref-dev.
    • Manifest. It is uploaded as the artifact hybrid-ref-capture-<key>, or ...-incomplete-<run> for an incomplete capture. It holds only names, sizes, sha256, paths and settings. No file content goes to this public repository.
  • Shared constant. The compiled-out browser-sources.json now comes from scripts/cmux-next/lib/browser-sources-compiled-out.json, read by both the compile and the capture, with byte-identical content (scheme-compile-browser-import.test.sh checks the bytes).

  • Settings. The decide and capture steps get the compile's settings as one HYBRID_REF_SETTINGS block, so only the compile step carries CMUX_NEXT_BROWSER_DATA_IMPORT (test_cmux_next_dev_release_import_out.py). A test keeps the block equal to the compile step's env.

Tests

tests/test_cmux_next_hybrid_ref_capture.py (checks group and the linux-guard lane) checks:

  • the key moves only with build-line inputs, and refuses an unknown setting;
  • an incomplete capture uploads nothing;
  • the packs are complete and deterministic, with nothing excluded leaking in;
  • a split pack's parts join back to the whole;
  • every upload carries its own sha256 and lane;
  • the manifest has no file contents;
  • the workflow invariants: off by default, continue-on-error with a timeout, capture after the compile on host mode only, only the EMIT setting added, and no permission change.

Also passing: test_cmux_next_dev_release_import_out.py, test_cmux_next_checks_workflow.py, test_cmux_next_dev_release_host_root.py, scheme-compile-browser-import.test.sh.

Risk and what is not measured yet

  • Off by default. Nothing changes until the variable is set.
  • When it is on: a capture landing's compile prints frontend lines, which makes the log larger. The job holds the push compile slot for the capture time (bounded at 10 min, expected to be a few minutes), so the next landing's compile and that run's DEV publish wait that long. That happens on about 9 landings a day out of about 440. I'll measure the log size and capture time on the first run and post them here. If they are too high, the follow-up is moving the pack and upload to a controller job on the same mini.
  • Not measured yet: whether a warm host-mode compile (cache hits) still prints a frontend line for every module. If it doesn't, the capture is marked incomplete, the DEV app is unaffected, and a follow-up adds a one-run cold path for that key.
  • No new workflow permissions, secrets or environments. The store upload uses the mini's tailnet identity, as every cmux-ci call on a mini does.

Agent session: agent session

🤖 Generated with Claude Code

… REF (off by default)

The Linux Darling build of the cmux NEXT DEV app plans its Swift compiles and links from
the rules of one real Mac build of the same shape (its REF). The only REF was assembled by
hand from a 10-18 nightly build, and the Linux path stopped as soon as Package.resolved
moved. With CMUX_NEXT_HYBRID_REF_CAPTURE=1 the per-landing DEV compile records one, on a
mini in host mode, when its REF key (Package.resolved, cmux.xcodeproj, the macOS
xcconfigs, the DEV build settings, the toolchain) has no complete capture yet:

- a decide step before the compile turns on EMIT_FRONTEND_COMMAND_LINES=YES for that
  compile only (the one added xcodebuild setting);
- a last step, continue-on-error with a 20 minute timeout, checks that every SwiftDriver
  module has a frontend compile line (else the capture is incomplete and uploads nothing),
  packs the log, XCBuildData, DerivedData without objects or the cache and the workspace,
  and puts them in the build fleet's private store (cmux-ci artifact-put); the manifest
  artifact names sizes and digests only;
- the compiled-out browser sources now come from one file both the compile and the capture
  read.

No new permission: the decide step reads artifacts with the job's actions: read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-c1e0761c3d79f629ab33
@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Oct 26, 2026
Review (cmux#25740 at 17829d4): the capture read the shared host tree and
DerivedData after compile-dev-release.sh had released the host lock, so another
runner on that Mac could replace them first. With the capture on, the script
now clones both (one clonefile(2) each, about a second) into RUNNER_TEMP while
it still holds the lock; the capture reads that snapshot, records the build's
own paths, and removes it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-c1e0761c3d79f629ab33
@blacksmith-sh

blacksmith-sh Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

Found 7 test failures on Blacksmith runners:

Failures

Test View Logs
/tmp/tmptilclehy/summary-fail.json (live):/summary-fail.json (live): View Logs
/tmp/tmpyssshwkm/summary-missing.json (live):/summary-missing.json (live): View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows
View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/
[tabs] debug.tab_switch timed 0 of 40 switches
View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames View Logs
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl
ights_validate)/
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl
ights_validate)
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Bead cx-tcpdv created for this PR by the cmux coordinator.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 75018d1: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr25740 -f ref=p0/hybrid-ref-capture -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); no finished base run on 2d14c08 to compare; do not merge until green; see lane-rules

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

high-signal review link / link is red on 12 of 16 fresh feat-cmux-next PR heads in the last 2:00:00, so it is most likely a base red, not this PR's. The base-reds lane has it; a base merge picks up the fix once it lands.

…ey on manifests and the recipe

Review (cmux#25740 at e459de9):
- The capture holds the push compile slot inside a 60 minute job: it now starts
  only while the job is under 30 minutes and stops after 10, and the decide step
  has 1 minute. A skipped capture is taken by the next landing with that key.
- A capture counts only while its packs are readable: unexpired artifacts from
  the last 13 days (the store keeps packs 14 days), and the manifest keeps 14
  days instead of 90.
- The key also covers the local macOS and shared package manifests and the
  compile recipe (check-cmux-scheme-compile.sh, compile-dev-release.sh,
  lib/compilation-cache.sh), which set build lines too.
- The DerivedData pack leaves out .git in SourcePackages checkouts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-c1e0761c3d79f629ab33
Review (cmux#25740 at 75640f8): the 30 minute guard counted from the decide
step, which runs after checkout, the web bundles and the Debug compile, so a
capture could still start near the job's 60 minute limit. The capture step now
reads this job's started_at from the run's jobs (its runner, in progress) and
starts only under 40 minutes; an unknown start skips it. The capture has 8
minutes and the manifest upload 2, so both end by minute 50.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-c1e0761c3d79f629ab33
…ument list and toolchain

- The decide step no longer tests the product lookup's hit: test_dev_release_product_reuse.py
  allows that only for steps a dev_daily run skips, and a reused product compiles nothing, so
  the capture step finds no build mode and skips anyway.
- The manifest adds xcodebuild_args (the wrapper's shell-quoted line split, so
  OTHER_SWIFT_FLAGS=... stays one argument, as the Linux DEV planner reads it) and toolchain
  (Xcode version and build, developer dir, SDK build).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-c1e0761c3d79f629ab33
@teamleaderleo
teamleaderleo merged commit 7a053f8 into feat-cmux-next Oct 27, 2026
111 of 123 checks passed
@teamleaderleo
teamleaderleo deleted the p0/hybrid-ref-capture branch October 27, 2026 00:37
@teamleaderleo teamleaderleo changed the title ci(cmux-next): the DEV compile can record the Linux whole-app build's REF (off by default) ci(cmux-next): the DEV compile can record the fleet whole-app build's REF (off by default) Oct 27, 2026
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

This branch was previously deployed

1 inactive deployment
ci-routing-agent — 75018d1a Deployed Oct 27, 2026 by teamleaderleo via wait for same-key tree publisher #61041
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant