[teamleaderleo]

Summary

Adds a "republish this verified build with tip tooling" path to the NEXT nightly. Today publish-nightly runs .publish-tooling from github.sha, the build commit, so every fix to publishing needs a new ~1.5 h build before it reaches a build that already passed.

Motivating runs:

• Run 41815641855 (nightly-next d96836b, build 4181564185501) built, signed, notarized and passed "Verify final nightly DMG", then publish-nightly published nothing. The cause was a supersede bug, since fixed by ci(nightly-next): a newer classic NIGHTLY build never supersedes a NEXT publication #35436.
• The run before it, 41814490749, failed its R2 upload, since fixed by ci(r2): stream PUT bodies so the 30 s timeout bounds idle gaps, not the whole nightly DMG upload #35337.

With this PR, that build can go out with the fixed tooling and no rebuild:

gh workflow run nightly.yml --ref nightly-next -f nightly_next_republish_run=41815641855 -f nightly_next_republish_attempt=1

The artifacts of that run expire on 2026-11-07 (3-day retention).

How it works

• New inputs nightly_next_republish_run and nightly_next_republish_attempt, built like nightly_next_unsigned_run and _attempt. decide refuses them in these cases:

  • outside a workflow_dispatch on refs/heads/nightly-next
  • a run or attempt that does not parse
  • combined with any other build or notary mode (continue-only, unsigned resume, fast, build-only, seed, notary tests or probe)

• A republish builds and submits nothing. It reuses continue_only (now "this run builds and submits nothing"), so every build job skips as it does for a continuation. find-nightly-next-recovery skips a republish. It forces should_build, so publication always runs and the recheck decides. The run title is Republish nightly-next run N attempt M.

• New job republish-nightly-next has no environment and no secrets, and actions: read only. It runs scripts/ci/republish-verified-nightly.py, which follows resume-unsigned-nightly.py and reuses its downloader. It refuses unless all of these hold:

  • The source is a completed protected nightly-next push of nightly.yml.
  • In the requested attempt, build-sign-notarize-nightly (arm64), Verify signed nightly app (arm64), Verify final nightly DMG (arm64) and build-nightly-app succeeded.
  • await-nightly-tree-marker passed or was skipped.
  • Each artifact (variant, tested digests, SSH daemon, plus the source archive and Chromium pack when present) is live, unique, uploaded by its producer job during that attempt, and downloaded against GitHub's recorded SHA-256.
  • The build commit comes from that build's dSYM receipt, and both it and the source head are ancestors of this run's nightly-next commit.
  • Every DMG matches a digest the source's verify job tested, and the daemon manifest is that build's.

  If all of these hold, it uploads the build in the recovered layout as cmux-nightly-republished-variant-arm64 and -tested-arm64, plus cef-runtime-pack if the source had one.

• publish-nightly gets a third way to start (needs.republish-nightly-next.result == 'success'). Its "Select the build to publish" step picks the republished build as source=recovered. Every guard then runs unchanged from the tip's .publish-tooling:

  • the lease and the recheck against both feeds
  • "Require every DMG to match a tested digest"
  • the final gate: the writer, the webview verdict read from the source run, known-bad, and the recovered lineage check
  • R2, appcasts and the GitHub release

  The gate's RECOVERED_RUN_ID now comes from the select step's new run_id output, so it names whichever run the published build came from.

• Same as for a recovered build: no deltas, no release-note signing and no published-in. The legacy-identity jobs are untouched. They read the live NEXT feed, so the next run signs the republished build as cmux NIGHTLY.

Testing

From tests/, python3 -m unittest test_nightly_next_recovery passes all 74 tests (new: RepublishSourceTests, RepublishDecideTests, RepublishWorkflowTests). The new tests cover:

• validate, for good and untrusted sources, failed or other-attempt verify jobs, the tree marker, producer windows, digests, ambiguity, optional artifacts and the dSYM receipt
• the DMG and daemon byte checks
• decide's outputs and refusals, through the existing node decide harness
• the job's secret-free shape
• the select step's republished branch and its run_id, and that no kept guard names the republish

Also passing: test_nightly_unsigned_resume, test_nightly_cmux_tui_resolve (the new job is exempt like the recovery jobs: it ships no source), test_nightly_trusted_routing, test_ci_runs_on_fallback, test_nightly_never_blacksmith, test_ci_nightly_off_blacksmith, test_workflow_file_size (421,724 bytes), test_gh_budget_wait, test_nightly_background_steps. Lint: actionlint, plus check-bash32-parse.sh over 423 inline scripts.

Observed / Not run

• Observed: the tests above. test_nightly_owned_minis_only, test_nightly_next_arm64_only and test_ci_macos_xcode_selection fail with the same failures on base, because they depend on the local environment.
• Observed: the source run's real jobs and artifacts (/actions/runs/41815641855/attempts/1/jobs, /artifacts) fit the producer windows the validator checks.
• Not run: the dispatch itself. It happens after merge, at the nightly-next tip.

Changelog

none

Risk: high (release publication path). Merges on an independent SHIP receipt at the exact head.

Agent session: agent session

🤖 Generated with Claude Code

---

^{Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.}

---

Summary by cubic

Summary

Adds a way to republish an already verified nightly-next build with the tip's publication tooling, so publication fixes reach builds that passed verification without a ~1.5-hour rebuild. Previously publish-nightly used the build commit's .publish-tooling, so fixing a failed publication of a finished build required rebuilding it (run 41815641855 verified build 4181564185501 and then published nothing, a supersede bug fixed by #35436). New dispatch inputs nightly_next_republish_run/nightly_next_republish_attempt make the run build nothing; a secret-free republish-nightly-next job fetches the source run's verified build and publish-nightly publishes it as a recovered build through every existing guard.

Safety

• The fetch job refuses anything but a completed nightly-next push whose signing and both verify jobs passed in the requested attempt.
• It checks artifact provenance and each download against GitHub's recorded SHA-256, and requires every DMG to match a digest the source's verify job tested.
• Like other recovered builds, a republished build gets no deltas, release-note signing, or published-in.

^{Written for commit b1d50f5. Summary will update on new commits.}

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review b1d50f5: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr35588 -f ref=nightly-next-republish-verified -f platform=macos -f backend_mode=local

Shipped in:

• cmux NEXT nightly 1.0.0-nightly.4187086980301 (df4454e051)
• cmux NEXT nightly 1.0.0-nightly.4188039004101 (3e464b08ff)
• cmux NEXT nightly 1.0.0-nightly.4190094575701 (036f8aa538)
• cmux NEXT nightly 1.0.0-nightly.4191131679101 (e01bbae5c9)

[teamleaderleo]

Owned by an agent session · session

[blacksmith-sh]

Found 18 test failures on Blacksmith runners:

Failures

Test	View Logs
main.R2UploadRequestsTests.test_a_slow_large_put_is_bounded_by_idle_gaps_not_total_ time/test_a_slow_large_put_is_bounded_by_idle_gaps_not_total_time	View Logs
cmux-vm.yml job deploy-production: its concurrency group does not isolate other label e vents/ cmux-vm.yml job deploy-production: its concurrency group does not isolate other label e vents	View Logs
cmux-vm.yml job deploy-production: its if does not end with the label guard !(github.ev ent_name == 'pull_request' && github.event.pull_request.base.ref == 'feat-cmux-next' && (github.event.action == 'labeled'
cmux-vm.yml job migrate-production: its concurrency group does not isolate other label events/ cmux-vm.yml job migrate-production: its concurrency group does not isolate other label events	View Logs
cmux-vm.yml job migrate-production: its if does not end with the label guard !(github.e vent_name == 'pull_request' && github.event.pull_request.base.ref == 'feat-cmux-next' & & (github.event.action == 'labeled'
cmux-vm.yml job smoke-production: its concurrency group does not isolate other label ev ents/ cmux-vm.yml job smoke-production: its concurrency group does not isolate other label ev ents	View Logs
cmux-vm.yml job smoke-production: its if does not end with the label guard !(github.eve nt_name == 'pull_request' && github.event.pull_request.base.ref == 'feat-cmux-next' && (github.event.action == 'labeled'
EXEMPT names nightly.yml job launch-smoke-nightly, which no longer exists (launches the app in the release-pool DMG beside signer-dmg; builds nothing, no Xcode)/ EXEMPT names nightly.yml job launch-smoke-nightly, which no longer exists (launches the app in the release-pool DMG beside signer-dmg; builds nothing, no Xcode)	View Logs
test_big_red (main.FleetRuleLints.test_big_red)/ test_big_red (main.FleetRuleLints.test_big_red)	View Logs
test_checked_in_tree_matches_its_ratchet (main.ShortTimeoutGuardTests.test_checked_ in_tree_matches_its_ratchet)/ test_checked_in_tree_matches_its_ratchet (main.ShortTimeoutGuardTests.test_checked_ in_tree_matches_its_ratchet)	View Logs

...and 8 more test failures. View all on Blacksmith

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[teamleaderleo]

Review (codex, router-review): SHIP at 68fc1a8

Reviewer: codex gpt-6-astra high, fast tier.

Follow-ups: 2. They land as checklist items in this PR's one follow-up tracking issue in this repo (Follow-ups: #N ..., label follow-up; add to an open one before filing, never one issue per finding), linking this receipt; the lane that opened the PR fixes them forward.

SHIP at 68fc1a8

Visual verdict: unchanged by this diff

• [P1] [follow-up] [category:correctness] [finding:F1] [outcome:pending] .github/workflows/nightly.yml:5424 gates the Chromium runtime pack on the current CMUX_NEXT_CEF_ON_DEMAND variable instead of the republished source artifact. If that setting changed, republishing can omit a pack the app requires or fail downloading a pack the source did not produce.

• [P2] [follow-up] [category:correctness] [finding:F2] [outcome:pending] .github/workflows/nightly.yml:5638 similarly gates source-archive handling on the current variable. If republishing a source without that artifact while the variable is enabled, mv nightly-out/source-archive fails after feed publication has already occurred.

Follow-ups:

• [finding:F1] Gate CEF publication on the source artifact’s presence.
• [finding:F2] Gate source-archive publication on the republished artifact’s presence.

[cmux-bot]

cmux-bot approves 68fc1a8: an independent review passed this exact head (router-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

[teamleaderleo]

Review (codex, router-review): SHIP at b1d50f5

Reviewer: codex gpt-6-astra high, fast tier.

Follow-ups: 3. They land as checklist items in this PR's one follow-up tracking issue in this repo (Follow-ups: #N ..., label follow-up; add to an open one before filing, never one issue per finding), linking this receipt; the lane that opened the PR fixes them forward.

SHIP at b1d50f5

Visual verdict: no images; Leo tries the build

The focused recovery and workflow test suites pass (74 and 157 tests).

• [P1] [follow-up] [category:correctness] [finding:F1] [outcome:pending] A republish can publish a CEF-enabled source build without its runtime pack if CMUX_NEXT_CEF_ON_DEMAND changed between the source run and republish. The fetch job uploads the pack only when the source artifact exists, while publish-nightly downloads/publishes it solely from the current variable (.github/workflows/nightly.yml:5113, .github/workflows/nightly.yml:5424). Preserve the source artifact’s enabled state or refuse the republish when the modes differ.

• [P2] [follow-up] [category:correctness] [finding:F2] [outcome:pending] The validator explicitly allows a missing source archive, but the recovered GitHub release path unconditionally runs mv nightly-out/source-archive source-archive when source-archive publishing is enabled (.github/workflows/nightly.yml:5637). Republish of an older verified run without that optional artifact will therefore fail after the feed/assets publication path has started.

• [P2] [follow-up] [category:reliability] [finding:F3] [outcome:pending] republish-nightly-next is absent from report-nightly-failure’s needs and result checks (.github/workflows/nightly.yml:7065). A failed fetch can leave publish-nightly skipped without opening/updating the nightly failure issue, unlike other publication-path failures. Add the job to the reporting dependency and failure predicate.

Follow-ups:

• F1: Preserve or validate the source CEF runtime-pack mode.
• F2: Handle an absent optional source archive.
• F3: Include republish failures in nightly failure reporting.