Skip to content

nightly-next: new builds go to the Dev ring first and move to Nightly on a canary pass - #23835

Merged
teamleaderleo merged 1 commit into
feat-cmux-nextfrom
ci/next-dev-ring-publish
Oct 21, 2026
Merged

teamleaderleo merged 1 commit into
feat-cmux-nextfrom
ci/next-dev-ring-publish

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Picking Dev in cmux NEXT's Update Channel does nothing today: every item on the cmux NEXT feed is tagged cmux-next, which every ring accepts. With this change and repository variable NIGHTLY_NEXT_DEV_RING=on, each new nightly-next build goes out as cmux-next-dev first, so only apps on Dev get it right away (same app, com.cmuxterm.app.nightly.cmux-next, updating in place). It moves to cmux-next (Nightly) once the canary passes it. Unset or off keeps every item on cmux-next, which is today's behavior, so merging this changes nothing until someone sets the variable.

How a build moves from Dev to Nightly

  • Published: build-sign-notarize-nightly, the recovery path and generate-nightly-deltas tag new items cmux-next-dev when the variable is on.
  • Qualifies: the durable canary verdict verdicts/<build>.json is pass. It's the signal that already exists, and the train already uses it for its base. I didn't add a soak timer: nothing in the workflows has one, and a timer would promote builds the canary never saw.
  • Promoted: nightly-next-verdict.yml sets every feed item's channel from the verdicts after it writes each verdict. The write is If-Match on the bytes it read, like the rollback. republish-nightly-deltas, which lists earlier builds again, tags them from the same verdicts before publishing. The verdict documents are the only source of truth, so any later writer corrects a missed promotion. Turning the variable off puts every item back on cmux-next at the next verdict or publication.
  • Old feed: legacy-identity-target (the cmux NIGHTLY-identity feed for pre-NEXT installs) copies only builds already on cmux-next.
  • Logic: all of it lives in scripts/ci/nightly_next_ring.py (channel, retag, select).

Old builds never see an unpromoted item

Builds from before rings allow only cmux-next: UpdateFeedResolver.Channel.allowedSparkleChannels is allowedSparkleChannels(ring: .nightly), and UpdateDriver+SPUUpdaterDelegate.allowedChannels(for:) returns it to Sparkle. Sparkle drops an item whose sparkle:channel isn't in allowedChannels, and the app's own model of this (AppcastItem.isOffered(allowing:) on the release-rings branch: an untagged item is always offered, a tagged one only when allowed) works the same way. Main's builds allow no channel at all. The new test test_nightly_and_older_builds_never_see_an_unpromoted_build checks that rule for Dev, Nightly, pre-ring and main updaters on a retagged feed.

The guard

nightly-next-guard.py appcast used to require exactly [cmux-next]. It now requires exactly one tag per item, from the four ring tags (cmux-next-stable, cmux-next, cmux-next-canary, cmux-next-dev). An untagged item, a second tag or a foreign tag is still refused. A test checks that the guard, nightly_next_ring.py and UpdateRing.swift/UpdateFeedResolver.swift name the same four tags. Nothing publishes canary or stable items yet.

How much faster Dev is

Not much, in the best case:

  • Build cadence: builds landed about every 21 minutes on 2026-10-21 (30 builds, 11:59 to 22:09 UTC).
  • Canary latency: the canary runs at :03/:18/:33/:48 (15 minutes apart) and has a 25-minute job timeout. The one durable verdict so far came 9 minutes after its build published. So promotion trails publication by about 10 to 40 minutes.
  • App check interval: the app checks hourly (UpdateSettings.scheduledCheckInterval = 60 * 60). For a Mac checking hourly, Dev is ahead of Nightly by less than one check interval.
  • Builds the canary skips: the canary checks only the newest build at each tick. A build that's superseded before its tick never gets a verdict and stays on Dev; Nightly moves on with the next build that passes.

Right now, on would freeze Nightly. The only verdict written so far is a fail (build 3968001295301, 22:18 UTC). Every scheduled canary run I opened that actually ran the canary failed (6 of them between 17:25 and 22:13 UTC); the others skipped already-checked builds. That's why the switch defaults to off and the doc says to turn it on only while the canary passes.

A Nightly app can be offered a promoted build only while that build is on the feed. The first publication writes a single-item feed with only the new build, and the delta republish adds up to 5 earlier builds back. Between those two writes, Nightly is offered nothing new. That's safe, but it's a delay.

Risk

risk:high: release/updater publication. Mitigations:

  • Default off: with the variable unset, the only changes are a guard that accepts four tags instead of one, a no-op select on the old feed, and an extra feed read in the verdict job that writes only if a channel differs (it won't).
  • When on, the failure direction is safe: an unreadable verdict counts as none, so the item stays on Dev. Nightly is never offered a build early; at worst it gets one late.
  • Concurrency: the feed write in the verdict job is If-Match. When it loses a race it writes nothing, and the next writer derives the same channels from the verdicts.
  • Tooling version: republish-nightly-deltas now runs the guard and the ring script from the workflow commit (.publish-tooling, as publish-nightly already does), because the build ref can predate them.
  • Not covered by this PR: a promoted build can drop off the feed (see above). The legacy feed waits while nothing is promoted.

Changelog

none

Observed / Not run

Observed, on c169198:

  • python3 tests/test_nightly_next_ring.py: 17 tests OK. The workflow test for the verdict promotion was red before the verdict workflow change.
  • python3 tests/test_nightly_next_guard.py: PASS. Against the base guard it fails on the three new ring-tag cases and the tag-sync check (red on base, green here).
  • These neighbors pass: test_nightly_next_feed_rollback, test_nightly_next_recovery, test_nightly_next_post_publish, test_nightly_next_train, test_nightly_republish_delta_guard, test_ci_test_execution_registry, test_ci_workflow_guards_are_wired, test_ci_run_guards, test_ci_linux_guard_routing, test_cmux_next_guards_route, test_feat_ci_guard_group, test_pr_lane_coverage, test_ci_actionlint_covers_every_workflow, test_ci_workflow_run_sources, test_ci_reusable_workflow_permissions, test_ci_release_product_reuse.
  • actionlint on the three changed workflows: the only finding is base's 17-input workflow_dispatch on nightly.yml, which is the same on base. Runner-label noise is ignored.
  • python3 scripts/cmux-next/prepush-gate.py: no new violations.
  • The live feed, verdict documents and canary run history quoted above were read on 2026-10-21.

Not run:

  • No nightly or verdict workflow was dispatched, so the retag and promotion steps haven't run against the real feed or R2.
  • test_ci_guard_workflow_structure.py and test_ci_change_areas.py failed only because files were missing from my sparse checkout (agent-chat/package.json, cmux.xcodeproj). CI runs them.
  • No app build. The Swift claims come from reading the code.

Agent session: huge-blue

🤖 Generated with Claude Code


Summary by cubic

Picking Dev in cmux NEXT's Update Channel does nothing today: all items on the feed are tagged cmux-next, which every ring accepts. With this change, setting repository variable NIGHTLY_NEXT_DEV_RING=on publishes each new nightly-next build as cmux-next-dev (Dev ring only) and moves it to cmux-next (Nightly) once its durable canary verdict is pass. Unset or off keeps every item on cmux-next, so merging this changes nothing until someone enables the variable.

  • Promotion happens in nightly-next-verdict.yml (retags the live feed with If-Match) and in republish-nightly-deltas; the verdict documents are the single source of truth.
  • Nightly and pre-ring builds accept only cmux-next, so they never see an unpromoted build; the old cmux NIGHTLY-identity feed copies only Nightly items.
  • The feed guard now requires exactly one of the four ring tags per item instead of exactly cmux-next; a test keeps the guard, the ring script, and UpdateRing.swift in sync.
  • With on, Nightly gets nothing new while the canary passes no build, so the variable should be enabled only while the canary is green.

Written for commit c169198. Summary will update on new commits.

View guided diff Turn on auto-fix

…ghtly on a canary pass

With repository variable NIGHTLY_NEXT_DEV_RING=on, every new cmux NEXT
build is tagged sparkle:channel cmux-next-dev, so only apps on the Dev
ring get it. Its durable canary verdict (verdicts/<build>.json) of pass
moves it to cmux-next: nightly-next-verdict.yml retags the live feed
after each verdict (If-Match), and republish-nightly-deltas tags the
earlier builds it lists from the same verdicts. Nightly and builds from
before rings accept only cmux-next, so they never see an unpromoted
build. The old cmux NIGHTLY-identity feed copies only Nightly builds.
Unset or off keeps every item on cmux-next, as today.

nightly-next-guard.py accepts exactly one of the four ring tags per
item; a test keeps the guard, the ring script and UpdateRing.swift on
the same tags.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a15c1c-9365-71b3-b73b-dc961610d618&host=huge-blue
@teamleaderleo teamleaderleo added the risk:high Risk tier (docs/ci/risk-tiers.md) label Oct 21, 2026
@github-actions

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review c169198: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr23835 -f ref=ci/next-dev-ring-publish -f platform=macos -f backend_mode=local

@blacksmith-sh

blacksmith-sh Bot commented Oct 21, 2026

Copy link
Copy Markdown

Found 2 test failures on Blacksmith runners:

Failures

Test View Logs
nightly cmux-tui resolve/nightly cmux-tui resolve View Logs
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl
ights_validate)/
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl
ights_validate)
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (independent subagent): SHIP at c169198

Off by default (NIGHTLY_NEXT_DEV_RING unset/off): the first publish, recovered build, delta republish, old-identity feed and verdict retag all leave every item on cmux-next. The reviewer ran select and retag on today's live feed: the output is byte-identical and changed: false. When on, the verdict retag writes with a real compare-and-swap (sha256-matched ETag plus If-Match; on a mismatch a newer writer wins). No new pull_request_target or workflow_run triggers, and no new secrets. The guard refuses untagged, doubled and unknown tags. tests/test_nightly_next_ring.py (17) and tests/test_nightly_next_guard.py pass. Fleet cmux-ci compile-check --app at c169198 passed.

Reds on this head that are not this PR's: release-notary on restored-terminals-come-back.md (also red on base), the fast tier (runner Swift 6.3.3 vs SDK 6.4, infrastructure), and the localization debt ratchet (base).

@teamleaderleo
teamleaderleo merged commit a99742b into feat-cmux-next Oct 21, 2026
@teamleaderleo
teamleaderleo deleted the ci/next-dev-ring-publish branch October 21, 2026 23:21
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Bead cx-hakzz created for this PR by the cmux coordinator.

@github-actions

Copy link
Copy Markdown
Contributor

Shipped in:

teamleaderleo added a commit that referenced this pull request Oct 22, 2026
… republish's tooling checkout (#24273)

* ci(nightly guards): release-notary checks the renamed legacy step and republish's tooling checkout

Two base reds in "workflow-guard-tests / release-notary", both from tests
that did not follow a workflow change:
- #24019 renamed "Skip a build older than the old feed's" to "Skip a build
  the old feed already has"; test_nightly_universal_build.sh still asked for
  the old name and exited there, so every check after line 1265 never ran.
- #23835 gave republish-nightly-deltas a sparse checkout of its publication
  tooling at github.sha into .publish-tooling, as publish-nightly has;
  test_nightly_cmux_tui_resolve.py exempted that checkout only in
  publish-nightly. The exemption now applies to any job, and only to a
  sparse checkout of scripts into .publish-tooling, never the build tree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a160a7-80b6-7101-bcbe-c94cbb747f12&host=cmux16

* ci(nightly guards): the tooling exemption holds only a whole sparse value of scripts paths

From the Codex review of e572b24: the exemption matched a prefix, so a
sparse list with a trailing /* (the whole tree at github.sha) slipped
through. The whole value must now name paths under scripts/, with no dot
segments; rejection cases for /*, a trailing /* and scripts/../ pin it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a160a7-80b6-7101-bcbe-c94cbb747f12&host=cmux16

* ci(nightly guards): a blank line in the tooling sparse list does not end it

From the Codex review of 17db2ce: the scan stopped at a blank line, so a
/* after one slipped through. The block scalar now runs to the first
non-blank line indented no deeper than its key; an explicit case with an
empty line before /* pins it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a160a7-80b6-7101-bcbe-c94cbb747f12&host=cmux16

* nightly resolve test: an inline sparse value continued on deeper lines is not exempt

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a160a7-80b6-7101-bcbe-c94cbb747f12&host=cmux16

* nightly resolve test: the tooling checkout's indent is spaces only, so blank lines can't hide a continuation

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a160a7-80b6-7101-bcbe-c94cbb747f12&host=cmux16

* nightly resolve test: drop only the validated tooling checkout, by its span

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a160a7-80b6-7101-bcbe-c94cbb747f12&host=cmux16

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
austinywang added a commit that referenced this pull request Oct 22, 2026
With NIGHTLY_NEXT_DEV_RING on (#23835), every new build is published as
cmux-next-dev, and the verdict moved only appcast-arm64.xml to Nightly. Now
that each nightly-next build also ships x86_64, its appcast-x86_64.xml item
would have stayed on Dev, so a Mac on the Nightly ring would never be
offered an Intel update. The promotion step retags both cmux NEXT feeds from
the same verdicts. The old cmux NIGHTLY-identity feed stays arm64 only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
teamleaderleo added a commit that referenced this pull request Nov 1, 2026
…ed purple) (#31288)

* Agent chat: a pull request chip's mark takes its state's colour (merged purple)

Leo 2026-10-23, of a merged pull request's blue #23835 chip: "I feel like this
should be purple". The chip's GitHub mark now reads the pull request's state
from the host's cache (one cache-only pr.inspect when it first draws, never a
GitHub read) and colours it as the hover card's glyph does: open green, merged
purple, closed red; a draft or an unknown one keeps the link colour. Colour
only, so the chip never reflows.

Tests: bun test test/bare-pr-refs.test.tsx test/url-chip-hover.test.tsx
test/bare-ref-repo.test.tsx test/pr-hover-card.test.tsx (+ link/pr card suites):
green; the new chip-state test fails on base (no data-pr-state).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-9cf9f3a64099b06033f4

* Agent chat: a card's read waits behind a chip's cache-only ask instead of being dropped

Review follow-up on the chip state colour: a card opened while the chip's
cache-only pr.inspect was in flight lost its read to the one-ask-per-link
rule. It now reads once that ask settles. The chip test asserts the state
without a hover and checks all three state rules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-9cf9f3a64099b06033f4

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant