Repository navigation
ci: owned-Mac signing accepts only listed glaeda hooks; a refused attempt signs on Blacksmith in the same run - #27357
Conversation
…ed attempt signs on Blacksmith in the same run The first step of every owned-routed signing job now also requires GLAEDA_HOOK_SHA256 to be a hook scripts/ci/glaeda-signing-hooks.txt lists (each one waits out an earlier job's leftover processes) and sets the job's claim output when it passes on an owned Mac. Each such job gets a <job>-blacksmith twin with the same steps, env, outputs and permissions that signs in the same run when the owned attempt never passed its gate, and every reader of the signing job reads the twin first. The guard tests check the twin's shape, the gate (refuses fb0442d3, passes 2695209c), and that every listed hook is one glaeda-disk pins. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…; the notary receipt step moves to a script for room Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
|
cmux status (one comment, edited in place; each line is updated by the job that owns it) Review App of this head (app.zip, kept 1 day)
red: cmux-next checks (god files, concurrency, crash safety, l10n); no finished base run on 7e93d81 to compare; do not merge until green; see lane-rules |
|
Owned by an agent session · session |
|
Bead |
|
Found 6 test failures on Blacksmith runners: Failures
|
…t script keeps the step's errexit Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…marker outcome step to a script (the notary receipt step stays inline) The build-sign-notarize-nightly checkout is the build's commit, so a new script there would be missing for older builds; the publication marker step runs from .publish-tooling, the workflow commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
|
The feat-cmux-next push run https://github.com/manaflow-ai/cmux/actions/runs/40855479350 at 8c5ab04 failed: cmux-next checks (god files, concurrency, crash safety, l10n). |
|
Shipped in:
|
…egacy-blacksmith job (#27485) * ci(owned-pool rescue): cover feat-cmux-next's sign-nightly-identity-legacy-blacksmith job #27357 added a same-run Blacksmith signer for a refused owned-Mac legacy identity attempt on feat-cmux-next. Main's NIGHTLY_SIGN_ALTERNATES did not list it, so test_ci_owned_pool_sources fails in CI fast guards on every main PR. Its runs-on already takes the legacy alternate on a re-run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-cc1df27cbb6156fd16c2 * ci(owned-pool rescue): say what a rescue re-run does for the legacy Blacksmith signer Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-cc1df27cbb6156fd16c2 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…tional; decide waits out rate limits Nightlies never use Blacksmith, ever (Lawrence, 2026-10-30). Blacksmith's macOS queue stalled and held the cmux NIGHTLY-identity copy (the old feed, the user's NIGHTLY) on build 4093446036501: sign-nightly-identity-legacy and generate-nightly-deltas waited on blacksmith-6vcpu-macos-26 for hours. - sign-nightly-identity-legacy, verify-nightly-identity-legacy and generate-nightly-deltas fall back to the owned nightly slot (vars.CI_NIGHTLY_NEXT_BUILD_RUNNER), then the owned side lane, never a Blacksmith label (the re-run alternate and vars.MACOS_RUNNER_26 are gone). - sign-nightly-identity-legacy-blacksmith (the twin, #27357) is removed: it could only sign on Blacksmith. A refused or failed owned sign is retried by the next nightly run, on owned Macs. The signing guard now refuses a Blacksmith twin in any nightly workflow. - generate-nightly-deltas and republish-nightly-deltas are optional (continue-on-error, NIGHTLY_DELTAS=off skips them): the feed already serves the full DMG, so deltas never fail a run or hold publication. - decide's github-script waits out GitHub API rate limits (Retry-After or x-ratelimit-reset, at most 10 min a wait, 20 min in all) instead of failing (run 41153241702: "API rate limit exceeded for installation"). - glaeda-signing-hooks.txt lists glaeda's newest hook (485d1dcc, #1764), so minis on it may sign. - AGENTS.md/CLAUDE.md state the rule. Leo: this reverses the "owned Macs, then Blacksmith" nightly routing (#26253, #26299, #26355, #27357, #26936, #27125, #28209). Lawrence's rule: nightlies run on our own Mac minis only, never Blacksmith, on any fallback, re-run or overflow path. Please don't add Blacksmith labels to nightly jobs. The other nightly jobs move in the next commit, with a guard test over every nightly workflow.
* ci guards: depth-limited fetches only in shallow checkouts (base red) scripts/measure/fresh-home-launch.sh and workers/cmux-vm/scripts/test-mesh-label-compat.ts passed --depth to a fetch in the caller's checkout, which turns a full clone shallow. Both now pass it only when git rev-parse --is-shallow-repository says the checkout is already shallow. Regressed by #30495 (fresh-home-launch.sh) and #30501 (test-mesh-label-compat.ts). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: map the transcript's TurnRows.tsx in place of the deleted MessageGroupRow.tsx (base red) #29800 removed MessageGroupRow.tsx (every sent message is its own card); TurnRows.tsx now draws each sent message row in order (SentMessageLine.tsx is already mapped), so chat-transcript maps it. Regressed by #29800 (deleted the file #28698 had mapped). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: scan cmux-next-ios/verify/ for errno-after-assertion (base red) cmux-next-ios/verify/imsg-replies/GestureServerUITests.swift is an XCTest source outside every TEST_ROOTS entry. Scan cmux-next-ios/verify/ (it has no errno use, so nothing else changes). Regressed by 201101a (added the verify/imsg-replies rig; landed by a safe-push train, no PR). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: list the four manual-lane workflow readers in UNWIRED (base red) tests/test_ci_blacksmith_linux_sizes.py, test_ci_overflow_pick_reads_pool.py, test_nightly_next_control_rescue.py and test_ci_cloud_probe_rescue.py read .github/workflows/ but are lane = "manual" in tests/test-execution.toml (no new CI tests; wiring them as linux-guard reddened tooling-tests on every PR). They join UNWIRED with the same reason as the other manual-lane entries, and each says when to run it by hand. Regressed by #30360, #30878, #30657 and #30796 (each added its test without an UNWIRED entry). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: fleet rule lints pass again (base red) "ci: Validate fleet rule lints" failed on feat-cmux-next in three rules. big-red: the personal machine's name left product code and tests. Doc comments say "one lane VM" / "<host>"; the webview tests use my-vm (its machine hue still differs from gpu-box's). Regressed by #28105 (peer_handshake_tests.rs, list_reply.rs, list_reply_tests.rs), #29495 (host_update_tests.rs), #29126 (chatOpenPacedList.test.ts), #31108 (project-chooser-machine-tag). pkill: the iOS verify rigs and the tool-evals remote daemon stop only what they started. imsg-replies records the conversation-sim, the recorders (simctl itself, not the xcrun shim) and the gesture servers in pidfiles and signals those PIDs after checking the command line still matches; pq_browser_events records its two site servers' PIDs; the tool-evals remote stop kills the setsid group whose PID the launcher recorded beside the socket. Regressed by 201101a (imsg-replies, first in #22328), #25915 (pq_browser_events.py), #29080 (tool-evals/session.mjs). Allowlisted as safe, with reasons: scripts/ci/macos-stubs/Security.tbd (a linker stub listing symbol names, no keychain read; #27603) and tests/tool-evals/cases/hard.ndjson (an eval case forbidding pkill; #28738). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: determinize python tests flagged by test-determinism (base red) Regressed by the commits introducing each assertion (see report). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: nightly Xcode selection, unused anchors, daemon retention key (base red) - Validate macOS jobs select a pinned Xcode: the nine flagged nightly.yml jobs already run select-ci-xcode.sh through the `- *select-xcode` step alias; the guard did not expand step aliases. It now resolves `- *name` to the `- &name` step, with self-tests for both an alias to the selector and an alias to another step. Regressed by #30471 (steps turned into aliases to keep nightly.yml under its size limit). - Lint every workflow: drop the five sign-nightly-identity-legacy-* anchors left after the Blacksmith twin that aliased them was removed (a7b6a61, "legacy sign and deltas run on owned Macs only"; the twin came from #27357). No behaviour change. - upload-artifact retention: the 14-day cmux-nightly-remote-daemon upload moved from build-nightly-app to build-nightly-remote-daemon in #25935 with its retention unchanged; move the allowlist key with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: media tours and visual-gate state match their validators (base red) Drop the unknown step key "record" (re-added after #26888 removed it by #27933 and f283a2e), the unimplemented top-level frameInterval in home-chief-shimmer (c022070), the focusLeft right after closePane in palette-chat-toggles (#25405), split pane-polish (31 steps) into pane-polish and pane-polish-themes, and give the app-store-consent-deploy state a synthetic host that is not an email shape (4a05792). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: allowlist remaining test-determinism findings with reasons (base red) Swift/web/tool-evals findings cannot be determinized without Mac or webview runs; each entry names its reason and adding PR. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: Linux pool for the push kicks; record pool readers as unrescued (base red) - cmux-next-base-tier-kick, cmux-next-dev-kick and ios-internal-nightly-trigger waited on one fixed Blacksmith Linux size (!startsWith(LINUX_RUNNER) || overflow 8 || 2 vCPU). They now read fromJSON(vars.RUNNER_POOLS).linux after the fork branches. Their pinning tests (base tier, DEV kick, iOS internal builder) assert the pool read instead. Regressed by #30837. - owned_pool_rescue.py: feat-cmux-next's UNRESCUED_WORKFLOW_PATHS records the secretless Linux pool readers the owned-pool sources test found unlisted (fast gates, gallery health, PR checks, cloud backend check, web validation, and the three kicks above), with the existing _POOL_READ_WHY. Regressed by #29844 and #31138. - test_ci_linux_single_size.py: drop cmux-next-fast-gates.yml from FIXED_PAIR (moved by #31138); test_ci_overflow_pick_reads_pool.py: drop the two kicks from PICKS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: signing-runner and retry-attempt guards follow the owned-Mac signing rule (base red) Signing keychain guard (rule 7): accept owned-pool runner variables only where safety is proven in the job itself: - 7a: a job in glaeda's exclusive signing class whose first step is the verbatim claim gate may fall back to CI_NIGHTLY_NEXT_BUILD_RUNNER / CI_SIDE_LANE_RUNNER (nightly.yml sign-nightly-identity-legacy and generate-nightly-deltas, owned-only by design, Lawrence a7b6a61; deltas' Sparkle key added in #29261). test_ci_signing_owned_mac.py holds the claim script, always() keychain delete and twin/picker rules for them. - 7b: stable-next.yml sign (#27898) reads every signing secret only under env.SIGN == 'developer-id', whose condition picks a literal Blacksmith label before any variable; its owned dry-run holds no key. Proven structurally. Retry guard: cmux-next.yml swift-test/daemon-test/generated-files' side-lane fallback (#29814) is now gated at run_attempt <= 3 for the bot, so the bot's attempt 4 takes blacksmith-6vcpu-macos-26 like every other cmux-next fleet fallback (#22565). The three owned-only nightly jobs are exempt while they keep exactly their owned route. The textual check now forbids naming attempts after 2 (== 3+), as its comment says, instead of also == 2 (#29847 nightly.yml, #29130 cmux-next-guards.yml name the bot's attempt 2, which may stay owned). Regressed by #29814, #27898, #29261, #29847, #29130 and a7b6a61. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: owned Mac pools through the picker/vars pattern; follow nightly step scripts for sentry-cli (base red) check_no_self_hosted_fleet_runners The guard told every workflow to "use a cloud label so required jobs never land on a mini". That is outdated policy: owned fleet Macs now come first for macOS jobs (Linux, then owned Macs through the picker, Blacksmith last), release signing may run on owned Macs, display tests gate nothing, and the NEXT nightly runs on the fleet with the nightly-only minis dedicated to it. The guard kept up only through exact per-workflow selector strings, which drifted with every routing change and turned the guard red on base: - cmux-next.yml swift-test/daemon-test/generated-files: the glaeda-side deferred fallback (regressed by #29814) and the attempt-1 gui selector (#28605, #27853) - cmux-next-base-autofix.yml / cmux-next-pr-autofix.yml: the gui selector gained a re-run predicate (#27853) - nightly.yml sign/prepare jobs: owned pool as the final fleet fallback (#29154) Fix the guard instead of moving those jobs to Blacksmith: an owned POOL label (glaeda-[<lane>-]<class>-xcode-<version>, served by many minis, never one host) is accepted as a quoted literal in a ${{ }} runs-on that routes through the declared picker/vars pattern (reads needs.<job>.outputs.* or vars.*). In a workflow a pull request can trigger, the fork pull request branch (fork heads to Blacksmith) must still come first on the line, so fork code never reaches an owned Mac (runner trust unchanged). Still refused: bare self-hosted/macOS/ARM64, single-host labels (cmux-macos, mac-mini, slot-N, tart-*, ...), macos-26 outside the fork branch, and an owned label named directly with no picker or variable route. The exact gui selector strings are replaced by this rule; self-tests cover each accept and refuse case. check_sentry_cli_install_portability #31013 moved nightly.yml's "Upload dSYMs to Sentry" body into scripts/ci/nightly-steps/app-upload-dsyms.sh (size budget), so the guard no longer saw the ./scripts/upload-sentry-dsyms.sh call. The guard now reads the step's step script too. Regressed by #31013. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: change-area tests follow the RUNNER_POOLS and devEngines routing; helper walk skips non-running namers (base red) - release-admission and remote-daemon-admission read fromJSON(vars.RUNNER_POOLS).linux by design since #30878; the two tests now assert that instead of vars.LINUX_RUNNER. - fast-checks.yml's setup-bun reads the bun pin from webviews/package.json devEngines at run time (#29757); the setup-bun test accepts a steps.<id>.outputs.version whose step reads devEngines, and still requires the literal pin everywhere else. - indexnow.yml pinned bun 1.3.14 when #30697 backported it; it now installs devEngines' 1.4.4. - ci_helper_areas() failed open on files that name a helper without running it, so owned_pool_rescue.py, owned_build_state.py and e2e_warm_derived_data.py stopped routing by where they run. Regressed by #28528 (test_cmux_next_route.py names owned_pool_rescue.py as a routing input and itself), #28513 (docs/ci/cmux-next-tiers.md names that test), #30657 (nightly_next_control_rescue.py's only test is manual-lane), #28568 (.github/warm-sources.json prose) and #30467 (base_red_excuse.py's guard-test table and its test's fixtures). The walk now ignores a test naming itself, docs, a Python test naming it only in a docstring, the listed path tables and warm-sources.json, and treats a test the registry declares in an inert lane that nothing imports as a dead end. An unregistered test nothing names still fails open. - cache_restore_receipt.py now routes precisely (macOS + Release, via ci-macos.yml's cache-restore action) instead of failing open; its assertion says so. python3 tests/test_ci_change_areas.py: 4 of 234 failed before, 0 after. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: review fixes for #31366 (daemon pgid, fork-first owned pools, stale pidfiles) - tests/tool-evals/session.mjs: the remote launcher recorded $! of the backgrounded && chain (the wrapping subshell), so the stop signalled a group that did not exist. setsid now starts a shell that writes its own PID and execs the daemon, so the recorded PID is the daemon's process group. Checked locally: the recorded PID equals the daemon's PGID and SID, and kill -TERM -- -PID stops it. - tests/test_ci_self_hosted_guard.sh (security): the owned-pool exemption in pull request workflows only required the fork branch somewhere before the owned literal, so `needs.x.outputs.runner || <fork branch> || 'glaeda-...'` passed. The fork branch must now be the first selector of the expression (after only the hosted repository_owner clause). New negative self-tests for reordered expressions; red with the old match, green with the new. - cmux-next-ios/verify/parity/pq_browser_events.py: stop_site() now stops only servers this run started (serve() reports each pidfile it wrote), kills a PID only while its command line is still that server's, and removes the pidfile after. Checked with a local stub harness. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 * ci guards: universal nightly helper runs-on match works under mawk (base red) The helper runs-on expected string passed to awk -v carried `\&\&`, which gawk reads as `&&` (with a warning) but mawk, the default awk on Ubuntu runners, keeps as `\&\&`, so the whole expression never matched and "nightly must build and hand off the macOS 26 helper" failed. Plain `&&` means the same in both. Regressed by #30824. bash tests/test_nightly_universal_build.sh: FAIL before, PASS after (mawk 1.3.4). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-fc739fd2f4e557ae7189 --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
![Fix with [code]smith](https://pr-comments-assets.blacksmith.sh/codesmith/fix-with-codesmith-light-3x.png)
Leo, 2026-10-28, on the release signing jobs: "ok. idk what the "risk" was before but like.. we can just read the code and stuff. anyways, what about that signing on linux boxes thing? all those other ones basically yes but what do i need to do". Leo, 2026-10-24, on nightly freshness: "it is unacceptable to have nightlies >30min out of date. shouldn't be cancelling unless existing run would've literally been busted".
A signing job on an owned Mac needs the whole mini. Some runner directories still carry a glaeda hook from before the leftover-process check (teamleaderleo/glaeda#1738), and that hook can admit a signing job on a mini where an earlier job's processes are still running. glaeda's claim now names the hook that granted it (teamleaderleo/glaeda#1739,
GLAEDA_HOOK_SHA256). This PR makes every owned-routed signing job accept only listed hooks, and has the same run sign on Blacksmith whenever the owned attempt is refused.Claim gate. The first step, before checkout and before any secret:
GLAEDA_JOB_CLAIM=exclusiveand aGLAEDA_HOOK_SHA256thatscripts/ci/glaeda-signing-hooks.txtlists. It fails closed on a missing or unlisted sha.claim=owned.signing-runner-claim.shchecks the same thing against the file.Same-run fallback. Each owned-routed signing job gets a Blacksmith twin,
<job>-blacksmith:sign-nightly-identity-legacy, releasebuild-sign-notarize, repairrepairand dev-dailysign.Guard tests (
tests/test_ci_signing_owned_mac.py):ifis exact, and every reader also reads the twin.fb0442d3(the hook before the check), a missing sha, a prefix and junk, leavingclaimempty so the twin signs. It passes2695209cand every other listed hook withclaim=owned.CMUX_NEXT_DRAIN_HOOK_SHA256, at or after2695209c. It warns when glaeda's newest hook isn't listed yet; signing then stays on Blacksmith for those minis.Before: a refused owned attempt failed the signing job, and only a re-run moved it to Blacksmith. A nightly's legacy sign could go stale meanwhile. After: the same run signs on Blacksmith.
Owned signing stays off (
vars.CI_SIGN_OWNED=0) until this and teamleaderleo/glaeda#1739 land and the hook reaches the minis.Risk: the four signing workflows change shape. Release, repair and dev-daily jobs keep their status, so a refused owned attempt still shows red in their run while the twin signs. The nightly job already has
continue-on-error.Also: the twins are added to
.github/runners.json, the nightly step-coverage registry and the nightly guards. To keep nightly.yml under its size margin, publish-nightly's "Record nightly publication marker outcome" step moves, unchanged, intoscripts/ci/nightly-steps/publish-tag-outcome.sh. It runs from.publish-tooling(the workflow commit) like the job's other moved steps.Tests: the signing-owned-mac, keychain, Xcode-selection, artifact-retention, dev-daily dispatch, step-coverage, sign-runner, routing-table, size and universal-build guards pass locally, and actionlint is clean. Local reds that also fail on base: fleet_rule_lints, cmux_next_tour, sparkle_build_monotonic, delta_since_green, notarize_nightly_dmg, install_cmux_tui_client, ghostty_zig_version_sync, nightly_next_arm64_only.
Agent session: agent session
🤖 Generated with Claude Code