Skip to content

CI: pin signing/notarization jobs to WarpBuild (fix self-hosted codesign errSecInternalComponent) - #6264

Merged
lawrencecchen merged 1 commit into
mainfrom
feat-ci-sign-on-warp
Jun 16, 2026
Merged

lawrencecchen merged 1 commit into
mainfrom
feat-ci-sign-on-warp

Conversation

@lawrencecchen

@lawrencecchen lawrencecchen commented Jun 16, 2026 •

Copy link
Copy Markdown
Contributor

Why

The nightly build-sign-notarize-nightly job fails repeatedly at "Codesign apps":

Warning: unable to build chain to self-signed root for signer "***"
cmux NIGHTLY.app/.../bin/cmux: errSecInternalComponent

Root cause: repo vars MACOS_RUNNER_15 = cmux-aws-macos-15 and MACOS_RUNNER_26 = cmux-macos-26 resolve to self-hosted minis that aren't provisioned for codesigning (their keychains lack the Developer-ID/WWDR chain). The Warp fallback label never triggers because the var is set. Same code signs fine on Warp (the 03:12 nightly succeeded); it fails whenever the job lands on a mini. (Same runner-routing root cause as the earlier npm: command not found nightly failure.)

Fix

Pin the two jobs that codesign + notarize directly to the Warp images, bypassing the self-hosted vars:

  • nightly.yml build-sign-notarize-nightly → warp-macos-15-arm64-6x
  • release.yml build-sign-notarize → warp-macos-26-arm64-6x

The non-signing build-ghostty-cli-helper job stays on the shared var (it doesn't sign). This is "for now" — revert to the vars indirection once the minis are provisioned for signing (or moved behind runner groups). Workflow-only.

🤖 Generated with Claude Code


View with Codesmith Autofix with Codesmith
Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.


Note

Low Risk
Workflow-only runner routing for two jobs; no app, signing secrets, or build script changes—only where CI executes existing sign/notarize steps.

Overview
Nightly and release codesign/notarize jobs no longer honor vars.MACOS_RUNNER_15 / vars.MACOS_RUNNER_26, which were routing those jobs to self-hosted minis where codesign fails with missing Developer-ID/WWDR chain (errSecInternalComponent).

nightly.yml build-sign-notarize-nightly now uses warp-macos-15-arm64-6x; release.yml build-sign-notarize uses warp-macos-26-arm64-6x. Inline comments document that this is temporary until self-hosted runners are provisioned for signing.

Non-signing work (e.g. release build-ghostty-cli-helper) is unchanged and still uses the repo runner vars.

Reviewed by Cursor Bugbot for commit 2208eea. Bugbot is set up for automated code reviews on this repo. Configure here.


Summary by cubic

Pin the signing and notarization CI jobs to WarpBuild macOS runners to fix codesign failures on self-hosted minis. Nightly and release signing should now pass reliably.

  • Bug Fixes
    • Pinned signing jobs to warp-macos-15-arm64-6x (nightly) and warp-macos-26-arm64-6x (release).
    • Self-hosted minis (from vars) lack the Developer-ID/WWDR chain, causing "unable to build chain to self-signed root" / errSecInternalComponent.
    • The non-signing build-ghostty-cli-helper job stays on the shared var.

Written for commit 2208eea. Summary will update on new commits.

Review in cubic

Summary by CodeRabbit

  • Chores
    • Updated macOS build runner configurations to use pinned runner labels for improved build reliability and consistency across nightly and release pipelines.

…ernalComponent)

vars.MACOS_RUNNER_15 (cmux-aws-macos-15) and MACOS_RUNNER_26 (cmux-macos-26)
resolve to self-hosted minis that are NOT provisioned for codesigning: their
keychains lack the Developer-ID/WWDR chain, so `codesign --sign` fails with
'unable to build chain to self-signed root' / errSecInternalComponent (nightly
build-sign-notarize failed repeatedly; the same code signs fine on Warp).

Pin the two signing+notarization jobs to the Warp images directly:
- nightly.yml build-sign-notarize-nightly -> warp-macos-15-arm64-6x
- release.yml  build-sign-notarize        -> warp-macos-26-arm64-6x

The non-signing helper build (release build-ghostty-cli-helper) stays on the
shared var. Revert to the vars indirection once the minis are provisioned for
signing.
@vercel

vercel Bot commented Jun 16, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
cmux Building Building Preview, Comment Jun 16, 2026 9:09pm
cmux-staging Building Building Preview, Comment Jun 16, 2026 9:09pm

@coderabbitai

coderabbitai Bot commented Jun 16, 2026 •

Copy link
Copy Markdown

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: dca85d07-74ee-445e-a060-12a58c8f1a8a

📥 Commits

Reviewing files that changed from the base of the PR and between ce67127 and 2208eea.

📒 Files selected for processing (2)
  • .github/workflows/nightly.yml
  • .github/workflows/release.yml

📝 Walkthrough

Walkthrough

Two CI workflow files (nightly.yml and release.yml) replace variable-with-fallback runs-on expressions with hard-pinned WarpBuild macOS runner labels for their respective signing/notarization jobs. Surrounding comments are updated to reflect the codesigning requirement.

Changes

macOS Runner Pinning

Layer / File(s) Summary
Hard-pin WarpBuild runner labels in nightly and release jobs
.github/workflows/nightly.yml, .github/workflows/release.yml
build-sign-notarize-nightly in nightly.yml replaces ${{ vars.MACOS_RUNNER_15 || 'warp-macos-15-arm64-6x' }} with the literal warp-macos-15-arm64-6x; build-sign-notarize in release.yml replaces ${{ vars.MACOS_RUNNER_26 || 'warp-macos-26-arm64-6x' }} with the literal warp-macos-26-arm64-6x. Comments in both files are updated to describe the codesigning chain constraint.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Possibly related PRs

  • manaflow-ai/cmux#4926: Adjusts the same macOS runner labels for build-sign-notarize-nightly and build-sign-notarize jobs in the same two workflow files.
  • manaflow-ai/cmux#4984: Introduced the vars.MACOS_RUNNER_*-with-fallback pattern in nightly.yml and release.yml that this PR is now replacing with pinned labels.
  • manaflow-ai/cmux#5022: Modifies runs-on runner selection in the same signing/notarization jobs across nightly.yml and release.yml.

Poem

🐇 Hop, hop, no more vars to chase,
The runner label's pinned right in place.
No fallback dance, no expression to parse,
Just warp-macos — steady and sparse.
A notarized build on a hardcoded lane,
This bunny loves CI that's perfectly plain! ✨

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat-ci-sign-on-warp

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 2208eea. Configure here.

# runners lack the Developer-ID/WWDR chain in their keychain, so codesign
# fails with errSecInternalComponent. Signing must run on the Warp image
# that carries the chain until the minis are provisioned for signing.
runs-on: warp-macos-26-arm64-6x

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Release SDK lane test mismatch

Medium Severity

Pinning build-sign-notarize to runs-on: warp-macos-26-arm64-6x drops the vars.MACOS_RUNNER_26 expression that tests/test_ci_release_sdk_lane.sh still requires in that job, so workflow-guard-tests fails even though the job still targets macOS 26.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 2208eea. Configure here.

@lawrencecchen
lawrencecchen merged commit 38cdb93 into main Jun 16, 2026
18 of 22 checks passed
@greptile-apps

greptile-apps Bot commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Pins the two codesigning/notarization jobs in nightly.yml and release.yml directly to their WarpBuild runner images, bypassing the vars.MACOS_RUNNER_* indirection that routes those jobs to self-hosted minis whose keychains lack the Developer-ID/WWDR chain.

  • nightly.yml build-sign-notarize-nightly: ${{ vars.MACOS_RUNNER_15 || 'warp-macos-15-arm64-6x' }} → warp-macos-15-arm64-6x
  • release.yml build-sign-notarize: ${{ vars.MACOS_RUNNER_26 || 'warp-macos-26-arm64-6x' }} → warp-macos-26-arm64-6x
  • Non-signing jobs (build-ghostty-cli-helper) are intentionally left on the shared vars; comments document the temporary nature of the pin and the provisioning prerequisite for reverting.

Confidence Score: 5/5

Safe to merge — workflow-only change that removes a flaky code path without touching any production build logic.

Both changed lines replace a conditional expression whose non-Warp branch was actively breaking production signing. The new hardcoded labels are identical to the existing fallback values, so the Warp jobs continue to run exactly as before when the var was unset. Comments clearly document the temporary nature of the pin and the prerequisite for reverting. No production code, Swift, or runtime logic is touched.

No files require special attention.

Important Files Changed

Filename Overview
.github/workflows/nightly.yml Hardcodes runs-on: warp-macos-15-arm64-6x for the codesigning job, replacing the `vars.MACOS_RUNNER_15
.github/workflows/release.yml Hardcodes runs-on: warp-macos-26-arm64-6x for the codesigning/notarization job, replacing the `vars.MACOS_RUNNER_26

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Nightly / Release trigger] --> B{Job type}
    B -->|build-ghostty-cli-helper\nnon-signing| C["vars.MACOS_RUNNER_* ||\n'warp-macos-*'\n(unchanged — no codesign)"]
    B -->|build-sign-notarize\ncodesign + notarize| D["BEFORE: vars.MACOS_RUNNER_*\n|| 'warp-macos-*'"]
    D -->|var IS set| E["self-hosted mini\n(no WWDR chain ❌)"]
    E --> F["errSecInternalComponent\nbuild fails"]
    B -->|build-sign-notarize\ncodesign + notarize| G["AFTER: warp-macos-*-arm64-6x\n(pinned, no var lookup)"]
    G --> H["WarpBuild image\n(full keychain ✅)"]
    H --> I["codesign + notarize\nsucceeds"]
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[Nightly / Release trigger] --> B{Job type}
    B -->|build-ghostty-cli-helper\nnon-signing| C["vars.MACOS_RUNNER_* ||\n'warp-macos-*'\n(unchanged — no codesign)"]
    B -->|build-sign-notarize\ncodesign + notarize| D["BEFORE: vars.MACOS_RUNNER_*\n|| 'warp-macos-*'"]
    D -->|var IS set| E["self-hosted mini\n(no WWDR chain ❌)"]
    E --> F["errSecInternalComponent\nbuild fails"]
    B -->|build-sign-notarize\ncodesign + notarize| G["AFTER: warp-macos-*-arm64-6x\n(pinned, no var lookup)"]
    G --> H["WarpBuild image\n(full keychain ✅)"]
    H --> I["codesign + notarize\nsucceeds"]
Loading

Reviews (1): Last reviewed commit: "CI: pin signing jobs to WarpBuild (fix s..." | Re-trigger Greptile

lawrencecchen added a commit that referenced this pull request Jun 16, 2026
…6264 follow-up) (#6265)

#6264 pinned release.yml build-sign-notarize to warp-macos-26-arm64-6x (the
self-hosted minis lack the signing chain). This guard still asserted the old
vars.MACOS_RUNNER_26 indirection string, so workflow-guard-tests failed on main
and every PR after #6264 merged. Update the expected needle to the pinned Warp
runner. nightly.yml is intentionally not covered by this guard; the self-hosted
guard already accepts the warp-macos-NN-arm64 label.
teamleaderleo added a commit that referenced this pull request Sep 26, 2026
…h as fallback (#14821)

* ci: build the nightly app on the trusted owned minis first, Blacksmith as fallback

build-nightly-app was hard-coded to blacksmith-12vcpu-macos-26, so a push
to main never considered an owned mini (run 36239331473). Attempt 1 of a
push or schedule run on main now asks for vars.CI_SEED_TRUSTED_POOL, the
trusted owned pool (no pull request runners; the glaeda hook admits only
main's push and schedule jobs), because the job holds the ci-cache-writer
R2 keys and the Sentry token and its app is signed and shipped. Forks,
rc/**, dispatches, fast dogfood and re-runs keep Blacksmith.

- owned_pool_rescue.py / ci-owned-pool-rescue.yml: watch nightly.yml runs
  like a picker-less side lane (NIGHTLY_WORKFLOW_PATH). Recognise
  glaeda-trusted-* labels. Allow one queue round behind a DerivedData seed.
  A stuck or refused build is re-run on Blacksmith, unless a newer nightly
  run on main is still pending.
- nightly.yml: an owned runner folds its workspace path into the release
  compile-cache key, so the mini and Blacksmith lineages never evict each
  other (Blacksmith's key is unchanged). Clear build outputs a persistent
  runner kept.
- runner_label_policy.py: CI_SEED_TRUSTED_POOL may only name a
  glaeda-trusted-* label; the health report and variable check read it.
- Signing and notarization stay on Blacksmith (unproven on owned Macs;
  #6264). Docs list every macOS job's route.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: send the nightly app build to cmux15's trusted runner only

Both trusted minis carry glaeda-trusted-std-xcode-26.6, but
cmuxs-mac-mini-6 also runs the team dev-build worker as the same user, so
the shipped build should never land there. Seeding keeps both, so
CI_SEED_TRUSTED_POOL is unchanged. build-nightly-app now asks for
["<CI_SEED_TRUSTED_POOL>", "<CI_NIGHTLY_TRUSTED_RUNNER>"]: the trusted pool
and one runner's own glaeda-runner-<name> label (glaeda-cmux-runner's
runner_label()). Either variable empty means Blacksmith, so this is inert
until cmux15's runner is re-registered with that label and the variable is
set.

- runner_label_policy.py: CI_NIGHTLY_TRUSTED_RUNNER may only be a
  lowercase glaeda-runner-* label (also safe inside the JSON runs-on).
- ci-health-report.yml / ci-repo-variables.yml: report it.
- ci-owned-pool-rescue.yml: watch nightly runs only while both are set.
- Docs and tests follow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Preview – cmux — 2208eeaf Deployed Jun 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant