Skip to content

release signer: wait out Apple notary rate limits instead of failing the nightly - #33028

Merged
teamleaderleo merged 2 commits into
feat-cmux-nextfrom
lane/notary-rate-limit-wait
Nov 3, 2026
Merged

teamleaderleo merged 2 commits into
feat-cmux-nextfrom
lane/notary-rate-limit-wait

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Nightly run 41571567978 (the tip build Leo's PRs were waiting on) failed at signer-app. Apple refused the Computer Use helper's notarization:

"code": "RATE_LIMIT", "status": "429", "title": "Too many notarizations",
"detail": "Exceeded daily limit for notarizations. Try again in 0:04:11."

The signer treated this like a rejection and failed the run, so nothing published. Apple only asked for four minutes.

Change

  • scripts/ci/release-signer/signer-env.sh gets notary_submit, a wrapper around notary-submit. On a 429 RATE_LIMIT it waits the time Apple names plus 15 s, then submits again. All waits share one budget, CMUX_NOTARY_RATE_LIMIT_SECONDS, 600 s by default, which fits inside the signer jobs' 40/75 min timeouts beside their notary waits.

    • Apple keeps nothing from a refused submission, so the retry duplicates nothing.
    • Any other failure still fails at once.
  • sign-app.sh (the helper submission), notarize-app.sh and notarize-dmg.sh submit through it. Both notarizations stay, and so does the stapled app inside the DMG.

  • tests/test_release_signer_scripts.py: the fake signer can now answer with Apple's 429, and a stand-in sleep records each wait. The new tests cover:

    • one refusal, then success (waits 266 s, submits twice);
    • a wait longer than the budget (no wait, fails);
    • the budget spent across several refusals;
    • a failure that is not a rate limit (no retry).

    The three rate-limit tests fail without the change.

Waiting adds time only when Apple has already refused, a case that failed the whole nightly before.

Follow-up, not in this PR: we are near Apple's daily cap. Each nightly now notarizes the helper, app and DMG for two identities (since #32140). Skipping app-identical promotions (#32689) cuts the count; the Mac fallback scripts (notarize-computer-use-helper.sh, notarize-nightly-dmg.sh) do not wait on a 429 yet.

Agent session: agent session

🤖 Generated with Claude Code


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

The release signer now waits out Apple's notary rate limits instead of failing the nightly. Previously a 429 RATE_LIMIT response aborted the run even when Apple only asked for minutes.

  • Adds notary_submit in signer-env.sh, wrapping notary-submit: on a 429 it waits the time Apple names plus 15 s and submits again, within CMUX_NOTARY_RATE_LIMIT_SECONDS (600 s by default).
  • All three notarization paths (helper, app, DMG) now submit through it. Refused submissions leave nothing at Apple, so retries are safe; other failures still fail immediately.
  • Adds tests covering a single rate limit, budget exhaustion across refusals, and non-rate-limit failures.

Written for commit 0621804. Summary will update on new commits.

View guided diff Turn on auto-fix

… the nightly

Nightly 41571567978 failed at signer-app: Apple answered the Computer Use
helper submission with 429 RATE_LIMIT, "Exceeded daily limit for
notarizations. Try again in 0:04:11." sign-app.sh, notarize-app.sh and
notarize-dmg.sh now submit through notary_submit (signer-env.sh), which
waits the time Apple names plus 15 s and submits again, within
CMUX_NOTARY_RATE_LIMIT_SECONDS (600 s by default). A refused submission
leaves nothing at Apple, so the retry duplicates nothing; any other failure
still fails at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-f724bb1ff9295c18fee7
@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review c026616: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr33028 -f ref=lane/notary-rate-limit-wait -f platform=macos -f backend_mode=local

web and size checks: green

Shipped in:

@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 3, 2026
@blacksmith-sh

blacksmith-sh Bot commented Nov 3, 2026

Copy link
Copy Markdown

Found 6 test failures on Blacksmith runners:

Failures

Test View Logs
/tmp/tmp9kq8awwb/summary-fail.json (live):/summary-fail.json (live): View Logs
/tmp/tmpg_umamm0/summary-missing.json (live):/summary-missing.json (live): View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows
View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/
[tabs] debug.tab_switch timed 0 of 40 switches
View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex, router-review): SHIP at c026616

Reviewer: codex gpt-6-astra high, fast tier.

SHIP at c026616
Visual verdict: no images; Leo tries the build

Reviewed all three call sites and retry/error handling. All 52 signer tests passed across runs after using OPENSSL_CONF=/dev/null to bypass the box’s missing OpenSSL configuration; shell syntax checks passed.

Follow-ups: none

@teamleaderleo teamleaderleo removed the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 3, 2026
@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 3, 2026
cmux-bot[bot]
cmux-bot Bot previously approved these changes Nov 3, 2026

@cmux-bot cmux-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cmux-bot approves c026616: an independent review passed this exact head (router-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

@cmux-bot
cmux-bot Bot temporarily deployed to ci-routing-agent November 3, 2026 05:03 Inactive
@cmux-bot
cmux-bot Bot temporarily deployed to ci-routing-agent November 3, 2026 05:04 Inactive
@cmux-bot
cmux-bot Bot dismissed their stale review November 3, 2026 05:13

A newer verdict blocks 0621804 (no independent SHIP at the head).

@teamleaderleo
teamleaderleo merged commit f739716 into feat-cmux-next Nov 3, 2026
151 of 156 checks passed
@teamleaderleo
teamleaderleo deleted the lane/notary-rate-limit-wait branch November 3, 2026 05:25

This branch was previously deployed

1 inactive deployment
ci-routing-agent — 0621804a Deployed Nov 3, 2026 by cmux-bot[bot] via wait for same-key tree publisher #104045
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant