Skip to content

ci(nightly): signer jobs take GitHub-hosted Ubuntu first again, Blacksmith 4 vCPU on a re-run - #31831

Merged
teamleaderleo merged 1 commit into
feat-cmux-nextfrom
lane/nightly-signers-hosted-first
Nov 2, 2026
Merged

teamleaderleo merged 1 commit into
feat-cmux-nextfrom
lane/nightly-signers-hosted-first

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

The four NEXT nightly signer jobs (signer-build, signer-app, signer-app-notary, signer-dmg) go back to the routes #29886 gave them, and that the hosted-runner guard (tests/test_ci_self_hosted_guard.sh, signer_route and signer_build_route) still allowlists:

  • attempt 1: GitHub-hosted ubuntu-24.04
  • re-run: blacksmith-4vcpu-ubuntu-2404, never 16 or 32 vCPU
  • signer-build takes decide's early fleet placement first, then the same route

Each runs-on line now matches the guard's allowlisted line exactly. #30081 had moved the four jobs onto the sign picker's Blacksmith Linux 4/16/32 output, citing the older 2026-10-27 no-hosted rule. The 2026-11-01 rule below overrides that rule for signing.

The rule, verbatim:

"One measured exception to the two rules above: a small job that is not CPU-bound, such as signing and its runner pick, takes GitHub-hosted Ubuntu first. It's trusted like Blacksmith and free on a public repo. The fleet, or the smallest Blacksmith size that fits, is its re-run route, never 16 or 32 vCPU. It still needs a fallback so a hosted queue never stalls it: a re-run onto that route, or a twin that takes over (the nightly's macOS signer). Each such job is one exact line in the hosted-runner guard's allowlist (cmux tests/test_ci_self_hosted_guard.sh), with its measured wall time beside it."

The fallback is unchanged. A hosted signer that GitHub drops unstarted fails the pool path, and the macOS signer (build-sign-notarize-nightly) signs instead. A re-run takes Blacksmith 4 vCPU.

What does not change:

Tests:

Run and passing: test_nightly_signer_jobs, test_nightly_trusted_routing, test_ci_nightly_off_blacksmith, test_nightly_sign_runner, test_ci_linux_single_size, test_runner_routing_table, test_runner_label_policy, test_supplychain_security, test_ci_signing_keychain_is_deleted_before_bundle_code and bash tests/test_ci_self_hosted_guard.sh. Of the 114 test files that mention nightly.yml or the signers, the 3 that fail here (missing bashlex, the scripts import path, no lipo) fail the same way on base.

Performance

metric before after delta harness or run link
queue per serial signer job (4 jobs) ~2-4 min each on blacksmith-4vcpu (picker placed 'unverified') measured on the first nightly after merge TBD runs 41383305465, 41385114223
signer queue added to landing-to-feed time 8-15 min per nightly (30-min target) measured on the first nightly after merge TBD runs 41383305465, 41385114223
Blacksmith minutes/day for signers all four jobs on Blacksmith Linux (4 vCPU tonight, 16 vCPU earlier) re-runs only not measured yet (estimate: Blacksmith only on re-runs) first nightly runs after merge

Agent session: agent session

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Route the four NEXT nightly signer jobs to GitHub-hosted Ubuntu on the first attempt and Blacksmith 4 vCPU on re-runs, replacing the sign picker’s 4/16/32 vCPU route. This reduces runner wait and Blacksmith use for signing jobs that are not CPU-bound.

  • signer-build keeps decide’s early fleet placement before the signer route.
  • If GitHub drops a hosted job before it starts, the existing macOS signer remains the fallback.
  • The sign picker and other jobs that use its route are unchanged.
  • Tests verify the runner expressions match the hosted-runner guard’s allowlist.

Written for commit c783a28. Summary will update on new commits.

View guided diff Turn on auto-fix

…smith 4 vCPU on a re-run

signer-build, signer-app, signer-app-notary and signer-dmg go back to the exact routes the
hosted-runner guard allowlists (signer_route, signer_build_route): attempt 1 on GitHub-hosted
ubuntu-24.04, a re-run on Blacksmith Linux 4 vCPU, never 16 or 32 vCPU; signer-build keeps
decide's early fleet placement first. #30081 had moved them onto the sign picker's Blacksmith
Linux 4/16/32 output, which queued each serial signer 2-4 min on blacksmith-4vcpu tonight.
The other picker-routed jobs and the picker's own route are unchanged.

Leo, 2026-11-01: "a small job that is not CPU-bound, such as signing and its runner pick,
takes GitHub-hosted Ubuntu first."

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-f724bb1ff9295c18fee7
@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review c783a28: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr31831 -f ref=lane/nightly-signers-hosted-first -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); base red: the same failure on the base b5060b4; does not block

Shipped in:

@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 2, 2026
@blacksmith-sh

blacksmith-sh Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Found 8 test failures on Blacksmith runners:

Failures

Test View Logs
/tmp/tmpn81jdso1/summary-fail.json (live):/summary-fail.json (live): View Logs
/tmp/tmpw3r_7pa9/summary-missing.json (live):/summary-missing.json (live): View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows
View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/
[tabs] debug.tab_switch timed 0 of 40 switches
View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames View Logs
test_every_compile_job_names_its_warm_sources (main.ThisTree.test_every_compile_job
_names_its_warm_sources)/
test_every_compile_job_names_its_warm_sources (main.ThisTree.test_every_compile_job
_names_its_warm_sources)
View Logs
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl
ights_validate)/
test_the_repository_highlights_validate (main.DigestTests.test_the_repository_highl
ights_validate)
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6.1-sol xhigh, router-review): SHIP at c783a28

Focused routing tests pass: 92 signer, trusted-routing and off-Blacksmith tests, and 88 routing and admission tests.

Follow-ups (cmux#31420):

  1. [P2][follow-up] signer-build's early-fleet branch (nightly.yml:4716) can never fire. place-early-mac's JOBS lists only pick-nightly-build-runner and its FLEET_LINUX is empty, so signer-build always takes hosted Ubuntu on attempt 1. Either wire it into a placement that emits a vetted Linux route, or drop the branch and the claim.

This branch was previously deployed

1 inactive deployment
ci-routing-agent — c783a28c Deployed Nov 2, 2026 by teamleaderleo via wait for same-key tree publisher #99326
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant