Skip to content

ci(nightly): the Mac signer jobs read the signed app and final DMG in parallel ranges - #32498

Merged
teamleaderleo merged 2 commits into
feat-cmux-nextfrom
lane/nightly-ranged-artifact-fetch
Nov 2, 2026
Merged

teamleaderleo merged 2 commits into
feat-cmux-nextfrom
lane/nightly-ranged-artifact-fetch

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

Why

The NEXT nightly's path from landing to feed should fit in 30 minutes and takes about 45. Two of the losses are actions/download-artifact on the nightly's Mac runners. It reads the artifact's blob in one stream with no stall timeout.

  • signer-dmg-image, "Download the signed app" (355 MB): in the last 26 completed nightly-next runs the first try took a p50 of 133 s and a p90 of 300 s. It hit its 5-minute limit in 4 runs. In run 41478567483 both tries ran out (16:43:41 to 16:53:41, with no bytes reported after "Starting download"). The macOS signer covered the build about 11 minutes later, and by then a newer build had published, so publish-nightly skipped it as superseded. That tip never reached the feed. The same artifact downloads in a p50 of 9 s (max 98 s, n=51) in signer-app-notary and signer-dmg on hosted Linux.
  • verify-final-dmg-nightly, "Download final DMG" (257 MB): p50 89 s, p90 274 s, max 446 s. 6 of 26 runs took 200 s or more. In run 41481821056 this step took 446 s of the job's 537 s; it is the 9-minute verify. The other steps were normal: verify 33 s, Apple's checks 24 s.

From a Linux VM, the same 257 MB blob (its signed URL taken through the REST redirect, which points at the same blob host) took 30.0 s in one stream, 4.6 s in 8 ranges and 2.9 s in 16 ranges, with the digest matching every time. The limit is per connection.

What

  • New scripts/ci/fetch_run_artifact.py (stdlib only):
    • Asks the artifact service that download-artifact itself uses (twirp ListArtifacts and GetSignedArtifactURL at ACTIONS_RESULTS_URL) with the job's runtime token. It reads the run and job backend ids from the token's Actions.Results scope. No GITHUB_TOKEN permission is needed, and no job's permissions change.
    • A run step does not see the runtime token, so a pinned actions/github-script step (v9.0.0, the pin used elsewhere in the repo) starts the script and hands it its own environment. Nothing is written to $GITHUB_ENV except the SIGNER_APP_FETCHED flag. The token goes to the artifact service only, never to the blob host.
    • Reads the signed blob URL in 16 parallel byte ranges (at least 16 MiB each). It is the same Azure blob SAS host download-artifact reads from (*.blob.core.windows.net), and that host answers Range requests with 206.
    • If a range sends nothing for 20 s, it asks again from the byte where it stopped, with a fresh signed URL. Each range gets up to 5 tries, and the whole download has a 240 s budget.
    • Checks the zip against the SHA-256 digest the service recorded at upload, then unpacks it into the same directory download-artifact uses, refusing members that would land outside it.
    • On any failure (no runtime token included) it exits non-zero and leaves the directory empty.
  • signer-dmg-image: the ranged step runs first (continue-on-error, 5 min). The existing download-artifact step and its retry run only when the ranged step did not set SIGNER_APP_FETCHED. It is an env flag rather than a step outcome because signer-dmg reuses those two anchored steps, has no ranged step, and so keeps its current route.
  • verify-final-dmg-nightly: the same pattern. "Download final DMG" (download-artifact) runs only when steps.fetch-final-dmg did not succeed.
  • Can the final-DMG download be dropped? No. The final DMG is signed, notarized and stapled on Linux by signer-dmg, and verify has to read those exact bytes on a Mac. The image is already smoked in parallel (launch-smoke-nightly, smoked_in_job), so the transfer is the only part left to cut.

Every gate stays: both notarizations, the static verify, Apple's checks, and the launch smoke that publish needs. The digest checks downstream (APP_SHA256 on signed-app.tar, the tested DMG digest) are unchanged.

Performance

metric before after (expected) delta run link
signer-dmg-image signed-app download, p50 133 s ~25 s ~-108 s 26 runs from 41448288706 to 41485637444
signer-dmg-image signed-app download incl. retry, p90 384 s ~60 s ~-324 s same 26 runs
signer-dmg-image runs that hit the 5 min limit 4 / 26 (1 lost its tip) ~0 (stalled ranges resume) -4 41478567483
verify-final-dmg-nightly final-DMG download, p50 89 s ~20 s ~-69 s same 26 runs
verify-final-dmg-nightly final-DMG download, max 446 s ~60 s ~-386 s 41481821056
same 257 MB blob from a Linux VM: 1 stream / 16 ranges 30.0 s 2.9 s -27.1 s measured locally against artifact 13164292369

The "after" column is an estimate: about 6 times the single-stream rate, from the Linux measurement. The first nightly-next runs after this lands will give the real numbers, which the ranged step prints (MB, seconds, MB/s, range retries). Blacksmith minutes are unchanged: both jobs run on owned Macs.

Tests

  • python3 tests/test_fetch_run_artifact.py (new, added to ci-guards' "Validate the release signer scripts"). Against a fake API and blob host it covers: a ranged read and unpack; a stalled range resuming with a fresh URL; a digest mismatch leaving nothing behind; a range that never finishes failing within the budget; and a zip member that escapes the destination being refused.
  • python3 tests/test_nightly_signer_jobs.py (new wiring test: ranged step first, fallback conditions, permissions, signer-dmg unchanged), tests/test_nightly_step_coverage.py (two new coverage entries), tests/test_nightly_sign_runner.py, tests/test_nightly_next_launch_smoke.py, tests/test_nightly_trusted_routing.py, tests/test_nightly_dedicated_mini.py, tests/test_nightly_next_post_publish.py, tests/test_ci_self_hosted_guard.sh, tests/test_ci_upload_artifact_retention.py: all pass.
  • The pinned actionlint (1.7.12-manaflow.1) passes on nightly.yml and ci-guards.yml.
  • tests/test_supplychain_security.py fails the same way on base (sentry-dsyms-nightly), so it is not this PR's red.

Agent session: agent session

🤖 Generated with Claude Code

… parallel ranges

download-artifact reads an artifact's blob in one stream with no stall
timeout. On the nightly's Macs the 355 MB signed app took a p50 of 133 s
in signer-dmg-image (hosted Linux reads it in 9 s) and ran out its
5-minute limit in 4 of the last 26 nightly-next runs; in run 41478567483
both tries ran out and that tip never published. verify-final-dmg-nightly's
final-DMG download took a p50 of 89 s and up to 446 s (run 41481821056).

scripts/ci/fetch_run_artifact.py reads the blob in 16 parallel ranges,
asks again for a range that sends nothing for 20 s from where it stopped
with a fresh signed URL, checks the zip against its upload digest and
unpacks it like download-artifact. Both jobs run it first and keep
download-artifact as the fallback when it fails. The two jobs gain
actions: read to list this run's artifacts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-f724bb1ff9295c18fee7
@cursor

cursor Bot commented Nov 2, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 8d67fd0: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr32498 -f ref=lane/nightly-ranged-artifact-fetch -f platform=macos -f backend_mode=local

Shipped in:

@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 2, 2026
…ce, with no new permission

Drop the actions: read grants. fetch_run_artifact.py now asks the artifact
service download-artifact itself uses (twirp ListArtifacts and
GetSignedArtifactURL at ACTIONS_RESULTS_URL) with the job's runtime token,
reading the run and job backend ids from the token's Actions.Results
scope. A run step does not see that token, so a pinned
actions/github-script step spawns the script and hands it its own
environment; nothing goes to $GITHUB_ENV except the SIGNER_APP_FETCHED
flag. The signed URL is the same Azure blob SAS URL, which takes Range
requests. download-artifact stays the fallback in both jobs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-f724bb1ff9295c18fee7
@blacksmith-sh

blacksmith-sh Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Found 8 test failures on Blacksmith runners:

Failures

Test View Logs
main.DeveloperIdAppTests.test_download_prints_the_active_profile_only/
test_download_prints_the_active_profile_only
View Logs
stubs: /System/Library/Frameworks/Security.framework/Versions/A/Security has no stub/
Security has no stub
View Logs
stubs: CoreFoundation.tbd: install-name /System/Library/Frameworks/
CoreFoundation.framework/Versions/A/CoreFoundation, missing 1: ['_CFRetain']/
CoreFoundation, missing 1: ['_CFRetain']
View Logs
stubs: CoreFoundation.tbd: install-name /System/Library/Frameworks/
CoreFoundation.framework/Versions/A/CoreFoundation, missing 1: ['_log2']/
CoreFoundation, missing 1: ['_log2']
View Logs
test_nightly_keeps_a_margin_under_the_limit (main.WorkflowFileSizeTest.test_nightly
_keeps_a_margin_under_the_limit)/
test_nightly_keeps_a_margin_under_the_limit (main.WorkflowFileSizeTest.test_nightly
_keeps_a_margin_under_the_limit)
View Logs
test_no_workflow_adds_a_single_size_linux_job (main.WorkflowTests.test_no_workflow_
adds_a_single_size_linux_job)/
test_no_workflow_adds_a_single_size_linux_job (main.WorkflowTests.test_no_workflow_
adds_a_single_size_linux_job)
View Logs
test_no_workflow_falls_back_to_blacksmith_outside_manaflow_ai (main.ForkRunnerRouti
ngTests.test_no_workflow_falls_back_to_blacksmith_outside_manaflow_ai)/
test_no_workflow_falls_back_to_blacksmith_outside_manaflow_ai (main.ForkRunnerRouti
ngTests.test_no_workflow_falls_back_to_blacksmith_outside_manaflow_ai)
View Logs
the short nightly decide gate must use the pooled Linux route with a Blacksmith fallbac
k/
the short nightly decide gate must use the pooled Linux route with a Blacksmith fallbac
k
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (subagent claude-opus-5-5 medium): SHIP at 8d67fd0

The runtime token is sent only to ACTIONS_RESULTS_URL, and nothing is written to $GITHUB_ENV except SIGNER_APP_FETCHED. The signed URL must be https, zip-slip is blocked, artifact names reach the script only through env, download-artifact runs after every failure, and every publish gate is unchanged. 51 tests pass.

Follow-ups:

  • Require a recorded sha256 digest, and fall back when one is missing.
  • Give the twirp calls a no-redirect opener, so Authorization is never sent on a cross-host redirect.
  • Catch HTTPException, ValueError, zlib.error and EOFError in the range threads and in main, so the fallback always starts from an empty folder.
  • Refuse an https→http redirect on the blob fetch.

@teamleaderleo
teamleaderleo merged commit 385345b into feat-cmux-next Nov 2, 2026
152 of 161 checks passed
@teamleaderleo
teamleaderleo deleted the lane/nightly-ranged-artifact-fetch branch November 2, 2026 18:18
teamleaderleo added a commit that referenced this pull request Nov 2, 2026
…ghtly.yml back under its size cap (#32513)

#32490, #32501 and #32498 together took nightly.yml to 493,218 bytes, past the
490,000-byte cap in tests/test_workflow_file_size.py. Move "Build and sign the
release notes" and "Embed channel provisioning profile" into
scripts/ci/nightly-steps/, the same way earlier steps were moved; the second
runs under bash -e like its inline block did. The workflow expanded by
tests/nightly_workflow_text.py is byte-identical to base. nightly.yml is now
485,819 bytes.

Base-red-job: test_workflow_file_size (nightly.yml over its 490,000-byte cap)


Agent-Session: https://cmux.com/deeplink/agent?id=aid-f724bb1ff9295c18fee7

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
teamleaderleo added a commit that referenced this pull request Nov 2, 2026
…ness PRs (#32526)

- Validate pull request macOS pool choice: #32501 moved the light-job
  placement's portable_job_placement.py call into
  scripts/ci/nightly-steps/light-placement.sh, so the step that carries
  CI_RUNNER_STATE no longer names the picker in its run text. The leak
  check now also reads the nightly-steps script a step calls; a step
  whose script runs no picker still fails.
- Validate supply-chain security boundaries: #32455's sentry-dsyms-nightly
  job takes SENTRY_AUTH_TOKEN with no environment. It now has the same
  environment expression build-nightly-app gave the in-job Sentry upload
  it replaced; no environment or its protection changes.
- Validate test determinism gate: #32498's test asserted on elapsed wall
  time. It now asserts the failure names the stalled range, which shows
  the fetch ended on its own bound.

Base-red-job: Validate pull request macOS pool choice
Base-red-job: Validate supply-chain security boundaries
Base-red-job: Validate test determinism gate


Agent-Session: https://cmux.com/deeplink/agent?id=aid-f724bb1ff9295c18fee7

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

This branch was previously deployed

1 inactive deployment
ci-routing-agent — 8d67fd03 Deployed Nov 2, 2026 by teamleaderleo via cmux-tui tree preflight #102215
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant