Skip to content

ci(nightly): the build job's tail uploads send 32 artifact blocks at a time, not 5 - #32343

Merged
teamleaderleo merged 1 commit into
feat-cmux-nextfrom
lane/nightly-build-upload-tail
Nov 2, 2026
Merged

teamleaderleo merged 1 commit into
feat-cmux-nextfrom
lane/nightly-build-upload-tail

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

The NEXT nightly's build-nightly-app ends with two serial uploads: the unsigned app (413 MB) and the dSYM handoff (221 MB). prepare-nightly-variant-primary and the signers wait for that job to end. actions/upload-artifact sends 8 MB blocks 5 at a time by default. Across five recent runs these two uploads ran at only 2.1-6.4 MB/s, with a block finishing every 3-17 s.

This PR sets ACTIONS_ARTIFACT_UPLOAD_CONCURRENCY: "32" (one YAML anchor) on:

  • Upload unsigned nightly app
  • its retry
  • Keep the nightly-next dSYMs for publication

ci-macos.yml's product upload already uses the same setting (#14375). The action caps the value at 16 per CPU.

Nothing else changes: artifacts, names, paths, compression levels (app 0, dSYMs 6), retention (1 and 21 days), overwrite, step order, timeouts and the retry are all the same. No step was added or renamed, so .github/nightly-step-coverage.json needs no change. No secret, permission, environment or concurrency-group change. nightly.yml grows to 480,200 bytes, still under the 490,000 budget.

Why not move the dSYMs off the job

The dSYMs exist only on the build Mac's disk. A separate job would have to download an artifact this job uploads first, so it saves nothing.

Running the dSYM upload in the background would mean handing the job's Actions runtime token to a run step. That changes how a secret is handled, so it is out of scope here.

The app upload already runs first, at compression-level: 0, and nothing slow comes before it. That leaves the upload's block concurrency as the safe cut.

Performance

Metric: from the start of Upload unsigned nightly app to the start of signer-app (the build job's tail, plus prepare and the signer pick).

run metric before projected after delta uploads before (app + dSYM)
41444184160 349 s 317 s -32 s 97 + 34 s
41439348303 311 s 263 s -48 s 92 + 55 s
41437704107 600 s 410 s -190 s 181 + 108 s
41432827903 622 s 435 s -187 s 188 + 98 s
41430382728 463 s 258 s -205 s 211 + 93 s
median 463 s 317 s -187 s 286 s

How "after" is projected: the best throughput these builders reached at 5 blocks was 6.4 MB/s (the dSYM upload in 41444184160), so the uplink carries at least that. Each upload is taken at min(measured, size / 6.4 MB/s), which is about 65 s for the app and 35 s for the dSYMs. Everything else in the window is kept as measured.

This is a projection, and it is meant to be conservative. More blocks in flight can go faster than 6.4 MB/s if the uplink has room, and no slower than today if it does not. The first nightly after landing measures it from the steps' own timings.

Blacksmith minutes per day: unchanged. These steps run on the owned build Mac, and no job moved.

Risk

Low. This only changes how many 8 MB blocks the action holds in flight (up to about 256 MB of buffers on the build Mac); the uploaded bytes are the same. If 32 connections stalled, the existing 10-minute timeout and one retry still bound the app upload, and the dSYM step stays continue-on-error. A revert is one commit.

Tests

  • New test_tail_uploads_send_blocks_in_parallel in tests/test_nightly_stop_if_publish_doomed.py: fails 3/3 on base, passes at head.
  • Pass: test_nightly_stop_if_publish_doomed, test_nightly_background_steps, test_ci_upload_artifact_retention, test_nightly_step_coverage, test_ci_fork_runner_routing, test_nightly_build_state, test_nightly_signer_jobs, test_nightly_delivery_concurrency (its functions called directly), test_nightly_workflow_size, test_nightly_unsigned_resume, test_nightly_helpers_beside_release_build, test_nightly_trusted_routing, test_nightly_build_runner.
  • Pass: bash tests/test_nightly_universal_build.sh, bash tests/test_ci_self_hosted_guard.sh.
  • actionlint 1.7.12-manaflow.1 with SHELLCHECK_OPTS="-S warning" reports nothing on nightly.yml, the same as base.

Agent session: agent session
🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Speeds up the nightly build job's final artifact uploads by raising the artifact block concurrency from 5 to 32, matching ci-macos.yml's product upload.

The build-nightly-app job's unsigned app (413 MB) and dSYM (221 MB) uploads kept the signers waiting; at the default 5 concurrent 8 MB blocks they ran at 2.1-6.4 MB/s. Setting ACTIONS_ARTIFACT_UPLOAD_CONCURRENCY: "32" on the app upload, its retry, and the dSYM handoff shortens the tail without changing what is uploaded: same bytes, artifact names, compression, retention, and step order. A new test pins the setting on all three steps.

Written for commit 10d74bb. Summary will update on new commits.

View guided diff Turn on auto-fix

…a time, not 5

build-nightly-app ends with the unsigned-app upload (413 MB, 92-211 s) and
the dSYM handoff (221 MB, 34-108 s); the signers and prepare wait on the
job's end. upload-artifact sends 8 MB blocks 5 at a time by default, and
these uploads ran at 2.1-6.4 MB/s, a block every 3-17 s. Raise
ACTIONS_ARTIFACT_UPLOAD_CONCURRENCY to 32 on both uploads and the app
retry, as ci-macos.yml's product upload already does. Same artifacts,
names, compression, retention and step order; a test pins the setting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-f724bb1ff9295c18fee7
@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 10d74bb: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr32343 -f ref=lane/nightly-build-upload-tail -f platform=macos -f backend_mode=local

Shipped in:

@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 2, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (subagent claude-opus-5-5 medium): SHIP at 10d74bb

Checked: the pinned upload-artifact v7.0.1 reads ACTIONS_ARTIFACT_UPLOAD_CONCURRENCY (getConcurrency, default 5). 32 is in range, and ci-macos.yml uses the same value. The anchor gives exactly that env on all three steps, and no keys are lost. Nothing else changes. These tests pass: test_nightly_stop_if_publish_doomed (10), step_coverage, fork_runner_routing, universal_build, upload_artifact_retention.

Follow-ups:

  • Record the app and dSYM upload times at 32 from the next nightly in the Performance table.

@teamleaderleo
teamleaderleo merged commit 74f5ee9 into feat-cmux-next Nov 2, 2026
150 of 160 checks passed
@teamleaderleo
teamleaderleo deleted the lane/nightly-build-upload-tail branch November 2, 2026 13:38
teamleaderleo added a commit that referenced this pull request Nov 2, 2026
…me, not 5 (#32490)

Signing waits on prepare-nightly-variant-primary's end, and its last step
uploads the 346 MB prepared app. On a dedicated nightly Mac that upload took
65-156 s (median 85 s over 21 runs, 41455121554-41488515134) at
upload-artifact's default 5 blocks at a time; the same Macs move the 413 MB
unsigned app in a median 53 s at 32 since #32343. Reuse that step's
ACTIONS_ARTIFACT_UPLOAD_CONCURRENCY anchor on the prepared-app upload, which
the fallback leg (prepare-nightly-variant) shares by alias. Same artifact,
name, compression and retention; a test pins it on both legs.


Agent-Session: https://cmux.com/deeplink/agent?id=aid-f724bb1ff9295c18fee7

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

This branch was previously deployed

1 inactive deployment
ci-routing-agent — 10d74bbd Deployed Nov 2, 2026 by teamleaderleo via wait for same-key tree publisher #101469
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant