Skip to content

ci: full-tree blob:none checkouts leave out cmux-next-ios/verify media - #34235

Merged
teamleaderleo merged 2 commits into
feat-cmux-nextfrom
ci-checkout-exclude-ios-verify
Nov 3, 2026
Merged

teamleaderleo merged 2 commits into
feat-cmux-nextfrom
ci-checkout-exclude-ios-verify

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Since about 16:00Z on 2026-11-03, full-tree filter: blob:none checkouts went from about 50 s to 3-5 min. The cause is the git cache in front of the Linux runners. Each such checkout downloads about 570 MiB of blobs at the tip, and about 470 MiB of that is mp4/png evidence under cmux-next-ios/verify/. No CI job reads that folder.

Change

  • One declared table, .github/checkout-lean.json. It holds the sparse patterns (/*, !/cmux-next-ios/verify/), the jobs proven not to read that path (lean, each with what it does read) and the jobs that keep the full tree (full, each with the reason).
    • A composite action can't hold the patterns: actions/checkout runs before any repository file exists, so a local action isn't there yet.
    • Instead, tests/test_ci_checkout_lean.py (linux-guard lane) keeps every lean step's sparse-checkout equal to the table, in non-cone mode.
    • The test also fails on a new full-tree blob:none checkout that is in neither list, and on stale or unreasoned entries.
    • python3 tests/test_ci_checkout_lean.py --write adds the patterns to a newly listed job.
  • 17 checkout steps in 13 workflows go lean:
    • backend deploy-staging and deploy-development
    • ci-compile-attribution
    • cmux-next-batch
    • car-guards plan and mac
    • generated-catch-up
    • pr-autofix action-areas, action-exports and repair
    • cmux-next-web web-bisect
    • cmux-tui-testbox-warmup
    • cmux-tui go-pane
    • gallery-advisory
    • gallery-health
    • nightly-next-train
    • r2-binaries-prune
  • web/public stays included.
  • Two jobs keep the full tree:
    • car-guards linux: it runs tests/test_ci_fleet_rule_lints.py, whose allowlist names scripts under cmux-next-ios/verify/. With those files absent, its git grep skips them and it reports 20 stale entries. I ran it in a sparse tree and it failed.
    • ci-guard-attribution: its guards run in scratch worktrees that would inherit the patterns. It checks out main, which has no cmux-next-ios/, so leaving the folder out would save nothing there.

Proof per job

I traced each job's run: steps, its local actions and the scripts it calls, transitively. The table's lean entries record what each job reads.

Points that matter in a sparse tree:

  • git ls-files still lists the skip-worktree paths. Every ls-files pathspec in these jobs is scoped (webviews, scripts/cmux-next, cmux-tui/bindings, *Cargo.lock, **/Package.swift), and none of them matches a file in verify.
  • git worktree add and checkout --force keep the patterns.
  • git add -A -- . and git restore -- . skip skip-worktree paths.
  • autofix repair's git read-tree clears the bits, but its patch diff and add are limited to paths it computed earlier.

Performance

Before: the checkout step's own time from the jobs list, for the 6 newest pull_request runs at 20:25-20:40Z, all on Blacksmith 4 vCPU. After: this PR's own run (pull_request jobs use the merge ref's workflows).

metric before (p50, n) after (p50, n) delta harness or run link
pr-autofix action export areas checkout 247 s (6) 214 s, 233 s (2) about -10%: this job's cost is its all-branch fetch, not blobs; a follow-up PR fetches only base and head GraphQL CheckRun.steps times; after: this PR's runs 41716913976 and 41722171269
pr-autofix autofix repairs checkout 290 s (6) 133 s (1) -157 s (-54%) same

Blacksmith minutes per day: these jobs spend about 3-4 fewer minutes each in checkout. Measured so far: about 2.5 fewer Blacksmith minutes per autofix repairs run. The daily total scales with PR pushes, and NOW's new checkout-time line tracks it.

Observed / Not run

  • Ran:
    • python3 tests/test_ci_checkout_lean.py: OK.
    • The 75 tests that name these workflows, sparse-checkout or blob:none: all pass except tests/test_check_ghostty_zig_workflows.py, which fails the same way on base.
    • actionlint on the 13 workflows: the same 2 findings as on base.
    • In a sparse tree I also ran test_ci_no_shallow_dev_fetch, test_swift_testing_require_not_nested, test_app_store_rules, test_ci_warm_sources, test_send_path_caps and others: they pass. The first two scan fewer files.
  • Not run: the lean jobs other than those this PR's own CI triggers.

Moving cmux-next-ios/verify/ out of the repo is left to the iOS owners.

Changelog

none

Agent session: agent session

🤖 Generated with Claude Code

Each full-tree blob:none checkout downloaded about 570 MiB of blobs, about
470 MiB of it mp4/png evidence under cmux-next-ios/verify/ that no job reads.
.github/checkout-lean.json declares the patterns and the 17 jobs proven not to
read that path (two keep the full tree, with their reason);
tests/test_ci_checkout_lean.py keeps every listed step equal to it and fails
on a new undeclared full-tree blob:none checkout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-8707001042ea634e8771
@teamleaderleo teamleaderleo added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Nov 3, 2026
@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 6aa3c55: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr34235 -f ref=ci-checkout-exclude-ios-verify -f platform=macos -f backend_mode=local

Shipped in:

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by an agent session · session

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (subagent claude-opus-5-5 medium): SHIP at c574a29
Visual verdict: unchanged by this diff

Checked: all 17 inserted steps are on actions/checkout v6.0.2 with consistent indentation; python3 tests/test_ci_checkout_lean.py passes (4 tests), and it is registered in tests/test-execution.toml (linux-guard). I traced the riskiest jobs myself: autofix repair (base-autofix.sh and its generators, ls-files scoped to cmux-tui/bindings and the cloud clients, the patch built only from the paths computed before read-tree), next_batch.py (merge, add -A, worktree add), generated-catch-up (merge and push), car-guards plan/mac (focused Swift package tests; e2e_warm_derived_data walks only files that exist) and gallery-advisory (webviews only). None of them reads cmux-next-ios/verify/. I also reproduced the actions/checkout non-cone setup locally: git add -A stages no deletions for skip-worktree paths, a merge that changes verify/ files commits them intact, git worktree add inherits the patterns, and read-tree clears the bits as the PR body says.

Follow-ups: none blocking. Any future lean job that runs tests/test_ci_fleet_rule_lints.py, or another tree-wide git grep allowlist check, has to stay in full; the table's reasons already say so. This includes testbox snapshots used for such tests if warmup ever hydrates feat-cmux-next.

@blacksmith-sh

blacksmith-sh Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Found 3 test failures on Blacksmith runners:

Failures

Test View Logs
test_every_compile_job_names_its_warm_sources (main.ThisTree.test_every_compile_job
_names_its_warm_sources)/
test_every_compile_job_names_its_warm_sources (main.ThisTree.test_every_compile_job
_names_its_warm_sources)
View Logs
test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt
epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)/
test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt
epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)
View Logs
test_the_attestation_reserves_budget_and_its_retry_shares_the_ledger (main.Workflow
Tests.test_the_attestation_reserves_budget_and_its_retry_shares_the_ledger)/
test_the_attestation_reserves_budget_and_its_retry_shares_the_ledger (main.Workflow
Tests.test_the_attestation_reserves_budget_and_its_retry_shares_the_ledger)
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@github-actions github-actions Bot added risk:high Risk tier (docs/ci/risk-tiers.md) and removed risk:normal Risk tier (docs/ci/risk-tiers.md) labels Nov 3, 2026

@teamleaderleo teamleaderleo left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review (codex gpt-6-astra high, next-review, since=20261103T210538Z): BLOCK at c574a29

BLOCK at c574a29

Visual verdict: no images; Leo tries the build

  1. [P1] [blocking] tests/test-execution.toml:2404: Registering linux-guard does not schedule the test. No workflow invokes test_ci_checkout_lean.py, so the existing registry validator reports “linux-guard lane is not run by any workflow,” and the workflow-wiring guard also rejects it. Add an invocation in ci-guards.yml so existing CI passes and the new invariant is enforced.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex, router-review): SHIP at c574a29

Reviewer: codex gpt-6-astra high, fast tier.

Follow-ups: 1. Each lands as an issue in this repo that links this receipt; the lane that opened the PR fixes it forward.

SHIP at c574a29

Visual verdict: unchanged by this diff

  1. [P2][follow-up] .github/checkout-lean.json:10-12 classifies compile attribution as safe for sparse checkout, but main_compile_attribution.py:290-291 runs git show, which renders full patches and lazily fetches missing blobs. A historical commit touching cmux-next-ios/verify/ can therefore download the excluded media during analysis; constrain the diff read or keep this job full-tree.

Follow-ups:

  • [P2][follow-up] Prevent git show in compile attribution from fetching excluded verify/ blobs.

@teamleaderleo teamleaderleo removed the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 3, 2026
@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 3, 2026

@cmux-bot cmux-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cmux-bot approves c574a29: an independent review passed this exact head (router-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (subagent claude-opus-5-5 medium): SHIP at 6aa3c55
Visual verdict: unchanged by this diff

Checked the new commit: it runs tests/test_ci_checkout_lean.py in ci-guards.yml's guard step beside test_ci_warm_sources.py, matching its linux-guard registry entry, and adds .github/checkout-lean.json to CI_ONLY. Only workflows and that test read the file, so CI_ONLY is the right tier. At this head, test_ci_workflow_guards_are_wired.py, validate_test_execution_registry.py, test_cmux_next_route.py, test_cmux_next_route_replay.py and test_ci_checkout_lean.py all exit 0. I also skimmed the full 18-file diff stat.

Follow-ups: none

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex, router-review): SHIP at 6aa3c55

Reviewer: codex gpt-6.1-sol xhigh, fast tier.

SHIP at 6aa3c55
Visual verdict: no images; Leo tries the build

Checked all 17 checkout sites, guard/routing wiring, and sparse Git behavior. All 219 targeted tests passed; no actionable correctness findings.

Follow-ups: none

@teamleaderleo teamleaderleo removed the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 3, 2026
@github-actions github-actions Bot added the risk:high Risk tier (docs/ci/risk-tiers.md) label Nov 3, 2026
cmux-bot[bot]
cmux-bot Bot previously approved these changes Nov 3, 2026

@cmux-bot cmux-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cmux-bot approves 6aa3c55: an independent review passed this exact head (router-review receipt by teamleaderleo). A push needs a new review; a later BLOCK at this head dismisses this approval.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review (codex gpt-6.1-sol high, sweep, via the review router; full diff from merge base fabc4a1 to this head): BLOCK at 6aa3c55

[P1] [blocking] tests/test_ci_checkout_lean.py:77-84 - the new guard rejects the existing full-tree blob:none checkout in test-ios.yml :: detect-ios-changes, which is absent from both manifest lists and therefore fails CI - add that job to checkout-lean.json (with the declared sparse patterns or a full-tree justification).

@cmux-bot
cmux-bot Bot dismissed their stale review November 3, 2026 21:52

A newer verdict blocks 6aa3c55 (a wait as new as the SHIP or newer: comment by teamleaderleo: Review (codex gpt-6.1-sol high, sweep, via the review rout).

@teamleaderleo
teamleaderleo merged commit cf0fdde into feat-cmux-next Nov 3, 2026
240 of 250 checks passed
@teamleaderleo
teamleaderleo deleted the ci-checkout-exclude-ios-verify branch November 3, 2026 23:15

This branch was previously deployed

1 inactive deployment
ci-routing-agent — 6aa3c551 Deployed Nov 3, 2026 by teamleaderleo via cmux-tui tree preflight #107709
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:high Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant