Skip to content

fix(ios): INTERNAL builder trims its caches before disk refuses archive jobs - #24251

Merged
azooz2003-bit merged 2 commits into
feat-cmux-nextfrom
ios-builder-disk-prune
Oct 22, 2026
Merged

azooz2003-bit merged 2 commits into
feat-cmux-nextfrom
ios-builder-disk-prune

Conversation

@azooz2003-bit

@azooz2003-bit azooz2003-bit commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

cmux23, the cmux INTERNAL iOS TestFlight builder, fell to 39.9 GiB free and its admission hook refused every archive job (run 39731717858, 2026-10-22). The watch loop never cleaned the pre-rename ~/cnx-ios-builds root, kept failed builds' .work (source and DerivedData) for 3 days, and never trimmed xcframework keys or the cargo target.

  • Build folders: also capped to the newest 5 (in addition to the 3-day rule).
  • Below 60 GiB free, and only while no runner job runs (Runner.Worker), the watch loop removes the legacy root, .work, xcframeworks beyond the newest 3 keys, and the cargo target. SwiftPM checkouts stay.
  • admit: when disk is the only reason it would refuse an admitted archive job, it trims once and checks again (the hook runs before the job's steps, and the concurrency group allows one build).
  • Design doc's watch-loop section updated.

The mini runs its own copy of this script; after merge it must be copied per the doc's Operations section (scp scripts/ci/ios_internal_builder.py cmux23:ios-internal-builder/).

Testing

python3 -m unittest tests.test_ios_internal_builder: 35 tests pass, 5 new (newest-N build folders, pressure trim keeps SwiftPM and outputs, tick trims only when low and idle, admit trims and re-checks only for an admitted job).

Changelog

  • Fixed: the cmux INTERNAL iOS builder frees its own caches before low disk refuses builds.

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Fixes the cmux INTERNAL iOS TestFlight builder (cmux23) so it frees its own caches before low disk makes its admission hook refuse archive jobs (cmux23 hit 39.9 GiB free on 2026-10-22 and refused every archive job).

  • Caps build folders to the newest 5 and, below 60 GiB free, trims the legacy ~/cnx-ios-builds root, failed builds' .work, xcframeworks beyond the newest 3 keys, and the cargo target; SwiftPM checkouts stay. Trims run only while no Runner.Worker (from any runner bin), testflight-build.sh, or build-xcframework.sh runs, and victims are renamed into .trash before deletion so a build that starts mid-delete gets a fresh directory.
  • The admission hook trims once and re-checks when disk is the only reason it would refuse an admitted archive job, logging a trim error if one occurs.
  • Updates the design doc's watch-loop section; after merge, copy the updated script to the mini runner per the doc's Operations section (scp scripts/ci/ios_internal_builder.py cmux23:ios-internal-builder/).

Written for commit c9d6eda. Summary will update on new commits.

View guided diff Turn on auto-fix

…ve jobs

The watch loop kept the pre-rename ~/cnx-ios-builds root, every failed
build's .work (source and DerivedData) for 3 days, every xcframework key and
an unbounded cargo target, so cmux23 fell to 39.9 GiB free and the admission
hook refused every archive job (run 39731717858).

- build folders: also capped to the newest 5
- below 60 GiB free and no runner job: remove the legacy root, .work,
  xcframeworks beyond the newest 3, the cargo target (SwiftPM stays)
- admit: when only disk refuses an admitted archive job, trim once and re-check

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review c9d6eda: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr24251 -f ref=ios-builder-disk-prune -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); NEW: red check: Checks job structure (+1 more) is not red on the base 6657e27; do not merge until green; see lane-rules

@github-actions github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 22, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator

Bead cx-8yz2i created for this PR by the cmux coordinator.

@blacksmith-sh

blacksmith-sh Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Found 11 test failures on Blacksmith runners:

Failures

Test View Logs
/tmp/tmpl00t5mb2/summary-missing.json (live):/summary-missing.json (live): View Logs
/tmp/tmpm0yo5nk2/summary-fail.json (live):/summary-fail.json (live): View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows
View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/
[tabs] debug.tab_switch timed 0 of 40 switches
View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames View Logs
convention violations found (ERROR lines above)./
convention violations found (ERROR lines above).
View Logs
nightly cmux-tui resolve/nightly cmux-tui resolve View Logs
test_every_submodule_free_checkout_on_an_owned_runner_resets_submodules (main.Reuse
dWorkspaceSubmodules.test_every_submodule_free_checkout_on_an_owned_runner_resets_submo
dules)/
test_every_submodule_free_checkout_on_an_owned_runner_resets_submodules (main.Reuse
dWorkspaceSubmodules.test_every_submodule_free_checkout_on_an_owned_runner_resets_submo
dules)
View Logs
test_every_submodule_free_checkout_on_an_owned_runner_resets_submodules (main.Reuse
dWorkspaceSubmodules.test_every_submodule_free_checkout_on_an_owned_runner_resets_submo
dules) (job='web-bundles')/
test_every_submodule_free_checkout_on_an_owned_runner_resets_submodules (main.Reuse
dWorkspaceSubmodules.test_every_submodule_free_checkout_on_an_owned_runner_resets_submo
dules) (job='web-bundles')
View Logs

...and 1 more test failures. View all on Blacksmith

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

…ight-only cap

- job_running matches Runner.Worker from any runner bin (self-updates use
  bin.<version>/) and manual testflight-build.sh / build-xcframework.sh runs
- pressure trim renames victims into .trash before deleting, so a build that
  starts mid-delete gets a fresh directory; _prune empties leftover .trash
- the newest-5 cap counts only *-testflight folders
- admit logs a trim error and falls through to the plain refusal

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@azooz2003-bit

Copy link
Copy Markdown
Collaborator Author

Subagent correctness review findings and resolution (fixed in the second commit):

  • High: job_running matched only <runner>/bin/Runner.Worker; after a runner self-update the Worker runs from bin.<version>/, so a running build looked idle. Now matches Runner.Worker anywhere (cmux23 has no other runner) plus manual testflight-build.sh / build-xcframework.sh.
  • Medium: race between the idle check and a slow rmtree. Victims are renamed into .trash (atomic, same volume) first, so a build starting mid-delete gets a fresh directory; _prune empties leftover .trash.
  • Medium/low: the newest-5 cap counted every folder; now only *-testflight. Doc notes the root belongs to the builder on cmux23.
  • Low: an OSError in the admit trim now logs and falls through to the plain refusal instead of a traceback.
  • Low: tests added for each (39 pass). _remove count overstating with ignore_errors is cosmetic; left as is.
    The fast-tier red is the base branch's CmuxNext test compile break (HoleRecordingPage / PageChromiumPage, fixed by open cmux-next tests: the tab-row holes test page answers setSeeThrough, so CmuxNext tests compile again #24227/fix(next tests): CmuxNext test targets compile again (HomeChiefLinkInk selector, HoleRecordingPage setSeeThrough) #24211), unrelated to this Python-only change.

@azooz2003-bit

Copy link
Copy Markdown
Collaborator Author

Merging per the standing directive for this effort. Every red check here is unrelated to this PR, which only touches scripts/ci/ios_internal_builder.py, its tests and the design doc:

@lawrencecchen this only takes effect once scripts/ci/ios_internal_builder.py is copied to cmux23 (~/ios-internal-builder/), and cmux23 is currently refusing every cmux INTERNAL build at 39.8 GiB free (< 40 GiB). Neither Aziz's laptop key nor the fleet key is authorized there (cmuxterm-hq REPAIR row from #2996). Could you or someone with a key run, as cmux on cmux23, while no Runner.Worker is running:

rm -rf ~/cnx-ios-builds ~/cmux-mobile-builds/.work/* ~/cmux-mobile-builds/.cache/cmux-mobile-rust-target
scp scripts/ci/ios_internal_builder.py cmux23:ios-internal-builder/   # from this merge

or install Aziz's key (allowed by the fleet manifest) or apply the hq#1973 ACL so we can do it ourselves.

@azooz2003-bit
azooz2003-bit merged commit 233be56 into feat-cmux-next Oct 22, 2026
@azooz2003-bit
azooz2003-bit deleted the ios-builder-disk-prune branch October 22, 2026 07:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants