[teamleaderleo]

"centralized cache store... always non-brittle, fast, auto-healing" (Leo)

What

manaflow-ai/cmux#25147 (merged ebbefce2) builds the cmux NEXT DEV Release compile on each glaeda mini from one tree, ~/Library/Caches/cmux-next-ci/dev-daily-release/host/{src,Release,lock}, so Xcode's compilation-cache keys, which include absolute paths, match whichever runner instance builds. A host flock on host/lock guards it. When the lock is busy, the build falls back to the old per-instance state dev-daily-release/<16-hex key>/, which is now only that fallback.

glaeda-disk now handles that layout from one declarative rule in disk-policy.json:

• cmux_next_ci_host_roots (default ["dev-daily-release/host"], [] turns it off) and cmux_next_ci_host_idle_hours (default 48).
• The host root goes last. It is listed whole and is never a state root, so the 40 GiB size cap never takes it. It gets its own final apply pass, after every other item and after idle runners' whole checkouts. Inside apply() it also sorts below the floor slots. It goes only:
  • once it has been idle 48 h (in an idle sweep or a pressure run), or
  • under the runner floor hold, while free space is still below floor_gib itself after everything else.
• Never while its lock is held. The survey and apply probe flock on host/lock. The removal takes LOCK_EX|LOCK_NB on it and, while holding it, removes every child except lock itself (each renamed aside, then deleted). The lock file keeps its inode, so a build that had already opened it still excludes every other build. While glaeda-disk holds the lock, a build that starts gets cmux's normal busy-lock fallback (it builds per instance). A build that starts afterwards sets up a fresh tree. When the root is taken for the floor alone, free space is read again under the lock, and nothing goes if the floor was met in the meantime. A host root that is a link or a mount point is refused. A process that has the tree open or names it also keeps it. An interrupted removal's aside tree is finished by the interrupted-deletions family under the same lock veto.
• Per-instance fallback states go first. These are dev-daily-release/<hex> states, a kind that now has a host root. They sort first inside the warm band (cap or floor) and inside the normal band. When the cache is over the cap, such a state idle for cmux_next_ci_unowned_idle_hours (6 h) goes whole, inputs included, the same way an unowned state does. Every veto still holds, including its runner's Runner.Worker.
• docs/CMUX_NEXT_CI_RETENTION.md documents the rule. No change to cmux.

Review

An independent Codex review (gpt-6-astra, xhigh) ran over several rounds, and every finding is fixed:

• the lock inode was replaced, which could let two builds hold two inodes;
• an owned fallback state the cap listed whole was misread as a floor slot and skipped;
• host asides were not debited from the cap budget;
• a mounted host root was not refused;
• bytes were double-counted across main's two apply passes;
• free space was not re-read under the lock at the floor boundary.

The final head has VERDICT: SHIP.

Evidence (measured 2026-10-26)

• The host cache works. On cmux14 a warm build had 424 of 502 compile jobs hit the cache and took 548 s of xcodebuild, against 794 s cold. The host tree there is 14 GiB: src 3.0G, Build 4.0G, CompilationCache.noindex 2.3G, SourcePackages 4.1G, ModuleCache 1.1G. The idle per-instance fallback dev-daily-release/3558a20294057360 beside it is 11 GiB.
• On cmux13s-mac-mini, dev-daily-release was 16 GiB (host) at 01:53Z and is 0 B at 08:05Z. Pressure runs there use target 170 GiB and floor 30 GiB, and blocked shows library-caches 43.8 / runner-checkout-build 36.5. Attribution caveat: no glaeda-disk receipt on cmux13s (~/Projects/recovery/disk-reclaim/receipts.jsonl) or in its size snapshot names dev-daily-release/host. The deployed glaeda-disk (d9bc5c62…, current main) cannot list it either, because its state-root shapes require a hex key. The directory's mtime (2026-10-25 20:06 PDT) matches a moment when free space jumped about 12 GiB during a job-completed glaeda-disk pass (pid 84397) that wrote no run line. So the wipe is likely, but not proven, glaeda-disk's. One other candidate: cmux's scripts/cmux-next/update-e2e.py cleanup removes every path containing cmux-next that was created under ~/Library/Caches up to depth 3 during its run, and cmux-next-ci/dev-daily-release/host is depth 3. What this PR changes either way: before it, glaeda-disk never bounded the host tree, not even below the hard floor, and it counted the host tree against the 40 GiB cap while evicting only other states. The cap would therefore take every per-runner state's output first, and an abandoned host tree stayed forever.

Tests

scripts/test-glaeda-disk.py:

• test_the_per_host_cmux_next_root_goes_last_and_only_under_the_floor:
  • Over the cap without a floor hold, the host root stays recent.
  • At 35 GiB free (under the hold's 50 GiB stop, above the 30 GiB floor), everything else goes in order: the fallback state's Build and then the fallback state whole, unowned states, the other Builds, the other slots. The host tree stays.
  • At 10 GiB free the host tree is also emptied, last. Its lock file stays and no aside is left.
  • An idle fallback state is a stale root over the cap. An enrolled runner's other state is not.
• test_the_per_host_cmux_next_root_is_kept_while_its_lock_is_held_and_goes_once_idle:
  • A held flock keeps the tree in both the survey and the removal.
  • In an idle sweep the tree goes at 49 h and stays at 47 h.
  • An aside tree is vetoed while its lock is held.
  • [] turns the rule off.
• test_main_takes_the_per_host_cmux_next_root_after_idle_runners_checkouts: main() applies the host root in its own pass after the others.
• test_the_host_lock_keeps_its_inode_so_a_build_that_opened_it_still_excludes_others
• test_over_the_cap_an_idle_owned_fallback_state_goes_whole_without_a_floor_hold
• test_a_finished_host_aside_counts_against_the_cap, which also covers the shared tally, the re-read under the lock and the mount refusal.
• Locally, the cmux-next, slot, main and interrupted subset is 279 tests. The only failures are the same three that fail on main on this host (an xattr clone test, a timeout-summary test, a read-error test).

Risk

Medium. The change is CI-host disk policy only.

• Failure mode 1: the host tree is deleted while a DEV build uses it. This is guarded by the flock probe, by the exclusive lock held across the removal, and by the open-file and command vetoes. A build that loses the race falls back per instance; it does not fail.
• Failure mode 2: a mini sits under its floor because the host tree is kept. This cannot happen: under the floor after everything else, the host tree goes.
• Failure mode 3: an owned fallback state goes whole over the cap. Its runner's Runner.Worker vetoes that, and the next fallback build re-fetches SourcePackages.

Signals

• Success: on minis with DEV compiles, dev-daily-release/host survives pressure runs that are not under the floor. Receipts show dev-daily-release/<hex> (fallback) items reclaimed before other cmux-next items, and host reclaimed only with the reason "idle 48h+" or below the floor. Consecutive DEV compiles on one host stay warm (xcodebuild well under the 794 s cold time).
• Failure: a receipt that reclaims dev-daily-release/host with free above floor_gib and idle under 48 h, or a DEV compile step log that shows host tree setup failed right after a glaeda-disk pass.

Not fixed here: eviction_busy skips

Of the last 400 pressure run lines, most are skipped: eviction_busy: cmux14 390/400, cmux13s 385/400. The owners are mostly other pressure passes. On cmux14 the last 400 skip receipts show glaeda-disk --pressure --apply --top 0 (the runner hook's job-completed detached pass) 342 times and --floor-gib 40 (the gate's floor pass) 36 times. That is 87 distinct owner pids with at most 15 skips each, and owner age p50 23 s / p90 107 s. So these are not overlapping launchd starts. Every job completion on 11-12 runner instances starts a detached pass, and the one that holds evict.lock makes the rest skip, as designed. The real cost is that the pressure target (170 GiB) is never reached, free space sits at 50-75 GiB, every pass runs its full 3-9 minutes, and a pass is almost always running. A fix is a policy question about whether to coalesce job-completed passes, skip without a run line, or lower the target. It is a different code path from this change (the runner hook's disk_pressure_detached and the target), so it is left for its own PR.

Agent session: agent session

🤖 Generated with Claude Code

[teamleaderleo]

Owned by an agent session · session