fix(worktree-care): persist and bound squash checks by teamleaderleo · Pull Request #27107 · manaflow-ai/cmux

teamleaderleo · 2026-10-28T22:50:58Z

Summary

Persist finished answers across daemon restarts and bound squash detection without changing archive/reclaim policy or which worktrees are cared for.

Requested behavior: “Persist the FINISHED answers” and “A restart then doesn't redo every pair.”

The memo stores worktree path, HEAD, base and answer, bounded to 16,384 entries and 16 MiB. Failed checks and skipped checks remain unknown and are not cached; conflict answers retain existing behavior. Writes occur after the first answer, every 1,024 answers and at pass end. A restart during a partial batch can therefore recompute that batch. The shared store lock prevents concurrent merge-tree checks using that store; contention skips the check. History enumeration stops at 129 commits and skips squash detection above 128. Each merge-tree retains the 20-second deadline and receives a Linux-only 768 MiB address-space ceiling. Scratch objects are removed after each attempt; process/file locks release when the attempt ends.

Validation

Head: c07f3a7e6f72b84383f9044b4f246c44ee4f2259.
40812339966: passed clippy with -D warnings for cmux-git and cmux-worktree-care, compiled and passed all 49 package/lifecycle/regression tests, then explicitly passed the ignored performance harness.
40820436555: final-head cargo clippy --locked -p cmux-worktree-care --all-targets -- -D warnings passed.
Restart regression runs scans in separate processes and checks zero merge-tree launches for unchanged persisted pairs.
History-limit regression checks that the capped revision walk runs and merge-tree does not.
Independent Astra high/fast review: SHIP at c07f3a7e6f72b84383f9044b4f246c44ee4f2259.
Base-red comparison: the CI merge 7e4eba4dc80bd17b404a925666132a15d683a867 (40814964087) and its actual base 6e50cde73c10b018d64682f379c3c0a0e0cb2627 (40815111040) both fail only test_the_committed_package_declares_its_imports in the dependency-list suite: CmuxNextAppTests imports CmuxNextAppCore without declaring it. The checker also emits that same error on both revisions; the base confirmation harness succeeds by asserting the expected checker/test failures. This unrelated Swift dependency error does not hide the passing Rust compile, clippy, or focused tests.
Schedule regression: before the explicit claim unlock, the full parallel library run 40817758045 failed a_second_daemon_waits_while_one_runs_a_pass; after the fixture serial and explicit Unix unlock, three repeated full library runs pass in 40819081983. The same three-run harness passes on base 25220cca8a1be757304543c7096814b272c1ead8 in 40819399705.

Performance

Same 16-worktree repository and hosted run, comparing a fresh process with memory-only memo against a fresh process loading the memo from a completed pass. The harness flushes the final batch just as the daemon does. The fixture resolves native Git via its executable directory, avoiding hosted Go caching wrappers whose virtual-memory reservation exceeds the child cap. GNU time measures only merge-tree child RSS. These are small synthetic-repository measurements, not a reproduction of the loaded-host memory incident.

metric	before	after	delta	harness or run link
Peak merge-tree child RSS	4,864 KiB	0 (no child launched)	-100%	40812339966
Pairs computed with merge-tree after restart	16	0	-16 (-100%)	40812339966
Inventory pass wall time	282.344 ms	206.486 ms	-75.858 ms (-26.87%)	40812339966

Agent session: agent session

Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302

cursor · 2026-10-28T22:51:03Z

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

teamleaderleo · 2026-10-28T22:54:51Z

Bead cx-o65qgw created for this PR by the cmux coordinator.

Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302

cursor · 2026-10-28T23:01:44Z

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

github-actions · 2026-10-28T23:06:18Z

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review c07f3a7: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr27107 -f ref=lane/hb-care-bound -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); NEW: green on the base c660264; do not merge until green; see lane-rules

…ocesses Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302

…hecks Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302

Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302

cursor · 2026-10-29T00:56:02Z

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302

teamleaderleo · 2026-10-29T05:16:08Z

Review (codex gpt-6-astra high fast, independent box review-27107-c07f): SHIP at c07f3a7

SHIP

No findings.

github-actions · 2026-10-29T06:20:17Z

Shipped in:

cmux NEXT nightly 1.0.0-nightly.4082537602501 (6ac0d64eb5)
cmux NEXT nightly 1.0.0-nightly.4082702109901 (f565f4a4ae)

github-actions · 2026-10-29T06:34:33Z

The feat-cmux-next push run https://github.com/manaflow-ai/cmux/actions/runs/40823474936 at c8b6445 failed: cmux-next checks (god files, concurrency, crash safety, l10n).
Those jobs last passed at fb5c0aa. This pull request is one of 3 merged since: #27172, #24212, #27107.
If the failure is in your change, fix forward on feat-cmux-next. Pull requests run only the tiers their paths reach (docs/ci/cmux-next-tiers.md); label a batch PR full-ci to run them all before merging.

* fix(worktree-care): persist and bound squash checks Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302 * fix(worktree-care): silence non-unix memory limit path Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302 * fix(worktree-care): use portable resource limit fields Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302 * fix(worktree-care): flush finished memo once per pass Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302 * fix(worktree-care): avoid resource-limit warning Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302 * fix(worktree-care): bound memo reads and serialize children across processes Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302 * test(worktree-care): measure persisted process restarts and skipped checks Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302 * fix(worktree-care): keep memo out of manifest enumeration Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302 * fix(worktree-care): amortize memo writes and honor custom stores Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302 * fix(worktree-care): adapt memo lock path for PathBuf Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302 * fix(worktree-care): retain memo path for shared lock readers Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302 * test(worktree-care): check memo writer ownership directly Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302 * test(worktree-care): flush final batch in restart measurements Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302 * test(worktree-care): gate GNU time harness to Linux Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302 * test(worktree-care): measure native git instead of runner wrappers Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302 * test(worktree-care): isolate parallel schedule fixtures Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302 * test(worktree-care): trace schedule fixture paths Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302 * test(worktree-care): remove fixture diagnostics Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302 * test(worktree-care): trace busy schedule fixture Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302 * test(worktree-care): inspect schedule lock holder Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302 * fix(worktree-care): release schedule claims explicitly Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302 * fix(worktree-care): preserve stamp error during claim drop Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302

github-actions · 2026-10-29T06:50:43Z

The feat-cmux-next push run https://github.com/manaflow-ai/cmux/actions/runs/40823842282 at 6ac0d64 failed: cmux-next checks (god files, concurrency, crash safety, l10n).
Those jobs last passed at fb5c0aa. This pull request is one of 3 merged since: #27172, #24212, #27107.
If the failure is in your change, fix forward on feat-cmux-next. Pull requests run only the tiers their paths reach (docs/ci/cmux-next-tiers.md); label a batch PR full-ci to run them all before merging.

github-actions · 2026-10-29T06:53:14Z

The feat-cmux-next push run https://github.com/manaflow-ai/cmux/actions/runs/40823925460 at 698b3ef failed: cmux-next checks (god files, concurrency, crash safety, l10n), cmux-next generated files.
Those jobs last passed at fb5c0aa. This pull request is one of 3 merged since: #27172, #24212, #27107.
If the failure is in your change, fix forward on feat-cmux-next. Pull requests run only the tiers their paths reach (docs/ci/cmux-next-tiers.md); label a batch PR full-ci to run them all before merging.

…LIMIT_AS RLIMIT_AS counts address space a child only reserves: a Go git wrapper's runtime (FastActions' gitwrapper on the hosted Linux runners) and Git's read-only pack mappings. Under the 768 MiB cap such a git died at random with almost nothing in use, and worktree care read the squash check as unknown. RLIMIT_DATA counts the writable memory the child allocates (brk and private writable mappings), which is what #27107 meant to bound; 1 GiB of writable memory is still refused. The limit is renamed memory_limit in cmux-git and cmux-worktree-care. macOS is unchanged (no limit there).

…(private test lock, RLIMIT_DATA) (#29729) * test(worktree-care): a squash-merged branch is finished while the host runs a merge-tree care_a_squash_merged_branch_is_finished failed in the full Linux suite at d3cf70b (finished = None after 0.146 s) and passed alone. The scan helper uses the library's in-memory memo, whose merge-tree lock is the host-wide $HOME/.cmux/worktree-archive/.merge-tree.lock. When any other process holds it (another test's merge-tree, the person's cleanup daemon), the check is "busy, unknown" and the branch reads as unfinished. This test now holds that host lock through the scan, which makes the failure deterministic. Red on this commit; the fix follows. * fix(worktree-care): lifecycle test scans take their merge-tree lock in their own folder The care_lifecycle scan helper used inventory::scan, whose in-memory memo takes the host-wide merge-tree lock in $HOME/.cmux/worktree-archive. That lock is nonblocking by design: busy means "unknown, ask again next pass". In the full suite every care_lifecycle test process (and the person's own cleanup daemon) shares it, so a squash check that met another merge-tree returned no answer and care_a_squash_merged_branch_is_finished saw finished = None (feat d3cf70b, job 01ac6d0216103926556037f5). The helper now scans with a per-process FinishedMemo in the test's own scratch folder, so its lock is private, the way pass::run already keeps its lock beside its store. The product behavior does not change. * fix(worktree-care): use the imported Arc in care_lifecycle (no unnecessary qualification) The fleet macOS run of 0a877e3 printed 9 unused_qualifications warnings: the new `use std::sync::{Arc, LazyLock}` made the existing std::sync::Arc paths redundant. Behavior is unchanged. * test(worktree-care): a git that reserves unused address space still gets a squash answer Second cause of the care_a_squash_merged_branch_is_finished base red, measured with diagnostics on the Linux runner (job fec1f9b5): the `git` on PATH there is /usr/local/bin/git, FastActions' gitwrapper (github.com/FastActions/fa/agent/gitwrapper, go1.27.3) in front of /usr/lib/git-core/git. merge-tree runs under a 768 MiB RLIMIT_AS, and the Go runtime reserves address space it never uses; under load it starts more threads and dies at init ("runtime: out of memory: cannot allocate 4194304-byte block (3932160 in use)", exit 2), so the squash check has no answer and the branch reads as unfinished (2 of 80 copies per full suite, jobs 76cc4bf6, 7925207d, d625e10d, fec1f9b5). On a person's Linux machine the same holds for any such git (a Go wrapper, or Git's own read-only pack mappings past 768 MiB in a big repository): the check is unknown on every pass, so a squash-merged worktree is never seen as finished. The new test puts a `git` on PATH whose merge-tree first maps 1 GiB of read-only anonymous memory (reserved, never written), then runs native Git: red under RLIMIT_AS. A second test holds the cap: 1 GiB of writable memory is still refused, and the check stays unknown. * fix(worktree-care): bound merge-tree's memory with RLIMIT_DATA, not RLIMIT_AS RLIMIT_AS counts address space a child only reserves: a Go git wrapper's runtime (FastActions' gitwrapper on the hosted Linux runners) and Git's read-only pack mappings. Under the 768 MiB cap such a git died at random with almost nothing in use, and worktree care read the squash check as unknown. RLIMIT_DATA counts the writable memory the child allocates (brk and private writable mappings), which is what #27107 meant to bound; 1 GiB of writable memory is still refused. The limit is renamed memory_limit in cmux-git and cmux-worktree-care. macOS is unchanged (no limit there).

fix(worktree-care): persist and bound squash checks

f9b99b3

Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302

fix(worktree-care): silence non-unix memory limit path

038cbe8

Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302

github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 28, 2026

teamleaderleo added 2 commits October 28, 2026 22:58

fix(worktree-care): use portable resource limit fields

9dddc86

Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302

fix(worktree-care): flush finished memo once per pass

e2e85b6

Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302

teamleaderleo temporarily deployed to ci-routing-agent October 28, 2026 23:01 — with GitHub Actions Inactive

fix(worktree-care): avoid resource-limit warning

0666154

Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302

teamleaderleo temporarily deployed to ci-routing-agent October 28, 2026 23:06 — with GitHub Actions Inactive

teamleaderleo temporarily deployed to ci-routing-agent October 28, 2026 23:13 — with GitHub Actions Inactive

azooz2003-bit mentioned this pull request Oct 28, 2026

cmux NEXT iOS: native paste of images and files in the composers #25900

Merged

3 tasks done

fix(worktree-care): bound memo reads and serialize children across pr…

285970c

…ocesses Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302

github-actions Bot added risk:high Risk tier (docs/ci/risk-tiers.md) and removed risk:normal Risk tier (docs/ci/risk-tiers.md) labels Oct 28, 2026

test(worktree-care): measure persisted process restarts and skipped c…

d042154

…hecks Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302

teamleaderleo temporarily deployed to ci-routing-agent October 29, 2026 00:23 — with GitHub Actions Inactive

teamleaderleo temporarily deployed to ci-routing-agent October 29, 2026 00:31 — with GitHub Actions Inactive

teamleaderleo added 3 commits October 29, 2026 00:45

fix(worktree-care): keep memo out of manifest enumeration

7689653

Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302

fix(worktree-care): amortize memo writes and honor custom stores

003f511

Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302

fix(worktree-care): adapt memo lock path for PathBuf

f61354e

Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302

teamleaderleo temporarily deployed to ci-routing-agent October 29, 2026 01:11 — with GitHub Actions Inactive

teamleaderleo added 2 commits October 29, 2026 01:16

fix(worktree-care): retain memo path for shared lock readers

16ca27b

Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302

test(worktree-care): check memo writer ownership directly

146b761

Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302

teamleaderleo temporarily deployed to ci-routing-agent October 29, 2026 01:58 — with GitHub Actions Inactive

teamleaderleo temporarily deployed to ci-routing-agent October 29, 2026 01:59 — with GitHub Actions Inactive

teamleaderleo temporarily deployed to artifacts October 29, 2026 04:19 — with GitHub Actions Inactive

test(worktree-care): isolate parallel schedule fixtures

106cdc2

Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302

teamleaderleo temporarily deployed to ci-routing-agent October 29, 2026 04:26 — with GitHub Actions Inactive

teamleaderleo temporarily deployed to ci-routing-agent October 29, 2026 04:28 — with GitHub Actions Inactive

teamleaderleo added 6 commits October 29, 2026 04:28

test(worktree-care): trace schedule fixture paths

a537d2f

Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302

test(worktree-care): remove fixture diagnostics

1c073f1

Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302

test(worktree-care): trace busy schedule fixture

938c320

Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302

test(worktree-care): inspect schedule lock holder

4075ddc

Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302

fix(worktree-care): release schedule claims explicitly

c115a7e

Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302

fix(worktree-care): preserve stamp error during claim drop

c07f3a7

Agent-Session: https://cmux.com/deeplink/agent?id=aid-46b0ecaccb08c94fe302

teamleaderleo temporarily deployed to ci-routing-agent October 29, 2026 05:08 — with GitHub Actions Inactive

teamleaderleo temporarily deployed to ci-routing-agent October 29, 2026 05:09 — with GitHub Actions Inactive

teamleaderleo merged commit 4b4cea9 into feat-cmux-next Oct 29, 2026
192 of 194 checks passed

teamleaderleo deleted the lane/hb-care-bound branch October 29, 2026 05:17

teamleaderleo temporarily deployed to artifacts October 29, 2026 05:21 — with GitHub Actions Inactive

teamleaderleo temporarily deployed to sentry-dsyms October 29, 2026 05:21 — with GitHub Actions Inactive

teamleaderleo temporarily deployed to artifacts October 29, 2026 05:26 — with GitHub Actions Inactive

teamleaderleo temporarily deployed to artifacts October 29, 2026 05:34 — with GitHub Actions Inactive

teamleaderleo temporarily deployed to artifacts October 29, 2026 05:37 — with GitHub Actions Inactive

This was referenced Oct 29, 2026

cmux-next: overlay host panels close with their window #27172

Merged

cmux NEXT: per-PR kill switches (registry, expiry lint, signed Swift reader) #24212

Merged

lawrencecchen mentioned this pull request Oct 31, 2026

fix(worktree-care): squash checks answer under load and in big repos (private test lock, RLIMIT_DATA) #29729

Merged

Navigation Menu

Repository navigation

fix(worktree-care): persist and bound squash checks - #27107

fix(worktree-care): persist and bound squash checks#27107
teamleaderleo merged 22 commits into
feat-cmux-nextfrom
lane/hb-care-bound

teamleaderleo commented Oct 28, 2026 •

edited

Loading

Uh oh!

cursor Bot commented Oct 28, 2026

Uh oh!

teamleaderleo commented Oct 28, 2026

Uh oh!

cursor Bot commented Oct 28, 2026

Uh oh!

github-actions Bot commented Oct 28, 2026 •

edited

Loading

Uh oh!

cursor Bot commented Oct 29, 2026

Uh oh!

teamleaderleo commented Oct 29, 2026

Uh oh!

Uh oh!

github-actions Bot commented Oct 29, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 29, 2026

Uh oh!

github-actions Bot commented Oct 29, 2026

Uh oh!

github-actions Bot commented Oct 29, 2026

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

teamleaderleo commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Performance

Uh oh!

cursor Bot commented Oct 28, 2026

Bugbot needs on-demand usage enabled

Uh oh!

teamleaderleo commented Oct 28, 2026

Uh oh!

cursor Bot commented Oct 28, 2026

Bugbot needs on-demand usage enabled

Uh oh!

github-actions Bot commented Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cursor Bot commented Oct 29, 2026

Bugbot needs on-demand usage enabled

Uh oh!

teamleaderleo commented Oct 29, 2026

Uh oh!

Uh oh!

github-actions Bot commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 29, 2026

Uh oh!

github-actions Bot commented Oct 29, 2026

Uh oh!

github-actions Bot commented Oct 29, 2026

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

teamleaderleo commented Oct 28, 2026 •

edited

Loading

github-actions Bot commented Oct 28, 2026 •

edited

Loading

github-actions Bot commented Oct 29, 2026 •

edited

Loading