[lawrencecchen]

Summary

Fixes the base red cmux-worktree-care::care_lifecycle care_a_squash_merged_branch_is_finished. It failed in the full Linux suite at feat d3cf70bb (job 01ac6d0216103926556037f5, hq run 41232603829: care_lifecycle.rs:261 panicked with None after 0.146 s) and passed when it ran alone. The flake had two causes. The second is a product defect.

Cause 1 (test isolation). The test's scan helper called inventory::scan, whose in-memory memo takes the host-wide merge-tree lock $HOME/.cmux/worktree-archive/.merge-tree.lock. That lock does not wait, by design (#27107): when it is busy, the check is "unknown, ask again next pass". All care_lifecycle test processes start in the same 100 ms and use that lock. The helper now scans with one FinishedMemo per test process in its own scratch folder, so its lock is private, as pass::run keeps its lock beside its store.

Cause 2 (product). merge-tree ran under a 768 MiB RLIMIT_AS (Linux only, from #27107). RLIMIT_AS counts address space that a process only reserves. Measured:

• On the hosted Linux runner, the git on PATH is /usr/local/bin/git, FastActions' gitwrapper (go version -m: go1.27.3, github.com/FastActions/fa/agent/gitwrapper), in front of /usr/lib/git-core/git. Its Go runtime reserves address space and, under load, died at start with runtime: out of memory: cannot allocate 4194304-byte block (3932160 in use), exit 2 (diagnostic jobs 76cc4bf6, 7925207d, d625e10d, fec1f9b5: 2 of 80 copies in each full suite). git-lfs is configured in the runner's /etc/gitconfig, but merge-tree --write-tree runs no filters, and the process that threw is the wrapper.
• With plain C git in a big repository: in this cmux checkout (one pack of 7.1 GB), /usr/lib/git-core/git merge-tree --write-tree --no-messages HEAD~50 HEAD under ulimit -v 786432 exits 128 with packfile ... cannot be mapped ... Cannot allocate memory. Under ulimit -d 786432 it exits 0. Git maps pack windows of up to 1 GiB read-only.

Product impact. On Linux, when the git on PATH reserves address space that it does not use (a wrapper written in Go, or C git in a repository with a pack bigger than about 768 MiB, such as cmux), the squash check gets no answer. A squash-merged worktree then never shows as finished, and worktree care never cleans it up. The check is not remembered, so each pass tries again, but with a big pack it fails each time. A Linux or Cloud machine with a cmux checkout gets this. Branches merged as ancestors are not affected (merge-base --is-ancestor runs without the limit).

Fix. cmux-git sets RLIMIT_DATA instead of RLIMIT_AS (the parameter is renamed memory_limit). RLIMIT_DATA counts the writable memory that the child allocates (brk and private writable mappings). It does not count unused reservations or read-only pack mappings. Thus it still bounds merge-tree's memory, as #27107 intended.

• macOS: the limit is set only on Linux (#[cfg(target_os = "linux")]). macOS is unchanged and has no limit.
• The memory bound still holds: the new guard test a_merge_tree_that_writes_past_the_memory_cap_has_no_answer maps 1 GiB of writable memory in merge-tree. The limit refuses it, and the check stays unknown.
• Callers of the limited spawn: in cmux-git/src/run.rs, only run_git_scratch_objects_within_limited passes a limit to run. Its only caller is cmux-worktree-care git::read_scratch_objects, whose only caller is finish() for merge-tree. All other run calls (run_git_within, the windowed and records readers, run_git_scratch_objects_within) pass None, so they do not change.

@teamleaderleo: this changes the limit that you added in #27107 (4b4cea908645) from RLIMIT_AS to RLIMIT_DATA, for the reasons above. The 768 MiB value, the deadline and the locks do not change. cmux-worktree-care and cmux-git have no row in OWNERS.md.

Testing

All runs use scripts/ci/cmux-tui-rust-check.sh test <filter> on the fleet. I read each log and checked that each one compiled cmux-git and cmux-worktree-care at its ref. The gate reruns a failed test alone and then passes it as FLAKY with exit 0. Thus I count a pass only from the first full run, never from the exit code.

Focused commands: --arg test --arg care_a_squash_merged_branch_is_finished (cause 1) and --arg test --arg a_git_that_reserves_unused_address_space (cause 2).

job	commit	what	result
0cebe155	red 1 2c53e4ef	squash test, Linux	failed at the squash assertion (None), in the full run and alone
dadb7d6f	red 2 a18ea61a	a_git_that_reserves_unused_address_space, Linux	failed: "merge-tree's 1 GiB address-space reservation was refused"
bf8e7ddd	base 3d509f63 + 20 copies (scratch)	care_, Linux	12 of 21 copies failed at :261 with None. The gate rerun passed them as FLAKY, exit 0
3f7652d5	head 93261448	care_, Linux	41 of 41 passed (this filter does not select the two new a_* tests; their evidence is the full suites below)
7d075945	head 93261448	care_, macOS 26 (aws-m4pro-4)	41 of 41 passed
9dcfd3d7	head 93261448	clippy -D warnings, Linux	pass
94e96255, 59079680, b40a7a31, fe07c0d7, 282a225f (infra rerun of 4d43fd11)	head + 20 copies (scratch 11e4364b)	full Linux suite	5 of 5 suites: all 105 copies (the squash test and 20 copies each), a_git_that_reserves_unused_address_space_still_gets_an_answer and a_merge_tree_that_writes_past_the_memory_cap_has_no_answer passed on the first run, 0 FLAKY

Before the product fix (head 0a877e3f + 20 copies), 5 full suites showed 2 failures in 105 copies (181afddd, ca4719fc). This found cause 2.

Every full suite also shows the same 22 acpmux failures (cli:: SIGABRT, plus daemon_memory and ephemeral_run in some runs) and once cmux-tui::cli apps_store. They are base reds: the same 22 failures appear on 0a877e3f, which changed only a test file in cmux-worktree-care.

Review receipt: #29729 (comment) (SHIP at 932614489c3f). Its nits are not fixed here: the per-process care-lifecycle-finished-* temp folder is not removed, and the shims' PATH/HOME restore is skipped if a scan panics (the file's existing pattern).

Observed / Not run

Observed: the runs in the table, and the ulimit -v / ulimit -d measurement of native git in this checkout (above).

Not run / known limits:

• Job 4d43fd11 (load run 1): NOT RUN (stale warm target, hq#5737): its warm cargo target did not recompile cmux-worktree-care and ran a stale binary (13787 tests, base line numbers). I reran it once (282a225f) and reported the fault to the lead.
• With plain cargo test (one process, many threads), tests can still collide on the in-process MERGE_TREE mutex in inventory.rs, which also does not wait. CI does not see this, because nextest runs each test in its own process. This PR does not change that.
• The regression helper host_merge_tree_lock uses the real $HOME/.cmux/worktree-archive/.merge-tree.lock by design: it holds the lock that the product uses. If a person's cleanup daemon runs a pass at the same moment, that pass has no answer for one check and asks again in its next pass.
• The new Linux tests need python3 on the host (they fail, not skip, without it). The RLIMIT_DATA behavior on a real Cloud machine is not tested separately. The measurement above used this VM's kernel.

Performance

No perf numbers: the bound value and the code path do not change. Only the kind of limit changes. The perf harness finished_scan_perf_harness was not run.

Changelog

Fixed: On Linux, worktree cleanup now finds squash-merged worktrees in large repositories and with git wrappers on PATH

[github-actions]

cmux status _{(one comment, edited in place; each line is updated by the job that owns it)}

Review 9326144: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr29729 -f ref=fix/base-red-worktree-care-squash -f platform=macos -f backend_mode=local

Shipped in:

• cmux NEXT nightly 1.0.0-nightly.4130052256501 (28aa638015)

[cursor]

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

[lawrencecchen]

Review (claude-review-subagent): SHIP at 9326144

I read the diff 3d509f63bb7f..932614489c3f and the code around it. I did not build or run tests. I found no correctness defect. The nits below do not block.

1. RLIMIT_DATA in place of RLIMIT_AS (cmux-git/src/run.rs:316-339): correct.

• On Linux 4.7 and later, RLIMIT_DATA counts brk plus private writable mappings that are not stack (VM_WRITE & ~VM_SHARED & ~VM_STACK). The kernel checks it on mmap, on brk, and when mprotect makes a mapping writable. Git's heap (malloc through brk or anonymous RW mmap) is counted. Pack windows, the index and the commit-graph are mapped PROT_READ, MAP_PRIVATE, and Go PROT_NONE reservations are also not counted. So the 768 MiB still bounds what merge-tree allocates. When malloc fails, git dies with exit 128. run (run.rs:378) then returns Unavailable and finish gives no answer, so nothing is remembered.
• Only one call gives a limit: run_git_scratch_objects_within_limited (run.rs:166), through git::read_scratch_objects (git.rs:40), from inventory::finish for merge-tree (inventory.rs:669). The other calls of run pass None: run.rs:110, run.rs:123, run.rs:142 through the limited wrapper, and run.rs:210.
• Other platforms do not change. let _ = memory_limit; and the check that an exit code other than 1 is Unavailable are the same code as before, with a new name.
• No stale address_space_limit or RLIMIT_AS text remains outside the new explanatory comments.

2. care_lifecycle helper (care_lifecycle.rs:72-85, 251-266): correct.

• Each process has one FinishedMemo::load in its own scratch folder. Thus lock_dir() is that folder, which is private to the process. Memo keys include the worktree root, and each test's root is unique, so tests do not share answers. With nextest (one process per test), the in-process MERGE_TREE mutex is not shared. The PR states this limit for plain cargo test.
• host_merge_tree_lock makes the red deterministic. File::lock is flock, and flock on a second open file description in the same process conflicts. Thus the base helper (memory memo, lock in Store::default_dir()) always gets try_lock busy, then None. The head helper never touches that lock. The fleet red (0cebe155: it failed alone too) agrees.

3. care_bounds Linux tests (care_bounds.rs:254-330): correct, and red/green is shown.

• With the PROT_READ shim, reached and an answer that contains origin/main are both asserted, so the test cannot pass vacuously. I read red job dadb7d6f at a18ea61: it panics at care_bounds.rs:306 "reservation was refused". At 11e4364b (its parent is the head), full-suite job 282a225f shows both new tests PASS.
• The shim argv is correct: os.execv(sys.argv[2], sys.argv[2:]) runs native git with argv[0] = the git path and then the original arguments. With fileno=-1, Python adds MAP_ANONYMOUS. With the PROT_WRITE shim, Python exits 1 with a traceback on stderr. That gives GitFailure::Exit with non-empty stderr (git.rs:66-67), then Err, then unknown. This agrees with the doc comment.
• PATH and HOME come back before any assert or unwrap that can fail after the scan (care_bounds.rs:289-291). The existing tests use the same pattern.

Nits (they do not block):

• care_lifecycle.rs:79-80: the care-lifecycle-finished-<pid>-<ms> folder (finished.json, finished.lock, .merge-tree.lock) is never removed. Every test process that calls scan leaves one in $TMPDIR (REVIEW.md section 10).
• care_lifecycle.rs:254-266: host_merge_tree_lock creates ~/.cmux/worktree-archive/.merge-tree.lock in the real HOME. It uses umask mode and no O_NOFOLLOW, but the product uses 0600 | O_NOFOLLOW (inventory.rs:543-548). This is outside REVIEW.md section 1 "isolate the test from the machine". The PR discloses it, and the base helper already used this lock. Consider the same 0600/O_NOFOLLOW options.
• care_bounds.rs:296-299: if .expect("listed") or .expect("readable") fails, the panic occurs before remove_dir_all(root) and the temp repository stays. If scan_with_memo panics, PATH and HOME are not restored and ENV_LOCK is poisoned. This is not important with nextest, and existing tests have the same pattern.
• care_bounds.rs:313-330, a_merge_tree_that_writes_past_the_memory_cap_has_no_answer: answer == None together with !reached also passes when merge-tree is never reached (a busy MERGE_TREE or host claim, or Python failing for another reason). It also cannot tell "unknown" from a remembered negative. It does catch the important regression: with no limit, reached becomes true. The "never a remembered answer" claim is true only from reading the code, not from the test.
• care_bounds.rs:274-279: the shim puts paths in single quotes, so a temp path that contains ' breaks it. The existing trace shim has the same limit.
• Kernels older than 4.7 count only brk for RLIMIT_DATA, and glibc then falls back to mmap, so there is no bound. This is for information only, because supported hosts are newer.
• PR evidence table: the care_ filter for head job 3f7652d5 does not select the two new a_* tests. Their head evidence is the full-suite runs at 11e4364b. Please say this in the table.