Skip to content

acpmux: machine join and consent locks unlock on drop (flock flakes) - #27435

Merged
teamleaderleo merged 1 commit into
feat-cmux-nextfrom
fix/acpmux-consent-lock-unlock
Oct 29, 2026
Merged

teamleaderleo merged 1 commit into
feat-cmux-nextfrom
fix/acpmux-consent-lock-unlock

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Flake fix for reds seen on base: cli::machine_consent::tests::tags_share_enrollment_and_live_grants_and_revocation failed at machine_consent_tests.rs:18 with "consent update in progress" in 7 of 60 full cargo test -p acpmux --lib runs at base 13531e3. cli::cloud_join::tests::concurrent_join_is_refused_before_a_second_redeem failed once in those runs at cloud_join.rs:557. Both come from #27216. Fixing base reds; never blocked.

The cause

Both locks are a nonblocking flock on a file that is taken, dropped by closing its descriptor, then taken again. flock belongs to the open file description, and a process spawned at that moment shares the description from its fork until its exec. Other lib tests spawn processes all the time, and so does the daemon (agents, tools, login shells). Closing our descriptor therefore left the lock held for that moment, and the next take failed as "consent update in progress" or "another machine join is in progress". #27407 fixed the same race in a test of host_approvals.

The daemon hits it too. record_auth_result takes the join lock on each authenticated cloud response, and a consent change takes the write lock, both while the daemon spawns. A change from the Settings switch could fail with "in progress" when nothing else was changing consent.

The fix

  • join_lock and the consent write lock now return HeldLock, which unlocks explicitly on drop. The upload record (attach/upload.rs) and the Codex auth broker already do this. The unlock acts on the open file description, so it releases the lock whatever copies remain.
  • Each test now holds a copy of the lock's descriptor across the drop, as a spawned child does. The race is now taken on every run. The assertions are unchanged: a second take still fails while the lock is held and succeeds after the drop. Consent changes from both tags still apply.

Evidence (Linux, cmux-ci test, 32 vCPU)

run tree result
3 x 20 full cargo test -p acpmux --lib base 13531e3 tags_share_… failed 2, 4 and 1 of 20; concurrent_join_… failed 1 of 20 (1, 2, 3)
10x a 2000-step grant-then-revoke loop, 8 threads spawning beside it (instrumented, not committed) base 13531e3 524-606 of 2000 steps failed with "consent update in progress", in every run (run)
5x the two tests with a held copy, production unchanged base 13531e3 + this PR's tests both failed 5/5 (run)
10x the same loop and both test files, with the spawning threads this head 0 of 2000 steps failed, 10/10 (run)
30x full cargo test -p acpmux --lib this head machine_consent and cloud_join passed 30/30 (run)
cargo clippy -p acpmux --all-targets -- -D warnings, cargo fmt --check this head clean (same run)

The other reds in those runs fail the same way at base, and each has its own PR:

Risk: low. A drop-time unlock on two short-lived locks. HeldLock keeps the file open for exactly as long as the File did.

Agent session: agent session

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Fixes intermittent flock failures in acpmux where machine join and consent locks appeared held after being dropped.

  • Closing the lock's descriptor left the lock held whenever a spawned process (agent, tool, login shell) shared the open file description from its fork until its exec, so the next nonblocking take failed with "consent update in progress" or "another machine join is in progress".
  • Locks now unlock explicitly on drop via a new HeldLock type, releasing the lock for the open file description regardless of copies.
  • Tests hold a copy of the lock's descriptor across the drop to reproduce the race on every run.

Written for commit 36954f0. Summary will update on new commits.

View guided diff Turn on auto-fix

`cli::machine_consent::tests::tags_share_enrollment_and_live_grants_and_revocation`
(about 1 in 5 full lib runs) and
`cli::cloud_join::tests::concurrent_join_is_refused_before_a_second_redeem`
took a nonblocking flock, dropped it and took it again. A process spawned
meanwhile shares the lock's open file description from its fork until its
exec, so closing our descriptor left the lock held, and the next take failed
with "consent update in progress" / "another machine join is in progress".

The daemon hits the same race: `record_auth_result` takes the join lock and
consent changes take the write lock while the daemon spawns agents and tools.

Both locks now return `HeldLock`, which unlocks explicitly on drop (as the
upload record and the Codex auth broker already do). Each test now holds a
copy of the lock's descriptor across the drop, the way a spawned child does,
so the race is taken on every run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-1288bbb2b5d2f5c6b66a
@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 36954f0: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr27435 -f ref=fix/acpmux-consent-lock-unlock -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); base red: the same failure on the base 5889445; does not block

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by an agent session · session

@blacksmith-sh

blacksmith-sh Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Found 8 test failures on Blacksmith runners:

Failures

Test View Logs
/tmp/tmp3rd9j_lp/summary-fail.json (live):/summary-fail.json (live): View Logs
/tmp/tmpvx4dfepv/summary-missing.json (live):/summary-missing.json (live): View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows
View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/
[tabs] debug.tab_switch timed 0 of 40 switches
View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames View Logs
test_live_raw_v12_layout_and_history_contracts_are_exact (test_ir.IrTests.test_live_raw
_v12_layout_and_history_contracts_are_exact) (command='new-pane-right')/
test_live_raw_v12_layout_and_history_contracts_are_exact (test_ir.IrTests.test_live_raw
_v12_layout_and_history_contracts_are_exact) (command='new-pane-right')
View Logs
test_live_raw_v12_layout_and_history_contracts_are_exact (test_ir.IrTests.test_live_raw
_v12_layout_and_history_contracts_are_exact) (command='new-pane-right')/
test_live_raw_v12_layout_and_history_contracts_are_exact (test_ir.IrTests.test_live_raw
_v12_layout_and_history_contracts_are_exact) (command='new-pane-right')
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@github-actions github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 29, 2026
@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Bead cx-tmh605 created for this PR by the cmux coordinator.

@teamleaderleo teamleaderleo left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review (gpt-6-astra, high, fast; independent Codex reviewer): SHIP at 36954f0

No correctness blockers:

  • The diagnosis and fix are correct. On Unix, File::unlock() is flock(LOCK_UN), which releases the lock held by the shared open file description even when copies from fork or dup survive.
  • try_clone() keeps that description open across the guard's drop, so the regression is now deterministic.
  • HeldLock's visibility allows its construction from machine_consent. File::unlock() exists on Windows, and acquiring the lock on non-Unix still returns an error.
  • Callers keep their guards for the original critical sections, and none needs a bare File.
  • No new clippy issue and no production harm.

Reds at this head, all base reds that fail the same way on feat-cmux-next, none this PR's:

  • cmux-tui SDKs / protocol contract and cmux-tui spec inventory / inventory: test_ir test_live_raw_v12_layout_and_history_contracts_are_exact, command new-pane-right.
  • cmux-next checks: No new SwiftUI.

@teamleaderleo
teamleaderleo merged commit 7f7ddbc into feat-cmux-next Oct 29, 2026
161 of 170 checks passed
@teamleaderleo
teamleaderleo deleted the fix/acpmux-consent-lock-unlock branch October 29, 2026 14:29
@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

This branch was previously deployed

3 inactive deployments
artifacts — 36954f04 Deployed Oct 29, 2026 by teamleaderleo via add the Windows daemon to the tree #77171
sentry-dsyms — 36954f04 Deployed Oct 29, 2026 by teamleaderleo via upload cmux-tui dSYMs to Sentry #77171
ci-routing-agent — 36954f04 Deployed Oct 29, 2026 by teamleaderleo via cmux-tui tree preflight #77171
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant