Skip to content

fix(acpmux): surface outbound queue gaps (#21000) - #24811

Merged
teamleaderleo merged 7 commits into
feat-cmux-nextfrom
nx-21000
Oct 24, 2026
Merged

teamleaderleo merged 7 commits into
feat-cmux-nextfrom
nx-21000

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #21000

ACPMUX silently drops outbound messages when a slow client fills its bounded output queue.

The bounded outbound queue now reserves capacity for an RPC response and a recovery marker. When notification delivery is rejected by queue pressure, ACPMUX emits _acpmux/lagged with reason: output_queue_full and a cumulative dropped count so clients can replay from their cursor. Inventory-watch notifications follow the same reserve policy.

Changelog

  • ACPMUX reports outbound queue gaps and preserves response/control capacity for reconnect recovery.

Verification

  • python3 scripts/cmux-next/prepush-gate.py — passed on 4849c970882c9dd7b2cefc4b9e359152e7541a95.
  • cargo fmt --manifest-path cmux-tui/Cargo.toml --all -- --check — passed.
  • Hosted focused verification run 40068104184 is bound to the exact head; Linux test is still running. Other jobs have independent runner/base failures (MSRV, macOS setup, cross-build, clippy/panic ratchet) and are recorded as base/infrastructure findings.

Agent session: huge-blue


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.


Summary by cubic

Fixes outbound queue backpressure in ACPMUX so slow clients get a replay signal instead of silently dropped notifications, while always reserving capacity for RPC responses.

  • Reserves queue slots for responses and the _acpmux/lagged recovery marker so notifications can't crowd out replies.
  • Queues _acpmux/lagged with reason: output_queue_full and a cumulative dropped count when notifications are dropped; oversized notifications report reason: message_too_large.
  • Serializes capacity checks and enqueues across request, inventory-watch, and inventory-disable paths so gap delivery and the disable reply stay ordered; when the queue is full, the marker waits for the next free slot so even a quiet reader eventually receives it.
  • Closes pending gap waiters when the connection tears down, so no task leaks after disconnect.

Written for commit 3c72e64. Summary will update on new commits.

View guided diff Turn on auto-fix

Fixes #21000

An output queue that fills must leave a replay signal while preserving response capacity.

Agent-Session: https://cmux.com/deeplink/agent?harness=codex&session=01a16c6c-0725-73b2-8d5e-1e50d0cd8545&host=huge-blue
@teamleaderleo
teamleaderleo temporarily deployed to cmux-tui-merge-tree October 24, 2026 06:04 — with GitHub Actions Inactive
@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review 3c72e64: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr24811 -f ref=nx-21000 -f platform=macos -f backend_mode=local

red: cmux-next checks (god files, concurrency, crash safety, l10n); no finished base run on eac20cf to compare; do not merge until green; see lane-rules

@github-actions github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 24, 2026
@blacksmith-sh

blacksmith-sh Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Found 12 test failures on Blacksmith runners:

Failures

Test View Logs
/tmp/tmphxdelwmr/summary-missing.json (live):/summary-missing.json (live): View Logs
/tmp/tmpnytm78rc/summary-fail.json (live):/summary-fail.json (live): View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows
View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/
[tabs] debug.tab_switch timed 0 of 40 switches
View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames View Logs
convention violations found (ERROR lines above)./
convention violations found (ERROR lines above).
View Logs
test_base_passes (main.NoticesPairTests.test_base_passes)/
test_base_passes (main.NoticesPairTests.test_base_passes)
View Logs
test_bot_attempt_three_drops_stale_owned_pool_placement (main.PathRoutingStructure.
test_bot_attempt_three_drops_stale_owned_pool_placement)/
test_bot_attempt_three_drops_stale_owned_pool_placement (main.PathRoutingStructure.
test_bot_attempt_three_drops_stale_owned_pool_placement)
View Logs
test_notices_for_another_ghostty_next_commit_fail (main.NoticesPairTests.test_notic
es_for_another_ghostty_next_commit_fail)/
test_notices_for_another_ghostty_next_commit_fail (main.NoticesPairTests.test_notic
es_for_another_ghostty_next_commit_fail)
View Logs

...and 2 more test failures. View all on Blacksmith: run 40085833211, run 40085833162

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Bead cx-g1dnw created for this PR by the cmux coordinator.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Review note: Conn::send uses a bounded Sender::capacity() check before try_send; concurrent fan-out/request producers can race at the reserve boundary. The focused regression exercises the slow-reader path with one producer. I am preserving this exact head while the ACPMUX review broker is unavailable; no speculative redesign is included.

@cursor

cursor Bot commented Oct 24, 2026

Copy link
Copy Markdown

Bugbot needs on-demand usage enabled

Bugbot uses usage-based billing for this team and requires on-demand usage to be enabled.

A team admin can enable on-demand usage in the Cursor dashboard.

@teamleaderleo
teamleaderleo temporarily deployed to cmux-tui-merge-tree October 24, 2026 08:21 — with GitHub Actions Inactive
@teamleaderleo
teamleaderleo temporarily deployed to cmux-tui-merge-tree October 24, 2026 08:27 — with GitHub Actions Inactive
@teamleaderleo
teamleaderleo temporarily deployed to cmux-tui-merge-tree October 24, 2026 10:11 — with GitHub Actions Inactive
@teamleaderleo
teamleaderleo temporarily deployed to cmux-tui-merge-tree October 24, 2026 10:25 — with GitHub Actions Inactive
@teamleaderleo
teamleaderleo merged commit 0b7ca80 into feat-cmux-next Oct 24, 2026
137 of 153 checks passed
@teamleaderleo
teamleaderleo deleted the nx-21000 branch October 24, 2026 10:40
@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

This branch was previously deployed

2 inactive (1 outdated) deployments
cmux-tui-merge-tree — 3c72e649 Deployed Oct 24, 2026 by teamleaderleo via publish the merge tree #5536
ci-routing-agent — 18f39686 Deployed Oct 24, 2026 by github-actions[bot] via cmux-next macOS placement #33482
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant