Skip to content

acpmux: keep session rows stable across short peer reconnects - #27005

Closed
teamleaderleo wants to merge 9 commits into
feat-cmux-nextfrom
lane/hb-peer-flap-grace
Closed

teamleaderleo wants to merge 9 commits into
feat-cmux-nextfrom
lane/hb-peer-flap-grace

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

HOLD: Final rebased-head review and paired soak are pending (#27048 has landed).

A peer restart currently marks every session unreachable, including closed chats, then announces the entire list again. This change gives row updates a 10-second disconnect grace and keeps closed rows closed. Reconnects announce only rows whose contents changed. Forwarded requests still fail immediately when the link is down.

Leo: "I do a hell of a lot of routing to remote machines, right? ... Are we covering the edge cases for when, you know, something isn't going right, if there's recovery, whatever, stuff like that."

A single notice loop owns the grace deadlines. A fresh list or ready notice cancels its peer's deadline; repeated failures do not extend it; generation checks reject deadlines for replaced peers. A dedicated persistent stop signal cancels the loop at daemon shutdown without consuming the main daemon's notification. Logging, purge fencing, and transcript persistence still happen at the real disconnect. No daemon update timing, forwarding refusal, or list_guard changes.

Reconnect still saves the cache because a preceding session notice may have changed a row while its save was throttled. Already-stale and closed rows avoid redundant row writes on subsequent failures.

Validation

  • Baseline 61028a9: 40770951012, job 123107681551, compiled and executed three regressions; all failed on the expected row/event assertions.
  • Fixed core at 54b8597: 40771795001 passed ten tests across grace, cache, purge fencing, and stalled-peer integration.
  • Independent Astra high/fast review found a shutdown-notification race. The dedicated stop signal and its regression fixed it; re-review SHIP covers aed5f94. Head 18f5813 differed only by rustfmt module order and had exact-head SHIP confirmation. Current head 372a60f rebases this change onto perf(acpmux): stream session summaries into list snapshots #27048 without conflicts; fresh independent review and remote paired checks are running.
  • Final-head focused checks and soak 40772511771 stopped before compiling/measuring: fetching the baseline recursively fetched an unrelated submodule with a missing object. The prepared runner now disables recursive submodule fetch and restores Cargo.lock normalization. Final shutdown-regression execution and the one-hour rerun remain outstanding. No local builds or test suites ran.

Performance

Metric Before After Delta Harness
Row watcher events per short flap Not measured Not measured — peer_flap_soak, 30 fake peers × 20 rows
CPU seconds over 1 h Not measured Not measured — same workload, 17-minute flap interval
Sampled maximum RSS over 1 h Not measured Not measured — same workload

The Linux-only ignored harness measures the daemon Hub in a test process with fake notices and a memory store. It excludes SSH, app rendering, and disk-cache I/O. The prepared paired runner builds both versions with the identical harness, then runs isolated processes on the same Linux worker.

Remaining integration and landing block

This draft overlaps #27002 in disconnect cache handling. When integrating, preserve that PR's first/hourly outage-save condition and clearing of saved outage state on Connected.

The broker refused a normal merge of companion #26998 because its credential is limited to publishing. The relay also could not reach the authorized landing owner. No merge override was attempted. This PR remains draft pending the final checks, measurements, and an authorized landing path.

Agent session: agent session

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Bead cx-v6macs created for this PR by the cmux coordinator.

@github-actions github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 28, 2026
@teamleaderleo
teamleaderleo force-pushed the lane/hb-peer-flap-grace branch from 18f5813 to fee7af0 Compare October 28, 2026 22:03
@teamleaderleo
teamleaderleo force-pushed the lane/hb-peer-flap-grace branch from fee7af0 to 372a60f Compare October 28, 2026 22:06
@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

red: cmux-next checks (god files, concurrency, crash safety, l10n); no finished base run on 538ccd0 to compare; do not merge until green; see lane-rules
red: cmux-next crash ratchet; NEW: green on the base 538ccd0; do not merge until green; see lane-rules

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by an agent session · took over from an earlier session · session

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Carried onto current feat-cmux-next as #27809 (this PR's eight peer-flap commits with their authorship, plus one commit keeping #27214's planned-restart notice settling in the moved notice loop). The paired soak and review continue there. Leaving this PR open for its owner.

@blacksmith-sh

blacksmith-sh Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

Found 8 test failures on Blacksmith runners:

Failures

Test View Logs
--from must install a complete prebuilt tree/
--from must install a complete prebuilt tree
View Logs
/tmp/tmpjxzt3nwa/summary-fail.json (live):/summary-fail.json (live): View Logs
/tmp/tmpnwehoany/summary-missing.json (live):/summary-missing.json (live): View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows
View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/
[tabs] debug.tab_switch timed 0 of 40 switches
View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames View Logs
test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt
epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)/
test_every_listed_hook_is_one_glaeda_released_with_the_check (main.SigningHooksInSt
epWithGlaedaTests.test_every_listed_hook_is_one_glaeda_released_with_the_check)
View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Superseded by #27809, which carries its eight peer-flap commits with their authorship. The soak and review continue there (comment at 01:29Z).

This branch had an error being deployed

1 failed and 2 inactive deployments
sentry-dsyms — 2544faff Deployed Oct 30, 2026 by teamleaderleo via upload cmux-tui dSYMs to Sentry #79881
artifacts — 2544faff Deployed Oct 30, 2026 by teamleaderleo via publish to R2 #79881
ci-routing-agent — 2544faff Deployed Oct 30, 2026 by teamleaderleo via cmux-tui tree preflight #79881
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant