Skip to content

acpmux: a working lane whose model went quiet with no tool call running says it has stalled (lane.stalledSince) - #25006

Merged
teamleaderleo merged 6 commits into
feat-cmux-nextfrom
hb-acpmux/lane-stalled
Oct 25, 2026
Merged

teamleaderleo merged 6 commits into
feat-cmux-nextfrom
hb-acpmux/lane-stalled

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

Part 2 of goal 8 (liveness), from the design doc leo-workspace research/2026-10-24-acpmux-tasks-liveness-urgency.md. After #24992 (tasks).

Leo, 2026-10-24: "shouldn't you be able to tell more metadata regarding... when things are completed, when things are not completed? Just like how you have your sub-agents... if things are hanging..."

What it does

A working lane whose model has gone quiet says so: lane.stalledSince (and turn.stalledSince in the session summary) is set when a turn is running, no tool call is in flight, and the model has sent nothing (text, thought, tool call) for stalledTurnMs, default 10 minutes. Setting it to 0 turns it off.

  • Session.last_model_at is set only from model output, by the tap next to the heard counter. updatedAt isn't used because it moves on any record, Codex bookkeeping included.
  • The running turn (TurnActivity) keeps the ids of its tool calls in flight. An id is added when a call starts or an update says it runs, and removed when the call completes or fails; the set ends with the turn. So a 15-minute build inside one tool call is never stalled, as in silent_turn.rs. At most 256 ids are kept; calls past that are counted and still end.
  • A turn whose agent host is adopted after a daemon restart gets its tool calls in flight back from the log (hosts.rs open_work). Before this, it came back with none, so a tool call that ran across the restart read as stalled. It also gets its latest tool and plan step back, which were empty after adoption before. A subagent's logged updates are left out, as on the live path.
  • _acpmux/schema and the README describe stalledSince and stalledTurnMs.
  • acpmux task ls shows the lane as (working, stalled 12m). The sub-session rows (list_sessions, get_session) carry stalledSince.

Departure from the doc: the doc said lane.state gains stalled. The state stays working and gains the stalledSince field instead. The Mac app (AgentActivityLane.of) and the webview (LANES) drop a lane whose state they don't know, so a new state would hide stalled lanes from them.

Timers and observers

None. The check runs on read (session_summary), as the doc says. There's no new timer, task or observer, so there's nothing to stop.

Tests (huge-blue)

  • tests/hub_integration/lanes.rs::a_quiet_turn_with_no_tool_running_stalls_and_a_long_tool_call_does_not: with stalledTurnMs = 300, a turn quiet with no tool call shows stalledSince. A turn whose tool call is still running does not.
  • hub::activity unit tests: a tool call is in flight until it completes or fails; calls past the cap end too (red at 4c61f76, the Codex finding); and the stall rule itself.
  • tests/durable_agents.rs::a_tool_call_running_across_a_restart_keeps_its_turn_from_stalling: a tool call running when the daemon is killed is still in flight once the next daemon adopts the host. Red at 7671f78 (the Codex finding), green now. a_subagents_tool_call_across_a_restart_is_not_its_parent_turns (fake mode child-gate:): a subagent's running tool call is not its parent turn's after a restart. Red at 55d4811, the Codex finding. durable_agents 41/41.
  • The full hub_integration run had 10 failures under load on huge-blue (transfer, transfer_guard, silent_turn, wait/turn-marker tests); each passes when run on its own.
  • hub_integration lanes 6/6, hub::activity 8/8. cargo fmt and clippy are clean. The protocol spec was blessed (protocol_ir_is_current) and the webview types regenerated (acpmux-protocol:codegen).

Capture

No capture: no UI changes. The new field is in the protocol and the CLI only; the app and webview ignore it until a later PR draws it.

Review

Codex xhigh via acpmux (session review-24939 on huge-blue): SHIP at d2d8084, after five rounds. Its findings, each fixed with a test: an open-call flag past the 256 cap never cleared; an adopted turn after a restart came back with no tool calls in flight; _acpmux/schema didn't list the field; the rebuild counted a subagent's tool calls as the parent turn's; and it then trusted an update's own _meta.acpmux, which an agent can write.

Overlap

pr-overlap lists five open PRs that touch the same files (#24150 lineage, #21944 sidebar rows, #24021, #24802, #24155). They share Session/make_session/session_summary and the protocol spec; none of them detects stalls.

Agent session: big-red

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

teamleaderleo and others added 6 commits October 24, 2026 16:33
Count calls past the cap instead of a flag that never cleared, so a turn
with 257+ tool calls can still stall once they all end (Codex review).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a16c8c-54ec-71b2-a0fa-125d1404dd72&host=big-red
…ADME say stalledSince

The adopting daemon rebuilt the running turn with nothing in flight, so a
tool call that ran across a restart read as stalled. open_work now notes
the open turn's logged tool_call/plan updates (TurnActivity::note), which
also brings back the turn's activity and plan step. _acpmux/schema and the
README describe stalledSince and stalledTurnMs (Codex review).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a16c8c-54ec-71b2-a0fa-125d1404dd72&host=big-red
The rebuild after a restart fed a subagent's logged updates into its
parent turn, so a child's running tool kept the parent from stalling and
took its activity. Skip records with _meta.acpmux.subagent, as the live
path does (Codex review).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a16c8c-54ec-71b2-a0fa-125d1404dd72&host=big-red
… subagents

open_work rebuilds the subagent tree from the log (SubagentTree::annotate,
as the tap does live) instead of trusting an update's _meta.acpmux, which
the agent can write (Codex review).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a16c8c-54ec-71b2-a0fa-125d1404dd72&host=big-red
@teamleaderleo
teamleaderleo temporarily deployed to cmux-tui-merge-tree October 25, 2026 00:46 — with GitHub Actions Inactive
@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

cmux status (one comment, edited in place; each line is updated by the job that owns it)

Review d2d8084: HQ feed card · gallery before/after runs · checks

App of this head (app.zip, kept 1 day)

gh workflow run reload-build.yml -R manaflow-ai/cmux --ref feat-cmux-next -f tag=pr25006 -f ref=hb-acpmux/lane-stalled -f platform=macos -f backend_mode=local

Gallery before/after for d2d8084: 16 states rendered, 8 images differ, 1826 more touched states not rendered. Images are in the run's artifact, not on this public PR.

gh run download 40148009507 -R manaflow-ai/cmux -n gallery-advisory-25006-d2d8084a7a7ff0697a19da22af9030a16361c178 (then read gallery-review.json)

States
State Theme Status Diff Picked by
agent-pane.chat-width/standard Apple System Colors changed 23.4% imports types.ts (2 hops)
agent-pane.chat-width/standard Apple System Colors Light changed 1.53% imports types.ts (2 hops)
agent-pane.chat-width/wide Apple System Colors changed 23.4% imports types.ts (2 hops)
agent-pane.chat-width/wide Apple System Colors Light changed 1.53% imports types.ts (2 hops)
agent-pane.chat-width/wide-blocks Apple System Colors changed 23.4% imports types.ts (2 hops)
agent-pane.chat-width/wide-blocks Apple System Colors Light changed 1.53% imports types.ts (2 hops)
agent-pane.composer-floats/scrolled-to-end Apple System Colors Light unchanged 0% imports types.ts (2 hops)
agent-pane.composer-floats/scrolled-to-end Apple System Colors unchanged 0% imports types.ts (2 hops)
agent-pane.composer-floats/composer-grown Apple System Colors Light unchanged 0% imports types.ts (2 hops)
agent-pane.composer-floats/composer-grown Apple System Colors unchanged 0% imports types.ts (2 hops)
agent-pane.docked-chat/new-chat Apple System Colors Light unchanged 0% imports types.ts (2 hops)
agent-pane.docked-chat/new-chat Apple System Colors unchanged 0% imports types.ts (2 hops)
agent-pane.docked-chat/after-turn Apple System Colors Light unchanged 0% imports types.ts (2 hops)
agent-pane.docked-chat/after-turn Apple System Colors unchanged 0% imports types.ts (2 hops)
agent-pane.docked-chat/working Apple System Colors Light unchanged 0% imports types.ts (2 hops)
agent-pane.docked-chat/working Apple System Colors unchanged 0% imports types.ts (2 hops)
agent-pane.docked-chat/typing-in-dock Apple System Colors Light unchanged 0% imports types.ts (2 hops)
agent-pane.docked-chat/typing-in-dock Apple System Colors unchanged 0% imports types.ts (2 hops)
agent-pane.docked-chat/long-title Apple System Colors Light unchanged 0% imports types.ts (2 hops)
agent-pane.docked-chat/long-title Apple System Colors unchanged 0% imports types.ts (2 hops)
agent-pane.gutter-select/from-left-margin Apple System Colors Light unchanged 0% imports types.ts (2 hops)
agent-pane.gutter-select/from-left-margin Apple System Colors unchanged 0% imports types.ts (2 hops)
agent-pane.gutter-select/from-right-margin Apple System Colors Light unchanged 0% imports types.ts (2 hops)
agent-pane.gutter-select/from-right-margin Apple System Colors unchanged 0% imports types.ts (2 hops)
agent-pane.inspector/chat-menu Apple System Colors Light unchanged 0% imports types.ts (2 hops)
agent-pane.inspector/chat-menu Apple System Colors unchanged 0% imports types.ts (2 hops)
agent-pane.inspector/wire-log Apple System Colors Light changed 0.0198% imports types.ts (2 hops)
agent-pane.inspector/wire-log Apple System Colors changed 0.01% imports types.ts (2 hops)
agent-pane.orchestration-lab/local-queue Apple System Colors Light unchanged 0% imports types.ts (2 hops)
agent-pane.orchestration-lab/local-queue Apple System Colors unchanged 0% imports types.ts (2 hops)
agent-pane.orchestration-lab/build-box Apple System Colors Light unchanged 0% imports types.ts (2 hops)
agent-pane.orchestration-lab/build-box Apple System Colors unchanged 0% imports types.ts (2 hops)

Pick other states with a Gallery: entry.id, entry.id/variant line in the PR body.

red: cmux-next checks (god files, concurrency, crash safety, l10n); no finished base run on e869761 to compare; do not merge until green; see lane-rules

Tour of d2d8084a7a (run): no media. The tour job failed before the tour ran (build or setup); the run log has the output.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Owned by hb-acpmux on big-red · session

@github-actions github-actions Bot added the risk:normal Risk tier (docs/ci/risk-tiers.md) label Oct 25, 2026
@blacksmith-sh

blacksmith-sh Bot commented Oct 25, 2026 •

Copy link
Copy Markdown

Found 14 test failures on Blacksmith runners:

Failures

Test View Logs
/tmp/tmp8ack6j7l/summary-missing.json (live):/summary-missing.json (live): View Logs
/tmp/tmpkqvik1ze/summary-fail.json (live):/summary-fail.json (live): View Logs
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows/
[new_workspaces] debug.new_workspace timed 0 of 50 new workspace rows
View Logs
[new_workspaces] the window draws no frames/[new_workspaces] the window draws no frames View Logs
[tabs] debug.tab_switch timed 0 of 40 switches/
[tabs] debug.tab_switch timed 0 of 40 switches
View Logs
[tabs] the window draws no frames/[tabs] the window draws no frames View Logs
convention violations found (ERROR lines above)./
convention violations found (ERROR lines above).
View Logs
test_a_branch_that_merges_feat_cmux_next_in_is_judged_on_its_own_changes (main.PreP
ushGateTests.test_a_branch_that_merges_feat_cmux_next_in_is_judged_on_its_own_changes)/
test_a_branch_that_merges_feat_cmux_next_in_is_judged_on_its_own_changes (main.PreP
ushGateTests.test_a_branch_that_merges_feat_cmux_next_in_is_judged_on_its_own_changes)
View Logs
test_a_branch_that_merges_feat_cmux_next_in_is_judged_on_its_own_changes (main.PreP
ushGateTests.test_a_branch_that_merges_feat_cmux_next_in_is_judged_on_its_own_changes)/
test_a_branch_that_merges_feat_cmux_next_in_is_judged_on_its_own_changes (main.PreP
ushGateTests.test_a_branch_that_merges_feat_cmux_next_in_is_judged_on_its_own_changes)
View Logs
test_a_merge_cannot_use_an_upstream_removal_to_pay_for_the_branch_s_own_debt (main.
PrePushGateTests.test_a_merge_cannot_use_an_upstream_removal_to_pay_for_the_branch_s_ow
n_debt)/
test_a_merge_cannot_use_an_upstream_removal_to_pay_for_the_branch_s_own_debt (main.
PrePushGateTests.test_a_merge_cannot_use_an_upstream_removal_to_pay_for_the_branch_s_ow
n_debt)
View Logs

...and 4 more test failures. View all on Blacksmith: run 40148009522, run 40148009625

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@teamleaderleo

Copy link
Copy Markdown
Collaborator Author

Bead cx-40pc6 created for this PR by the cmux coordinator.

@teamleaderleo
teamleaderleo merged commit c72f0c9 into feat-cmux-next Oct 25, 2026
158 of 170 checks passed
@teamleaderleo
teamleaderleo deleted the hb-acpmux/lane-stalled branch October 25, 2026 00:59
@github-actions

Copy link
Copy Markdown
Contributor

The feat-cmux-next push run https://github.com/manaflow-ai/cmux/actions/runs/40148827822 at c72f0c9 failed: cmux-next generated files.
Those jobs last passed at c6d2865. This pull request is one of 1 merged since: #25006.
If the failure is in your change, fix forward on feat-cmux-next. Pull requests run only the tiers their paths reach (docs/ci/cmux-next-tiers.md); label a batch PR full-ci to run them all before merging.

@github-actions

Copy link
Copy Markdown
Contributor

The feat-cmux-next push run https://github.com/manaflow-ai/cmux/actions/runs/40149382418 at a4c4d46 failed: cmux-next generated files.
Those jobs last passed at c6d2865. This pull request is one of 4 merged since: #25006, #24997, #25011, #24938.
If the failure is in your change, fix forward on feat-cmux-next. Pull requests run only the tiers their paths reach (docs/ci/cmux-next-tiers.md); label a batch PR full-ci to run them all before merging.

@github-actions

Copy link
Copy Markdown
Contributor

The feat-cmux-next push run https://github.com/manaflow-ai/cmux/actions/runs/40149412279 at 19a5504 failed: cmux-next generated files.
Those jobs last passed at c6d2865. This pull request is one of 5 merged since: #25006, #24997, #25011, #24938, #24942.
If the failure is in your change, fix forward on feat-cmux-next. Pull requests run only the tiers their paths reach (docs/ci/cmux-next-tiers.md); label a batch PR full-ci to run them all before merging.

@github-actions

Copy link
Copy Markdown
Contributor

The feat-cmux-next push run https://github.com/manaflow-ai/cmux/actions/runs/40149315230 at f3895fa failed: cmux-next generated files.
Those jobs last passed at 8125a40. This pull request is one of 3 merged since: #25006, #24997, #25011.
If the failure is in your change, fix forward on feat-cmux-next. Pull requests run only the tiers their paths reach (docs/ci/cmux-next-tiers.md); label a batch PR full-ci to run them all before merging.

@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

teamleaderleo added a commit that referenced this pull request Oct 25, 2026
#25006 added a display-only stalledTurnMs (lane.stalledSince, default 10 min).
The turn-ending reconciler's key is now stalledTurnEndMs (default 6 h, 0 off)
so the two never share a setting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?harness=claude&session=01a16c8c-54ec-71b2-a0fa-125d1404dd72&host=big-red

This branch was previously deployed

3 inactive deployments
artifacts — d2d8084a Deployed Oct 25, 2026 by teamleaderleo via add the Windows daemon to the tree #51338
sentry-dsyms — d2d8084a Deployed Oct 25, 2026 by teamleaderleo via upload cmux-tui dSYMs to Sentry #51338
cmux-tui-merge-tree — d2d8084a Deployed Oct 25, 2026 by teamleaderleo via publish the merge tree #5624
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk:normal Risk tier (docs/ci/risk-tiers.md)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant