Agent pane: a turn whose model can't be reached says so, not Thinking (acpmux turn_retrying)

cmux-tui/crates/acpmux/README.md

@@ -449,6 +449,10 @@ responding model produces, such as Claude's stream and thinking or Codex's moder
 notice; the agent's own usage and turn notices do not count) for that long after its
 prompt (a stalled model) logs `turn_silent` {turnId, silentMs} once, so a client can say
 the model has not answered. The model's first sign after it logs `turn_heard` {turnId}.
+A turn whose harness retries a failed model request (Claude Code's `system/api_retry`, Codex's
+stream error with `willRetry`) logs `turn_retrying` {turnId, attempt, maxRetries?, error?,
+errorStatus?, retryDelayMs?, source} for each retry, so a client can say the model cannot be
+reached instead of "Thinking"; the model's next sign logs `turn_heard` {turnId}.
 `stalledTurnMs` (default 600000; 0 turns it off): a running turn with no tool call in
 flight whose model has shown no sign for that long has stalled: `lane.stalledSince` (and
 `turn.stalledSince`) say since when, and the lane's state stays `working`. A long build in

cmux-tui/crates/acpmux/docs/acpmux-schema.json

@@ -167,11 +167,11 @@
     "Event": {"sessionId": "string", "seq": "number", "at": "number (unix ms)", "dir": "in|out|mux|peer", "kind": "string (method, sessionUpdate value, or mux kind)", "msg": "object"},
     "MessageEvent": {"sessionId": "string (the journal containing this copy)", "sender": {"name": "string", "sessionId": "string", "host": "string? (redacted for peer-origin readers)"}, "text": "string", "at": "number (unix ms)", "messageId": "string (delivery id, or prompt id for direct/report sends)"},
     "ToolRow": {"id": "string (mcp:<server>, skill:<name> or plugin:<name>)", "name": "string", "kind": "mcp|skill|plugin|cmux (cmux: cmux's own tools)", "source": "cmux|claude (what gives it)", "enabled": "boolean (false: the chat turned it off, or the harness reports it disabled)", "canToggle": "boolean (an MCP server of a Claude Code chat that its own config did not turn off)"},
-    "muxEventKinds": ["created", "_acpmux/message", "user_message", "queued", "turn_started", "turn_silent", "turn_heard", "turn_end", "turn_error", "turn_result", "status", "permission_request", "permission_auto", "permission_decision", "permission_group", "permission_chat_allowance", "mode", "model", "config", "policy", "rules", "tools", "tags", "renamed", "titled", "forked", "imported", "resumed", "resume_failed", "failover", "policy_unenforced", "host_notice", "reopened", "stopped", "exited", "stderr", "dequeued", "message_superseded", "queue_removed", "queue_approved", "queue_declined", "prompt_retracted", "process_update", "git", "background_turn"],
+    "muxEventKinds": ["created", "_acpmux/message", "user_message", "queued", "turn_started", "turn_silent", "turn_heard", "turn_retrying", "turn_end", "turn_error", "turn_result", "status", "permission_request", "permission_auto", "permission_decision", "permission_group", "permission_chat_allowance", "mode", "model", "config", "policy", "rules", "tools", "tags", "renamed", "titled", "forked", "imported", "resumed", "resume_failed", "failover", "policy_unenforced", "host_notice", "reopened", "stopped", "exited", "stderr", "dequeued", "message_superseded", "queue_removed", "queue_approved", "queue_declined", "prompt_retracted", "process_update", "git", "background_turn"],
     "background_turn": {"state": "started|ended", "id": "string (shared by a turn's two records)", "trigger": "task_notification|scheduled|system (what started the turn: a background task or subagent ended, a scheduled wakeup or loop tick, anything else such as a hook)", "quiet": "boolean? (ended only: its reply is at most one short line of 160 characters with no question and no link; clients draw it as one muted line that opens on a click)", "reason": "result|prompt|exit (ended only: its result, a prompt that started an acpmux turn while it ran (what follows is that turn's), or its agent exited)", "note": "A turn Claude Code ran on its own, which neither the person nor acpmux started (hub/background_turn.rs). Its own updates are logged between its two records. It never marks the chat unread, never moves lastTurn, and its words never become preview or lastMessage, so nothing notifies of it."},
     "turn_result": {"status": "completed|cancelled|failed", "stopReason": "string?", "turnId": "string (shared with turn_started)", "turnSeq": "number (seq of turn_started)", "promptId": "string", "error": "string?", "code": "number?", "errorText": "string? (why the turn failed; a terminal error the harness reported in-band, such as Codex _meta.codex.error without willRetry, fails the turn and the prompt is answered with that error)", "errorCode": "number|object|null", "errorSource": "agent|codex|acpmux (acpmux: a stalled turn acpmux ended)", "errorChunkSeqs": "number[]? (the trailing agent_message_chunk records whose text is exactly errorText: hide them, show the error)", "detail": "string? (outcome_unknown when the daemon restarted mid-turn; stalled when its model showed no sign of work for config stalledTurnEndMs, default 21600000, 0: never, four times that inside a tool call that had not ended, and nobody held the turn: hub/stalled_turn.rs)", "stalledMs": "number? (with detail stalled: how long the model had been silent)", "model": "string? (the session's model when an answered turn ended)", "costUsd|inputTokens|outputTokens|cacheReadTokens|cacheWriteTokens": "TurnUsage fields of an answered turn, as its harness reported them (a fork's log copies its parent's records before its forked record: count those once)", "gitBranch": "string? (the branch checked out in the session's cwd when the turn ended, answered or failed; absent on a detached HEAD, outside a repository, when git could not be read within 2 s, and in older logs)", "gitHead": "string? (the full HEAD commit at that moment, as gitBranch)", "turnCount": "number? (captured lane turn count; older logs omit it; never inferred from a newer running turn)"},
     "git": {"gitBranch": "string|null (the branch the session's cwd has checked out now; null on a detached HEAD or outside a repository)", "note": "Written when the branch changes while a client holds the chat (a watch on the folder's HEAD, hub/git_watch.rs; no polling), and on the next attach for a change made while none did. Never for a branch the log's last git record already told, a daemon restart included. The branch the session started on (session.json gitBranch) never changes."},
-    "EventKinds": "string[]: each entry is a category or an exact record kind; a record passes when any entry matches. Categories: transcript (agent session/update except .replay, usage_update, available_commands_update, current_mode_update, config_option_update; plus mux _acpmux/message, user_message, queued, dequeued, turn_started, turn_silent, turn_heard, turn_result, permission_request, permission_auto, permission_decision, permission_group, permission_chat_allowance, message_superseded, failover, policy_unenforced, host_notice, forked, imported, mode, model, stopped, exited), mux (every mux record), wire (in/out requests, responses, harness lines and .replay records), all.",
+    "EventKinds": "string[]: each entry is a category or an exact record kind; a record passes when any entry matches. Categories: transcript (agent session/update except .replay, usage_update, available_commands_update, current_mode_update, config_option_update; plus mux _acpmux/message, user_message, queued, dequeued, turn_started, turn_silent, turn_heard, turn_retrying, turn_result, permission_request, permission_auto, permission_decision, permission_group, permission_chat_allowance, message_superseded, failover, policy_unenforced, host_notice, forked, imported, mode, model, stopped, exited), mux (every mux record), wire (in/out requests, responses, harness lines and .replay records), all.",
     "LaneReportProvenance": {"kind": "lane_report", "from": "string (lane session id)", "fromName": "string", "to": "string (authenticated starter session id)", "turn": "number (lane turn count)", "state": "string", "details": {"session": "string (lane session id)", "commits": "string[] (full 40-hex SHAs named in the reply)"}, "note": "The first line is [NAME] VERDICT at SHA12 · finding counts · turn N, STATE when the reply contains Review (who): SHIP|BLOCK|FIX|HOLD at a full SHA, including after prose. Counts cover following P1/P2/P3 finding starts; None means no findings. Otherwise the first sentence is limited to 120 characters at a word boundary. The body keeps whole lines up to about 600 characters and every P1/P2 finding's first sentence even past that budget. Session and commits are structured details, collapsed by the UI, not visible text. The daemon sends this at each turn end of a session the parent started; answer only if it needs you. Identity and turn/state come from provenance, never from the title. Legacy lane report: turn N ended titles remain readable. An authenticated parent waiting explicitly for the lane's completed turn, or reading its completed result through last/history, receives the same report as already read: archived without a new notification or queued turn. Anonymous/other-session reads do not suppress it; active parent turns are never cancelled."},
     "user_message": {"text": "string", "client": "string", "promptId": "string", "turnId": "string", "steer": "boolean? (steered into the running turn)", "images": "[{id, mimeType, name?}]? (the prompt's images, kept under their ids; _acpmux/attachment reads one back)", "inlineParts": "[{kind: text, text, pasted?: true} | {kind: image, id, mimeType, name?, persisted?: false} | {kind: pasted|file, name, mimeType?, size?, text?} | {kind: document|audio, name, mimeType?, size?, pageCount?} | {kind: file, name, uri?, mimeType?}]? (ACP prompt blocks in original order; attachment data is never stored; text on file/pasted parts is copied from the corresponding ACP text block)", "pasted": "[{start: number, end: number}]? (each text block the client marked _meta.cmux.pasted, as UTF-16 code-unit offsets into text, so text.slice(start, end) is the paste; absent when nothing was pasted)", "_meta.provenance": "{kind: \"agent_message\", from, fromName?, to, msgId, threadId, replyTo?}? (the prompt's _meta.acpmux.provenance: an agent's message, not the person's words); hub-set {kind: \"turn_cap\", turnId, capMs} on the one notice steered into a turn past its turn cap (_acpmux/set_turn_cap), never a client's; hub-set LaneReportProvenance for automatic lane results (never accepted from a client)"},
     "queued": {"text": "string", "client": "string", "position": "number", "promptId": "string", "turnId": "string", "images": "[{id, mimeType, name?}]? (as user_message: the prompt's images, kept under their ids when it was queued; an _acpmux/queue_edit update records its own on queue_updated)", "inlineParts": "as user_message.inlineParts (fresh ordered metadata from queued blocks)", "pasted": "[{start, end}]? (as user_message; an _acpmux/queue_edit update records its own on queue_updated)", "_meta.provenance": "object? (as user_message; a restart keeps it, an _acpmux/queue_edit update drops it)", "fromAgent": "string? (another agent's message: its sender)", "class": "block|question|result|fyi? (another agent's message: what it asks of the chat, session/prompt _meta.acpmux.class, result when it named none; a restart keeps it, an _acpmux/queue_edit update drops it)", "awaitingOwner": "true? (it waits for the owner, _acpmux/queue_approve; its author's queue_updated carries it again with the next revision)", "revision": "number? (with awaitingOwner)"},
@@ -189,6 +189,7 @@
     "prompt_retracted": {"promptId": "string (the withdrawn prompt)", "msgId": "string", "turnId": "string (its running turn)", "via": "steer|queued (how the notice went to the agent)", "noticePromptId": "string (the notice's promptId)"},
     "turn_started": {"prompt": "string", "client": "string", "promptId": "string", "turnId": "string", "note": "turnSeq of the turn is this record's seq"},
     "turn_silent": {"turnId": "string", "silentMs": "number (config silentTurnMs, default 90000; 0: never logged)", "note": "Logged once when a turn's model has shown no sign of work (no message, thought, tool call, plan, compaction, subagent progress or request, and no harness line only a responding model produces: claude.stream_event, claude.assistant, claude.system.thinking_tokens, claude.tool_progress, codex.turn/moderationMetadata; the agent's own usage, command, mode and turn notices and its host's stderr do not count) for silentMs since the prompt went out (hub/silent_turn.rs): a stalled model. turn_heard ends it; a turn that answered and then works quietly never logs it."},
+    "turn_retrying": {"turnId": "string", "attempt": "number (when known)", "maxRetries": "number (when known)", "error": "string (the harness's error, when known)", "errorStatus": "number (HTTP status, when known)", "retryDelayMs": "number (when known)", "source": "claude | codex", "note": "Logged for each retry of the turn's failed model request (hub/turn_retry.rs): Claude Code's system/api_retry line, or Codex's session_info_update _meta.codex.error with willRetry. A client says the model cannot be reached, not Thinking. The model's next sign logs turn_heard; a retry that runs out ends the turn failed."},
     "turn_heard": {"turnId": "string", "note": "Logged once, before the record that carries it, when a turn that logged turn_silent first shows its model at work (hub/silent_turn.rs): a client stops saying the model has not answered."},
     "turn_end": {"stopReason": "string|null", "turnId": "string", "turnSeq": "number", "note": "legacy pair of turn_result; turn_error likewise carries {error, code, turnId, turnSeq}"},
     "policy_unenforced": {"policy": "ask|approve-reads|approve-edits|deny-all (the chat's own policy, narrower than approve-all)", "modeId": "string|null (the mode the harness runs in instead)", "reason": "string (why, in English, for logs; a client names the harness and the policy in its own words)", "note": "Logged when a chat's policy is narrower than approve-all and its harness has no mode that asks (hub/policy_mode.rs): opencode, whose own settings decide what it asks; pi; an ACP agent with no modes. Written when the chat is created and when _acpmux/set_policy narrows a live chat's policy that its harness cannot keep. A transcript record: a chat client notes it where it happened."},

cmux-tui/crates/acpmux/src/hub/mod.rs

@@ -182,6 +182,7 @@ mod tool_paths;
 mod tool_shell;
 mod tools;
 pub(crate) mod turn_cap;
+mod turn_retry;
 mod watch;
 pub use new_request::{
     NewRequest, declared_model_json, profile_takes_model_at_spawn, terminal_harness_refusal,

cmux-tui/crates/acpmux/src/hub/paging.rs

@@ -53,6 +53,8 @@ pub const TRANSCRIPT_MUX_KINDS: &[&str] = &[
     // (`silent_turn.rs`).
     "turn_silent",
     "turn_heard",
+    // The harness retries the turn's failed model request (`turn_retry.rs`).
+    "turn_retrying",
     "turn_result",
     // A turn the person did not start, around its rows (`background_turn.rs`).
     "background_turn",

cmux-tui/crates/acpmux/src/hub/stream.rs

@@ -58,6 +58,7 @@ impl Hub {
             }
         }
         let mut superseded = None;
+        let mut retrying = None;
         {
             let mut st = session.stream.lock().unwrap();
             match kind {
@@ -79,6 +80,7 @@ impl Hub {
                 "session_info_update" => {
                     if let Some((will_retry, err)) = codex_error(update) {
                         if will_retry {
+                            retrying = Some(super::turn_retry::codex_retry(&err));
                             if let Some(open) = st.open_message.clone() {
                                 let notice = err
                                     .get("message")
@@ -115,6 +117,10 @@ impl Hub {
                 }
             }
         }
+        // The model request failed and Codex tries again (`turn_retry.rs`).
+        if let Some(retry) = retrying {
+            self.turn_retrying(session, retry);
+        }
         if let Some((old, new, notice)) = superseded {
             let turn_id = session.turn().map(|t| t.turn_id);
             self.append(

cmux-tui/crates/acpmux/src/hub/tap.rs

@@ -229,7 +229,15 @@ impl Hub {
                         let mut line = params.clone().unwrap_or(Value::Null);
                         crate::model_copies::scrub_logged(&mut line);
                         crate::claude_stdio::leave_out_bodies(&kind, &mut line);
-                        return tap_hub.append_logged(&tap_session, "in", &kind, line, host_seq).1;
+                        let retry = (!muted && method == "claude.system.api_retry")
+                            .then(|| super::turn_retry::claude_retry(&line));
+                        let logged =
+                            tap_hub.append_logged(&tap_session, "in", &kind, line, host_seq).1;
+                        // The model request failed and Claude Code tries again (`turn_retry.rs`).
+                        if let Some(retry) = retry {
+                            tap_hub.turn_retrying(&tap_session, retry);
+                        }
+                        return logged;
                     }
                     if method == crate::rpc::method::SESSION_UPDATE {
                         if let Some(su) = params