Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 12 additions & 8 deletions docs/design/2026-08-06-active-work-health.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,20 +8,22 @@

`GET /health?deep=1` gains three fields: `activeWork`, `activeWorkReporting`, and `activeWorkStaleMs`.

`activeWork` is true while any managed workspace has an accepted-but-unsettled prompt, a running background Agent, or an Agent terminal notification that is queued, awaiting acceptance, or being processed by its parent continuation. It deliberately does **not** cover background shells, Monitors, workflows, or cron. That exclusion is a scope decision, not an oversight: those categories have no equivalent signal today, and a controller that treats `activeWork: false` as "nothing at all is running" will be wrong about them.
`activeWork` is true while any managed workspace has an accepted-but-unsettled prompt, a running background Agent, an Agent terminal notification that is queued, awaiting acceptance, or being processed by its parent continuation, or Session-managed background shell work. Shell work covers a running registry entry and the terminal notification until its parent continuation settles. It deliberately does **not** cover Monitors, workflows, cron, or external processes the shell registry can no longer track.

It is also **Session-scoped, not channel-scoped**. Channel-level work with no Session attached yet — a spawn in flight, a pending restore, MCP discovery or authentication — is not counted, so `activeWork` can read false while the daemon's own `hasNoChannelWork` is simultaneously refusing to reclaim that channel. The two answer different questions and are allowed to disagree: this field describes work owned by Sessions, and widening it to cover channel setup would change what the boolean means for every existing reader. A controller that needs "is this daemon reclaimable" must combine the three-term rule below with a graceful-shutdown handshake, not read more into this one field than it claims.

Restart policy stays with the external controller. The daemon publishes facts; it does not publish `restartSafe`.

## Why holds, and why full snapshots

Each Session reports a set of named **holds**, each carrying a category (`agent`, `notification`). Two properties follow, and both are the point:
Each Session reports a set of named **holds**, each carrying a category (`agent`, `notification`, or `shell`). Two properties follow, and both are the point:

**Holds are derived, never maintained.** `Session.collectActiveWorkHolds()` reads the owners of the work — the background-task registry's unfinalized set, the notification queue, the in-flight acceptance and continuation state — on every call. There is no acquire/release ledger kept alongside the work, because a ledger can miss a release, and a leaked hold would pin its Session forever while every snapshot faithfully republished the leak.
**Holds are derived, never maintained.** `Session.collectActiveWorkHolds()` reads the owners of the work — the background-task registry's unfinalized set, the background-shell registry's running entries, the notification queue, and the in-flight acceptance and continuation state — on every call. There is no acquire/release ledger kept alongside the work, because a ledger can miss a release, and a leaked hold would pin its Session forever while every snapshot faithfully republished the leak.

The agent category uses `BackgroundTaskRegistry.hasUnfinalizedTasks()`'s predicate rather than `hasRunningTasks()`'. A cancelled agent still owes its terminal task-notification: `cancel()` flips status and emits a status change, but the notification arrives later from `finalizeCancelled()` or the 5s grace timer. Keying on "running" would make the Session look idle inside that window, and a detached Session would be closed with the notification still owed.

Shells use one aggregate hold, `{ "category": "shell", "id": "background-shells" }`, regardless of the number of running shells. The task registry and `/tasks` surface remain the detailed roster; active-work only needs the bounded retention fact. The aggregate also prevents an unbounded shell roster from exceeding the protocol's per-Session hold limit.

**Reports are complete snapshots at channel scope, not per-Session transitions.** One message per ACP channel carries every Session the child owns and every hold it holds:

```json
Expand All @@ -40,19 +42,20 @@ Prompts are absent from the child's report on purpose. The daemon accepts, queue

## Ordering

A snapshot is flushed ahead of the prompt response on the same stream. The daemon drops its pending-prompt count the instant that response lands, so a hold the prompt left behind — a background Agent it started — must already be on the wire, or the daemon briefly sees neither fact.
A snapshot is flushed ahead of the prompt response on the same stream. The daemon drops its pending-prompt count the instant that response lands, so a hold the prompt left behind — a background Agent or shell it started — must already be on the wire, or the daemon briefly sees neither fact.

## Three states, and closing atomically
## Reporting states, and closing atomically

Per Session the daemon holds one of:

- **unsupported** — the channel never negotiated. Contributes nothing; pre-existing cleanup behavior applies unchanged. Treating this as "unknown" would make every legacy Session permanently unreapable.
- **incomplete** — the channel negotiated but does not report every category the daemon currently requires. Health is graded `partial`, and ordinary automatic cleanup is disabled for that Session. Unlike unknown freshness, another round trip cannot make an older child understand a category it did not negotiate.
- **unknown** — negotiated, not yet heard from _recently enough_. Reads as busy on the health surface, but is not a state the daemon sits in: it asks.
- **known** — a fresh snapshot has been applied.

Never-reported and gone-quiet are the same state on purpose. A snapshot older than the grading window (`intervalMs × 3`) is not a report that the Session is idle, it is the absence of one — a background Agent could have started at any point since — so it stops counting as evidence.

**Unknown is a reason to ask, not a reason to skip.** The two consumers read it differently, and they have to: the health surface reports unknown as busy (a controller must never mistake "nobody told me" for "nothing is running"), while automatic cleanup treats it as a candidate and goes on to the conditional close below. Only _known_ work — daemon-owned, or a fresh report of held work — blocks the attempt outright. Skipping on unknown instead would look safe and in fact be the worse failure: nothing would ever resolve it, so a Session on a channel that went quiet would be retained forever with no path out. Asking costs one bounded round trip and still retains on any non-answer, and the child can answer authoritatively under its close gate whether or not its snapshots are arriving.
**Unknown is a reason to ask, not a reason to skip.** The two consumers read it differently, and they have to: the health surface reports unknown as busy (a controller must never mistake "nobody told me" for "nothing is running"), while automatic cleanup treats it as a candidate and goes on to the conditional close below. Only _known_ work — daemon-owned, or a fresh report of held work — blocks the attempt outright. Skipping on unknown instead would look safe and in fact be the worse failure: nothing would ever resolve it, so a Session on a channel that went quiet would be retained forever with no path out. Asking costs one bounded round trip and still retains on any non-answer, and the child can answer authoritatively under its close gate whether or not its snapshots are arriving. Incomplete coverage is different and does skip: a negotiated child that omits `shell` can truthfully answer according to its older predicate while missing a running shell, so its answer cannot authorize automatic destruction.

Reclaiming a channel that has stopped answering entirely is still not this mechanism's job; see below.

Expand All @@ -63,7 +66,7 @@ qwen/control/session/close { sessionId, onlyIfUnheld: true }
→ { closed: true, holds: [] } | { closed: false, holds: [...] }
```

The child evaluates it under its own close gate, before anything destructive runs. With the gate held the Session admits no new prompt and starts no new automatic turn, so a hold cannot appear between the check and the teardown **on the child side**. If holds exist, the gate is released and they are handed back; the daemon adopts them and backs off.
The child evaluates it under its own close gate, before anything destructive runs. It rejects known holds immediately, drains any turn that was already active when the gate closed, then evaluates the unfiltered collector again. The second read matters because an already-running out-of-scope turn such as cron can register a background shell while it drains. With the gate still held no new turn can start after that final read, so a hold cannot appear between final authorization and teardown **on the child side**. If either read finds holds, the gate is released and they are handed back; the daemon adopts them and backs off.

The daemon side needs its own cover, because the round trip is an await of up to ten seconds. A Session with a conditional close outstanding is marked in-flight, and every admission path — attach, prompt, rewind — refuses it exactly as it refuses one that is already closing. Without that, a prompt accepted during the round trip is lost when the teardown it raced completes; the previous synchronous guard-then-teardown sequence got this for free, and splitting it is what created the need to say so explicitly.

Expand All @@ -80,6 +83,7 @@ Four things can decide it is time to look at a Session: the last client detachin
| not already closing or close-in-flight | two paths racing the same teardown duplicate the round trip and race each other's guards |
| no SSE subscriber | someone is watching this Session's stream |
| nothing daemon-owned in flight | queued and dispatched prompts and notifications the daemon is pushing; never depends on the child reporting anything |
| negotiated reporting covers every category | an older predicate must not authorize teardown while work in a newer category exists |
| no fresh child report of held work | only _known_ work blocks; unknown is a candidate that goes on to ask |
| the child confirms under its own close gate | the cache says what _was_ true; only the child can say what is true now |

Expand All @@ -105,7 +109,7 @@ Killing a whole multiplexed channel is reasonable when the channel is _actually_
| `activeWorkReporting` | `full` / `partial` / `none` — how much of that boolean is vouched for |
| `activeWorkStaleMs` | Age of the oldest snapshot it rests on; `0` when nothing is covered |

Freshness is graded by the daemon, not the controller: the reporting cadence is negotiated per channel (the child proposes, the daemon clamps into an agreed range), so only the daemon can judge it. A stale snapshot or a child that omits a category degrades the grade to `partial` rather than silently narrowing what the boolean covers. `activeWorkStaleMs` is diagnostic, and it measures only the _covered_ Sessions — an uncovered one already shows up in the grade, so letting it also drag the age down would double-count it and produce a positive staleness next to a grade saying nothing is covered.
Freshness is graded by the daemon, not the controller: the reporting cadence is negotiated per channel (the daemon requests a cadence and category set; the child echoes the clamped cadence and the supported intersection), so only the daemon can judge it. A stale snapshot or a child that omits a category degrades the grade to `partial` rather than silently narrowing what the boolean covers. A v1 request without `categories` means the legacy `agent`/`notification` baseline, which lets a new child keep its wire report readable by an old daemon while its local collector still sees shell work for conditional close. `activeWorkStaleMs` is diagnostic, and it measures only the _covered_ Sessions — an uncovered one already shows up in the grade, so letting it also drag the age down would double-count it and produce a positive staleness next to a grade saying nothing is covered.

The grade is computed once over the whole daemon rather than per runtime and then combined, because grades do not compose: a runtime with no Sessions vouches for everything it has, and folding that vacuous `full` in as evidence let an empty workspace vouch for another workspace's unreported Sessions. Each runtime therefore exposes coverage counts and the route sums them before grading.

Expand Down
50 changes: 50 additions & 0 deletions docs/design/2026-08-13-active-work-background-shell.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# Background shell active-work coverage

## Problem

A Prompt can start a long-running background shell and finish immediately. Before this change the daemon then observed `activePrompts: 0` and `activeWork: false` even though `GET /session/:id/tasks` still reported a running shell. A restart controller could therefore treat the daemon as idle and terminate the Session before the shell's terminal notification reached the parent continuation.

## Decision

Session-managed background shells join the existing active-work snapshot protocol as category `shell`. A Session publishes one aggregate hold while its shell registry has a running entry, a shell terminal notification is queued, or that notification is driving the parent continuation:

```json
{ "category": "shell", "id": "background-shells" }
```

The hold is deliberately aggregate. The shell registry and task-status surfaces remain the detailed roster, while the retention protocol stays bounded even if a Session owns more than 1024 shells.

The Session collector remains an unfiltered statement of local truth. Category negotiation is applied only when the reporter serializes a wire snapshot. This distinction is required for compatibility: a new child talking to an old v1 daemon filters `shell` from the wire, but its conditional-close check still sees the running shell locally and answers `closed: false`.

## Negotiation and compatibility

The protocol version remains v1. The daemon initialize request advertises `agent`, `notification`, and `shell`; the child answers with the intersection it supports. A request with no `categories` is the pre-negotiation v1 baseline, `agent` and `notification`.

| Peers | Reporting result | Ordinary automatic cleanup |
| ------------------------------------------ | ---------------------------------------- | ----------------------------------------------------- |
| new daemon + new child | `full`; shell hold crosses the wire | existing conditional-close flow |
| new daemon + old v1 child | `partial`; `shell` is missing | disabled for that Session |
| old v1 daemon + new child | wire contains only the legacy categories | local conditional close still rejects a running shell |
| daemon + child with no active-work support | `none` | historical legacy cleanup |

Negotiated-but-incomplete and unsupported are intentionally different. An unsupported historical child keeps the behavior it had before active-work existed. A child that negotiated the protocol but omitted a currently required category has explicitly disclosed that its predicate is incomplete, so it cannot authorize an ordinary teardown. Explicit close, kill, daemon shutdown, channel exit, and condemned restore cleanup keep their force semantics.

## Lifecycle and ordering

The shell registry synchronously reports registration and terminal transitions. Session installs an identity-safe status callback that triggers the existing change-coalesced reporter and removes exactly that callback on dispose.

At shell completion, the registry invokes the notification callback before publishing the terminal status change. The notification is therefore already queued when the running entry becomes terminal. When the drain removes the queue item it marks the shell continuation active before yielding. These transitions ensure the derived aggregate hold has no false gap between running, queued, and executing states. Prompt teardown also retains the existing reporter flush-before-response ordering, so a shell started by the Prompt is visible before the daemon decrements its own prompt count.

`Session.isIdle()` consumes the same unfiltered collector. Workspace reload therefore skips a Session while a background shell or its terminal continuation is active.

Conditional close reads the unfiltered collector once before disturbing active turns and again after those turns drain, while the Session close gate remains held. The final read closes the window where an already-running, otherwise out-of-scope cron or automatic turn registers a shell during drain; the new shell refuses ordinary teardown without adding cron itself to `activeWork`.

## Boundaries

This change tracks the logical lifecycle owned by `BackgroundShellRegistry`; it does not use PID probes or sidecars to reconstruct process liveness. `task_stop` follows the registry's terminal status and does not promise an additional OS-level exit confirmation. A promoted or externally detached process that the registry no longer tracks is outside the signal.

Long-running development servers consequently keep `activeWork: true`. This is the intended retention fact, not shell-stall detection or a restart lease. Monitor, workflow, cron, and follow-up work remain out of scope, and the public health shape, persistence formats, shell admission policy, heartbeat behavior, and watchdog behavior do not change.

## Verification

Unit coverage pins aggregate cardinality, running-to-notification handoff, reporter filtering, legacy negotiation, bridge parsing, incomplete-child retention, post-drain conditional-close authorization, explicit force close, callback cleanup, and unchanged unsupported-child behavior. The E2E plan reproduces the released baseline with a running `sleep` shell and compares it with the local build through shell completion and parent continuation settlement.
Loading
Loading