Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions website/docs/user-guide/features/api-server.md
Original file line number Diff line number Diff line change
Expand Up @@ -375,6 +375,17 @@ Statuses are retained briefly after terminal states (`completed`, `failed`, or `

Server-Sent Events stream of the run's tool-call progress, token deltas, and lifecycle events. Designed for dashboards and thick clients that want to attach/detach without losing state.

When the agent delegates work to background subagents, the stream also carries
`subagent.start` and `subagent.complete` lifecycle events, so clients can
observe delegation outcomes — including timeouts and failures — instead of the
run going silent while a child works. The `subagent.complete` payload carries
the child's status, summary, duration, token/cost figures, and a
`child_session_id` for correlation; free-text fields pass forced secret
redaction before leaving the process. Per-tool child events
(`subagent.tool`, progress ticks) are intentionally **not** forwarded — they
are high-volume UI noise; use the per-child live transcript files for
play-by-play.

Unconsumed event buffers expire after five minutes so a detached client cannot
grow memory indefinitely. This expires transport state only: a run that is
still executing remains visible to status polling, approval, stop control, and
Expand Down
62 changes: 61 additions & 1 deletion website/docs/user-guide/features/delegation.md
Original file line number Diff line number Diff line change
Expand Up @@ -191,10 +191,54 @@ delegation:

A positive value enforces a hard wall-clock limit on each child; `0` or a negative value disables it.

When a configured cap fires, the child's result carries structured timeout
metadata alongside the error message so parents and hooks can distinguish a
stopwatch kill from other failures without parsing text: `timeout_seconds`
(the configured cap), `timed_out_after_seconds` (actual wall clock), and
`timeout_phase` (`before_first_llm_call` when the child never reached its
first request, `after_llm_calls` otherwise). All three are `null` on
non-timeout errors.

:::tip Diagnostic dump on zero-call timeout
With a hard cap configured, if a subagent times out having made **zero** API calls (usually: provider unreachable, auth failure, or tool-schema rejection), `delegate_task` writes a structured diagnostic to `~/.hermes/logs/subagent-timeout-<session>-<timestamp>.log` containing the subagent's config snapshot, credential-resolution trace, and any early error messages. Much easier to root-cause than the previous silent-timeout behavior.
With a hard cap configured, if a subagent times out having made **zero** API calls (usually: provider unreachable, auth failure, or tool-schema rejection), `delegate_task` writes a structured diagnostic to `~/.hermes/logs/subagent-timeout-<session>-<timestamp>.log` containing the subagent's config snapshot, credential-resolution trace, any early error messages, and stack traces for **all** live threads (not just the child's own) — a child parked waiting on a nested helper thread is indistinguishable from a slow provider without the full picture.
:::

## Stall Detection for Background Subagents

Background delegations (`delegate_task(background=true)`) are watched by a
**progress-based stall monitor** — on by default, zero config. Unlike a
wall-clock timeout, it never touches a child that is making progress, no
matter how long it runs.

The monitor samples each detached child's progress signals — API-call count,
current tool, and last-activity timestamp (which ticks on **every streamed
token**, tool transition, and API-call boundary, so a child mid-stream on a
long response always counts as alive):

1. **Progressing children are never touched.** Any advancing signal resets
the clock.
2. A child whose progress is completely frozen past the stale threshold
(450s idle, 1200s while inside a tool — legitimately slow terminal
commands and web fetches get the higher ceiling) is **interrupted** and
given a 120s grace window. A child that unwinds in time delivers its
partial results through the normal completion path.
3. A child that never returns is force-finalized with a terminal `stalled`
completion event, so the owning session hears an outcome instead of
going silent, and the async slot frees for new work.

The `stalled` event carries structured metadata mirroring the sync-path
timeout fields: `stalled_after_quiet_seconds`, `stall_threshold_seconds`,
`stall_phase` (`idle` / `in_tool`), and `stall_grace_seconds`.

This closed a long-standing failure mode where a wedged background child
left its session looking dead until a process restart. The underlying wedge
(children hanging at their first API call after multi-day gateway uptime)
was also fixed at the root: delegated children now run their OpenAI-wire
API requests inline on their own conversation thread instead of a nested
worker thread — the layer where the wedge lived. The stall monitor remains
as the safety net for anything else.


## Monitoring Running Subagents (`/agents`)

The TUI ships a `/agents` overlay (alias `/tasks`) that turns recursive `delegate_task` fan-out into a first-class audit surface:
Expand All @@ -206,6 +250,22 @@ The TUI ships a `/agents` overlay (alias `/tasks`) that turns recursive `delegat

The classic CLI just prints `/agents` as a text summary; the TUI is where the overlay shines. See [TUI — Slash commands](/user-guide/tui#slash-commands).

On the classic CLI and every gateway platform (Telegram, Discord, Slack, ...),
`/agents` also lists **background delegations with live per-child activity**,
sampled directly from each running child:

```
Background delegations: 1 running
- deleg_ab12cd34 · running · research the delegation stall monitor
- child 1: 4 api calls · in web_search · active 12s ago
- child 2: 7 api calls · between turns · active 3s ago
```

A delegation the stall monitor has flagged shows as
`stalling · no progress 450s — interrupting`, and long-quiet-but-healthy
children show their quiet time so you can tell "slow" from "stuck" at a
glance.

## Live Transcripts

Every `delegate_task` dispatch also creates one **append-only, human-readable log per task** so you (or the parent agent) can watch a subagent work in real time instead of waiting for the consolidated summary:
Expand Down
1 change: 1 addition & 0 deletions website/sidebars.ts
Original file line number Diff line number Diff line change
Expand Up @@ -750,6 +750,7 @@ const sidebars: SidebarsConfig = {
link: {type: 'doc', id: 'developer-guide/plugins/index'},
items: [
'developer-guide/plugin-llm-access',
'developer-guide/subagent-lifecycle-api',
'developer-guide/desktop-plugin-sdk',
'developer-guide/memory-provider-plugin',
'developer-guide/context-engine-plugin',
Expand Down
Loading