Skip to content
This repository was archived by the owner on Jul 22, 2026. It is now read-only.
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -16240,6 +16240,21 @@ def _signal_handler_q(signum, frame):
_exit_code = _RL_CODE
except Exception:
_exit_code = 1
# Drop the board-wide quota-wall sentinel
# BEFORE exiting: on hosts where the
# dispatcher can't see exit codes (Windows
# without handle capture) this row is the
# only signal that stops it from spawning
# more workers into the same wall.
try:
from hermes_cli.kanban_db import (
record_rate_limit_wall_for_worker,
)
record_rate_limit_wall_for_worker(
str(result.get("error") or "")
)
except Exception:
pass
sys.exit(_exit_code)

# Exit with error code if credentials or agent init fails
Expand Down
5 changes: 5 additions & 0 deletions docs/observability/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,11 @@ need to reconstruct agent execution without changing runtime behavior. This
contract supports trace, metrics, audit, replay, and export integrations such
as Langfuse, OpenTelemetry-style collectors, and NeMo Relay.

Native dashboard slash-worker process, queue, and command-lifecycle events use
a narrower privacy-bounded log contract documented in
[`dashboard-slash-worker.md`](dashboard-slash-worker.md). They intentionally do
not duplicate observer, generic process-supervisor, or gateway delivery events.

Observer hooks are intentionally backend-neutral. They expose stable lifecycle
events, correlation IDs, sanitized payloads, timing, status, and error fields.
They do not replace Hermes' planner, model providers, memory, tool registry,
Expand Down
56 changes: 56 additions & 0 deletions docs/observability/dashboard-slash-worker.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Dashboard slash-worker operational events

The native dashboard owns a persistent `tui_gateway.slash_worker` subprocess for
slash commands. Its operational events are emitted as one-line JSON records to
the dashboard's normal GUI log using the stable `HERMES_TELEMETRY ` marker and
the `hermes.dashboard.slash_worker.v1` schema. This keeps the records on the
existing host-log path used by local operators and log collectors; telemetry
failure is fail-open and never changes command or shutdown behavior.

## Privacy and cardinality contract

Records contain only:

- fixed lifecycle, state, reason, severity, command-category, and bucket enums;
- bounded counters and exit codes;
- stable fingerprints; and
- SHA-256-derived opaque references for the profile, launcher, parent process,
worker instance, session, command, and an optional gateway event.

They never contain command arguments, prompts, slash-command text, message or
response content, model/API tokens, credentials, authorization headers,
usernames, home paths, or raw URLs. Free-form exception and stderr text passes
through an allowlist-only summarizer before it reaches a record or the retained
stderr tail. Every wire record is bounded to 2048 bytes. Dynamic references
have a fixed cardinality budget; duplicate events, bursts, and over-budget
references are suppressed or collapsed, with bounded suppression counters.

## State transitions

The schema covers worker starting, initialization, readiness, graceful or forced
shutdown, shutdown failure, abrupt exit, and restart-loop recovery. Command
states cover accepted, rejected, deduplicated, dispatched, started, completed,
failed, timed out, and abandoned. Queue health uses fixed depth, oldest-age, and
latency buckets and emits edge-triggered pressure, stall, and recovery records.

`fingerprint` is stable for the event/state/reason/session tuple. Consumers
should open an incident on degraded, stalled, failed, timed-out, abandoned, or
crashed states and resolve it on the corresponding recovered/ready/stopped
transition. Restart-loop and queue-health transitions are emitted only on state
edges to avoid alert churn.

## Ownership boundary

These records own only dashboard/slash-worker semantics:

- They do **not** duplicate generic process-supervisor lifecycle events. Parent,
launcher, and instance references are correlation fields only.
- They do **not** claim channel or gateway delivery. A worker-local response or
delivery-handoff failure may reference an opaque gateway event ID, while the
gateway/channel event remains authoritative for actual delivery.
- They do **not** instrument Desktop shell, WebUI, MCP, or Telegram domain
events.

Collectors must parse only lines with the exact marker and schema, preserve the
bounded record as structured details, and must not enrich it with raw process
arguments, environment values, message text, or filesystem paths.
Loading
Loading