feat(ttm-control-plane): H6 lifecycle receiver — stop/pause/resume/expand_scope - #18854
Closed
ik-svc-oc wants to merge 9 commits into
Closed
feat(ttm-control-plane): H6 lifecycle receiver — stop/pause/resume/expand_scope#18854ik-svc-oc wants to merge 9 commits into
ik-svc-oc wants to merge 9 commits into
Conversation
* fix(gateway): guard stale compaction summaries * feat(gateway): persist session goal contracts --------- Co-authored-by: nyk <93952610+0xNyk@users.noreply.github.com>
* feat(ttm-control-plane): bundled dispatch receiver plugin
PR-F-H1 of the TTM/Hermes alignment plan. Lands the bundled dashboard
plugin that TTM's HermesAdapter.dispatch_run() targets, plus the
launchd plist that keeps the dashboard (and so the plugin) reachable
continuously.
Plugin layout:
plugins/ttm-control-plane/
README.md
dashboard/
manifest.json # tab.hidden: true; api: plugin_api.py
plugin_api.py # FastAPI APIRouter — mounted at
# /api/plugins/ttm-control-plane/
launchd/
ai.hermes.dashboard.plist # operator-installable
Wire contract per RUNTIME-ADAPTER-CONTRACT.md §Spawn-On-Launch
Dispatch + RUNTIME-PRINCIPAL-CONTRACT.md §Issuance:
GET /health — plugin metadata, unauthed
POST /runs/dispatch — initial run-spawn dispatch;
validates principal_token
+ runtime_id, idempotent
(409 on rebound run_id),
returns 202 + runtime_run_ref
GET /runs/{ref}/status — last-known status
POST /runs/{ref}/stop — drop binding for rebind
Auth: shared-secret X-TTM-Control-Plane-Secret matching the
TTM_CONTROL_PLANE_SECRET env var loaded from ~/.hermes/.env. The
dashboard's general auth middleware deliberately bypasses
/api/plugins/* so the plugin owns its own check. Empty env var means
"auth disabled" — dev/CI fallback only.
After 202 the plugin schedules two tasks off the request loop:
1. POST run.dispatched to ingress_base_url + /api/ingress/runtime/
hermes/events with Authorization: Bearer <principal_token>;
skipped (warning log) when ingress is empty so the plugin still
comes up cleanly in environments where the TTM ingress routes
are not yet reachable.
2. _spawn_headless_session — currently a logging stub; the
follow-up PR wires the actual `hermes chat --headless` (or
in-process equivalent) so a real agent picks up the run.
Idempotency: in-memory _BindingRegistry (thread-safe) keyed by
run_id. Dashboard restarts drop active bindings — sqlite persistence
is a deliberate follow-up.
Tests cover: shared-secret enforcement vs. dev fallback, missing/
mismatched principal_token (400), runtime_id mismatch, happy-path
runtime_run_ref minting, 409 on duplicate dispatch carrying the
prior ref, status round-trip, stop+rebind sequence, and that the
plugin loads the same way the dashboard mounts it (sys.modules
registration before exec_module).
* fix(ttm-control-plane): drop future annotations + fix plist --no-open flag
Without sys.modules pre-registration, from __future__ import annotations
causes dataclass field introspection to fail at dashboard load time.
Python 3.11 handles str | None and dict[str, Any] natively; no need for
the future import. Also corrects the plist flag from --no-browser to
--no-open (the actual hermes dashboard CLI flag).
---------
Co-authored-by: nyk <93952610+0xNyk@users.noreply.github.com>
…H2) (#3) The Hermes runtime calls this module to write canonical state back to TTM during a control-plane run: events, evidence, and approval requests. Per RUNTIME-PRINCIPAL-CONTRACT.md the per-run principal_token rides as Authorization: Bearer + X-Runtime-Id: hermes + X-Run-Id: <run_id> on every ingress call, never logged in plaintext (only an 8-char prefix). Wire contract matches ControlPlaneEventAppendRequest / ControlPlaneEvidenceAppendRequest / RuntimeApprovalRequest in TTM PR-F (NousResearch#646). bind_run() is called once per run by the dispatch receiver bootstrap (PR-F-H1, plugins/ttm-control-plane); subsequent post_event / post_evidence / request_approval calls resolve the bound context by run_id. On 401 the call raises IngressAuthError immediately and does NOT retry (token has been revoked). On non-401 4xx it raises IngressClientError. On 5xx or transport error it retries up to 3x with exponential backoff (0.5s base) before raising IngressServerError. All paths log at DEBUG on success and WARNING on failure with structured ctx including the redacted token prefix. 27 unit tests cover URL + header shape, body shape per the actual TTM schemas, scope_epoch defaulting + override, human-actor body rule (actor_id null), token redaction (plaintext never appears in any log record), 401 no-retry, 409 no-retry, 5xx retry-then-succeed, 5xx retry-exhaust, and connect-error retry-exhaust. Smoke: structural smoke against :8000 with TTM down confirms imports, URL construction, IngressNotBoundError on unbound runs, IngressServerError on connection refused, and token redaction in logs. Live 200/401 probe deferred until TTM backend is running. Co-authored-by: nyk <93952610+0xNyk@users.noreply.github.com>
Adds the bootstrap env-var contract between PR-F-H1's spawn shim and the agent process: the spawn writes TTM_RUN_ID + TTM_PRINCIPAL_TOKEN + TTM_INGRESS_BASE_URL (plus optional TTM_RUNTIME_ID, TTM_SCOPE_EPOCH) into the child process env, and the agent calls ttm_ingress.bind_run_from_env() once at session start instead of hand-rolling the lookup. Returns the bound run_id on success or None when env vars are not all set (lets H3+ skills detect "not running under TTM control" and skip ingress writes cleanly). Until _spawn_headless_session is finished the same env vars also work for local/CI dev — operators can export them before invoking a skill and the binding lands the same way it will in production. 6 new unit tests cover: success path, missing-required returns None (does not raise), empty env returns None, optional TTM_RUNTIME_ID + TTM_SCOPE_EPOCH plumb into the URL/header/body, invalid scope_epoch raises ValueError, defaults to os.environ when env arg is omitted. Tests: 33/33 pass (27 existing + 6 new). No public API change to post_event / post_evidence / request_approval. Co-authored-by: nyk <93952610+0xNyk@users.noreply.github.com>
…ent, #5) (#5) * Harden macOS gateway service recovery * Fix gateway-mode update restart for manual gateways * fix: harden tool publication diagnostics and MCP env * fix(gateway): guard stale compaction summaries * test: align gateway and terminal verification with new runtime behavior * fix: use supported codex auxiliary model * feat(ttm-ingress): add get_run_state + refactor _post into _request (#5) Adds TtmIngress.get_run_state(run_id) -> dict, the read-side complement to the existing write routes. Wire: GET {ingress_base_url}/api/ingress/runtime/ {runtime_id}/runs/{run_id}/state with same auth headers. Auto-applies the response scope_epoch to the binding so subsequent writes use the fresh epoch. Refactors _post into _request(method, path, body?, ...) to share the retry/ backoff/error logic with GET without duplicating 60 lines. _post stays as a thin shim for call-site compatibility. Adds 5 unit tests covering success, 401 auth error, 5xx retry, missing scope_epoch, and not-bound guard. --------- Co-authored-by: nyk <93952610+0xNyk@users.noreply.github.com>
…ation (#6) * Harden macOS gateway service recovery * Fix gateway-mode update restart for manual gateways * fix: harden tool publication diagnostics and MCP env * fix(gateway): guard stale compaction summaries * test: align gateway and terminal verification with new runtime behavior * fix: use supported codex auxiliary model * feat(ttm-ingress): add get_run_state + refactor _post into _request (#5) Adds TtmIngress.get_run_state(run_id) -> dict, the read-side complement to the existing write routes. Wire: GET {ingress_base_url}/api/ingress/runtime/ {runtime_id}/runs/{run_id}/state with same auth headers. Auto-applies the response scope_epoch to the binding so subsequent writes use the fresh epoch. Refactors _post into _request(method, path, body?, ...) to share the retry/ backoff/error logic with GET without duplicating 60 lines. _post stays as a thin shim for call-site compatibility. Adds 5 unit tests covering success, 401 auth error, 5xx retry, missing scope_epoch, and not-bound guard. * fix(ttm-control-plane): add rebind-token endpoint for principal token rotation - POST /runs/{run_id}/rebind-token accepts new principal credential from TTM HermesAdapter.notify_rebind() after a runtime rebind - _BindingRegistry.replace_token() atomically swaps stored bearer token - RebindTokenRequest schema validates new_binding_id + new_token fields - 3 new tests: happy path, unknown run 404, missing secret 401 - 15/15 plugin tests pass --------- Co-authored-by: nyk <93952610+0xNyk@users.noreply.github.com>
The rebind-token handler logged the first 8 characters of the new principal token. Even partial token material is unsafe for long-running production logs, so log only run_id and binding_id. Test: caplog assertion that the rebind log line contains neither the full token nor its prefix. Co-authored-by: nyk <93952610+0xNyk@users.noreply.github.com>
…stry
PR-F-H1 left two items deferred. With H3 closeout this PR completes them
and removes the partial-token logging that survived in tools/ttm_ingress.
- _spawn_headless_session now spawns `hermes chat -q <brief> -Q
--max-turns 200` with TTM_RUN_ID / TTM_PRINCIPAL_TOKEN /
TTM_INGRESS_BASE_URL / TTM_RUNTIME_ID injected as env vars. The
child is detached (start_new_session=True) and its stdout/stderr go
to ~/.hermes/logs/runs/<run_id>.log. Failure to resolve the binary
or empty token logs and skips — never crashes the dispatch route.
Set TTM_CONTROL_PLANE_DISABLE_SPAWN=1 to suppress (tests/dev). The
principal token is captured from the binding before
_post_run_dispatched can clear it, eliminating a token-clear race.
- _BindingRegistry is now SQLite-backed at ~/.hermes/state.db (override
via TTM_CONTROL_PLANE_DB_PATH). Bindings survive dashboard restarts
so re-dispatch is correctly idempotent and operators can audit the
history. The principal_token is intentionally NOT persisted — only
binding metadata. After a restart the operator triggers a TTM rebind
to issue a fresh token, which arrives via /runs/{run_id}/rebind-token.
- tools/ttm_ingress.py: replace _redact_token (8-char prefix) with
_token_present (boolean marker). The log line now reports token=set
or token=unset; never a value or prefix. Tests updated to assert
neither the full token nor its 8-char prefix appears in caplog.
- README updated: H1 deferrals marked landed; pause/resume/retry-slice
remain H4 territory.
Tests:
- 5 new plugin tests: persistence across registry instances, remove
clears persistence, spawn no-op when disabled, spawn log-and-skip
when binary missing.
- TestTokenPresenceLogging: token value and prefix never logged.
- All 58 tests pass.
…pand_scope
Adds POST /runs/{ref}/lifecycle (stop|pause|resume|expand_scope) and
keeps POST /runs/{ref}/stop as a stable compat alias for the current
TTM HermesAdapter until the adapter is updated to use /lifecycle.
Stop: SIGTERM process group → 10s wait → SIGKILL survivors; emits
task.updated{stopped} via TTM ingress if plugin holds a token; binding
is kept (status=stopped) so TTM drives canonical closure independently.
Hermes never self-approves run closure.
Pause: SIGSTOP process group; saves dossier state (pid, lane_id,
worktree_id, paused_at); emits task.updated{paused}. Degrades
explicitly via runtime.error if SIGSTOP is unavailable — never
silently claims paused when the suspend did not happen.
Resume: SIGCONT from saved pause state; restores dossier; emits
task.updated{active}.
Expand-scope: SIGUSR1 advisory hint; revokes old principal_token
immediately so plugin-level events stop using it; updates binding
status to scope_expanding. Existing rebind-token route restores the
token (and resets status to running) when TTM issues the new one.
Headless agent detects epoch change via 401 on next ingress write.
_ProcessRegistry tracks live headless session PIDs registered at
spawn time. _PauseState dossier persists per-run context for resume.
41 unit tests: action validation, SIGTERM→wait→SIGKILL fallback,
SIGSTOP degrade path, pause/resume dossier lifecycle, token never
logged, ingress event emission, expand_scope token revocation.
alt-glitch
added
type/feature
Author
|
Superseded by clean squash PR from upstream main — no file conflicts. |
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
POST /runs/{ref}/lifecycleacceptingstop | pause | resume | expand_scope(202 async, validated against known bindings)POST /runs/{ref}/stopas a stable compat alias for current TTMHermesAdapter.stop_run()until adapter is updated to use/lifecycle_ProcessRegistrytracking live headless session PIDs registered at spawn time_PauseStatedossier (pid, lane_id, worktree_id, paused_at) for pause/resumeTTM adapter mismatch (open)
Current TTM
HermesAdapter(backend/app/services/orchestration/hermes_adapter.pyas ofa007593b):stop_run()→ POSTs to/runs/{ref}/stop✓ (compat route handles this)pause_run()→ returnsunsupported(not wired)resume_run()→ returnsunsupported(not wired)A follow-up TTM PR must update
hermes_adapter.pyto POST to/runs/{ref}/lifecyclewith{"action": "pause"/"resume"/"stop"}and add adapter tests. That PR targets TTMdevelopand is separate from this one.Lifecycle semantics
Stop: SIGTERM process group → 10s wait → SIGKILL survivors. Emits
task.updated{status: stopped}via TTM ingress if plugin holds a token. Binding is kept (last_status=stopped); TTM drives canonical closure independently. Hermes never self-approves run closure.Pause: SIGSTOP process group; saves
_PauseStatedossier; emitstask.updated{status: paused}. Degrades explicitly viaruntime.errorif SIGSTOP raises — never silently claims paused.Resume: SIGCONT from saved dossier; clears
_PauseState; emitstask.updated{status: active}.Expand-scope: SIGUSR1 advisory hint; clears
principal_tokenfrom plugin registry (old token treated as revoked); setslast_status=scope_expanding. Existingrebind-tokenroute restores the new token and transitions back torunning. Headless agent's own env-var copy of the old token receives 401 on next ingress write and must stop emitting.Validation
Tests cover: action validation (422 on bad action), SIGTERM→wait→SIGKILL fallback, SIGSTOP degrade path, pause/resume dossier lifecycle, expand_scope token revocation, token never logged, ingress event emission mocked.
Gate