fix(gateway): make API runs durable and idempotent

[rm0nroe]

What does this PR do?

Makes the existing /v1/runs API recoverable when a client loses the 202 response. Hermes now reserves the caller's Idempotency-Key and request fingerprint in the profile-scoped state database before dispatch, so an identical retry returns the original run and a conflicting retry fails closed.

Run status and exact correlation lookup survive gateway restart. Nonterminal runs whose owner disappeared become interrupted rather than being replayed, and stop remains bound to the exact run until terminal cancellation.

Related Issue

No upstream issue. This fixes a confirmed lost-response ambiguity in the existing Runs API.

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

• Add a profile-scoped SQLite run ledger with atomic reservation, lifecycle transitions, retention, and restart reconciliation.
• Add exact Idempotency-Key lookup plus durable status and stop handling to gateway/platforms/api_server.py.
• Keep database work off the aiohttp event loop and serialize maintenance with reservation.
• Add regression coverage for lost responses, concurrent duplicates, conflicts, restart recovery, profile isolation, stop ordering, storage failures, lifecycle races, and terminal immutability.

How to Test

1. Run HERMES_PYTHON=/path/to/python scripts/run_tests.sh tests/gateway/test_api_server_runs.py tests/gateway/test_api_server.py -q.
2. Confirm 130 tests pass with no retries or failures.
3. Submit the same Idempotency-Key twice and verify the second response returns the same run_id without redispatch.

Checklist

Code

• I've read the Contributing Guide.
• My commit message follows Conventional Commits.
• I searched open PRs for an existing durable Runs correlation fix.
• My PR contains only changes related to this fix.
• The full hosted test suite passes.
• I've added tests for the fix.
• I've tested on macOS 15 with Python 3.11.

Documentation & Housekeeping

• Documentation update: N/A; the API advertises the contract through /v1/capabilities.
• cli-config.yaml.example update: N/A; no configuration keys added.
• CONTRIBUTING.md or AGENTS.md update: N/A; existing gateway architecture is extended.
• Cross-platform impact considered: SQLite, asyncio, and stdlib process ownership paths are platform-neutral.
• Tool descriptions/schemas update: N/A; no model tool changed.

Screenshots / Logs

Focused CI-parity wrapper: 130 passed, 0 failed. Ruff, py_compile, and git diff --check pass.

[Enough1122]

AI code review — automated review for reference, author can ignore or act on any point.

fix(gateway): make API runs durable and idempotent

1. Per-request ledger maintenance — _handle_get_run, _handle_run_events, _handle_run_approval, _handle_lookup_run and _handle_stop_run all call _maintain_run_ledger() (gateway/platforms/api_server.py ~7105, ~7135, ~7207, ~7260, ~7315), which runs recover_interrupted_runs + purge_terminal_runs — a full scan of active rows with per-row PID-liveness checks — on every request. A dashboard polling run status every second turns each poll into an O(active) table scan plus syscalls. Consider moving recovery/purge into the periodic sweep (_sweep_orphaned_runs) and keeping per-request lookups read-only.
2. gateway/run_ledger.py:44–77 — api_runs lives in the shared state.db, and _connect() re-runs CREATE TABLE/INDEX DDL on every connection (every transaction). A long-running session-store write can block BEGIN IMMEDIATE up to the 10s busy timeout, adding latency to run status updates. Consider a dedicated ledger DB, or hoisting the DDL to a one-time init.
3. Cross-scope interruption edge — recover_interrupted_runs marks a row interrupted when its run_id is not in the current scope's active set, even if the owning executor thread is still live in-process (a multi-profile process serving several HERMES_HOMEs). _clear_run_memory protects the live task from being dropped, but the durable row stays interrupted and the still-running executor's later updates are rejected by transition_allowed. Please verify the multi-profile-in-one-process topology can't strand a live run as interrupted.
4. reserve_run races across processes on the UNIQUE idempotency_key surface as sqlite3.IntegrityError, which the handler maps to 503 run_storage_unavailable (gateway/platforms/api_server.py:6719–6727) — misleading for what is really an idempotency conflict. Catching IntegrityError and returning 409 would match the single-process path.