Skip to content

feat: add sanitized Kanban REST API - #61982

Open
andriy4k07 wants to merge 17 commits into
NousResearch:mainfrom
andriy4k07:feat/kanban-rest-api
Open

feat: add sanitized Kanban REST API#61982
andriy4k07 wants to merge 17 commits into
NousResearch:mainfrom
andriy4k07:feat/kanban-rest-api

Conversation

@andriy4k07

@andriy4k07 andriy4k07 commented Jul 10, 2026

Copy link
Copy Markdown

Sanitized Kanban REST API for external control planes

Summary

Adds a narrow, authenticated, safe-by-default REST API over the existing Hermes Kanban store, mounted at
/api/plugins/kanban. It gives external control planes a stable integration boundary for creating, routing,
inspecting, and managing Kanban tasks — without reading raw SQLite state, touching internal dashboard routes, or
executing profiles directly.

Design constraints, all preserved end to end:

  • Reuses hermes_cli.kanban_db — same database, WAL/transaction behavior, task state machine, dependency promotion
    rules, and dispatcher lifecycle. No second schema, no second queue.
  • No profile-execution endpoint. Work is routed by task assignment; the existing gateway dispatcher claims and
    launches it.
  • Sanitized DTOs only — never raw internal rows.
  • No product-specific workflow types or business logic.

API surface

Area Endpoints
Status GET /health, GET /capabilities
Boards GET /boards, GET /boards/{id-or-name}
Profiles GET /profiles — sanitized assignee roster (read-only)
Tasks GET /tasks, POST /tasks, GET /tasks/{id}, PATCH /tasks/{id}
Actions POST /tasks/{id}/comment, /complete, /block, /unblock, /archive
Dependencies POST /tasks/{parent}/links/{child}, DELETE /tasks/{parent}/links/{child}
Observability GET /tasks/{id}/events, /runs, /log

All request models are strict Pydantic (extra="forbid"); unknown fields are rejected.

Authentication

External controllers authenticate with a dedicated service credential, wired through the dashboard-auth framework
rather than bolted onto the routes:

  • The generic token-auth seam gains prefix registration (exact-path matching cannot cover parameterised routes)
    and required scopes: a verified principal lacking the route's scope is refused with 403 + an audit event. With
    multiple service credentials stacked, one credential can no longer open another surface's routes — the drain secret
    cannot drive the kanban API and vice versa.
  • A new bundled dashboard_auth plugin (mirroring the drain plugin) verifies Authorization: Bearer $HERMES_KANBAN_API_SECRET with a constant-time compare and vouches for a kanban-scoped principal. A
    weak/short/low-entropy secret (< 256 bits) fails closed at load; the shared entropy gate now lives in
    dashboard_auth.secret_strength and is reused by both plugins.
  • The interactive dashboard subtree (see below) is excluded from the token seam and stays on cookie/session auth.
    Without the secret set, the plugin is a no-op and prior behavior is unchanged.

Idempotency (concurrency-safe)

POST /tasks accepts an Idempotency-Key header or idempotency_key body field. A repeat returns the existing live
task with HTTP 200 and created: false; a fresh insert returns HTTP 201 and created: true.

The guarantee holds under concurrency and is enforced at the storage layer:

  • A partial UNIQUE index on idempotency_key (live, non-archived tasks; legacy duplicates deduped during
    migration) closes the check-then-insert race. Archiving a task frees its key.
  • create_task_idempotent resolves a lost race to the winner's row instead of surfacing IntegrityError — every
    caller (CLI, tools, dispatcher, REST) gets one task id, satisfying the multi-process stress contract
    (idempotency_key_race).

Sanitization

Task and run DTOs expose workflow state plus routing attribution — assignee, created_by (which profile/surface
created the card), and the executing profile per run — and deliberately omit: task bodies and results, comment
text, workspace paths, branch names, claim locks, worker PIDs, session IDs, idempotency keys, run
summaries/metadata/errors, and raw event payloads.

GET /profiles returns only name, description, and has_description — no models, providers, paths, env, or
skill inventories. GET /tasks/{id}/log returns a bounded excerpt (8 KiB default, 32 KiB max) after Hermes secret
redaction, Authorization-header scrubbing, and absolute-path replacement with [PATH].

Error details follow an allowlist: known-safe validation messages pass through verbatim; anything else collapses to a
stable generic detail with the raw error logged server-side only.

Dashboard compatibility

The first-party dashboard needs a richer API than should be exposed externally, so its existing routes are namespaced
under /api/plugins/kanban/dashboard (bundled dashboard JS updated). Both routers share the same kanban_db
implementation. Third-party consumers of the previously undocumented dashboard routes should migrate to the sanitized
API or adopt the new namespace.

Documentation

website/docs/user-guide/features/kanban-rest-api.md — credential provisioning, the full endpoint contract,
idempotency semantics, and a worked example (parent operation → dependent children → dependency links → completion →
polling sanitized events/runs/logs).

Testing

  • Affected suites (REST adapter, kanban store, token-auth seam, auth middleware, both auth plugins):

    python -m pytest tests/hermes_cli/test_kanban_api.py tests/hermes_cli/test_kanban_db.py \                          
      tests/hermes_cli/test_dashboard_token_auth.py tests/hermes_cli/test_dashboard_auth_middleware.py \               
      tests/plugins/dashboard_auth/ -q                                                                                 
    # 509 passed                                                                                                       
                                                                                                                       
  • Auth is covered end-to-end through the real mounted dashboard app in gated mode: valid bearer drives the external
    surface, missing/wrong token → 401, a foreign-scoped service credential → 403, and the dashboard subtree never
    accepts the bearer.

  • Idempotency race coverage: a deterministic lost-race regression at the storage layer, a threaded concurrent-create
    regression, and the multi-process stress scenario. Exercised live as well: 6 concurrent POST /tasks with one key →
    exactly one 201 + five 200s, one row persisted.

  • Full atypical stress suite on this branch:

python tests/stress/test_atypical_scenarios.py

28 scenarios, 0 failures, 0 skips

▎ Note: one commit (test(stress): catch atypical scenarios up to current kanban contracts) repairs two stress
▎ scenarios that fail on current main independently of this PR — they had gone stale against the spawn_failures →
▎ consecutive_failures rename (#20410) and the archived-parents-are-terminal dependency fix. Included so the suite is
▎ green on this branch; happy to split it into its own PR.

Out of scope

Profile execution or configuration endpoints, raw worker/session state, product-specific workflow types, and any
second Kanban schema or storage layer.

Follow-ups

  • Cursor pagination for large boards
  • Finer-grained RBAC within the kanban scope (e.g. read-only credentials)
  • Explicit API versioning if additional external contracts are introduced

@alt-glitch alt-glitch added type/feature New feature or request comp/cron Cron scheduler and job management comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have labels Jul 10, 2026
andriy4k07 and others added 5 commits July 11, 2026 10:22
… index

The REST adapter's task creation did a SELECT-before-INSERT on
idempotency_key with no uniqueness constraint, so two concurrent POSTs
with the same key both passed the check and created duplicate tasks.

Add a partial UNIQUE index on tasks(idempotency_key) scoped to live
(non-archived) rows — matching exactly how create_task looks the key up
(WHERE idempotency_key = ? AND status != 'archived'). Boards are separate
SQLite files, so this gives board-scoped idempotency. Archiving a task
still frees its key for reuse.

The migration runs in the existing connect()/_migrate_add_optional_columns
path so legacy DBs upgrade on open: it first nulls out any duplicate live
keys (keeping the row the lookup would return) so the index can build, then
drops the old non-unique idx_tasks_idempotency and creates the partial
UNIQUE index. It uses plain statements (the CREATE UNIQUE INDEX DDL commits
the dedupe UPDATE) to avoid nesting inside the migration's implicit txn.

The adapter now catches the resulting IntegrityError on insert and returns
the winning task with created=false / HTTP 200, preserving the documented
idempotency semantics. Also removes the always-true `created = existing is
None` dead code (the existing case returns earlier).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The adapter reached into private kanban_db internals (_normalize_board_slug,
_append_event) and ran a raw UPDATE against the tasks table from the API
layer. Promote a public normalize_board_slug() wrapper and add a public
edit_task_fields() domain function that owns the title/body/priority UPDATE
and the "edited" event, so the adapter imports only public API.

edit_task_fields() also enforces state-machine honesty: editing the title or
body of a completed (done) or archived task is refused (RuntimeError ->
HTTP 409), consistent with the other transition guards. priority stays
editable. The adapter checks the terminal state up front so a mixed
assignee+title PATCH doesn't partially apply.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ections

list_boards opened two SQLite connections per board (init_db then connect)
just to count tasks; connect() already runs the schema/migration pass on
first open, so _board_counts now uses a single connect_closing() per board.

The events and runs endpoints loaded a task's full history and sliced
[-limit:] in memory. Add a `limit` parameter to list_events()/list_runs()
that applies the LIMIT in SQL (newest N by time, re-sorted oldest-first) so
long histories don't materialise every row.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…own link

Three response-hardening fixes to the REST adapter:

- kanban_db ValueError messages went straight into HTTP `detail`, which
  could leak internal specifics. Introduce _client_error(): it logs the raw
  exception server-side and returns a stable message — a known-safe
  validation string when recognised, otherwise a generic fallback. Applied
  to board-slug, list-filter, create, comment, and block errors.

- _AUTH_HEADER_RE only matched `Authorization:` at the start of a line
  (`^` multiline), so a token inside a dumped curl command survived
  redaction. Match the header anywhere in a line.

- link_tasks now _require_task's both parent and child, so linking a
  nonexistent task returns 404 (consistent with unlink) instead of a 400
  from kanban_db's ValueError. Genuine invalid links (self/cycle) stay 400.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… changes

Extend the adapter test suite: idempotency uniqueness under a duplicate live
insert (and archived-key reuse), the API returning the winner on an
IntegrityError race, link 404s for missing parent/child, PATCH 409 on
completed and archived tasks, events/runs limit correctness (newest N,
oldest-first), and sanitized vs. safe error detail.

Document the two deliberate contract improvements (link 404 instead of 400;
PATCH 409 on completed/archived) and the now-real idempotency guarantee
(partial UNIQUE index; concurrent duplicates resolve to one task).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for separating a narrow DTO-based contract from the rich operator dashboard surface. The premise is valid: current plugins/kanban/dashboard/plugin_api.py:158-176 serializes the complete task dataclass, and current main has no sanitized adapter.

Problems

  • website/docs/user-guide/features/kanban-rest-api.md:20-25 documents a bearer credential for external controllers, but current main accepts Bearer only when it is the ephemeral dashboard _SESSION_TOKEN (hermes_cli/web_server.py:319-336). On gated deployments, routes require a verified cookie session (hermes_cli/web_server.py:361-369). Generic service bearer auth is limited to exact registered paths (hermes_cli/dashboard_auth/token_auth.py:54-74,144-164), and this change registers none.
  • The advertised idempotency is not concurrency-safe. hermes_cli/kanban_db.py:2539-2552 checks before BEGIN IMMEDIATE at :2580, and idx_tasks_idempotency is non-unique (:1996-1999). The existing stress contract requires concurrent creates with one key to return one id and persist one row (tests/stress/test_atypical_scenarios.py:712-751).

Suggested changes

  • Wire a real service-auth path and test it through the mounted dashboard app, or scope the documentation to dashboard-session clients.
  • Make keyed creation atomic at the storage layer and add a concurrent-create regression.

Automated hermes-sweeper review.

returned. The log endpoint returns only a bounded excerpt after Hermes secret
redaction and absolute-path removal.

Use the same authentication required by the Hermes dashboard deployment. In

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This bearer flow is not available to an external controller on a gated dashboard: current main only accepts Bearer <_SESSION_TOKEN> in loopback mode (hermes_cli/web_server.py:319-336), while gated mode requires a cookie session. The generic service-token seam also only applies to exact registered paths (hermes_cli/dashboard_auth/token_auth.py:54-74), and this router registers none. Please add a real service-auth integration and E2E coverage, or change the contract/docs to dashboard-session authentication.

@teknium1 teknium1 added sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 11, 2026
The partial UNIQUE index made a lost create race surface as
sqlite3.IntegrityError: the retry loop only generated a fresh task id,
so the same idempotency_key failed again and the caller crashed —
violating the stress contract (idempotency_key_race) that concurrent
creators with one key must all get the winner's id.

Move the recovery into the storage layer: create_task_idempotent now
owns the full contract (fast-path lookup, insert, and IntegrityError
resolution to the live winner) and returns (task_id, created);
create_task stays a same-signature wrapper returning the id. The REST
adapter drops its duplicated lookup + IntegrityError handling and maps
created=False to HTTP 200.

Covered by a deterministic lost-race regression (winner injected via
the _new_task_id seam), a 4-thread concurrent-create regression, and
the existing stress scenario.
…n-auth seam

Exact-path registration cannot cover routers with parameterised paths,
so the seam gains register_token_route_prefix(prefix, exclude=...) with
excluded subtrees (an external plugin surface can be token-authable
while its interactive dashboard subtree stays on the cookie gate).

Registrations may now demand a required scope: a verified principal
lacking it is refused with 403 + an audit event. With a second service
credential joining the stack this is what keeps one credential (e.g.
the drain secret) from opening another surface's routes. The drain
plugin passes its configured scope through; unscoped registrations
keep the old accept-any-verified-principal behaviour.

Also extracts the shared secret entropy gate out of the drain plugin
into dashboard_auth.secret_strength (drain re-exports for compat).
New dashboard_auth plugin (mirroring the drain plugin) that gives
external control planes a real authentication path to the sanitized
kanban REST adapter — previously the documented bearer flow did not
exist: loopback only accepted the ephemeral SPA session token and
gated binds required a cookie session.

The operator provisions HERMES_KANBAN_API_SECRET (>=256-bit,
fail-closed entropy gate); the plugin registers a kanban-scoped token
provider plus the /api/plugins/kanban/ prefix in the token-auth seam,
excluding the interactive /dashboard subtree which stays on
cookie/session auth. E2E-covered through the mounted dashboard app in
gated mode: bearer drives the external surface, missing/wrong token is
401, the drain credential is refused with 403, and the dashboard
subtree never accepts the bearer.
Replace the fictional HERMES_DASHBOARD_TOKEN bearer flow with the real
contract: provisioning the service secret, its kanban-only scope, the
excluded interactive dashboard subtree, and the fallback/token-only
semantics with and without the secret.
…the REST API

Gives an external control plane what it needs to visualise and drive
orchestrator fan-out without touching the internal dashboard API:

- GET /profiles — sanitized assignee roster: name, description (the
  operator-facing profile.yaml text the built-in decomposer routes on),
  and has_description. Models, providers, filesystem paths, env/config
  state, and skill inventories stay private; 503 with a generic detail
  when listing fails. capabilities now advertises profiles_api: true
  (profile_execution stays false — routing happens via task assignment
  only).
- Task DTO gains created_by, so a consumer can attribute which
  profile/surface created each card, not just who executes it.
- Run entries gain profile — the executor of that specific attempt,
  which diverges from task.assignee when a task is reassigned between
  retries. The sanitization contract test now pins the widened run
  shape while still refusing summary/metadata/error leakage.

The new endpoint sits under the token-guarded /api/plugins/kanban/
prefix, so the service-credential scope rules apply unchanged
(E2E-covered through the mounted app).
Two scenarios in the atypical stress suite were stale against
deliberate kernel changes and failed on every run:

- workspace_nonexistent_path read task.spawn_failures, a field renamed
  to consecutive_failures when NousResearch#20410 unified the circuit-breaker
  counter across spawn/timeout/crash outcomes. It also never actually
  reached the workspace-failure path anymore: since the dispatcher
  crash-loop fix (2026-05-05) tasks whose assignee is not a real
  profile on disk are skipped before claiming, so the scenario ran
  zero of its assertions. The scenario now creates the profile dir
  first and asserts on consecutive_failures — the counter provably
  ticks to 1 on a nonexistent dir: workspace.

- parent_in_different_status_states expected a child of an archived
  parent to stay in todo, contradicting the documented recompute_ready
  contract ('done' OR 'archived' parents unblock children) introduced
  by the "treat archived parent tasks as terminal for dependency
  resolution" fix. The expectation now matches the kernel: archived,
  like done, is terminal.

Full suite: 28 scenarios, 0 failures, 0 skips.
@andriy4k07

andriy4k07 commented Jul 14, 2026

Copy link
Copy Markdown
Author

Thanks — both findings were valid. Addressed in four commits:

  1. Bearer auth for external controllers did not actually exist (docs advertised a credential no code path accepted)

Wired a real service-auth path instead of scoping the docs down:

  • feat(dashboard-auth): prefix routes and scope enforcement in the token-auth seam — the generic seam now supports
    register_token_route_prefix(prefix, scope=..., exclude=...), since exact-path registration can't cover
    parameterised routes like /tasks/{id}. Registrations may demand a required scope: a verified principal lacking it
    gets 403 + an audit event. This is what keeps one stacked service credential from opening another surface's
    routes — the drain plugin now passes its scope through as well.
  • feat(kanban): service bearer credential for the external REST API — a new dashboard_auth plugin mirroring the
    drain plugin: the operator provisions HERMES_KANBAN_API_SECRET (>=256-bit, fail-closed entropy gate shared with
    drain via the new dashboard_auth.secret_strength), and the plugin registers a kanban-scoped token provider plus
    the /api/plugins/kanban/ prefix — excluding the interactive /dashboard subtree, which stays on the cookie/session
    gate.
  • Docs rewritten around the real contract (HERMES_KANBAN_API_SECRET), including the fail-closed consequence: with
    the secret set the external surface is token-only on every bind.

E2E coverage goes through the real mounted dashboard app in gated mode (web_server.app + TestClient,
auth_required=True, providers registered through the plugins' own register()): valid bearer drives the external
surface, missing/wrong token → 401, the drain credential on kanban routes → 403 (and vice versa), and the kanban
bearer never opens the dashboard subtree. Also exercised live against a running server: 401/403/200 matrix plus the
documented workflow (links, complete, events, sanitized log).

  1. Idempotency was not concurrency-safe at the storage layer

Correct — the partial UNIQUE index made a lost race surface as sqlite3.IntegrityError: the retry loop only
regenerated the task id, so the same key failed again and the caller crashed, violating the idempotency_key_race
stress contract.

  • fix(kanban): resolve idempotency-key races inside create_task — recovery now lives in the storage layer.
    create_task_idempotent(...) -> (task_id, created) owns the full contract: fast-path lookup, insert, and
    IntegrityError resolution to the live winner (same created_at DESC, id DESC tiebreak as the migration's survivor
    choice). create_task remains a same-signature wrapper, so no existing caller changes. The REST adapter drops its
    duplicated lookup + IntegrityError handling and just maps created=False → HTTP 200.
  • Regressions added at the storage layer: a deterministic lost-race test (the concurrent winner is injected between
    the fast-path lookup and the INSERT via the _new_task_id seam) and a 4-thread concurrent-create test (one id, one
    row). The existing multi-process stress scenario passes.

Verified live as well: 6 concurrent POST /tasks with one Idempotency-Key against a running server → exactly one
201 and five 200s, all six responses carrying the same task id, one row persisted.

@andriy4k07

Copy link
Copy Markdown
Author

Note on the stress-suite commit (test(stress): catch atypical scenarios up to current kanban contracts)

While verifying this PR against tests/stress/test_atypical_scenarios.py, I found two scenarios that fail on current
main as well — they are pre-existing and unrelated to this PR, but since reviewers run the stress suite against
this branch, I've included the fix here so the suite is fully green (28 scenarios, 0 failures, 0 skips).

Both are cases of the stress suite going stale against deliberate kernel changes from May 2026:

  • workspace_nonexistent_path read task.spawn_failures, which fix(kanban): unify failure counter across spawn/timeout/crash outcomes #20410 renamed to consecutive_failures when
    the circuit-breaker counter was unified across spawn/timeout/crash outcomes (with a column migration). The scenario
    had also silently stopped testing anything: since the dispatcher crash-loop fix, tasks whose assignee is not a real
    profile on disk are skipped before claiming, so the workspace-failure path was never reached. The scenario now
    creates the profile dir first and asserts the counter actually ticks.
  • parent_in_different_status_states expected a child of an archived parent to stay in todo, contradicting
    the documented recompute_ready contract — the "treat archived parent tasks as terminal for dependency resolution"
    fix deliberately made archived terminal like done, so a cancelled parent can't wedge its children forever. The
    expectation now matches the kernel.

Happy to split this commit out into its own PR if you'd rather land the test repair independently — the failures
reproduce on main without any of this PR's changes.

@andriy4k07
andriy4k07 requested a review from teknium1 July 14, 2026 12:22
Resolves four collisions with a month of upstream drift:

- kanban_db.create_task_idempotent: keep the (task_id, created) return
  while picking up the new project_source_task_id parameter and the
  _inherit_notify_subs call upstream added.
- kanban_db.create_task: forward model_override, provider_override,
  reasoning_effort and project_source_task_id. The thin wrapper was
  introduced here after those parameters landed upstream, so the
  dashboard plugin_api call site broke on the merged tree.
- kanban_db: keep edit_task_fields alongside the set_model_override /
  set_reasoning_effort pair upstream inserted at the same spot.
- test_dashboard_token_auth: drop the two seam tests upstream pruned
  intentionally, keep the new prefix/scope coverage.
- kanban.md: keep the REST API pointer on top of the widened toolset list.
_connection called init_db() before connect(), which evicts the path from
_INITIALIZED_PATHS and so forces connect() down its slow path on every
single request: header validation, the full integrity probe, the schema
executescript and the whole optional-column migration, all behind the
cross-process init lock — plus a second connection for the actual work.

connect() already creates the parent directory and auto-runs schema and
migrations on a path's first open, which is why _board_counts dropped the
same redundant init_db call. Route the task endpoints through
connect_closing for the same reason: this is a polling surface for
external control planes, and it should not pay a migration per GET.
website/sidebars.ts enumerates docs explicitly, so the new page was
reachable only through the inline link from the Kanban page.
contributor-check resolves every non-merge commit author against this map
and fails the run on an unmapped, non-noreply address.
plugin_api.router is now the composed router: the sanitized external
surface at the root plus the operator dashboard under /dashboard. Tests
that mount it on a bare app and call /api/plugins/kanban/<route> get a
404 for every dashboard route.

The same three-line change was already applied to the dashboard tests
that existed when this branch was cut; these four files landed upstream
in the meantime and need it too.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cron Cron scheduler and job management comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants