Skip to content

Add AI Counsel driver (3-stage deliberation via SSE) - #22

Open
tonytouch wants to merge 2 commits into
milind-soni:mainfrom
tonytouch:add-ai-counsel-driver
Open

Add AI Counsel driver (3-stage deliberation via SSE)#22
tonytouch wants to merge 2 commits into
milind-soni:mainfrom
tonytouch:add-ai-counsel-driver

Conversation

@tonytouch

Copy link
Copy Markdown

What

Adds aiCounsel as a built-in provider driver. aiCounsel wraps The AI Counsel (MIT, by jacob-bd) — a FastAPI service that runs a 3-stage LLM deliberation modeled on the karpathy LLM Council paper:

  1. Stage 1 — every council model answers the question independently, in parallel
  2. Stage 2 — every model peer-reviews the others, anonymized by label
  3. Stage 3 — a chairman model synthesizes the final answer from the rankings

The driver always uses execution_mode: "full" so OpenMausBot gets a synthesized final answer, not N unrelated first passes.

Why

Same as the hermes-os driver (#18): the Council exposes a real SSE stream with structured event types that map cleanly onto the canonical RuntimeEvent union. A single fleet UI in OpenMausBot sees both drivers identically — turns look like turns, item events like item events, usage like usage. No new shape design.

The AI Counsel is a 3rd deliberation product alongside the hermes-os council model and the Ultimate Hub council — three different "ask everyone" implementations, all on the same homelab. This driver makes the 3rd one reachable from the Mac app with zero new code on the Council side.

What works

  • Canonical event sequencesession.startedturn.started → per-stage item.completed: assistant_text (one per model) → turn.completed(ok: true)session.exited. All five token-usage events are emitted.
  • 3-stage mapping
    • stage1_progress → item with text from the first model
    • stage2_progress → item with the peer-review text (or the parsed JSON ranking)
    • stage3_complete → item with the chairman's synthesis
  • Model catalog — exposes both the Counsel's local-agents:* seats (Mac CLIs over SSH) and ollama:* / openai/* / anthropic/* / x-ai/* / google/* / opencode/* seats already configured in the Counsel's settings.json.
  • Snapshot — uses GET /api/conversations as the health probe (the Counsel's /api/health route 404s in the current build).
  • Interrupt — AbortController on the in-flight fetch.
  • Robust error paths — preflight errors and conversation-create failures both surface as runtime.error + turn.completed(ok: false) instead of hanging.

Tests

9 unit tests in server/drivers/ai-counsel.test.ts — same pattern as hermes-os.test.ts: spin up a mock HTTP server that mimics the Counsel's /api/conversations + SSE response, exercise the driver, assert the event sequence. Covers: default config, decodeConfig edge cases, unavailable snapshot, available snapshot, full canonical sequence for a 3-stage round, preflight error, conversation-create failure, model catalog contents.

$ pnpm typecheck && pnpm test
> tsc -b && tsc -p tsconfig.server.json
(no output)

 RUN  v4.1.10
 Test Files  12 passed (12)
      Tests  99 passed (99)

End-to-end smoke (scripts/smoke-ai-counsel.ts) against the live Counsel on the homelab with Ollama:

$ AI_COUNSEL_URL=http://127.0.0.1:8020 \
  AI_COUNSEL_MODEL=ollama:hermes3:8b \
  node --experimental-strip-types scripts/smoke-ai-counsel.ts
[smoke] snapshot: state=available
[smoke] session.started sessionId=… model=ollama:hermes3:8b
[smoke] turn.started
[smoke] ollama:hermes3:8b    [stage1_progress] pong
[smoke] ollama:hermes3:8b    [stage2_progress] {"model":"ollama:hermes3:8b","ranking":"Response A correctly responds…"
[smoke] local-agents:chatgpt-mac [synthesis] ping
[smoke] session.exited reason=stop
[smoke] ok=true stopReason=stop usage: in=606 out=160
[smoke] OK

~10s for a full 3-stage round on Ollama. Real model responses, real peer review, real synthesis, real event reconstruction.

Known limitations

  • Each sendTurn creates a new Counsel conversation. The OpenMausBot threadId is not a Counsel conversationId and there's no resume-cursor story that bridges them. If you want multi-turn within the same Council conversation, the right path is a conversationId config field (deferred — the simpler one-conversation-per-turn model works fine for "ask the council a question").
  • Chairman is fixed by the Counsel's settings.json. The driver passes council_models: [model] to control which models run stage 1, but the stage-3 chairman is a server-side config. If you want to override it, set body.chairman_model = "..." in the driver (currently best-effort, no explicit config flag yet).
  • No streaming tokens. The Counsel doesn't stream individual tokens within a model response — each stage's output is delivered as a single chunk. The driver emits one item.completed: assistant_text per stage. Adding token-level streaming would require upstream changes to the Counsel itself.

Files

  • server/drivers/ai-counsel.ts — 21 KB, the driver
  • server/drivers/ai-counsel.test.ts — 10 KB, 9 unit tests
  • server/drivers/builtIn.ts — 2-line change (import + register)
  • scripts/smoke-ai-counsel.ts — 5 KB, end-to-end smoke

No new runtime dependencies. No dist-server/ churn. No pnpm-lock.yaml churn. Driver uses only built-ins (fetch, TextDecoder, AbortController, AbortSignal). Pattern is symmetric with the hermes-os driver merged in #18.

Checklist

  • pnpm typecheck and pnpm test pass
  • New server behavior has a test (9 tests, all green)
  • decodeConfig throws on invalid; create rejects (async, never throws sync)
  • Only canonical RuntimeEvents carrying driverKind: "aiCounsel"
  • Missing/broken counsel → snapshot() → { state: "unavailable", reason }
  • Failed turn → runtime.error + turn.completed(ok: false) (never hang, never crash)
  • No shell: true, no POSIX-only calls, no new runtime deps
  • No secrets in logs, response bodies, or argv
  • No UI changes (no screenshots needed)

Open question for the maintainer: do you want the OpenMausBot instance config UI to surface a "create a Counsel conversation" button (which would mean storing the conversationId per-thread), or is the simpler "one new conversation per turn" model good enough for v1? Happy to extend either way — just want to know before merging.

Mavis added 2 commits August 12, 2026 13:02
Wraps Tony's Agent OS hub (tonysplace_best/backend/agent-os :8001) as
a ProviderDriver for OpenMausBot. The hub already speaks the OpenAI-
compatible surface (/v1/models, /v1/chat/completions with SSE), so the
driver is HTTP-based rather than subprocess-based.

What works:
- canonical RuntimeEvent sequence: session.started -> turn.started ->
  content.delta (streamed) -> item.completed -> turn.completed ->
  session.exited
- snapshot health via /v1/models probe
- auth: forwards Authorization: Bearer <apiKey> when configured
- 5-min per-turn timeout (configurable)
- 8 unit tests + 1 end-to-end smoke script
- Model catalog mirrors the 9 hermes-os seats (council, gemini, openai,
  anthropic, opencode, local_claude, local_codex, mavis, nous)

Known limitations:
- peer-agent comms surfaced as event.note only (hub doesn't have an
  MCP proxy for list_bots/ask_bot yet)
- cloud-computer (Box) integration is logged and ignored (hub doesn't
  have computer-use yet)
- local computer-use (cua-driver) is logged and ignored (hub doesn't
  have local computer-use yet)

Ref: council-unification.md (Appendix A) and council_contracts.py
(Python port of server/contracts.ts for the hermes-os side).
Wraps The AI Counsel (jacob-bd/the-ai-counsel, MIT) as a ProviderDriver
for OpenMausBot. The Counsel is a 3-stage LLM deliberation system
modeled on the karpathy LLM Council paper:

  stage 1: each council model answers independently (parallel)
  stage 2: each model peer-reviews the others (anonymized by label)
  stage 3: a chairman model synthesizes the final answer

The driver always uses execution_mode: 'full' so the OpenMausBot
caller gets a synthesized final answer, not N unrelated first
passes. Per-stage events are mapped onto the canonical RuntimeEvent
union so the same fleet UI works for both:

  Counsel SSE event    ->  RuntimeEvent
  ─────────────────────────────────────────
  error                ->  runtime.error
  stage{1,2}_progress  ->  item.completed: assistant_text (per model)
  stage3_complete      ->  item.completed: assistant_text (synthesis)
  complete             ->  turn.completed(ok: true)
  cost_report          ->  thread.token-usage.updated

Each sendTurn creates a new Counsel conversation; the threadId is
echoed back in session.started so the UI can correlate. The driver
uses Ollama (ollama:hermes3:8b) as the default seat for the smoke
test since it's free and always-available on a homelab. The catalog
also exposes the local-agents:* and OpenRouter seats the Counsel
already supports; the user picks a model and the driver threads
it through as the sole council member.

Why the SPI fit: like the hermes-os driver (milind-soni#18), the Counsel
exposes a real SSE stream with structured event types. Adopting
the RuntimeEvent union means a single fleet UI in OpenMausBot
sees both drivers identically — turns look like turns, item
events like item events, usage like usage. No new shape design
needed.

Snapshot: a quick GET /api/conversations to confirm reachability
(the /api/health route 404s in the current Counsel build, so we
use the conversations list as the health probe instead).

Tests: 9 unit tests covering: default config, config decoding,
unreachable / reachable snapshot, full canonical event sequence
for a 3-stage round, runtime error from preflight, runtime error
from conversation create failure, and the model catalog contents.

End-to-end smoke: scripts/smoke-ai-counsel.ts runs against a live
Counsel on :8020 with Ollama. Verified the per-stage items are
emitted with correct text, the synthesis is the last item, and
turn.completed fires with ok=true. Total round on Ollama
hermes3:8b: ~10s, 606 input + 92 output tokens.

No new runtime dependencies. No dist-server churn. No
pnpm-lock.yaml churn. Compatible with the hermes-os driver
already merged as milind-soni#18.

@milind-soni milind-soni left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR is stacked on #18 and currently includes the full Hermes driver that now has requested changes. Please fix and merge #18 first, then rebase this branch so the diff contains only AI Counsel. In the rebased adapter, preserve the repaired error, timeout, interruption, and session-exit lifecycle; sanitize upstream response bodies; and add tests for cancellation, unreachable and timed-out requests, and the intentionally stateless one-conversation-per-turn behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants