Add AI Counsel driver (3-stage deliberation via SSE) - #22
Open
tonytouch wants to merge 2 commits into
Open
Conversation
added 2 commits
August 12, 2026 13:02
Wraps Tony's Agent OS hub (tonysplace_best/backend/agent-os :8001) as a ProviderDriver for OpenMausBot. The hub already speaks the OpenAI- compatible surface (/v1/models, /v1/chat/completions with SSE), so the driver is HTTP-based rather than subprocess-based. What works: - canonical RuntimeEvent sequence: session.started -> turn.started -> content.delta (streamed) -> item.completed -> turn.completed -> session.exited - snapshot health via /v1/models probe - auth: forwards Authorization: Bearer <apiKey> when configured - 5-min per-turn timeout (configurable) - 8 unit tests + 1 end-to-end smoke script - Model catalog mirrors the 9 hermes-os seats (council, gemini, openai, anthropic, opencode, local_claude, local_codex, mavis, nous) Known limitations: - peer-agent comms surfaced as event.note only (hub doesn't have an MCP proxy for list_bots/ask_bot yet) - cloud-computer (Box) integration is logged and ignored (hub doesn't have computer-use yet) - local computer-use (cua-driver) is logged and ignored (hub doesn't have local computer-use yet) Ref: council-unification.md (Appendix A) and council_contracts.py (Python port of server/contracts.ts for the hermes-os side).
Wraps The AI Counsel (jacob-bd/the-ai-counsel, MIT) as a ProviderDriver
for OpenMausBot. The Counsel is a 3-stage LLM deliberation system
modeled on the karpathy LLM Council paper:
stage 1: each council model answers independently (parallel)
stage 2: each model peer-reviews the others (anonymized by label)
stage 3: a chairman model synthesizes the final answer
The driver always uses execution_mode: 'full' so the OpenMausBot
caller gets a synthesized final answer, not N unrelated first
passes. Per-stage events are mapped onto the canonical RuntimeEvent
union so the same fleet UI works for both:
Counsel SSE event -> RuntimeEvent
─────────────────────────────────────────
error -> runtime.error
stage{1,2}_progress -> item.completed: assistant_text (per model)
stage3_complete -> item.completed: assistant_text (synthesis)
complete -> turn.completed(ok: true)
cost_report -> thread.token-usage.updated
Each sendTurn creates a new Counsel conversation; the threadId is
echoed back in session.started so the UI can correlate. The driver
uses Ollama (ollama:hermes3:8b) as the default seat for the smoke
test since it's free and always-available on a homelab. The catalog
also exposes the local-agents:* and OpenRouter seats the Counsel
already supports; the user picks a model and the driver threads
it through as the sole council member.
Why the SPI fit: like the hermes-os driver (milind-soni#18), the Counsel
exposes a real SSE stream with structured event types. Adopting
the RuntimeEvent union means a single fleet UI in OpenMausBot
sees both drivers identically — turns look like turns, item
events like item events, usage like usage. No new shape design
needed.
Snapshot: a quick GET /api/conversations to confirm reachability
(the /api/health route 404s in the current Counsel build, so we
use the conversations list as the health probe instead).
Tests: 9 unit tests covering: default config, config decoding,
unreachable / reachable snapshot, full canonical event sequence
for a 3-stage round, runtime error from preflight, runtime error
from conversation create failure, and the model catalog contents.
End-to-end smoke: scripts/smoke-ai-counsel.ts runs against a live
Counsel on :8020 with Ollama. Verified the per-stage items are
emitted with correct text, the synthesis is the last item, and
turn.completed fires with ok=true. Total round on Ollama
hermes3:8b: ~10s, 606 input + 92 output tokens.
No new runtime dependencies. No dist-server churn. No
pnpm-lock.yaml churn. Compatible with the hermes-os driver
already merged as milind-soni#18.
milind-soni
requested changes
Aug 14, 2026
milind-soni
left a comment
Owner
There was a problem hiding this comment.
This PR is stacked on #18 and currently includes the full Hermes driver that now has requested changes. Please fix and merge #18 first, then rebase this branch so the diff contains only AI Counsel. In the rebased adapter, preserve the repaired error, timeout, interruption, and session-exit lifecycle; sanitize upstream response bodies; and add tests for cancellation, unreachable and timed-out requests, and the intentionally stateless one-conversation-per-turn behavior.
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
aiCounselas a built-in provider driver.aiCounselwraps The AI Counsel (MIT, by jacob-bd) — a FastAPI service that runs a 3-stage LLM deliberation modeled on the karpathy LLM Council paper:The driver always uses
execution_mode: "full"so OpenMausBot gets a synthesized final answer, not N unrelated first passes.Why
Same as the hermes-os driver (#18): the Council exposes a real SSE stream with structured event types that map cleanly onto the canonical
RuntimeEventunion. A single fleet UI in OpenMausBot sees both drivers identically — turns look like turns, item events like item events, usage like usage. No new shape design.The AI Counsel is a 3rd deliberation product alongside the hermes-os
councilmodel and the Ultimate Hub council — three different "ask everyone" implementations, all on the same homelab. This driver makes the 3rd one reachable from the Mac app with zero new code on the Council side.What works
session.started→turn.started→ per-stageitem.completed: assistant_text(one per model) →turn.completed(ok: true)→session.exited. All five token-usage events are emitted.stage1_progress→ item with text from the first modelstage2_progress→ item with the peer-review text (or the parsed JSON ranking)stage3_complete→ item with the chairman's synthesislocal-agents:*seats (Mac CLIs over SSH) andollama:*/openai/*/anthropic/*/x-ai/*/google/*/opencode/*seats already configured in the Counsel'ssettings.json.GET /api/conversationsas the health probe (the Counsel's/api/healthroute 404s in the current build).runtime.error+turn.completed(ok: false)instead of hanging.Tests
9 unit tests in
server/drivers/ai-counsel.test.ts— same pattern ashermes-os.test.ts: spin up a mock HTTP server that mimics the Counsel's/api/conversations+ SSE response, exercise the driver, assert the event sequence. Covers: default config, decodeConfig edge cases, unavailable snapshot, available snapshot, full canonical sequence for a 3-stage round, preflight error, conversation-create failure, model catalog contents.End-to-end smoke (
scripts/smoke-ai-counsel.ts) against the live Counsel on the homelab with Ollama:~10s for a full 3-stage round on Ollama. Real model responses, real peer review, real synthesis, real event reconstruction.
Known limitations
sendTurncreates a new Counsel conversation. The OpenMausBotthreadIdis not a CounselconversationIdand there's no resume-cursor story that bridges them. If you want multi-turn within the same Council conversation, the right path is aconversationIdconfig field (deferred — the simpler one-conversation-per-turn model works fine for "ask the council a question").settings.json. The driver passescouncil_models: [model]to control which models run stage 1, but the stage-3 chairman is a server-side config. If you want to override it, setbody.chairman_model = "..."in the driver (currently best-effort, no explicit config flag yet).item.completed: assistant_textper stage. Adding token-level streaming would require upstream changes to the Counsel itself.Files
server/drivers/ai-counsel.ts— 21 KB, the driverserver/drivers/ai-counsel.test.ts— 10 KB, 9 unit testsserver/drivers/builtIn.ts— 2-line change (import + register)scripts/smoke-ai-counsel.ts— 5 KB, end-to-end smokeNo new runtime dependencies. No
dist-server/churn. Nopnpm-lock.yamlchurn. Driver uses only built-ins (fetch,TextDecoder,AbortController,AbortSignal). Pattern is symmetric with the hermes-os driver merged in #18.Checklist
pnpm typecheckandpnpm testpassdecodeConfigthrows on invalid;createrejects (async, never throws sync)RuntimeEvents carryingdriverKind: "aiCounsel"snapshot() → { state: "unavailable", reason }runtime.error+turn.completed(ok: false)(never hang, never crash)shell: true, no POSIX-only calls, no new runtime depsOpen question for the maintainer: do you want the OpenMausBot instance config UI to surface a "create a Counsel conversation" button (which would mean storing the
conversationIdper-thread), or is the simpler "one new conversation per turn" model good enough for v1? Happy to extend either way — just want to know before merging.