feat: single gateway, multiple agents (MVP) - #25660
Conversation
alt-glitch
added
type/feature
4d8a642 to
b856e06
Compare
|
Tracked follow-up technical debt from this PR:
|
CI Test Failure AnalysisThe Verified locallyAll tests related to this PR pass locally (313+ tests):
CI failure breakdown (all pre-existing)
None of these failures are related to the multi-agent changes introduced in this PR. |
|
@discolotus Thanks for tracking these follow-ups! All four items are already documented in the DESIGN.md file under the "Non-Goals (Future PRs)" section with the same issue numbers you listed. The design doc explicitly scopes them out of this MVP to keep the PR reviewable. |
E2E Test Report — Multi-Agent Routing ValidationWe completed end-to-end validation of the multi-agent routing feature. Here is the summary: Test Matrix
Configuration Useddefault_agent: main
agents:
main: {}
wecom-agent:
home_dir: /root/.hermes/profiles/wecom-agent
code:
model: kimi-for-coding
provider: moonshot
home_dir: /root/.hermes/profiles/code
routes:
- match: { platform: wecom }
agent: wecom-agent
- match: { platform: matrix }
agent: codeKanban Subsystem Impact AnalysisThe Kanban subsystem requires zero code changes. Key findings:
Configuration convention: Kanban task Full details: |
|
@alt-glitch This PR is ready for review. Here's a summary of what's been addressed since the initial submission: Changes since last review
Key design decisions for reviewer attention
Please let me know if you'd like any section expanded or if there are specific areas you'd like me to walk through. |
673123a to
48894d4
Compare
|
Force-pushed: rewrote commit history from 11 commits to 7 focused commits. Line count breakdown by category
Key point: tests + docs together account for 53.6% of the diff. Production code surface (1,422 lines)Only 16 files contain production code changes; the rest are tests, docs, or config:
What was removed vs the previous 11-commit version
Verification
|
48894d4 to
730d92c
Compare
|
Force-pushed (rebased onto latest main + fixed run_agent.py refactor migration). What changed in this pushRebased onto latest main —
Line count breakdown by category
PR status
|
|
Rebased onto latest main (519657a) and resolved conflicts from the run_agent.py refactor. What changed since last push:
Test results after rebase:
Commit breakdown (7 commits):
Ready for review. |
The OpenAI-compatible HTTP adapter was the one inbound surface from PR NousResearch#25660 that never called ``_attach_agent_id`` — every ``/v1/chat/completions``, ``/v1/responses``, and ``/v1/runs`` request fell through to ``default_agent`` regardless of the configured routes, silently undermining the multi-agent guarantee on any deployment that exposes the API server. Add a single routing entry point, ``_resolve_agent_profile``, that: * Reads ``X-Hermes-Chat-Id`` / ``X-Hermes-User-Id`` / ``X-Hermes-Thread-Id`` from the request (sanitised through the same length + control-char caps as the existing ``X-Hermes-Session-Id`` / ``X-Hermes-Session-Key``). * Builds a synthetic ``SessionSource(platform=API_SERVER, …)`` and pipes it through the shared ``_attach_agent_id`` hook so declarative routes *and* the ``select_agent`` plugin hook fire identically to every other adapter. * Looks up the resolved ``agent_id`` in ``self._gateway_ref._agent_registry`` and returns the matching ``AgentProfile`` (or ``None`` for legacy single-agent installs). The three agent-invoking handlers (chat completions, responses, runs) now resolve the profile up front and bind it via ``use_profile`` for the duration of the run. Binding happens twice — once on the asyncio side and once inside the executor thread — because asyncio's default executor does not propagate ContextVars. Behaviour is fully backward compatible: requests with no routing headers (the existing OpenAI-API contract) resolve to ``default_agent``, exactly the current behaviour. New tests in ``tests/gateway/test_api_server_routing.py`` cover: * Header sanitisation (CRLF rejection, length caps, whitespace). * Route resolution: matching, no-header fall-through, unmatched header fall-through, ``platform``-only catch-all, ``user_id`` and ``thread_id`` routes, route-order precedence. * Resilience: missing gateway reference, empty registry. * ContextVar isolation under ``asyncio.gather`` so two concurrent HTTP requests with different chat_ids stay isolated. Refs: PR NousResearch#25660 (single-gateway multi-agent).
The OpenAI-compatible HTTP adapter was the one inbound surface from PR NousResearch#25660 that never called ``_attach_agent_id`` — every ``/v1/chat/completions``, ``/v1/responses``, and ``/v1/runs`` request fell through to ``default_agent`` regardless of the configured routes, silently undermining the multi-agent guarantee on any deployment that exposes the API server. Add a single routing entry point, ``_resolve_agent_profile``, that: * Reads ``X-Hermes-Chat-Id`` / ``X-Hermes-User-Id`` / ``X-Hermes-Thread-Id`` from the request (sanitised through the same length + control-char caps as the existing ``X-Hermes-Session-Id`` / ``X-Hermes-Session-Key``). * Builds a synthetic ``SessionSource(platform=API_SERVER, …)`` and pipes it through the shared ``_attach_agent_id`` hook so declarative routes *and* the ``select_agent`` plugin hook fire identically to every other adapter. * Looks up the resolved ``agent_id`` in ``self._gateway_ref._agent_registry`` and returns the matching ``AgentProfile`` (or ``None`` for legacy single-agent installs). The three agent-invoking handlers (chat completions, responses, runs) now resolve the profile up front and bind it via ``use_profile`` for the duration of the run. Binding happens twice — once on the asyncio side and once inside the executor thread — because asyncio's default executor does not propagate ContextVars. Behaviour is fully backward compatible: requests with no routing headers (the existing OpenAI-API contract) resolve to ``default_agent``, exactly the current behaviour. New tests in ``tests/gateway/test_api_server_routing.py`` cover: * Header sanitisation (CRLF rejection, length caps, whitespace). * Route resolution: matching, no-header fall-through, unmatched header fall-through, ``platform``-only catch-all, ``user_id`` and ``thread_id`` routes, route-order precedence. * Resilience: missing gateway reference, empty registry. * ContextVar isolation under ``asyncio.gather`` so two concurrent HTTP requests with different chat_ids stay isolated. Refs: PR NousResearch#25660 (single-gateway multi-agent).
The OpenAI-compatible HTTP adapter was the one inbound surface from PR NousResearch#25660 that never called ``_attach_agent_id`` — every ``/v1/chat/completions``, ``/v1/responses``, and ``/v1/runs`` request fell through to ``default_agent`` regardless of the configured routes, silently undermining the multi-agent guarantee on any deployment that exposes the API server. Add a single routing entry point, ``_resolve_agent_profile``, that: * Reads ``X-Hermes-Chat-Id`` / ``X-Hermes-User-Id`` / ``X-Hermes-Thread-Id`` from the request (sanitised through the same length + control-char caps as the existing ``X-Hermes-Session-Id`` / ``X-Hermes-Session-Key``). * Builds a synthetic ``SessionSource(platform=API_SERVER, …)`` and pipes it through the shared ``_attach_agent_id`` hook so declarative routes *and* the ``select_agent`` plugin hook fire identically to every other adapter. * Looks up the resolved ``agent_id`` in ``self._gateway_ref._agent_registry`` and returns the matching ``AgentProfile`` (or ``None`` for legacy single-agent installs). The three agent-invoking handlers (chat completions, responses, runs) now resolve the profile up front and bind it via ``use_profile`` for the duration of the run. Binding happens twice — once on the asyncio side and once inside the executor thread — because asyncio's default executor does not propagate ContextVars. Behaviour is fully backward compatible: requests with no routing headers (the existing OpenAI-API contract) resolve to ``default_agent``, exactly the current behaviour. New tests in ``tests/gateway/test_api_server_routing.py`` cover: * Header sanitisation (CRLF rejection, length caps, whitespace). * Route resolution: matching, no-header fall-through, unmatched header fall-through, ``platform``-only catch-all, ``user_id`` and ``thread_id`` routes, route-order precedence. * Resilience: missing gateway reference, empty registry. * ContextVar isolation under ``asyncio.gather`` so two concurrent HTTP requests with different chat_ids stay isolated. Refs: PR NousResearch#25660 (single-gateway multi-agent).
The OpenAI-compatible HTTP adapter was the one inbound surface from PR NousResearch#25660 that never called ``_attach_agent_id`` — every ``/v1/chat/completions``, ``/v1/responses``, and ``/v1/runs`` request fell through to ``default_agent`` regardless of the configured routes, silently undermining the multi-agent guarantee on any deployment that exposes the API server. Add a single routing entry point, ``_resolve_agent_profile``, that: * Reads ``X-Hermes-Chat-Id`` / ``X-Hermes-User-Id`` / ``X-Hermes-Thread-Id`` from the request (sanitised through the same length + control-char caps as the existing ``X-Hermes-Session-Id`` / ``X-Hermes-Session-Key``). * Builds a synthetic ``SessionSource(platform=API_SERVER, …)`` and pipes it through the shared ``_attach_agent_id`` hook so declarative routes *and* the ``select_agent`` plugin hook fire identically to every other adapter. * Looks up the resolved ``agent_id`` in ``self._gateway_ref._agent_registry`` and returns the matching ``AgentProfile`` (or ``None`` for legacy single-agent installs). The three agent-invoking handlers (chat completions, responses, runs) now resolve the profile up front and bind it via ``use_profile`` for the duration of the run. Binding happens twice — once on the asyncio side and once inside the executor thread — because asyncio's default executor does not propagate ContextVars. Behaviour is fully backward compatible: requests with no routing headers (the existing OpenAI-API contract) resolve to ``default_agent``, exactly the current behaviour. New tests in ``tests/gateway/test_api_server_routing.py`` cover: * Header sanitisation (CRLF rejection, length caps, whitespace). * Route resolution: matching, no-header fall-through, unmatched header fall-through, ``platform``-only catch-all, ``user_id`` and ``thread_id`` routes, route-order precedence. * Resilience: missing gateway reference, empty registry. * ContextVar isolation under ``asyncio.gather`` so two concurrent HTTP requests with different chat_ids stay isolated. Refs: PR NousResearch#25660 (single-gateway multi-agent).
The OpenAI-compatible HTTP adapter was the one inbound surface from PR NousResearch#25660 that never called ``_attach_agent_id`` — every ``/v1/chat/completions``, ``/v1/responses``, and ``/v1/runs`` request fell through to ``default_agent`` regardless of the configured routes, silently undermining the multi-agent guarantee on any deployment that exposes the API server. Add a single routing entry point, ``_resolve_agent_profile``, that: * Reads ``X-Hermes-Chat-Id`` / ``X-Hermes-User-Id`` / ``X-Hermes-Thread-Id`` from the request (sanitised through the same length + control-char caps as the existing ``X-Hermes-Session-Id`` / ``X-Hermes-Session-Key``). * Builds a synthetic ``SessionSource(platform=API_SERVER, …)`` and pipes it through the shared ``_attach_agent_id`` hook so declarative routes *and* the ``select_agent`` plugin hook fire identically to every other adapter. * Looks up the resolved ``agent_id`` in ``self._gateway_ref._agent_registry`` and returns the matching ``AgentProfile`` (or ``None`` for legacy single-agent installs). The three agent-invoking handlers (chat completions, responses, runs) now resolve the profile up front and bind it via ``use_profile`` for the duration of the run. Binding happens twice — once on the asyncio side and once inside the executor thread — because asyncio's default executor does not propagate ContextVars. Behaviour is fully backward compatible: requests with no routing headers (the existing OpenAI-API contract) resolve to ``default_agent``, exactly the current behaviour. New tests in ``tests/gateway/test_api_server_routing.py`` cover: * Header sanitisation (CRLF rejection, length caps, whitespace). * Route resolution: matching, no-header fall-through, unmatched header fall-through, ``platform``-only catch-all, ``user_id`` and ``thread_id`` routes, route-order precedence. * Resilience: missing gateway reference, empty registry. * ContextVar isolation under ``asyncio.gather`` so two concurrent HTTP requests with different chat_ids stay isolated. Refs: PR NousResearch#25660 (single-gateway multi-agent).
|
+1 — strongly in favor of this landing. Adding a real-world data point: I've been running exactly this architecture in OpenClaw for months: a single gateway process hosting 8 agents, each with its own Telegram bot token, personality, model config, and isolated memory. One process polls all 8 bots, routes inbound by bot/chat, and operationally it's one daemon to install, watch, and restart instead of eight. I've started building agents in Hermes and want to migrate fully — but the one-gateway-per-profile model is the blocker. Recreating my setup today means 8 separate gateway services, 8 restart paths, and 8 chances for the PID/launchd races already reported elsewhere in the tracker. That's a hard sell when the single-gateway model demonstrably works at this scale day-to-day. The design here (per-agent profile + declarative routes, zero behavior change for existing single-agent installs) maps 1:1 to how I'd consolidate. Happy to test this MVP against a real 8-bot Telegram fleet if useful. |
alt-glitch
added
sweeper:risk-session-state
Related: supersedes #25008 (closed, same single-gateway/multi-agent MVP scope) and #34741 (closed rebase of this PR onto v0.15.0). Addresses feature requests #7517, #9514, #12099, #23735. Tracked follow-up tech debt from this PR: per-agent token bucket #25695, filesystem isolation guards #25696, per-agent process supervision #25697. Not a duplicate (the prior MVP PRs are closed); this is the active version. Maintainer to review as the canonical multi-agent gateway PR. |
alt-glitch
added
comp/tools
|
@02356abc @davidgut1982 — opened #62944: your 7 commits rebased onto current |
teknium1
left a comment
teknium1
left a comment
There was a problem hiding this comment.
Thanks for the substantial multi-agent implementation. The current main branch has since gained a different multiplexing architecture, but it deliberately requires separate credentials per polling profile (gateway/run.py:8566-8580), so it does not subsume this PR's metadata-routing use case.
Problems
cron/jobs.py:1131unconditionally scans every profile's due jobs. When cloned/profile-local schedules share an explicit delivery target, each copy is executed and delivered; this matches the duplicate-delivery report in this PR's discussion.
Suggested changes
- Establish an explicit cron execution-owner rule before aggregating profile jobs, and add a multi-profile fixed-target delivery regression test.
Automated hermes-sweeper review.
| This is the multi-agent equivalent of ``get_due_jobs()``. | ||
| """ | ||
| all_due: List[Dict[str, Any]] = [] | ||
| for agent_id, profile in registry.items(): |
There was a problem hiding this comment.
This scans and returns due jobs for every profile. If profile cron files contain equivalent schedules with the same explicit deliver: target, the scheduler runs and sends every copy. Define an execution-owner/deduplication rule before aggregation and cover the fixed-target multi-profile case.
teknium1
added
sweeper:risk-security-boundary
…pi_server_routing.py
This PR-authored test file ("ported from David Gutowsky's original
NousResearch#25660-era commit") predates two things upstream/main has since changed:
1. All 6 failing tests patched `adapter._ensure_session_db = lambda: ...`,
but every real request handler calls the async `_ensure_session_db_async()`
instead, whose actual test-override hook is the `_session_db` attribute
(checked first, independently of the sync method). The patch was
silently a no-op, so handlers fell through to opening a real on-disk
SessionDB. Fixed by setting `adapter._session_db = mock_db` (or a real
SessionDB, see below) instead of monkeypatching the sync method.
2. The two session-CREATE tests additionally assumed `_handle_create_session`
calls `db.create_session(...)` directly (matching an older implementation).
It no longer does: upstream rewrote the endpoint to a single atomic
check-insert-title SQL block (`_execute_write`) to close a TOCTOU window
on concurrent same-ID creates -- a real, deliberate safety improvement
that must be kept, not reverted to make the test pass. A MagicMock can't
meaningfully observe a raw SQL INSERT, so both tests now use a real
temp-file SessionDB and assert on the persisted row's agent_id via
db.get_session() instead of asserting call_args on a mocked method.
All 31 tests in the file pass; the fork and session-chat tests needed only
fix NousResearch#1, since _handle_fork_session/_handle_session_chat still call
db.create_session()/db.get_session() directly and were never affected by
the atomic-INSERT rewrite.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
GottZ
left a comment
GottZ
left a comment
There was a problem hiding this comment.
This was generated by AI during triage.
Summary
Six PRs address the single-gateway multi-agent request: #25660 and its rebase #34741 implement metadata-based routing with isolated agent state, while #37497, #37498, #37500, and #37502 decompose the same cumulative feature into routing/runtime, CLI, cron/API, and documentation layers. The route-table design directly targets same-credential metadata routing, but current main now provides a different profile-multiplexing lifecycle and the remaining diffs require integration with its validated profile and credential scopes.
Related pull requests
- #25660
related— (+2983/-82) — keep open for targeted salvage, not merge-ready: implements the original metadata-routing cause across sessions, profiles, platforms, hooks, cron, delivery, CLI, and docs, a use case the keep_open review says current credential-per-profile multiplexing does not subsume. Its unconditional cross-profile cron scan can duplicate fixed-target deliveries, so an execution-owner rule and regression test are required before any merge. - #34741 [closed]
duplicate— (+3518/-125) — superseded reference implementation: this closed rebase carries essentially #25660 plus API-server header routing and executor-thread profile propagation, so it remains useful as the most complete historical implementation. It was explicitly superseded by the six-part chain ending in #37502 and should not be reopened. - #37497
related— (+1749/-31) — do not merge as-is: adds route-table selection, session namespacing, profile binding, runtime overrides, and hook propagation, but selected IDs are not validated before session-key construction, allowing an unknown agent namespace to execute with main-agent state. Despite the keep_open review on #37497, current main's multiplex-profile lifecycle means this diff should be mined for metadata-selection behavior rather than merged directly. - #37498 [closed]
related— (+2489/-32) — implemented on main / superseded: adds the cumulative routing base plus a separatehermes agentmanagement CLI, but current main already exposes profile management throughhermes profileand multiplexed gateways. This closed PR remains relevant as the operator-UX portion of the abandoned route-table series, not as a candidate to reopen. - #37500
related— (+3225/-113) — targeted salvage only: extends the cumulative series with per-agent cron and delivery propagation plus API-server header routing, directly covering scheduled and HTTP ingress. Despite the keep_open review on #37500, the diff captures the local output directory before entering the selected profile and omits current main's credential scope, so only missing API-routing behavior should be ported onto the multiplex-profile boundary. - #37502
related— (+3528/-117) — documentation salvage only: represents the full cumulative chain and adds configuration docs plus broaderuse_profilescope, but documents obsolete top-levelagents/routes/default_agentsemantics and introduces a second profile ContextVar without current main's fail-closed secret scope. Despite the keep_open review on #37502, it should not merge as-is; rewrite any reusable documentation and ingress behavior againstgateway.multiplex_profiles,SessionSource.profile, and_profile_runtime_scope.
Duplicates
#34741 is a rebased and extended duplicate of #25660; #37497, #37498, #37500, and #37502 are cumulative slices of that same implementation, with #37502 containing the complete chain rather than an independent solution.
Suggested consolidation
Do not merge any listed PR as-is. Keep #25660 as the canonical metadata-routing salvage tracker because its keep_open review establishes that same-credential metadata routing remains distinct, but first address the blocking cron execution-owner defect; port only the still-missing metadata/API-routing delta into current main's validated multiplex-profile and credential scope. #34741 and #37498 can remain closed, and #37497, #37500, and #37502 can be closed as superseded duplicates once that current-main-based replacement preserves their relevant tests and documentation.
Complex graph
flowchart LR
classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
classDef best stroke-width:3px,stroke:#b45309
classDef target stroke-width:3px,stroke:#4338ca
subgraph Dup25660 ["PRs duplicating each other"]
P25660["PR #25660 (open)"]
P34741["PR #34741 (closed)"]
P37502["PR #37502 (open)"]
end
class P25660 open
class P34741 closed
class P37502 open
class P25660 target
click P25660 "https://github.com/NousResearch/hermes-agent/pull/25660"
click P34741 "https://github.com/NousResearch/hermes-agent/pull/34741"
click P37502 "https://github.com/NousResearch/hermes-agent/pull/37502"
Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed or no verify verdict yet (state tag in the node label).
Cross-PR triage: Reviewed 6 pull requests and 0 issues in this complex. Each diff was read against this issue; Assessment working set: 1064 kB of PR diffs, 12 kB of issue/PR text, 22 kB of discussion (25 comments), 2 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.
alt-glitch
added
needs-decision
|
I really need this and hope a PR will be created for this feature. |
10 participants
Summary
Enable a single
hermes gateway runprocess to host N isolated AI agents,routing inbound messages by platform/chat/thread/user metadata while keeping
each agent's memory, skills, SOUL.md, and model config fully separate.
Fixes the bottleneck behind #23735, #7517, #9514, and #12099.
Deployment scenario matrix
Architecture (8 commits)
agent_idinSessionSource/SessionEntry,build_session_keyprefix, SQLite migrationuse_profile()propagates through async chainsroutes:list with 9 match keys, first-match-wins;select_agentplugin hook override_apply_profile_runtime_overrides,_apply_profile_toolsetsCronJob.agent_id, per-profile storage,DeliveryTarget.agent_idhermes agent list/add/remove/showPrecedence chain
Session
/modeloverride → Profile override → Gateway defaultThe default
"main"profile is a no-op overlay; existing single-agentinstalls see zero behavior change.
Migration Guide
Existing single-agent users (no action required)
No configuration changes needed. The default
default_agent: mainensuresall existing behavior is preserved. Your existing
~/.hermes/directorycontinues to work as the
mainagent profile.Adding a second agent
Consolidating multiple gateway processes
Before this PR:
hermes -p coder gateway run+hermes -p research gateway runAfter this PR:
~/.hermes/profiles/<name>/config.yamlPerformance Impact
_agent_cacheagent:main:...(+9 chars)agent:<id>:...No measurable throughput regression for single-agent configs.
Tests
tests/agent/test_profile_contextvar.pytests/gateway/test_agent_routing.pytests/gateway/test_session.pybuild_session_keywithagent_idacross all chat typestests/gateway/test_profile_overrides.pytests/hermes_cli/test_agent_cli.pyhermes agentlist/show/add/remove commandstests/gateway/test_session_boundary_hooks.pyagent_idassertionstests/test_model_tools.pyagent_idMulti-agent suite: 181 passed
Full regression: 22677 passed / 38 failed (pre-existing env issues) / 105 skipped
E2E Validation
Matrix →
codeagent routing validated with local Dendrite homeserver:codeagentmain/wecom-agent)agent:code:matrix:dm:...session keysFull report:
docs/plans/2026-05-15-multi-agent-matrix-e2e-report.mdNon-goals (future PRs)
Verification commands
Manual smoke checklist
main, session_keyagent:main:...coder, session_keyagent:coder:.../newin topic 42 →on_session_finalizereceivesagent_id="coder"profiles/coder/cron/jobs.json"research"fromselect_agenthook → overrides route matchagent_idagents:androutes:from config → all messages route tomain"main"