feat(gateway): session model pool — concurrency-aware auto-assignment with auxiliary slot tracking

[ricardo-camilo-programador-frontend-web]

Problem or Use Case

Users running multiple concurrent sessions (Discord threads, Telegram topics, WhatsApp chats, etc.) must manually run /model and /yolo in every new session. Without intentional distribution, multiple sessions hit the same model, exhausting provider rate limits.

Additionally, auxiliary tasks (vision, compression, curator, session_search, approval, etc.) make independent API calls that are invisible to session management — meaning even a perfect session distribution can cause rate-limit collisions when an auxiliary call saturates a model already used by active sessions.

Concrete scenario: With the Z.AI Coding Plan, 7+ GLM models are available with separate concurrency pools totaling 28+ slots. Today only 1 session uses capacity before collisions start, and auxiliary calls further erode available slots unpredictably.

Proposed Solution

Add a session_model_pool section to config.yaml that:

1. Auto-assigns the next available model when the gateway creates a new session, respecting per-model concurrency limits
2. Tracks auxiliary tool calls against reserved slots per model, preventing auxiliary calls from stealing session capacity
3. Provides multiple allocation strategies (round-robin, least-loaded, priority)

New files

• gateway/session_model_pool.py — Thread-safe SessionModelPool class with PoolModelEntry dataclass, 3 allocation strategies, session + auxiliary slot tracking, inactive timeout, and module-level singleton (get_session_model_pool())
• tests/gateway/test_session_model_pool.py — 35 unit tests covering pool entry logic, all 3 strategies, session/auxiliary slot management, thread safety, edge cases

Modified files

• gateway/run.py — 3 integration points:
  • _resolve_session_agent_runtime: pool assigns model when no manual /model override exists (stores in _pool_assigned_models dict)
  • _handle_message_with_agent: releases pool slot on session reset
  • _handle_model_command: releases pool slot when user makes manual /model override (override always prevails)
• agent/auxiliary_client.py — call_llm() acquires auxiliary slot before API call, releases in finally block
• hermes_cli/config.py — session_model_pool section added to DEFAULT_CONFIG

Proposed config schema

session_model_pool:
  enabled: true
  strategy: round-robin              # round-robin | least-loaded | priority
  inactive_timeout: 1800             # seconds — slot released after inactivity
  pool:
    - model: glm-5-turbo
      provider: zai
      max_concurrent: 1
      reserved_for_auxiliary: 0
      context_length: 203000
    - model: glm-5
      provider: zai
      max_concurrent: 2
      reserved_for_auxiliary: 1
      context_length: 200000
    - model: glm-4.6
      provider: zai
      max_concurrent: 3
      reserved_for_auxiliary: 2
      context_length: 200000
    - model: glm-4.5
      provider: zai
      max_concurrent: 10
      reserved_for_auxiliary: 0
      context_length: 128000

Field definitions

• max_concurrent (required) — Hard limit on simultaneous uses (sessions + auxiliary)
• reserved_for_auxiliary (required) — Guaranteed minimum slots for auxiliary tool calls; sessions can only use max_concurrent - reserved_for_auxiliary slots
• context_length (optional) — If set, gateway may warn on context overflow
• priority (optional, 1–10, default 5) — Higher = preferred when multiple models have free slots

Behavior

• session_model_pool.disabled → current behavior (global model.default), no overhead
• Manual /model override always prevails — pool slot is released and session is marked as "manually overridden"
• All session slots saturated → falls through to model.default
• Auxiliary call blocked → retries with backoff, then skips (non-critical) or surfaces error (critical like vision)
• Gateway restart → in-memory state lost; sessions reassigned on first message

Test results

• 35/35 new unit tests passing
• 2426/2427 existing gateway tests passing (1 pre-existing Matrix failure unrelated)

Closes #37511

[rafaumeu]

Code Review — PR #37519: Session Model Pool

Veredito: APPROVE com 1 HIGH + 4 MEDIUM + 3 LOW

---

HIGH

H1 — async_call_llm sem integracao com pool
gateway/auxiliary_client.py tem TODO referenciando issue #37744, mas esse issue nao existe. O path async bypassa todo o pool (round-robin, aux count, override, health check). Se alguem chamar async antes do pool ser integrado, vai direto num provider aleatorio.

• Acao: Criar o issue #37744 e linkar no TODO, ou remover a referencia.

---

MEDIUM

M1 — Config e runtime state na mesma dataclass
PoolModelEntry mistura campos de config (provider_config, pool_key) com runtime state (_failure_count, _last_failure). Qualquer teste que instancia PoolModelEntry precisa mockar ambos os mundos. Separar em PoolModelConfig (frozen) + runtime state facilita testes e evita mutabilidade acidental.

• Acao: Considerar PoolModelConfig frozen dataclass + runtime wrapper.

M2 — call_llm retorna None quando slot auxiliar bloqueado
Quando _max_auxiliary_count e atingido, call_llm retorna None. Todos os callers esperam Response. Isso pode causar AttributeError downstream se o caller nao verificar.

• Acao: Documentar o contrato (return type Response | None) ou levantar exception dedicada.

M3 — Inconsistencia de config loading
_mark_pool_override le config do disco (self._load_config()), enquanto _resolve_session_agent_runtime usa um singleton {} como fallback. Dois caminhos diferentes pro mesmo dado.

• Acao: Unificar em um unico config loader.

M4 — _manual_overrides: set sem type hint generico
Deveria ser set[str] para consistencia com o resto do codebase tipado.

• Acao: _manual_overrides: set[str] = set().

---

LOW

L1 — time.sleep(0.5) em tests
Pode causar flakiness em CI lento. Considerar mockar ou usar timeout maior.

• Acao: Avaliar se time.sleep pode ser substituido por asyncio.sleep + mock.

L2 — Round-robin e semanticamente LRU
O nome round_robin confunde porque a implementacao e mais proxima de LRU (reordena no acesso). Isso pode enganar quem le o codigo.

• Acao: Renomear ou documentar a semantica.

L3 — _aux_pool_cache nunca resetado
O cache de pool auxiliar cresce indefinidamente. Em sessoes longas, pode consumir memoria.

• Acao: Adicionar TTL ou limite de tamanho.

---

Positivos

• Thread safety robusta (locks bem colocados, sentinel pattern elegante)
• Config validation completa com erros claros
• 46 testes solidos cobrindo casos de borda
• Integration points bem isolados (pool nao vaza pro gateway)
• Commit history transparente e incremental

---

Review por @rafaumeu via Hermes Agent

[ricardo-camilo-programador-frontend-web]

All issues from previous CHANGES_REQUESTED reviews have been addressed in subsequent commits. All 9 inline comments resolved.

[teknium1]

Thanks for the detailed pool implementation and the follow-up fixes. The feature premise is still present on current main, but this needs rework before it can be safely salvaged.

Problems

• agent/auxiliary_client.py:571 returns the _UNSET sentinel when no pool is configured. The sync and async callers then access _pool.enabled outside the guarded acquisition block (agent/auxiliary_client.py:5017 and 5495 on the PR head). Since session_model_pool.enabled defaults to false, this breaks normal auxiliary calls rather than becoming a no-op.
• The PR resolver predates current routing precedence. Current main applies channel_overrides in gateway/run.py:3811-3844, whereas the PR applies its pool provider at gateway/run.py:2658. Salvage must not let a pool overwrite an explicit channel provider or bypass persisted /model restoration.

Suggested changes

• Return None rather than _UNSET from the auxiliary helper when no enabled pool is available; add sync and async disabled-pool regressions.
• Integrate allocation into current main's resolver after preserving /model and channel-override precedence, with focused gateway routing tests.

Automated hermes-sweeper review.

[teknium1]

When no pool is configured, _aux_pool_cache remains _UNSET, so this returns a truthy sentinel rather than the documented None. The callers then evaluate _pool.enabled outside their acquisition try blocks (lines 5017 and 5495), breaking auxiliary calls under the default-disabled configuration. Return None here when the cache is still _UNSET, and add disabled-pool sync/async coverage.

[teknium1]

Current main resolves channel_overrides after the global runtime and documents /model → channel_overrides → global precedence at gateway/run.py:3750-3844. When salvaging this allocation, apply it only below both explicit override layers; otherwise the later pool provider write can pair a channel-selected model with the pool provider.

[DeamonDev888]

Complementary approach to #62467. This PR handles session-level model+pool assignment, while #62467 handles the credential resolution layer underneath.

What this PR does: auto-assigns models to sessions with concurrency-aware pool slot tracking. This is orthogonal to credential resolution — it decides WHICH pool entry a session uses, not HOW that entry's base_url is resolved.

How they stack:

• This PR (session pool): "Session A uses pool entry Terminal tool #1, Session B uses pool entry Support passing morph snapshot id #2"
• feat(auth): unified credential resolver + pool management API + security hardening #62467 (credential resolver): "Pool entry Terminal tool #1 has base_url=coding/paas/v4 — here's the resolved credential"

Both are needed. This PR's session-level assignment would consume the ResolvedCredential returned by #62467's unified resolver. If this PR's auxiliary slot tracking reads entry.base_url, it will automatically benefit from #62467's fixes (proper endpoint per entry, no stale cache, config override when pool_url matches registry default).

No conflict — these two PRs are complementary layers.