[RGaonkar26]

An agent can name a model_pool of deployments instead of one model and client. Each rollout binds, at its first model request, to a deployment whose request gate has room, and stays on it. The trace records the deployment for every call.

Why
Spreading rollouts over several replicas or models within each one's concurrency cap (for example several replicas of one model, or GLM-5.3 and GLM-5.3-flash) needs a binding step. Each deployment's request gate already holds its cap host-wide, so a pool is just several gates in the same daemon, and the daemon needs no change.

Change

• ModelPoolConfig(deployments, same_model=False) and ModelDeployment(id, model, client) in configs/model_pool.py, exported from verifiers.v1, plus AgentConfig.model_pool. Every deployment must set client.request_gate. Its scope defaults to the deployment id, so two deployments behind one URL and key keep separate caps.
• model_routing.py: at a session's first model request, the interception server shuffles the deployments and binds to the first one that grants a non-queuing admission (ModelGate.try_acquire). Such an admission yields to requests already queued at that gate. When every deployment is full, it retries with a jittered backoff from 30 ms up to 1 s.
• The slot taken while choosing is handed to that first request (reserved_permit), so a session is admitted once, not twice. An unused slot is returned.
• Later requests use the bound deployment's model and client. On resume, a saved deployment that is still in the pool is kept; otherwise the session binds again.
• Recorded on the trace: trace.info["model_binding"], trace.agent.config.model and .client, and a new ModelCall.deployment.
• same_model = true rejects a pool whose deployments serve different model ids. It is meant for replica pools whose outputs must be attributable to one model.

Behaviour

• Off by default: with no model_pool, no binding code runs. When unset, model_pool and deployment are left out of dumps, so configs and traces still load in older verifiers.
• No failover mid-rollout: a deployment that errors fails that rollout's calls, as today, and a retried rollout chooses again.
• In a pool of different models, a resumed session whose deployment has left the pool can change model. same_model rules this out.

Tests

• Unit: 10 tests in tests/unit/request_gate/test_model_pools.py, with fake providers. They cover the spread under caps of 1, 2 and 3, the backoff bounds, one admission for the choice plus the first request, resume, config validation, and off by default. Unit suite: 283 passed, 18 skipped.
• Live (Prime Inference, 2026-10-23):
  • 6 rollouts × 3 turns over two deployments with cap 2 each: both deployments were used, each rollout's calls recorded one deployment matching trace.info["model_binding"], and each deployment's upstream peak was 2.
  • With one deployment held at its cap, 3 of 3 new rollouts bound to the other.
  • same_model rejected a mixed pool. Two replicas at one URL and key kept separate caps (peak 1 + 1).
  • On resume, a saved binding was kept. A binding whose deployment had left the pool rebound, with one admission covering both the choice and the first request.
  • 20 waiting sessions settled at about 1 poll per second per session per deployment.
• Not tested live: load at scale.

Notes

• Stacked on feat(v1): adapt the request gate's limit to 429/529 #2970, because the pool's permit hook sits inside the gate's 429 retry loop. Merge after feat(v1): a host-wide request gate daemon #2968–feat(v1): adapt the request gate's limit to 429/529 #2970. It could be rebased onto feat(v1): gate model requests with client.request_gate, and vf-gate #2969 with a small conflict in GatedAsyncClient.send.
• Ported from Parker's feat/pipelines f04a56a14 (plus try_acquire from The request gate admits through one daemon per host #2946 and dad67e0b4). Differences: no live model switch (steering.switch_model needs a steering registry, which main lacks) and no AgentConfig.session_id. New here: ModelCall.deployment, same_model, the backoff on the bind poll, and tests.

Parker, as with the gate: this could land on main and reach feat/pipelines at its next main merge, with switch_model staying on your branch until main has steering. Or you could open it from this branch yourself. Which do you prefer?

🤖 Generated with Claude Code

[RGaonkar26]

Closing in favour of Parker's #2946, which is the same request-gate / model-pool work and where it should land. Nothing on verifiers main or our pipeline needs it right now. The branch stays up: it has shared-host hardening (private 0700 dir, peer-uid checks, single-daemon lock, restart window), live tests against the Prime Inference gateway, and the pool poll backoff, all of which apply to #2946 and can be cherry-picked.