fix(api-server): pop runtime model before AIAgent kwargs unpack (#27540)

[briandevans]

What does this PR do?

The api_server gateway adapter crashes with TypeError: AIAgent() got multiple values for keyword argument 'model' on the first turn after a fallback provider takes over. _try_resolve_fallback_provider() returns a "model" key in its kwargs dict, but _create_agent() was **-unpacking that dict alongside an explicit model= argument. This PR pops "model" from runtime_kwargs first and treats it as an override for the gateway-resolved model — mirroring the canonical runtime_kwargs.pop("model", None) pattern already used in GatewayRunner._resolve_model_runtime (gateway/run.py:1867). With the pop, the fallback's model wins, there's no double-bind, and the override path is logged.

Related Issue

Fixes #27540

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

• gateway/platforms/api_server.py — in _create_agent, runtime_model = runtime_kwargs.pop("model", None); if present, log "API server: runtime provider supplied explicit model override: %s -> %s" and model = runtime_model before the AIAgent(model=model, **runtime_kwargs, ...) call. Mirrors GatewayRunner._resolve_model_runtime (gateway/run.py:1866-1874).
• tests/gateway/test_api_server.py — new test_create_agent_handles_runtime_model_override_from_fallback (runtime returns {"model": "openai/gpt-5", "provider": "openrouter", ...} → _create_agent() must not raise, AIAgent(model=...) receives the fallback model). Plus happy-path test_create_agent_keeps_gateway_model_when_runtime_omits_model (runtime omits "model" → gateway-resolved model passes through unchanged).

How to Test

1. uv run --with pytest --with pytest-xdist --with pytest-asyncio python3 -m pytest tests/gateway/test_api_server.py tests/gateway/test_api_server_runs.py -v — 173 passed.
2. Regression guard: with the production fix reverted, test_create_agent_handles_runtime_model_override_from_fallback reproduces the exact TypeError: ... got multiple values for keyword argument 'model' from the issue. With the fix applied, all 3 test_create_agent_* tests pass.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run focused tests for the touched code and all pass
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: macOS 15.x

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings) — or N/A
• I've updated cli-config.yaml.example if I added/changed config keys — or N/A
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
• I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
• I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Sibling Audit

Sibling code paths that may need the same fix: gateway/run.py:7137 and gateway/run.py:8299 use _resolve_runtime_agent_kwargs() defensively via .get() calls only, so they are unaffected. gateway/platforms/feishu_comment.py:_resolve_model_and_runtime is consumed by explicit per-key forwarding (no **-unpack) so it's also unaffected. No widening needed.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Fixes a crash in the API server agent creation path when fallback provider resolution supplies a runtime model override, ensuring model is not passed twice to AIAgent.

Changes:

• Pop model from runtime kwargs and treat it as an override for the gateway-resolved model.
• Add regression tests covering both the override (fallback) and non-override (primary provider) paths.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
gateway/platforms/api_server.py	Prevents TypeError by removing model from runtime_kwargs before **-unpacking into AIAgent, optionally overriding the gateway model.
tests/gateway/test_api_server.py	Adds regression tests validating model override behavior and ensuring no duplicate-model crash.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[andreab67]

Confirmed reproduction on a second environment + cherry-picked the patch successfully — sharing in case it helps push this through.

Reproduction: Ubuntu 24.04 / Python 3.12, hermes-agent running as a long-lived systemd unit (hermes.service on a VPS, :8642 gateway, single-user). Primary provider is openai-codex (ChatGPT Codex), fallback_providers includes an openrouter entry with model: set. After a routine git pull brought in the recent main, the Codex access_token went stale and _resolve_runtime_agent_kwargs() started returning a fallback dict that contains "model". Every subsequent /v1/chat/completions, /v1/responses, and tools-probe call returned the exact TypeError: run_agent.AIAgent() got multiple values for keyword argument 'model' from #27540 — same line 893 of gateway/platforms/api_server.py, same traceback shape. Gateway was effectively down for ~3 hours until I noticed (workers kept reporting success because they swallow the 500 — separate issue, not for this PR).

Patch verification: Cherry-picked the production change from this PR onto the VPS in-place. After systemctl restart hermes.service:

• POST /v1/chat/completions ({"model":"gpt-5.5","messages":[{"role":"user","content":"PONG"}]}) → 200, clean chat.completion with usage.total_tokens: 1342
• POST /v1/responses with structured input → 200, clean response object with output[0].content[0].text: "PONG"
• gateway.run still logs Primary provider auth failed: Codex auth is missing access_token. — trying fallback (expected — separate hermes auth issue), but the fallback path now completes instead of crashing
• No more TypeError in journalctl -u hermes.service post-restart

The runtime provider supplied explicit model override info-log doesn't fire in my repro because my specific fallback chain in config.yaml resolves to a provider whose entry doesn't carry model: — so the pop returns None and the explicit model passes through unchanged. That's a useful confirmation that the patch is correctly defensive on the "fallback returned no model" branch too.

On the red CI check: the failing test is tests/hermes_cli/test_tui_npm_install.py::test_make_tui_argv_skips_build_only_on_termux_when_fresh — completely unrelated to this PR. It's a Termux argv assertion that expects [node, entry.js] but main now emits [node, --expose-gc, entry.js] (the --expose-gc flag was added by the Termux cold-start salvage in #30609 / #30618 the same day this PR was last pushed). Confirmed by looking at the latest Tests workflow run on main (commit 2233b8b2, 2026-05-22 21:32 UTC) — that one is green. A rebase of this branch against current main should clear the red.

Happy to keep running with the cherry-pick locally on production until this merges. Thanks for the fix.

[Koraji95-coder]

Additional production verification — Windows + Foundry broker through Tailscale-mesh reverse proxy on two machines, hermes-agent v0.14.0, Python 3.11.15. Cross-posting from my (now-closed) duplicate #31139:

Repro before this fix: POST /v1/chat/completions returns HTTP 500 within ~50ms regardless of payload:

{"error": {"message": "Internal server error: run_agent.AIAgent() got multiple values for keyword argument 'model'", "type": "server_error", ...}}

After cherry-picking #27678's runtime_kwargs.pop('model', None): HTTP 200 with a real OpenAI-shaped Chat Completion, finish_reason: stop, 33-second end-to-end response with real agent tool-calls:

{
  "id": "chatcmpl-f67cabfe3f6146b1a870ab6b5ac54", "object": "chat.completion",
  "created": 1779567828, "model": "hermes-agent",
  "choices": [{"index": 0, "message": {"role": "assistant", "content": "pong"}, "finish_reason": "stop"}],
  "usage": {"prompt_tokens": 14816, "completion_tokens": 29, "total_tokens": 14845}
}

So adding to @andreab67's Ubuntu 24.04 confirmation: also works on Windows 11 Pro under the Foundry production deployment, across two independent machines. No regression observed against the existing model-name advertisement at /v1/models, default-profile behavior, or any prior working /v1/chat/completions requests that happened to dodge the collision.

If a regression test for the kwargs collision would help this PR through review, I have one ready (test_create_agent_pops_colliding_model_from_runtime_kwargs mirroring the existing test_create_agent_forwards_config_reasoning_effort pattern in tests/gateway/test_api_server.py::TestCreateAgent) — happy to PR it on top of this branch.

[briandevans]

Thanks @andreab67 and @Koraji95-coder — much appreciated. Rebased onto current main (a7a58af9d) to clear the unrelated Termux --expose-gc red; that test was updated by commits 3d2f14646 / 2ea7cf287 after this branch was last pushed.

@Koraji95-coder — for the regression coverage, the PR already includes test_create_agent_handles_runtime_model_override_from_fallback and test_create_agent_keeps_gateway_model_when_runtime_omits_model in tests/gateway/test_api_server.py::TestAdapterInit, both confirmed passing locally on the rebased branch alongside the existing test_create_agent_forwards_config_reasoning_effort pattern you referenced — so no duplicate needed.

[briandevans]

Closing to focus the queue on security/file-safety work where civilian merges are landing. Happy to reopen if maintainers want this picked up.

[briandevans]

Reopening — on reflection this is a P1 with two independent civilian production repros, which keeps it in the salvage-priority lane rather than dead-weight. Apologies for the churn.

[briandevans]

Housekeeping: closing to keep my open-PR set focused on actively-reviewed work. This has been open ~14d without maintainer review and the surrounding code has continued to move, so it's unlikely to land as-is. The underlying fix still stands — happy to reopen and rebase if it would be useful. Thanks!

[briandevans]

Reopening — closed this in error during queue housekeeping. This is in an active salvage niche (security / provider-SDK-drift / install-correctness / civilian-repro'd P1), not stale plumbing. Keeping it open.

[teknium1]

Thanks for the focused fix. I verified the premise against current main and do not see blocking problems.

The current API server path still calls _resolve_runtime_agent_kwargs() at gateway/platforms/api_server.py:1033, resolves model separately at gateway/platforms/api_server.py:1035, then calls AIAgent(model=model, **runtime_kwargs) at gateway/platforms/api_server.py:1046-1048. The fallback resolver can return a runtime dict containing model at gateway/run.py:1388-1397, so the duplicate-keyword crash is still reachable.

The proposed direction matches the existing native gateway path: _resolve_session_agent_runtime already pops runtime_kwargs.pop("model", None) and lets that model override at gateway/run.py:2825-2833; _resolve_turn_agent_config then builds a sanitized runtime dict without model at gateway/run.py:2893-2905.

I also checked the named sibling Feishu path: gateway/platforms/feishu_comment.py:1074-1080 forwards individual runtime fields instead of **runtime_kwargs, so it does not have the same double-bind shape.

Automated hermes-sweeper review.

[briandevans]

Rebased onto current main — still conflict-free (mergeable/clean) and CI is fully green (0 failures across all checks). The fix is unchanged: it mirrors the established runtime_kwargs.pop("model", None) pattern from gateway/run.py so the api_server path doesn't pass model= twice after fallback-provider resolution. With @teknium1's keep_open verdict and two independent production repros (@andreab67, @Koraji95-coder), this is purely awaiting a human merge.

[teknium1]

Closing — issue #27540 could not be reproduced on current main; the behavior this PR targets is already correct in current code, so the change is no longer needed. Thanks for the contribution.