fix(agent-init): forward custom_headers on OpenAI-wire branch

[shailensobhee]

Summary

The anthropic_messages branch in init_agent() resolves custom_headers from custom_providers entries (matching by base_url) and forwards them to the Anthropic client as default_headers, so APIM-style gateways that require auth via a subscription header (e.g. Ocp-Apim-Subscription-Key) authenticate correctly.

The OpenAI-wire branch (chat_completions / codex_responses) never had this lookup. It only sets default_headers for a fixed list of providers (OpenRouter, NVIDIA NIM, Routermint, Copilot, Kimi, Qwen Portal, Codex Cloudflare) plus a profile.default_headers fallback. For any other custom_providers entry served over the OpenAI wire — including the AMD LLM Gateway, Azure OpenAI behind APIM, and similar enterprise proxies — the subscription header was silently dropped and the gateway rejected every request with HTTP 401:

Access denied due to missing subscription key. Make sure to include subscription key when making requests to an API.

Because the fallback chain often cycles through several custom_providers entries that all share the same gateway, this manifests as every fallback failing with the same 401. Visible in ~/.hermes/sessions/request_dump_*.json — outbound headers contain only Authorization and Content-Type, never the configured custom_headers.

Fix

Mirror the existing anthropic_messages lookup on the OpenAI-wire branch:

1. Match base_url against custom_providers entries (case-insensitive, trailing slash normalized).
2. Copy custom_headers into client_kwargs["default_headers"].
3. Forward verify: false via an explicit httpx.Client(verify=False) for gateways behind self-signed certs.

Guarded by if "default_headers" not in client_kwargs so it never overrides the existing per-host branches (OpenRouter, Copilot, etc.) or a profile.default_headers set earlier.

Repro

config.yaml:

model:
  provider: custom:my-gateway
  base_url: https://gateway.example.com/openai
  api_mode: chat_completions
  api_key: dummy

custom_providers:
- id: my-gateway
  name: My APIM Gateway
  api_mode: chat_completions
  base_url: https://gateway.example.com/openai
  api_key: dummy
  custom_headers:
    Ocp-Apim-Subscription-Key: <your-key>
  model: gpt-4o

Before this PR: every request returns HTTP 401 because Ocp-Apim-Subscription-Key is never sent.
After this PR: request succeeds, header is forwarded.

Test Plan

• Existing tests pass: pytest tests/run_agent/test_provider_attribution_headers.py tests/run_agent/test_callable_api_key.py — 26/26 passing.
• Verified end-to-end against an APIM-fronted gateway across all three surfaces (Anthropic-wire, OpenAI-wire, OnPrem). All return HTTP 200.
• Verified guard: when default_headers is already set by an earlier per-host branch (e.g. OpenRouter), the new block is a no-op.

Notes

• This is the mirror of the lookup already present at lines 564–592 of agent/agent_init.py (the anthropic_messages branch).
• Symmetric with PRs Named custom providers with api_mode: anthropic_messages fail with 404 in auxiliary tasks #15033 / fc1b74a which originally added the lookup to the Anthropic branch — same rationale, same shape.

[teknium1]

Thanks for chasing this. I verified the core premise on current main: the explicit OpenAI-wire init path still never reads custom_providers[*].custom_headers (agent/agent_init.py:750-781), and the later model.default_headers hook is only the separate model-level override (agent/agent_init.py:892-898). The PR is in the right neighborhood, but this patch is not complete yet.

Problems

• The lookup is only added to the explicit api_key and base_url init branch. OpenAI-wire custom clients built through the shared resolver still construct clients without custom provider headers (agent/auxiliary_client.py:3567-3612, agent/auxiliary_client.py:3684-3725), so fallback/auxiliary/named-custom paths can still drop the subscription header.
• Credential swaps rebuild headers through _apply_client_headers_for_base_url, which still only applies host/profile/model.default_headers and would clear a custom-provider subscription header on an unknown gateway (run_agent.py:3890-3933).
• No regression test covers a custom_providers entry with custom_headers; the existing tests cover provider attribution and model.default_headers, not this config shape (tests/run_agent/test_provider_attribution_headers.py:180-273).

Suggested changes

• Centralize custom-provider header/verify resolution and call it from init, resolver/fallback, auxiliary/named-custom, and credential-swap rebuild paths.
• Add tests for legacy custom_providers.custom_headers on the main OpenAI-wire init path and at least one resolver/fallback path.

Automated hermes-sweeper review.

[teknium1]

Because this lookup lives only in the explicit startup branch, resolver-built custom clients and credential-swap rebuilds can still drop the same gateway header. This should probably be a shared custom-provider header resolver used by init, fallback/resolver, auxiliary, and _apply_client_headers_for_base_url rather than a one-off block here.

[teknium1]

Reading raw load_config()["custom_providers"] bypasses the repo's compatible custom-provider view, so any equivalent provider entry represented through the newer providers schema will be missed unless the normalization layer is extended to preserve these fields.

[teknium1]

Thanks for the gateway-auth fix. Current main now provides this capability through a broader shared configuration path, so this PR is superseded.

• b98baa3039e357d97ccb8d84e6d70b1902a03582 added endpoint-scoped extra_headers for both providers and legacy custom_providers; agent/agent_init.py:1101-1125 applies them to OpenAI-wire client construction.
• hermes_cli/config.py:5042-5097 supplies case-insensitive, trailing-slash-normalized matching and merges endpoint headers with existing defaults.
• run_agent.py:4470-4484 reapplies those headers on credential swaps and client rebuilds, covering paths a one-time init block would miss.
• Per-provider TLS is separately supported by 3a2ba959ce2f09cbf3ed58d26f2526c6a98643c6 via ssl_verify / ssl_ca_cert (agent/agent_runtime_helpers.py:1689-1691).

For the configuration in this PR, use extra_headers: rather than custom_headers: and ssl_verify: false rather than verify: false. This is an automated hermes-sweeper review.