Conversation
jonpol01
force-pushed
the
fix/doctor-custom-provider-key-env
branch
2 times, most recently
from
September 3, 2026 07:59
4bbceae to
1d904cc
Compare
Contributor
PR 101350 — fix(doctor): flag a custom provider whose key_env names an unset var. Extends doctor's credential check to named custom providers ( Non-blocking observations:
|
jonpol01
force-pushed
the
fix/doctor-custom-provider-key-env
branch
from
September 6, 2026 15:34
1d904cc to
9a7bf96
Compare
jonpol01
force-pushed
the
fix/doctor-custom-provider-key-env
branch
2 times, most recently
from
September 18, 2026 18:53
5f1eb6e to
20fc1d5
Compare
…able doctor's credential check runs only for providers in PROVIDER_REGISTRY with auth_type api_key. A named custom provider (a `providers:` entry or a legacy `custom_providers` item) is not in the registry, so the check skipped it and doctor reported the configuration healthy. At runtime the key resolves key_env → inline api_key → key_cmd (hermes_cli/runtime_provider.py); when key_env names a variable that is not set and the entry has no other source, every request fails with an auth error and the user is sent to inspect the provider instead of their .env — the tracker has a long tail of "custom provider key_env → 401" reports for exactly this. After the registry check, resolve the active model.provider to its custom entry through the same alias set the runtime uses and, when its key_env is declared but unset and no api_key / key_cmd is present, fail with the variable's name and where to set it. Only the active route is checked here; the bare `provider: custom` + model.key_env form is left alone because the runtime does not resolve that key yet (NousResearch#67554).
jonpol01
force-pushed
the
fix/doctor-custom-provider-key-env
branch
from
September 20, 2026 05:18
20fc1d5 to
fec1fed
Compare
…e doctor warning Six tests become one table. An unset key_env on the active provider is flagged, for both the providers: mapping and the legacy custom_providers list; a set variable, an inline api_key, a key_cmd and an inactive entry stay quiet. The two red rows fail on the pre-fix doctor, and the api_key/key_cmd rows fail when those fallbacks stop exempting the entry. Docs: the named custom providers section says hermes doctor reports such an entry.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
hermes doctornow fails when the active custom provider'skey_envnames an environment variable that is not set.Problem
The credential check in
hermes_cli/doctor.py(~:1685) is deliberately limited to API-key providers inPROVIDER_REGISTRY, to avoid false positives fromget_auth_status(). A named custom provider — aproviders:entry or a legacycustom_providersitem — is not in the registry, so the check skips it and doctor reports the configuration healthy.At runtime (
hermes_cli/runtime_provider.py:830-870) such a provider's key resolves askey_env→ inlineapi_key→key_cmd. Whenkey_envnames a variable that is not set and the entry has no other source, every request fails with an auth error, and the log points at the provider rather than at the missing variable. The tracker carries a long tail of "custom providerkey_env→ 401" reports (#43586, #44349, #46690, #62254, #67453, #67935, #74561, #83847, #91683, #92124, #100858) — exactly the misconfiguration doctor exists to catch, and it said nothing.Change
After the existing registry credential check, doctor resolves the active
model.providerto its custom entry through the same alias set the runtime resolver uses (custom_provider_aliases: display name,providers.<key>,custom:<key>), and when that entry declareskey_env(or theapi_key_envalias) but the variable is unset and there is no inlineapi_keyorkey_cmd, it fails with the variable's name and where to set it:Scope, deliberately narrow:
key_envis not this section's business;provider: custom+model.key_envform is left alone because the runtime does not resolve that key yet (open fix(config): resolve model.key_env in bare custom provider path (#67453) #67554); validating it here would fail a key the runtime never reads;get_env_value(config.env+ environment), the same call the existing OpenRouter branch uses;providers:entry as well as the compatibility view forapi_key/key_cmd:_normalize_custom_provider_entrylistskey_cmdas a known key but does not carry it into the view, while the runtime reads it from the raw entry (runtime_provider.py:866) — without this akey_cmdprovider would have been flagged falsely (a test pins it).The registry check's own false-positive guard is untouched: this path never calls
get_auth_status.Related Issue
Related: #67453 (runtime resolution of
model.key_env, open PR #67554), #100858, and the "key_env → 401" family listed above. None asks for the doctor check.Type of Change
Changes Made
hermes_cli/doctor.py—_custom_provider_entry_for()helper;key_envcheck after the registry credential blocktests/hermes_cli/test_doctor.py— one invariant table throughrun_doctor: an unsetkey_envon the active provider is flagged, for both theproviders:mapping and the legacycustom_providerslist; a set variable, an inlineapi_key, akey_cmdand an inactive entry stay quiet.website/docs/integrations/providers.md— the Named Custom Providers section sayshermes doctorreports such an entry.How to Test
scripts/run_tests.sh tests/hermes_cli/test_doctor.py -k key_envproviders: {mylocal: {base_url: http://127.0.0.1:1234/v1, key_env: MYLOCAL_API_KEY}}andmodel.provider: mylocal, runhermes doctorwith the variable unset → the line above; export it → clean.Sabotage-checked: with the pre-fix doctor only the two flagged rows fail; with
api_key/key_cmdno longer exempting the entry only those two quiet rows fail. Merged onto currentmainlocally: no conflicts, all ninetests/hermes_cli/test_doctor*.pyfiles pass (155 passed).ruff checkis clean.Checklist
Code
key_env,_compatible_custom_providers, "doctor custom provider api key"; fix(config): resolve model.key_env in bare custom provider path (#67453) #67554 fixes the runtime path, nothing adds the doctor checkruff check, and the footgun linter; relying on CI for the full matrixDocumentation & Housekeeping
website/docs/integrations/providers.md(Named Custom Providers)cli-config.yaml.example— N/ACONTRIBUTING.md/AGENTS.md— N/A