fix: ignore provider config flags in model picker

[Michaelyklam]

Thinking Path

• The model picker should only list real model providers.
• providers.only_configured is a filtering flag, but the provider-detection loop treated every key under providers: as a candidate provider id.
• Because the previous condition accepted the raw key itself, scalar flags could be canonicalized into provider ids such as only-configured and rendered as fake groups.
• This PR narrows config-driven provider detection to known providers/aliases or dict-shaped provider configs, preserving custom provider config support while excluding scalar flags.

What Changed

• Filtered the cfg["providers"] detection loop so scalar config flags no longer seed detected_providers.
• Preserved canonical-to-raw key mapping for known providers and dict-valued provider/custom configs.
• Added regression coverage for providers.only_configured: true and another unknown scalar sibling under providers:.
• Added an Unreleased changelog note.

Why It Matters

• Fixes a confusing fake Only-Configured provider group in the picker.
• Keeps providers.only_configured as filtering behavior only, not a visible provider.
• Avoids future scalar provider-level flags accidentally becoming picker groups.

Verification

• env -u HERMES_CONFIG_PATH -u HERMES_WEBUI_HOST /home/michael/.hermes/hermes-agent/venv/bin/python -m pytest tests/test_issue2399_provider_config_flags.py tests/test_provider_mismatch.py -q — 65 passed
• /home/michael/.hermes/hermes-agent/venv/bin/python -m py_compile api/config.py
• git diff --check

Risks / Follow-ups

• Low-risk backend picker-data change. Known provider ids/aliases and dict-shaped provider configs still seed groups; only scalar unknown provider siblings are ignored.

Closes #2399

Model Used

AI-assisted change with repository inspection, targeted editing, and shell-based test verification.

[nesquena-hermes]

Maintainer pushed a small fix on top of your branch (force-pushed 9f404c3c).

CI was red on test_issue604_all_providers_model_picker.py::test_cfg_providers_only_adds_known — that test does a windowed grep (1500 chars from the "Also detect providers explicitly listed" anchor) for _PROVIDER_MODELS to confirm the documentation describes the gating. Your new gating logic correctly references _PROVIDER_MODELS, but the in-line comment block pushed the reference 1731 chars from the anchor — 231 chars past the test's window.

The fix is just a comment-block reorganization: I moved the _PROVIDER_MODELS / _PROVIDER_DISPLAY gating description into the top anchor comment (where it's now part of the documented contract that the test asserts on), and shortened the in-loop comment to a back-reference. Distance from anchor to _PROVIDER_MODELS is now 325 chars — comfortably inside the window.

Your Author: Michael Lam is preserved on the commit — I amended only the comment text, not behavior or test files.

Local verification:

• pytest tests/test_issue604_all_providers_model_picker.py tests/test_issue2399_provider_config_flags.py tests/test_provider_mismatch.py → 78 pass / 0 fail
• Full suite pytest tests/ → 5796 passed, 6 skipped, 3 xpassed, 8 subtests passed, 0 failed (112s)

If CI on the new commit comes back green (in progress), this is ready for nesquena independent review. Thanks for the careful fix — the gating against _PROVIDER_MODELS / _PROVIDER_DISPLAY / dict-shaped configs is exactly the right shape for #2399.

[nesquena]

Review — end-to-end ✅ (clean APPROVE, no fix pushed)

What this ships

21-line fix in api/config.py to gate the provider-detection loop on either (a) a canonical id that's in _PROVIDER_MODELS / _PROVIDER_DISPLAY or (b) a dict-shaped provider config. Before the fix, the loop accepted EVERY key under providers: because the existing condition _canonical in _cfg_providers or _pid_key in _cfg_providers was always true while iterating _cfg_providers. Scalar siblings like providers.only_configured: true got canonicalized to only-configured and rendered as fake picker groups labeled Only-Configured. Closes #2399.

Surfaces touched: api/config.py (+21/-3), tests/test_issue2399_provider_config_flags.py (+79 new, 2 tests), CHANGELOG.md (+4). 3.11/3.12/3.13 CI green.

Traced against upstream hermes-agent

Pulled a fresh tarball. No cross-tool surface — _build_configured_model_badges lives in webui-only api/config.py. The CLI doesn't read these picker groups. No config.yaml shape change.

End-to-end trace

The buggy condition (pre-fix) at api/config.py:2972-2974 (before this PR):

for _pid_key in _cfg_providers:
    _canonical = _canonicalise_provider_id(_pid_key)
    ...
    if _canonical in _PROVIDER_MODELS or _canonical in _cfg_providers or _pid_key in _cfg_providers:
        detected_providers.add(_canonical)

Since we iterate _cfg_providers, the last clause _pid_key in _cfg_providers is always True. So the condition collapses to "always add canonical to detected_providers". Any scalar key under providers: becomes a fake provider group.

Verified canonical mapping of the failure-shape keys:

_canonicalise_provider_id("only_configured") = 'only-configured'   (NOT in _PROVIDER_MODELS)
_canonicalise_provider_id("future_toggle")   = 'future-toggle'     (NOT in _PROVIDER_MODELS)
_canonicalise_provider_id("opencode_go")     = 'opencode-go'       (IS  in _PROVIDER_MODELS)
_canonicalise_provider_id("nous")            = 'nous'              (IS  in _PROVIDER_MODELS)

Pre-fix, all four would have been added. Only the last two should have been.

The new gate at api/config.py:2986-2993:

_is_known_provider = (
    _canonical in _PROVIDER_MODELS or _canonical in _PROVIDER_DISPLAY
)
_is_provider_config = isinstance(_provider_cfg, dict)
if not (_is_known_provider or _is_provider_config):
    continue

OR semantics:

• Known provider id (canonical or alias) → admit, regardless of value shape.
• Dict-shaped value → admit, regardless of name (covers custom providers with api_key/base_url/models).
• Everything else (scalar value with unknown name) → skip.

Iteration shape change

for _pid_key in _cfg_providers → for _pid_key, _provider_cfg in _cfg_providers.items(). Adds value access needed for the isinstance(dict) check. Net-zero for known iteration semantics.

Other audit — things that are correct already

Branch matrix (verified)

_pid_key	value	canonical	known?	dict?	added?
openai	{models: [...]}	openai	yes	yes	✅
openai	true	openai	yes	no	✅ (known wins)
opencode_go	{api_key: ...}	opencode-go	yes	yes	✅
nous	true	nous	yes	no	✅
only_configured	true	only-configured	no	no	❌ (skipped — the fix)
future_toggle	enabled	future-toggle	no	no	❌
my-llm-server	{base_url: ...}	my-llm-server	no	yes	✅ (custom config)
<unknown-flag>	false	<some>	no	no	❌

Security

• No SQL/shell/XSS. Provider id strings come from admin-controlled config.yaml. Even with malicious config (admin self-injection), the output is rendered through the picker's text/title pipeline which escapes via t()/textContent elsewhere. No new attack surface introduced.

Backward compat

• Existing canonical-to-raw-key dedup (#2245) preserved at api/config.py:2995.
• Known providers with scalar config (e.g. nous: true, openai: true) still picked up via the _is_known_provider short-circuit. No regression for users who used scalar shorthand.

Test coverage

• test_providers_only_configured_flag_does_not_create_picker_group pins the reported case + asserts openai is still present (positive control).
• test_unknown_scalar_provider_config_flags_are_ignored pins the general shape (future_toggle: enabled → not picked up).
• Both reset _available_models_cache before/after to avoid cross-test pollution.

Edge-case trace

Scenario	Expected	Actual
providers.only_configured: true + providers.openai: {...}	openai shown, only-configured hidden	✅ test
providers.future_toggle: enabled + openai	unknown scalar skipped	✅ test
Known provider with true scalar value	shown (known wins)	✅ harness trace
Custom provider with dict value	shown (dict wins)	✅ harness trace
Empty providers: map	no detected providers	✅ skipped iteration
providers: null	not a dict → outer guard skips	✅ pre-existing
Cross-tool: CLI reads same config.yaml	unaffected (webui-only picker)	✅

Tests

• PR-targeted: 2/2 pass.
• Full local suite (Python 3.14): 5675 passed, 0 failed.
• CI: 3.11/3.12/3.13 all green.

Minor observations (non-blocking)

1. The pre-fix bug was effectively a no-op condition — _pid_key in _cfg_providers was always True. The condition appeared to validate something but did nothing. A linter that flags self-tautologies could have caught this earlier.
2. _canonicalise_provider_id accepts any string. It normalizes underscores to dashes and lowercases. Combined with the old "always add" path, this produced Only-Configured style display labels (via _canonical.title() somewhere downstream). The new gate is the right shape; the underlying canonicaliser is fine.
3. Could grow _PROVIDER_DISPLAY allow-list. If future config schema adds new top-level scalar siblings (e.g. providers.retry_count: 3, providers.timeout_ms: 5000), they correctly skip. Good design — opt-in via dict shape rather than maintaining a per-flag denylist.

Recommendation

Approved. Surgical correction of an always-true condition. Gate semantics are clear (known canonical OR dict-shaped). Two regression tests pin both the reported failure case and the broader unknown-scalar shape. No backward compat concerns for users with scalar shorthand on known providers.

✅ Parked at approval — ready for the release agent's merge/tag pipeline.

[nesquena-hermes]

Shipped via stage-373 / PR #2420 / v0.51.80 (Release BD). Attribution preserved in the squashed stage commit and the v0.51.80 CHANGELOG entry. Thank you!