fix: fallback chain nous normalization + same-provider rate-limit dedup - #33996
Closed
OmarB97 wants to merge 16 commits into
Closed
fix: fallback chain nous normalization + same-provider rate-limit dedup#33996OmarB97 wants to merge 16 commits into
OmarB97 wants to merge 16 commits into
Conversation
Pre-filter fallback_providers when the agent starts up — entries whose provider has no credentials (no API key in env, no custom_providers definition, no base_url with env var set) are dropped silently from the chain and a startup diagnostic lists them with a hint naming the env var that would activate them. Before: runtime fallback loop discovered each failure one-by-one via 401/402/429 errors → wall of red logs, multiple seconds of latency, confusing UX (the #1 source of "why does my fallback keep 401ing"). After: chain is clean from the first failed API call; startup prints e.g. "Fallback chain: skipped 4 unconfigured: deepseek, minimax, zai, kimi-moonshot" with per-entry hints like "set DEEPSEEK_API_KEY in ~/.hermes/.env" or "`hermes auth add nous`".
- Add resolve_auto_skin() with 6-step terminal detection chain: HERMES_LIGHT/TUI_THEME/TUI_BACKGROUND env vars, COLORFGBG, OSC 11 query, default dark - Add _SKIN_VARIANT_PAIRS mapping dark/light skin pairs - Update load_skin(), set_active_skin(), init_skin_from_config() to handle 'auto' keyword - Update list_skins() to show 'auto' as virtual entry - Update /skin command in cli.py to display resolved auto skin - Works across all machines: dark terminal → dark variant, light terminal → light variant, no per-machine config needed Resolves: auto skin resolution for multi-device Hermes deployments
When config files (e.g. ~/.hermes/config.yaml) are symlinked to a different filesystem — common on WSL where dotfiles live under /mnt/c/ — os.replace() fails with EXDEV (errno 18: Invalid cross-device link). Fall back to shutil.copy2 + os.unlink when EXDEV is raised. copy2 preserves metadata (permissions, timestamps) so the result is equivalent. Non-EXDEV errors (EACCES, etc.) still propagate. This affects all atomic write paths (atomic_yaml_write, atomic_json_write, atomic_roundtrip_yaml_update) since they all route through atomic_replace. Added 4 regression tests: - Direct EXDEV fallback on plain file - EXDEV fallback preserving symlinks - Non-EXDEV errors are not swallowed - End-to-end atomic_yaml_write EXDEV recovery
…el rejected
When the user's main chat model is forwarded verbatim to a provider that
only supports a subset of its catalog for auxiliary (cheap / fast) tasks,
the operator currently sees a confusing HTTP 401 or HTTP 404 against a
model that clearly works for chat. Example: mimo-v2.5 on opencode-zen
returns HTTP 401: Model mimo-v2.5 is not supported for aux tasks,
while chat succeeds.
Root cause: call_llm() raises the error instead of trying the
provider's registered default aux model. The existing fallback chain
(payment / connection / rate-limit) correctly bypasses this case because
model-unsupported errors are neither quota-nor-connectivity-related.
Add is_model_unsupported_error() (agent/aux_unsupported_model.py) to
distinguish model rejection from credential / quota / connectivity
failures:
* 404 with a 'model' + 'not found/does not exist/unknown' marker
* 401 with explicit 'not supported / unsupported model' wording
(aggregators sometimes classify unsupported models as 401 instead of 404)
* 400/422 with 'not supported / unsupported model' wording
* Pure credential failures (401 without model wording) return False
so the existing auth-refresh chain still handles them.
Extend call_llm() with a 'Model-unsupported fallback' block inserted
between the existing payment/connection fallback and the teardown:
* For explicit (non-auto) providers: resolve the provider's default aux
model (ProviderProfile.default_aux_model or legacy fallback dict) and
retry the same request against it.
* Auto-users are skipped because their rejected model IS the aux model
— retrying would be a no-op.
* If the rejected model already equals the provider's default, no retry.
* If the retry also fails, re-raise the original error.
Operator impact: users on opencode-zen / Nous Portal / any provider whose
aux catalog differs from its chat catalog now get transparent recovery
for title generation, compression, session search, etc. — they no longer
need to manually pin auxiliary.<task>.provider/model in config.yaml.
When a user types `/model mimo-v2.5` on opencode-zen (or any aggregator),
the model switch pipeline falls through to `detect_provider_for_model()`
which finds the model in xiaomi's static catalog and switches providers —
requiring an API key the user doesn't have. The user gets a 401 auth error
instead of staying on their configured aggregator.
Root cause chain:
1. `/model mimo-v2.5` triggers `switch_model()` step d (aggregator catalog search)
2. `list_provider_models('opencode')` queries models.dev cache which has
`mimo-v2.5-free` (stale/free-tier slug) — no match for bare `mimo-v2.5`
3. Step d fails → `resolved_in_current_catalog` stays False
4. Step e `detect_provider_for_model('mimo-v2.5', 'opencode')` finds it in
xiaomi static catalog
5. Provider switches to xiaomi → needs `XIAOMI_API_KEY` → 401 → fallback
chain → abort
Changes:
- Aggregator catalog fallback (step d): After models.dev catalog search
fails to match, added two additional checks:
1. Check `custom_providers` model dicts from config.yaml — if the model
is listed there, it's authoritative for the user's configured aggregator
2. Query the live `/v1/models` endpoint via `fetch_endpoint_model_metadata()`
— the live API returns current model IDs (e.g. `mimo-v2.5`) not stale
models.dev slugs (e.g. `mimo-v2.5-free`)
- Validation override matching: Fixed the `custom_providers` override in
the validation rejection handler to also match `entry_name == target_provider`
and `entry_name == normalized_target` (e.g. both `opencode-zen` and
`opencode`), not just `custom:<name>`. Built-in provider aliases that
also appear in `custom_providers` were invisible to the old check.
- Import: Added `normalize_provider` to the imports from `hermes_cli.providers`.
Tests: 36 passed (1 pre-existing fixture failure unrelated to change).
…ast-fail
When Nous Portal returns HTTP 503 with 'upstream capacity limits', the
error classifier now marks it retryable=False, should_fallback=True
instead of retryable=True. This tells the conversation loop to
immediately fall back to the next provider in the chain (e.g., kimi-k2.6
via opencode-zen) instead of burning 2+ minutes on exponential backoff
retries that will all fail with the same capacity error.
Also added eager-fallback logic in conversation_loop.py (lines 2550-2563)
that explicitly checks for overload errors and triggers fallback,
mirroring the existing rate-limit eager-fallback path.
Changes:
- agent/error_classifier.py: classify 'upstream capacity limits' in
503/529 errors as non-retryable with should_fallback=True
- agent/conversation_loop.py: add eager fallback check for overload
errors (lines 2550-2563)
- tests/agent/test_error_classifier.py: 3 new tests validating the
capacity-pattern detection (153 total, all passing)
MeshBoard task: hermes-503-capacity-fast-fail
Nous Portal rejects requests with extra_body.tags from non-Portal aux clients (title generation, compression, vision). The tags field is not accepted by all auxiliary endpoints that the aux adapter falls back to. Returning empty list from nous_portal_tags() prevents the 400 errors while keeping the function signature intact for future conditional re-enablement. Fixes: 'Extra inputs are not permitted, field: tags' on aux title generation
When the agent has active tools, skip fallback providers marked with
supports_tools: false. This prevents silent failures when falling back
to local models or providers that reject tool schemas (e.g. kimi-k2.6
via opencode-zen returning 400, llama.cpp models that 'think about'
tools but never produce actual tool_calls).
Usage in config.yaml:
fallback_providers:
- provider: opencode-zen
model: kimi-k2.6
supports_tools: false
- provider: local-llama-swap
model: qwen3.6-27b
supports_tools: false
Fixes cascading agent death: Nous 503 → fallback to non-tool model
→ 400 or tool-less response → session hangs.
feat: filter fallback chain by tool capability (supports_tools flag)
The module-level `auxiliary_is_nous` flag is only reset inside _resolve_auto(), which is never called when a task has an explicit provider configured (e.g. auxiliary.title_generation.provider: opencode-zen). A stale True value from a previous call (e.g. compression via auto→Nous) leaks Nous Portal tags into requests for non-Nous providers, causing HTTP 400 'Extra inputs are not permitted, field: tags'. Reset the flag in both call_llm() and async_call_llm() when the resolved provider is not 'auto', so it accurately reflects whether the current call targets Nous Portal.
fix: reset auxiliary_is_nous for non-auto provider resolution paths
Add real-time macOS/Linux/Windows appearance change detection to the skin engine. When display.skin is set to 'auto', a background daemon thread polls the OS appearance every 2 seconds. When a change is detected (light->dark or dark->light), the skin is automatically switched without requiring user action or session restart. Changes: - Add macOS AppleInterfaceStyle detection to _detect_terminal_is_light() - Add AppearanceWatcher class with cross-platform polling (macOS/Linux/Windows) - Start watcher from TUI gateway entry point (emits skin.changed event) - Start watcher from CLI main loop (re-initializes skin) - Add get_current_os_appearance() standalone helper - Add 8 new tests (41 total pass)
feat: auto-detect OS appearance changes and switch skins mid-session
…oxes * fix: add tests for aggregator fallback + fix custom: prefix matching Address PR NousResearch#33716 review feedback: 1. Add tests for the new aggregator fallback code paths: - custom_providers model dict iteration resolves models not in models.dev - live fetch_endpoint_model_metadata fallback resolves missing models - live API exceptions are silently swallowed (except Exception: pass) - canonical model casing from live endpoint is preserved 2. Fix custom: prefix slug matching: - target_provider='Custom:OpenRouter' now matches entry_name='OpenRouter' - Added split(':',1)[-1].lower() suffix comparison as 5th condition - Normalise always lowercases so 'custom:openrouter' won't match 'OpenRouter' by name alone; the suffix strip handles arbitrary casing. Closes: NousResearch#33716 review feedback (liuhao1024) * fix: add HERMES_SKIP_PROFILE_OVERRIDE escape hatch for launcher sandboxes MeshBoard's stream-tap launcher creates a per-dispatch HERMES_HOME sandbox whose .env points at a local loopback proxy. Hermes' _apply_profile_override() was reading ~/.hermes/active_profile and clobbering the sandbox path, causing every dispatch to bypass the tap. Add an early return when HERMES_SKIP_PROFILE_OVERRIDE=1 is set in the environment. The MeshBoard launcher will set this alongside HERMES_HOME so the sandbox is honoured verbatim. Refs meshboard task: hermes-stream-tap-profile-override --------- Co-authored-by: Omar B <omar@kostudios.io>
Two systemic bugs in the fallback provider chain: 1. nous provider was misclassified as _AGGREGATOR_PROVIDERS, causing _prepend_vendor() to keep the nousresearch/ prefix when falling back to the Nous direct API (inference-api.nousresearch.com). The direct API expects bare model names like hermes-4-70b, not nousresearch/hermes-4-70b, resulting in HTTP 400 'Bad Request'. Fix: Move nous from _AGGREGATOR_PROVIDERS to _MATCHING_PREFIX_STRIP_PROVIDERS and add 'nousresearch' as a provider alias for 'nous' so _strip_matching_provider_prefix() correctly strips the nousresearch/ prefix. 2. _try_activate_fallback() only deduped entries matching the exact (provider, model) pair. When the primary provider (e.g. opencode-zen) was rate-limited, all fallback entries on the same provider would also hit the same 429, burning the retry budget pointlessly. Fix: When the trigger is a rate-limit (429) or billing error and an active cooldown is set, skip fallback entries on the same provider. Different providers are still tried normally. Files changed: - hermes_cli/model_normalize.py: Move nous from aggregator to matching-prefix-strip - hermes_cli/models.py: Add nousresearch → nous alias - agent/chat_completion_helpers.py: Add same-provider rate-limit dedup - tests/hermes_cli/test_model_normalize.py: Update nous normalization tests - tests/run_agent/test_provider_fallback.py: Add same-provider cooldown test Refs: fallback-chain-nous-normalization-same-provider-rate
OmarB97
deleted the
fix/fallback-chain-nous-normalization-same-provider-dedup
branch
May 28, 2026 17:11
Closed
3 tasks
alt-glitch
added
type/bug
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two systemic bugs in the fallback provider chain:
Bug 1: nous provider misclassified as aggregator → HTTP 400
The
nousprovider was in_AGGREGATOR_PROVIDERS, causing_prepend_vendor()to keep thenousresearch/prefix when falling back to the Nous direct API (inference-api.nousresearch.com). The direct API expects bare model names likehermes-4-70b, notnousresearch/hermes-4-70b, resulting in HTTP 400 "Bad Request".Fix: Move
nousfrom_AGGREGATOR_PROVIDERSto_MATCHING_PREFIX_STRIP_PROVIDERSand addnousresearchas a provider alias fornousso_strip_matching_provider_prefix()correctly strips thenousresearch/prefix.Bug 2: same-provider rate-limit loop → all fallbacks hit same 429
_try_activate_fallback()only deduped entries matching the exact(provider, model)pair. When the primary provider (e.g. opencode-zen) was rate-limited, all fallback entries on the same provider would also hit the same 429, burning the retry budget pointlessly.Fix: When the trigger is a rate-limit (429) or billing error and an active cooldown is set, skip fallback entries on the same provider. Different providers are still tried normally.
Files Changed
hermes_cli/model_normalize.py: Move nous from aggregator to matching-prefix-striphermes_cli/models.py: Addnousresearch→nousaliasagent/chat_completion_helpers.py: Add same-provider rate-limit deduptests/hermes_cli/test_model_normalize.py: Update nous normalization teststests/run_agent/test_provider_fallback.py: Add same-provider cooldown testTesting
Fork PR
OmarB97#19