fix(agent): distinguish absent credentials from payment errors in auxiliary unhealthy-marking

[awizemann]

What does this PR do?

_mark_provider_unhealthy() hardcoded "(payment / credit error)" at WARNING
into its log line, but four of its eight call sites quarantine a provider for
non-billing reasons: absent OpenRouter/Nous credentials, a stale fallback
credential, or a rate limit. A local-only setup with zero cloud credentials
(e.g. Ollama via a custom endpoint) saw repeated WARNINGs claiming billing
problems on providers it never configured. The skip line in
_log_skip_unhealthy() had the same defect ("recently returned payment error"
for every skip).

This PR threads a truthful reason and a log level through
_mark_provider_unhealthy() and echoes the recorded reason from the skip log.
The absent-credentials branches now log "no credentials configured" at DEBUG
(an unconfigured optional provider is an expected state); the confirmed-402
branches in call_llm / acall_llm keep "payment / credit error" at WARNING
via the default. Quarantine semantics — TTLs, cache, skip behavior — are
unchanged. This composes with #59985/#60357, which fix the retry TTL for the
same call sites; this PR fixes the wording and severity.

Related Issue

Fixes #64144

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

• agent/auxiliary_client.py
  • _mark_provider_unhealthy(provider, ttl=None, reason="payment / credit error", level=logging.WARNING) — new keyword-only-by-convention params; the reason is stored per label (_aux_unhealthy_reason) and interpolated into the mark log line via logger.log(level, ...). Default preserves the exact current message and level for the confirmed-payment callers.
  • _log_skip_unhealthy() — echoes the stored reason instead of hardcoding "recently returned payment error".
  • _is_provider_unhealthy() / _reset_aux_unhealthy_cache() — evict/clear the reason entry alongside the existing dicts.
  • Call sites updated with truthful reasons:
    • _try_openrouter absent key → reason="no credentials configured", level=logging.DEBUG
    • _try_nous no auth → reason="no credentials configured", level=logging.DEBUG
    • _try_nous no usable inference JWT → reason="no usable credentials", level=logging.DEBUG
    • _try_nous cross-session rate-limit guard → reason="rate limited" (stays WARNING)
    • _call_fallback_candidate_sync / _call_fallback_candidate_async → reason="stale/unrefreshable credential" (stays WARNING)
    • call_llm / acall_llm confirmed-402 branches → unchanged (default = payment wording at WARNING)
• tests/agent/test_auxiliary_client.py
  • New TestAuxUnhealthyReasonLabeling class (5 tests) pinning: absent OpenRouter key and absent Nous auth mark at DEBUG with "no credentials configured" and emit no payment-labeled record; default (confirmed-402) marking still says "payment / credit error" at WARNING; the skip log echoes the mark reason both ways.
  • Updated two existing call-arg assertions (test_try_openrouter_pool_exhausted_no_env_marks_unhealthy, the stale-fallback-credential test) for the new kwargs.

How to Test

1. Automated: scripts/run_tests.sh tests/agent/test_auxiliary_client.py — 305 passed, 0 failed (includes the 5 new tests).
2. Manual repro: configure a local-only model (provider: custom, base_url: http://127.0.0.1:11434/v1, no OPENROUTER_API_KEY, no Nous auth), start a session, send one prompt, and watch the log.
3. Before (0.17.0 and main tip 861d69c7b):

   [WARNING] agent.auxiliary_client: Auxiliary: marking openrouter unhealthy for 60s (payment / credit error). Subsequent auxiliary calls will skip it until 23:14:12.
   [WARNING] agent.auxiliary_client: Auxiliary: marking nous unhealthy for 60s (payment / credit error). Subsequent auxiliary calls will skip it until 23:14:12.
   

   After (this branch):

   [DEBUG] agent.auxiliary_client: Auxiliary: marking openrouter unhealthy for 60s (no credentials configured). Subsequent auxiliary calls will skip it until 23:14:12.
   [DEBUG] agent.auxiliary_client: Auxiliary: marking nous unhealthy for 60s (no credentials configured). Subsequent auxiliary calls will skip it until 23:14:12.
   

   No WARNING-level payment claims remain; a genuine 402 (e.g. a depleted
   OpenRouter key) still produces the payment WARNING unchanged.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate (fix(aux): use full 600s unhealthy TTL for permanently unavailable providers #59985/fix(auxiliary): use default 600s TTL when provider permanently unavailable (#59984) #60357 change the TTL at the same call sites; neither touches the label/level — the changes compose)
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run pytest tests/ -q and all tests pass (via scripts/run_tests.sh)
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: macOS 26.0 (Apple Silicon), Python 3.11.15

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings) — docstrings updated in _mark_provider_unhealthy
• I've updated cli-config.yaml.example if I added/changed config keys — N/A (no config changes)
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A
• I've considered cross-platform impact (Windows, macOS) per the compatibility guide — log-string change only, no platform surface
• I've updated tool descriptions/schemas if I changed tool behavior — N/A

Screenshots / Logs

Trimmed log from the reproduction session (hermes 0.17.0, local-only Ollama
config, one prompt):

2026-07-13 23:13:12 [WARNING] agent.auxiliary_client: Auxiliary: marking openrouter unhealthy for 60s (payment / credit error). Subsequent auxiliary calls will skip it until 23:14:12.
2026-07-13 23:13:12 [WARNING] agent.auxiliary_client: Auxiliary Nous client unavailable: no Nous authentication found (run: hermes auth).
2026-07-13 23:13:12 [WARNING] agent.auxiliary_client: Auxiliary: marking nous unhealthy for 60s (payment / credit error). Subsequent auxiliary calls will skip it until 23:14:12.
2026-07-13 23:13:12 [WARNING] agent.auxiliary_client: Auxiliary: marking openrouter unhealthy for 60s (payment / credit error). Subsequent auxiliary calls will skip it until 23:14:12.
2026-07-13 23:13:12 [WARNING] agent.auxiliary_client: Auxiliary: marking nous unhealthy for 60s (payment / credit error). Subsequent auxiliary calls will skip it until 23:14:12.

[teknium1]

Thanks for tracing the misleading unhealthy-provider logs. The premise remains valid on current main: absent OpenRouter credentials still call _mark_provider_unhealthy() at agent/auxiliary_client.py:2053, whose mark and skip logs hardcode payment wording at agent/auxiliary_client.py:2884-2890 and 2919-2922.

Problems

• agent/auxiliary_client.py:1994 also covers an existing but exhausted OpenRouter pool. _select_pool_entry() returns (True, None) when pool.select() cannot choose an entry (agent/auxiliary_client.py:800-813); the pool selector returns None when all entries are exhausted (agent/credential_pool.py:1543-1548). Calling that state no credentials configured is inaccurate.
• The changed assertion at tests/agent/test_auxiliary_client.py:1120-1134 encodes that same exhausted-pool state as absent credentials.

Suggested changes

• Split the no-key path by pool_present, retaining DEBUG/no-configured-credentials only when no pool exists and assigning a truthful non-payment reason to an exhausted pool.
• Add a caplog regression test for the exhausted-pool state, including absence of a payment label.

Automated hermes-sweeper review.

[teknium1]

This branch also handles _select_pool_entry() returning (True, None), which current CredentialPool._select_unlocked() uses when all pool entries are exhausted (agent/credential_pool.py:1543-1548). That is an existing configured credential pool, not "no credentials configured"; split by pool_present and use a distinct truthful non-payment reason.

[GottZ]

This was generated by AI during triage.

Summary

One PR addresses #64144. #64146 replaces hardcoded payment wording with reason-aware unhealthy-state and skip logging while preserving quarantine behavior, but its OpenRouter branch labels both an absent pool and an exhausted existing pool as "no credentials configured."

Related pull requests

• fix(agent): distinguish absent credentials from payment errors in auxiliary unhealthy-marking #64146 best fix — (+158/-16) — best existing partial fix: the diff adds per-quarantine reasons and log levels, uses DEBUG for genuinely absent OpenRouter/Nous credentials, preserves WARNING-level payment wording for confirmed 402 paths, and makes skip logs report the recorded reason. The contributor's COMMENTED keep_open review identifies the remaining flaw: _select_pool_entry() can return no key for an exhausted existing pool, so that state needs a distinct truthful reason and caplog regression coverage.

Suggested consolidation

Keep #64146 open with a salvage path: retain its reason-aware tracking, truthful skip logs, absent-credential DEBUG behavior, and confirmed-payment defaults, while splitting the OpenRouter no-key path by pool presence and adding a separate exhausted-pool caplog test as requested in the contributor keep_open review. There are no duplicate PRs to close.

Complex graph

flowchart LR
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I64144(["issue #64144 (open)"])
    P64146["PR #64146 (open)"]
    P64146 -->|best fix| I64144
    class I64144 open
    class P64146 open
    class P64146 best
    class P64146 target
    click I64144 "https://github.com/NousResearch/hermes-agent/issues/64144"
    click P64146 "https://github.com/NousResearch/hermes-agent/pull/64146"

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).

Cross-PR triage: Reviewed 1 pull request and 1 issue in this complex. Each diff was read against this issue; Assessment working set: 14 kB of PR diffs, 12 kB of issue/PR text, 1 kB of discussion (1 comments), 2 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.