Fix: add profile credential pool fallback to resolve_xai_http_credentials

[palebluewanders]

What does this PR do?

When the xAI OAuth provider state (providers.xai-oauth.tokens) is cleared by a terminal token-refresh failure (expired refresh token, tier gate, etc.), resolve_xai_http_credentials falls through to the bare XAI_API_KEY env fallback — even when valid tokens exist in the credential pool written by hermes auth add xai-oauth. This causes the x_search tool to silently degrade to synthesized responses with no real citations.

The breakage happens because hermes auth add writes fresh tokens to the credential pool (credential_pool.xai-oauth) but the x_search tool reads from the provider state singleton. xAI revokes the old refresh token on re-auth, the provider state gets cleared during the next refresh attempt, and the credential pool — which still has valid tokens — is never consulted.

Add a third resolution tier that walks <hermes_home>/profiles/*/auth.json and reads access_token + base_url from each profile's credential pool xai-oauth entries. Using get_hermes_home() (not HERMES_HOME env) avoids a silent CWD-relative Path("profiles") trap when the env var is unset or empty.

Related Issue

Similar to PR #38440: fix(auth): resolve xAI OAuth credentials across profiles but fills the gap of when tokens exist in profile-specific auth.json.

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

Added a third credential-resolution fallback to resolve_xai_http_credentials() in tools/xai_http.py: when the runtime provider and provider state both fail, walk <hermes_home>/profiles/*/auth.json and read access_token from each profile's credential_pool.xai-oauth entries. The first valid token wins (sorted alphabetically by profile name).

How to Test

How to reproduce the bug

1. Configure xAI OAuth (hermes auth add xai-oauth --no-browser)
2. Let the refresh token expire or get revoked (re-auth from another device, or wait ~1h for the access_token to expire naturally)
3. Run hermes -p ask "search X for [topic]" — observe degraded/synthesized response with no real citations despite valid tokens in the credential pool
4. Verify: cat ~/.hermes/profiles/<profile>/auth.json | jq '.credential_pool["xai-oauth"][0].access_token' shows a valid token, but providers["xai-oauth"].tokens.access_token is empty

How to test the fix

1. Clear the provider state
2. Run x_search — should return real results with citations
3. Alternatively, run tests/tools/test_xai_credential_fallback.py

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run pytest tests/ -q and all tests pass
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: Linux (Amazon Linux 2023, x86_64, Python 3.11.15), Cron worker context

Documentation & Housekeeping

• [N/A ] I've updated relevant documentation (README, docs/, docstrings) — or N/A
• [N/A] I've updated cli-config.yaml.example if I added/changed config keys — or N/A
• [N/A] I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
• [N/A] I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
• I've updated tool descriptions/schemas if I changed tool behavior — or N/A

[liuhao1024]

Verification: credential pool fallback looks correct

Reviewed the new _walk_profiles fallback path in resolve_xai_http_credentials. A few notes:

• The fallback correctly walks profiles/*/auth.json in sorted (alphabetical) order, returning the first valid xai-oauth credential pool entry — consistent with how other credential resolution paths work.
• Input validation is thorough: non-dict stores, missing credential_pool, non-list entries, empty/whitespace/None access tokens are all skipped gracefully.
• The except Exception: pass around the entire block is appropriate here — this is a best-effort fallback that must never crash the credential resolution pipeline.
• Default base_url (https://api.x.ai/v1) matches the existing fallback used in the env-var path.

One minor observation: the str(entry.get("access_token") or "").strip() pattern treats 0 and False as empty (via or ""), but access tokens are always strings so this is safe in practice.

[kshitijk4poor]

Thanks for this, @palebluewanders — same root cause, resolved a different way.

This adds the credential-pool/profile fallback directly in resolve_xai_http_credentials (+300). #46614 (merged as 8844e09) fixes the same symptom one layer up, at _read_xai_oauth_tokens — the single chokepoint that resolve_xai_http_credentials → resolve_xai_oauth_runtime_credentials already funnels through — so the x_search / image_gen / TTS HTTP paths get the pool + global-root fallback without a separate walk, and it also closes the rotating-refresh write-back gap (#43589) the read-only fix can't.

Closing as superseded by #46614. Your diagnosis of the credential-pool-vs-singleton mismatch was spot on and matches what landed.