fix(agent): resolve custom-provider key_env preferring .env over stale os.environ (#67935)

[PRATHAMESH75]

What does this PR do?

On a Desktop local gateway (or any long-lived hermes serve), a custom
OpenAI-compatible provider configured with key_env fails with 401 after
~/.hermes/.env is edited or the key is added mid-session. A fresh hermes chat
works, which pins this as a dotenv-reload / env-resolution problem, not a bad
key: the long-lived backend snapshots os.environ once at spawn, and the custom
key_env was resolved with plain os.getenv(), so a mid-session .env change was
never seen. The request then fell through to the no-key-required placeholder
and 401'd until the backend was restarted.

The codebase already has get_env_value_prefer_dotenv() (in hermes_cli/config.py)
built for exactly this ".env edited mid-session → stale os.environ → persistent
401" class — and the credential-pool seeding path already uses it. Custom-provider
resolution didn't, which is the inconsistency this fixes.

Related Issue

Fixes #67935

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

• agent/auxiliary_client.py: new _resolve_config_key_env() helper that resolves
  a config-declared credential key_env through get_env_value_prefer_dotenv()
  (fresh .env wins over stale inherited os.environ). Routed the three
  config-driven key_env → os.getenv() sites through it — all the same idiom,
  same bug class:
  • named custom-provider resolution in resolve_provider_client (the reported
    repro path — the no-key-required 401 site),
  • _fallback_entry_api_key (fallback-chain entries),
  • auxiliary-task key_env resolution.
• tests/agent/test_auxiliary_key_env_prefer_dotenv.py: new regression tests —
  key present only in .env resolves; a rotated .env value wins over a stale
  os.environ value; unset → empty (caller keeps no-key-required); and the
  named-custom-provider resolution shape picks up the live .env key.

How to Test

1. scripts/run_tests.sh tests/agent/test_auxiliary_key_env_prefer_dotenv.py -q → passes.
2. Regression suite for sibling paths stays green:
   scripts/run_tests.sh tests/agent/test_auxiliary_named_custom_providers.py tests/tools/test_credential_pool_env_fallback.py tests/hermes_cli/test_runtime_provider_resolution.py -q → 200 passed.
3. Manual: with a long-lived hermes serve, add/rotate LONGCAT_API_KEY in
   ~/.hermes/.env mid-session and send a request — the fresh key is now used
   without restarting the backend.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix
• I've run the affected tests and they pass
• I've added tests for my changes
• I've tested on my platform: macOS (arm64)

Documentation & Housekeeping

• I've updated relevant documentation — N/A (behavioral fix, no new config keys)
• I've updated cli-config.yaml.example if I added/changed config keys — N/A
• I've updated CONTRIBUTING.md or AGENTS.md — N/A
• I've considered cross-platform impact — platform-independent (env/dotenv resolution)

Note

Related to #67453 (same root-cause class on the Docker gateway path). This PR
scopes to the custom-provider key_env credential-resolution sites in
agent/auxiliary_client.py that the #67935 repro exercises.

[alt-glitch]

This was generated by AI during triage.

Related to #67935 and #55972. This PR covers declared custom-provider, fallback-chain, and auxiliary-task key_env resolution; #55972 covers other auxiliary default-key paths.

[teknium1]

Thanks for tracing the stale-environment behavior. The auxiliary defect is live: agent/auxiliary_client.py:5517-5519 still resolves a named custom provider's key_env with os.getenv(), while hermes_cli/config.py:3984-4011 already supplies the intended dotenv-first resolver.

Problems

• The stated Desktop/local-gateway primary-provider repro remains unfixed. hermes_cli/runtime_provider.py:692-705 and :1115-1136 resolve custom key_env through _getenv; _getenv reaches agent/secret_scope.py:176-177, which reads os.environ in the normal single-profile path. Extend the fix with a profile-safe dotenv-first read there, or narrow the PR to auxiliary resolution only.
• tests/agent/test_auxiliary_key_env_prefer_dotenv.py copies the named-provider selection logic rather than invoking resolve_provider_client, so it does not protect the production branch at agent/auxiliary_client.py:5515-5520 or the other two changed call paths.

Suggested changes

• Add production-path tests for named custom providers, _fallback_entry_api_key, and _resolve_task_provider_model, with stale os.environ and a fresh temporary .env.

Automated hermes-sweeper review.

[GottZ]

This was generated by AI during triage.

Summary

Two PRs address the stale-environment credential complex through different runtime paths: #67843 rebuilds the cached primary client at conversation-turn boundaries, while #67947 applies dotenv-first key resolution to auxiliary custom-provider and task-provider paths. Their overlap is the named custom-provider key_env case, but only #67843's visible diff updates the already-running primary client described in #67821 and #67935.

Related pull requests

• fix(agent): adopt .env credential/base-url edits at the turn boundary #67843 best fix — (+479/-24) — n/a: The diff adds per-turn primary-client credential and base-URL refresh, including named custom providers, transactional retry after rebuild failure, and route-derived TLS/header recalculation, directly covering Desktop: provider base URL / API key saved in Settings should apply to live sessions immediately #67821 and the primary-session path of [Bug]: Desktop local gateway: custom provider key_env fails 401 after .env edit (long-lived hermes serve) #67935. The contributor keep_open review identified missing custom-provider coverage, premature baseline advancement, scoped-secret regression, and stale route configuration; the current visible diff addresses each objection, but contributor re-review is still appropriate because the follow-up claims came from a non-contributor.
• fix(agent): resolve custom-provider key_env preferring .env over stale os.environ (#67935) #67947 best fix — (+125/-2) — n/a: The diff introduces dotenv-first key_env resolution in auxiliary named-custom-provider and task-provider paths, but it does not modify the primary runtime-provider resolution identified by the maintainer-bot as the Desktop repro path, and its named-provider test mirrors selection logic rather than invoking the production branch. Consistent with the maintainer-bot keep_open verdict, its high-salvage value is the narrower auxiliary-path fix, with production-path tests still requested.

Duplicates

#67843 and #67947 overlap on dotenv-first named custom-provider key_env refresh for #67935, but they are not complete duplicates: #67843 changes cached primary-client lifecycle behavior, whereas #67947 changes auxiliary resolution paths.

Suggested consolidation

Keep #67843 open with a salvage path centered on its per-turn primary-client refresh and obtain contributor re-review of the current diff against the four previously blocking concerns. Keep #67947 open with a salvage path narrowed to auxiliary resolution, adding production-path tests for the changed named-custom and task-provider branches; alternatively, its auxiliary changes can be split out, while #67843 remains the broader implementation for the primary live-session cause.

Complex graph

flowchart LR
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I67935(["issue #67935 (open)"])
    P67947["PR #67947 (open)"]
    P67947 -->|best fix| I67935
    class I67935 open
    class P67947 open
    class P67947 best
    class P67947 target
    click I67935 "https://github.com/NousResearch/hermes-agent/issues/67935"
    click P67947 "https://github.com/NousResearch/hermes-agent/pull/67947"

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).

Cross-PR triage: Reviewed 2 pull requests and 2 issues in this complex. Each diff was read against this issue; Assessment working set: 33 kB of PR diffs, 14 kB of issue/PR text, 6 kB of discussion (13 comments), 7 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.

[Enough1122]

AI code review — automated review for reference, author can ignore or act on any point.

fix(agent): resolve custom-provider key_env preferring .env over stale os.environ (#67935)

1. The dotenv cache is keyed by (path, mtime, size) — agent/auxiliary_client.py _resolve_config_key_env routes through get_env_value_prefer_dotenv, whose backing load_env cache can serve a stale value if a .env edit preserves size and mtime granularity (rare but possible on filesystems with 1s mtime). The tests carefully call invalidate_env_cache() after writes; production relies on mtime/size. Worth a sentence in the docstring documenting the cache boundary so future edits don't assume a fresh read every call.
2. The lazy import can raise mid-resolution — from hermes_cli.config import get_env_value_prefer_dotenv inside the function: if hermes_cli.config fails to import in a long-lived backend (the exact skew scenario this fix family guards against), the call site now raises instead of falling back to the previous os.getenv behavior. Consider wrapping in try/except and falling back to plain os.getenv(key_env) for resilience.
3. Behavioral intent is clear and consistent — dotenv-wins-over-stale-environ matches the credential-pool seeding path, so the auxiliary/custom-provider routes now agree with the main runtime. The tests (fresh-dotenv, rotated-dotenv-over-stale-env, empty-when-unset, named-custom-provider shape) pin the invariant well.