fix(auth): heal poisoned stored Nous inference URL on the no-refresh read path - #50265
fix(auth): heal poisoned stored Nous inference URL on the no-refresh read path#50265r266-tech wants to merge 1 commit into
Conversation
…read path NousResearch#49735 healed a poisoned inference_base_url only on the OAuth-refresh path. On the no-refresh read path (valid invoke JWT, the common steady state) resolve_nous_runtime_credentials read the stored value via _optional_base_url and served it unvalidated, so a poisoned auth.json 401s forever (a 401 from the dead host never triggers a refresh). Validate the stored (network-provenance) value on read, mirroring the refresh-site heal; the user-provided NOUS_INFERENCE_BASE_URL env override stays unvalidated.
|
Related: #49735 (heals a poisoned This PR fills the gap that #49735 does not cover: the no-refresh read path in |
…sessions (#52270) The host-allowlist hardening (#30611) plus the refresh heal (#49735) left the documented NOUS_INFERENCE_BASE_URL dev/staging escape hatch unreachable for OAuth sessions, despite three code comments asserting it still works. Root cause — resolution precedence in resolve_nous_runtime_credentials: inference_base_url = ( _optional_base_url(state.get("inference_base_url")) # stored — wins or os.getenv("NOUS_INFERENCE_BASE_URL") # env — unreachable or DEFAULT_NOUS_INFERENCE_URL ) A staging OAuth login persists its inference_base_url, but the allowlist rejects the staging host and the refresh heal rewrites the stored value to the production default. The stored (now prod) value is then read BEFORE the env var, so the override never takes effect — every request 401s against prod or is pinned to prod, and setting the env var does nothing. Fix: the user-set env override is the most-trusted source, so consult it FIRST for the URL used to build the client / returned to callers — while keeping the PERSISTED value the validated, network-provenance one (the override is a runtime overlay, never written to auth.json, so unsetting it cleanly reverts to prod). Applied at both chokepoints: - resolve_nous_runtime_credentials (no-refresh read path AND refresh path) - the nous_portal proxy adapter, which re-validates the resolver's returned base_url against the prod allowlist as defense-in-depth and would otherwise reject a legitimate staging override at the forward boundary. New _nous_inference_env_override() / split of stored-vs-effective URL keep the threat model intact: Portal-returned URLs are still allowlist-validated at every network site, and the env path stays ungated (trusted OS user). Also folds in the no-refresh read-path heal (supersedes the approach in the open #50265): a poisoned stored staging host now heals to the prod default on read even when no refresh fires. Tests: TestEnvOverrideWins (env wins on read + refresh paths; override never persisted; poisoned stored heals) and TestProxyAdapterEnvOverride. Verified the 4 behavioral tests fail against pre-fix code and pass with the fix; full inference-validation + nous-provider suites green (85 passed). E2E-validated against a real temp HERMES_HOME exercising the real resolver + proxy adapter: resolver→staging, persisted→prod, proxy→staging, unset→reverts to prod.
|
Superseded by #52270 (merged), which folds in the no-refresh-read-path heal you implemented here AND fixes the #52270 explicitly cites this PR as superseded and credits your read-path heal. Thanks for the fix — the core idea landed. |
…sessions (NousResearch#52270) The host-allowlist hardening (NousResearch#30611) plus the refresh heal (NousResearch#49735) left the documented NOUS_INFERENCE_BASE_URL dev/staging escape hatch unreachable for OAuth sessions, despite three code comments asserting it still works. Root cause — resolution precedence in resolve_nous_runtime_credentials: inference_base_url = ( _optional_base_url(state.get("inference_base_url")) # stored — wins or os.getenv("NOUS_INFERENCE_BASE_URL") # env — unreachable or DEFAULT_NOUS_INFERENCE_URL ) A staging OAuth login persists its inference_base_url, but the allowlist rejects the staging host and the refresh heal rewrites the stored value to the production default. The stored (now prod) value is then read BEFORE the env var, so the override never takes effect — every request 401s against prod or is pinned to prod, and setting the env var does nothing. Fix: the user-set env override is the most-trusted source, so consult it FIRST for the URL used to build the client / returned to callers — while keeping the PERSISTED value the validated, network-provenance one (the override is a runtime overlay, never written to auth.json, so unsetting it cleanly reverts to prod). Applied at both chokepoints: - resolve_nous_runtime_credentials (no-refresh read path AND refresh path) - the nous_portal proxy adapter, which re-validates the resolver's returned base_url against the prod allowlist as defense-in-depth and would otherwise reject a legitimate staging override at the forward boundary. New _nous_inference_env_override() / split of stored-vs-effective URL keep the threat model intact: Portal-returned URLs are still allowlist-validated at every network site, and the env path stays ungated (trusted OS user). Also folds in the no-refresh read-path heal (supersedes the approach in the open NousResearch#50265): a poisoned stored staging host now heals to the prod default on read even when no refresh fires. Tests: TestEnvOverrideWins (env wins on read + refresh paths; override never persisted; poisoned stored heals) and TestProxyAdapterEnvOverride. Verified the 4 behavioral tests fail against pre-fix code and pass with the fix; full inference-validation + nous-provider suites green (85 passed). E2E-validated against a real temp HERMES_HOME exercising the real resolver + proxy adapter: resolver→staging, persisted→prod, proxy→staging, unset→reverts to prod.
…sessions (NousResearch#52270) The host-allowlist hardening (NousResearch#30611) plus the refresh heal (NousResearch#49735) left the documented NOUS_INFERENCE_BASE_URL dev/staging escape hatch unreachable for OAuth sessions, despite three code comments asserting it still works. Root cause — resolution precedence in resolve_nous_runtime_credentials: inference_base_url = ( _optional_base_url(state.get("inference_base_url")) # stored — wins or os.getenv("NOUS_INFERENCE_BASE_URL") # env — unreachable or DEFAULT_NOUS_INFERENCE_URL ) A staging OAuth login persists its inference_base_url, but the allowlist rejects the staging host and the refresh heal rewrites the stored value to the production default. The stored (now prod) value is then read BEFORE the env var, so the override never takes effect — every request 401s against prod or is pinned to prod, and setting the env var does nothing. Fix: the user-set env override is the most-trusted source, so consult it FIRST for the URL used to build the client / returned to callers — while keeping the PERSISTED value the validated, network-provenance one (the override is a runtime overlay, never written to auth.json, so unsetting it cleanly reverts to prod). Applied at both chokepoints: - resolve_nous_runtime_credentials (no-refresh read path AND refresh path) - the nous_portal proxy adapter, which re-validates the resolver's returned base_url against the prod allowlist as defense-in-depth and would otherwise reject a legitimate staging override at the forward boundary. New _nous_inference_env_override() / split of stored-vs-effective URL keep the threat model intact: Portal-returned URLs are still allowlist-validated at every network site, and the env path stays ungated (trusted OS user). Also folds in the no-refresh read-path heal (supersedes the approach in the open NousResearch#50265): a poisoned stored staging host now heals to the prod default on read even when no refresh fires. Tests: TestEnvOverrideWins (env wins on read + refresh paths; override never persisted; poisoned stored heals) and TestProxyAdapterEnvOverride. Verified the 4 behavioral tests fail against pre-fix code and pass with the fix; full inference-validation + nous-provider suites green (85 passed). E2E-validated against a real temp HERMES_HOME exercising the real resolver + proxy adapter: resolver→staging, persisted→prod, proxy→staging, unset→reverts to prod.
…sessions (NousResearch#52270) The host-allowlist hardening (NousResearch#30611) plus the refresh heal (NousResearch#49735) left the documented NOUS_INFERENCE_BASE_URL dev/staging escape hatch unreachable for OAuth sessions, despite three code comments asserting it still works. Root cause — resolution precedence in resolve_nous_runtime_credentials: inference_base_url = ( _optional_base_url(state.get("inference_base_url")) # stored — wins or os.getenv("NOUS_INFERENCE_BASE_URL") # env — unreachable or DEFAULT_NOUS_INFERENCE_URL ) A staging OAuth login persists its inference_base_url, but the allowlist rejects the staging host and the refresh heal rewrites the stored value to the production default. The stored (now prod) value is then read BEFORE the env var, so the override never takes effect — every request 401s against prod or is pinned to prod, and setting the env var does nothing. Fix: the user-set env override is the most-trusted source, so consult it FIRST for the URL used to build the client / returned to callers — while keeping the PERSISTED value the validated, network-provenance one (the override is a runtime overlay, never written to auth.json, so unsetting it cleanly reverts to prod). Applied at both chokepoints: - resolve_nous_runtime_credentials (no-refresh read path AND refresh path) - the nous_portal proxy adapter, which re-validates the resolver's returned base_url against the prod allowlist as defense-in-depth and would otherwise reject a legitimate staging override at the forward boundary. New _nous_inference_env_override() / split of stored-vs-effective URL keep the threat model intact: Portal-returned URLs are still allowlist-validated at every network site, and the env path stays ungated (trusted OS user). Also folds in the no-refresh read-path heal (supersedes the approach in the open NousResearch#50265): a poisoned stored staging host now heals to the prod default on read even when no refresh fires. Tests: TestEnvOverrideWins (env wins on read + refresh paths; override never persisted; poisoned stored heals) and TestProxyAdapterEnvOverride. Verified the 4 behavioral tests fail against pre-fix code and pass with the fix; full inference-validation + nous-provider suites green (85 passed). E2E-validated against a real temp HERMES_HOME exercising the real resolver + proxy adapter: resolver→staging, persisted→prod, proxy→staging, unset→reverts to prod.
…sessions (NousResearch#52270) The host-allowlist hardening (NousResearch#30611) plus the refresh heal (NousResearch#49735) left the documented NOUS_INFERENCE_BASE_URL dev/staging escape hatch unreachable for OAuth sessions, despite three code comments asserting it still works. Root cause — resolution precedence in resolve_nous_runtime_credentials: inference_base_url = ( _optional_base_url(state.get("inference_base_url")) # stored — wins or os.getenv("NOUS_INFERENCE_BASE_URL") # env — unreachable or DEFAULT_NOUS_INFERENCE_URL ) A staging OAuth login persists its inference_base_url, but the allowlist rejects the staging host and the refresh heal rewrites the stored value to the production default. The stored (now prod) value is then read BEFORE the env var, so the override never takes effect — every request 401s against prod or is pinned to prod, and setting the env var does nothing. Fix: the user-set env override is the most-trusted source, so consult it FIRST for the URL used to build the client / returned to callers — while keeping the PERSISTED value the validated, network-provenance one (the override is a runtime overlay, never written to auth.json, so unsetting it cleanly reverts to prod). Applied at both chokepoints: - resolve_nous_runtime_credentials (no-refresh read path AND refresh path) - the nous_portal proxy adapter, which re-validates the resolver's returned base_url against the prod allowlist as defense-in-depth and would otherwise reject a legitimate staging override at the forward boundary. New _nous_inference_env_override() / split of stored-vs-effective URL keep the threat model intact: Portal-returned URLs are still allowlist-validated at every network site, and the env path stays ungated (trusted OS user). Also folds in the no-refresh read-path heal (supersedes the approach in the open NousResearch#50265): a poisoned stored staging host now heals to the prod default on read even when no refresh fires. Tests: TestEnvOverrideWins (env wins on read + refresh paths; override never persisted; poisoned stored heals) and TestProxyAdapterEnvOverride. Verified the 4 behavioral tests fail against pre-fix code and pass with the fix; full inference-validation + nous-provider suites green (85 passed). E2E-validated against a real temp HERMES_HOME exercising the real resolver + proxy adapter: resolver→staging, persisted→prod, proxy→staging, unset→reverts to prod.
…sessions (NousResearch#52270) The host-allowlist hardening (NousResearch#30611) plus the refresh heal (NousResearch#49735) left the documented NOUS_INFERENCE_BASE_URL dev/staging escape hatch unreachable for OAuth sessions, despite three code comments asserting it still works. Root cause — resolution precedence in resolve_nous_runtime_credentials: inference_base_url = ( _optional_base_url(state.get("inference_base_url")) # stored — wins or os.getenv("NOUS_INFERENCE_BASE_URL") # env — unreachable or DEFAULT_NOUS_INFERENCE_URL ) A staging OAuth login persists its inference_base_url, but the allowlist rejects the staging host and the refresh heal rewrites the stored value to the production default. The stored (now prod) value is then read BEFORE the env var, so the override never takes effect — every request 401s against prod or is pinned to prod, and setting the env var does nothing. Fix: the user-set env override is the most-trusted source, so consult it FIRST for the URL used to build the client / returned to callers — while keeping the PERSISTED value the validated, network-provenance one (the override is a runtime overlay, never written to auth.json, so unsetting it cleanly reverts to prod). Applied at both chokepoints: - resolve_nous_runtime_credentials (no-refresh read path AND refresh path) - the nous_portal proxy adapter, which re-validates the resolver's returned base_url against the prod allowlist as defense-in-depth and would otherwise reject a legitimate staging override at the forward boundary. New _nous_inference_env_override() / split of stored-vs-effective URL keep the threat model intact: Portal-returned URLs are still allowlist-validated at every network site, and the env path stays ungated (trusted OS user). Also folds in the no-refresh read-path heal (supersedes the approach in the open NousResearch#50265): a poisoned stored staging host now heals to the prod default on read even when no refresh fires. Tests: TestEnvOverrideWins (env wins on read + refresh paths; override never persisted; poisoned stored heals) and TestProxyAdapterEnvOverride. Verified the 4 behavioral tests fail against pre-fix code and pass with the fix; full inference-validation + nous-provider suites green (85 passed). E2E-validated against a real temp HERMES_HOME exercising the real resolver + proxy adapter: resolver→staging, persisted→prod, proxy→staging, unset→reverts to prod.
…sessions (NousResearch#52270) The host-allowlist hardening (NousResearch#30611) plus the refresh heal (NousResearch#49735) left the documented NOUS_INFERENCE_BASE_URL dev/staging escape hatch unreachable for OAuth sessions, despite three code comments asserting it still works. Root cause — resolution precedence in resolve_nous_runtime_credentials: inference_base_url = ( _optional_base_url(state.get("inference_base_url")) # stored — wins or os.getenv("NOUS_INFERENCE_BASE_URL") # env — unreachable or DEFAULT_NOUS_INFERENCE_URL ) A staging OAuth login persists its inference_base_url, but the allowlist rejects the staging host and the refresh heal rewrites the stored value to the production default. The stored (now prod) value is then read BEFORE the env var, so the override never takes effect — every request 401s against prod or is pinned to prod, and setting the env var does nothing. Fix: the user-set env override is the most-trusted source, so consult it FIRST for the URL used to build the client / returned to callers — while keeping the PERSISTED value the validated, network-provenance one (the override is a runtime overlay, never written to auth.json, so unsetting it cleanly reverts to prod). Applied at both chokepoints: - resolve_nous_runtime_credentials (no-refresh read path AND refresh path) - the nous_portal proxy adapter, which re-validates the resolver's returned base_url against the prod allowlist as defense-in-depth and would otherwise reject a legitimate staging override at the forward boundary. New _nous_inference_env_override() / split of stored-vs-effective URL keep the threat model intact: Portal-returned URLs are still allowlist-validated at every network site, and the env path stays ungated (trusted OS user). Also folds in the no-refresh read-path heal (supersedes the approach in the open NousResearch#50265): a poisoned stored staging host now heals to the prod default on read even when no refresh fires. Tests: TestEnvOverrideWins (env wins on read + refresh paths; override never persisted; poisoned stored heals) and TestProxyAdapterEnvOverride. Verified the 4 behavioral tests fail against pre-fix code and pass with the fix; full inference-validation + nous-provider suites green (85 passed). E2E-validated against a real temp HERMES_HOME exercising the real resolver + proxy adapter: resolver→staging, persisted→prod, proxy→staging, unset→reverts to prod.
Follow-up to merged #49735 (heal a poisoned Nous
inference_base_url).Problem
#49735 healed a poisoned stored URL only on the OAuth-refresh path (both refresh sites reset to the default when
_validate_nous_inference_url_from_network()rejects the Portal-returned URL). Butresolve_nous_runtime_credentialsreads the storedinference_base_urlvia the unrestricted_optional_base_url(auth.py:5612), and the heal lives insideif force_refresh or invoke_jwt_status is not None:.When the access token is still a usable invoke JWT — the common steady state — no refresh fires, the heal block is skipped, and the stored value flows straight to
state["inference_base_url"](persist) and the returnedbase_url. So a poisonedauth.json(e.g. astg-inference-api...host persisted before the allowlist existed — the exact incident #49735 cites) keeps 401ing forever: a 401 from the dead inference endpoint does not trigger an OAuth refresh, so the refresh-path heal is never reached.Fix
Validate the stored
inference_base_urlon read, before theor-chain, mirroring the refresh-site heal: reject → fall back to the default (which then persists, repairingauth.jsonon first read).The stored value is network-provenance (it was persisted from a Portal refresh response), so validating it is consistent with the existing trust model. The user-provided
NOUS_INFERENCE_BASE_URLenv override stays unvalidated — it's the documented dev/staging escape hatch — by validating only the stored branch and leaving theos.getenvline untouched.All three existing guard tests still hold:
test_no_unvalidated_inference_base_url_assignments_remain— onlystate.get(...)is read via_optional_base_url; the guardedrefreshed.get/mint_payload.getpatterns are untouched.test_validator_wired_at_all_known_call_sites— the 2 refresh-site validator calls are unchanged (the new call usesstored_inference_base_url).test_env_override_path_does_not_call_validator— theos.getenv("NOUS_INFERENCE_BASE_URL")line never gains the validator.Tests
New
TestNoRefreshReadPathHeals(3 cases, valid token so no refresh fires): poisoned stored URL → heals to default (fails on pre-fix code); allowlisted stored URL → preserved; poisoned stored +NOUS_INFERENCE_BASE_URLenv set → env value used unvalidated (escape hatch intact).Note / scope
refresh_nous_oauth_pureacceptsinference_base_urlas a caller-resolved parameter and likewise doesn't re-validate it on its own no-refresh path. I scoped this PR to the documented runtime resolver (resolve_nous_runtime_credentials, matching the #49735 incident); happy to fold a symmetric guard intorefresh_nous_oauth_pureif you'd prefer.Every symbol re-read against current
main. If this has already been addressed via a separate patch, feel free to close.