Skip to content

fix(tests): defer live module env load to test execution, not collection - #80350

Draft
wali-reheman wants to merge 1 commit into
NousResearch:mainfrom
wali-reheman:fix/upstream-test-env-leak
Draft

wali-reheman wants to merge 1 commit into
NousResearch:mainfrom
wali-reheman:fix/upstream-test-env-leak

Conversation

@wali-reheman

Copy link
Copy Markdown
Contributor

fixes #80343

What

Move _load_user_env() out of module scope and behind an autouse fixture that
runs after _hermetic_environment. The old module-level call ran at pytest
collection time, before the credential scrubber, leaking developer-local
env vars (e.g. SEARXNG_URL) into os.environ and causing non-deterministic
test failures.

Why

When test_sequential_chats_live.py was collected, _load_user_env() read
~/.hermes/.env and wrote into os.environ immediately — before any pytest
autouse fixtures had a chance to scrub credential-shaped variables. With
SEARXNG_URL leaked, _get_backend() in test_web_tools_config.py returned
searxng instead of firecrawl, causing those two tests to fail on machines
that had SEARXNG_URL configured locally.

How

The fix defers the env load to fixture time, not collection time. An autouse
fixture _live_test_env calls _load_user_env() after _hermetic_environment
has already run — so live tests still get OPENROUTER_API_KEY for execution,
but collection is clean.

Also adds tests/test_env_leak_guard.py: a regression test that walks every
test module's AST and fails if any module writes a file-read result into
os.environ at module scope.

Verification

# The original reproducer (before fix — 2 failures, 1 skip):
pytest tests/run_agent/test_sequential_chats_live.py \
  tests/tools/test_web_tools_config.py::TestBackendSelection::test_fallback_no_keys_defaults_to_firecrawl \
  tests/tools/test_web_tools_config.py::TestCheckWebApiKey::test_null_backend_value_does_not_crash -q
# 2 passed, 1 skipped

# Regression test catches the pattern:
pytest tests/test_env_leak_guard.py -q
# passes

Move _load_user_env() out of module scope and behind an autouse fixture
that runs after _hermetic_environment. The old module-level call ran at
pytest collection time, before the credential scrubber, leaking developer-local
env vars (e.g. SEARXNG_URL) into os.environ and causing non-deterministic
test failures that depended on what was in ~/.hermes/.env.

The autouse fixture is the right mechanism here: it fires for every test in
the module (skipped or not), but only after the hermetic scrub has already
run, so developer credentials can still flow in for live test execution
without contaminating the collection phase.

Also add tests/test_env_leak_guard.py: a regression test that walks every
test module's AST and fails if any module writes a file-read result into
os.environ at module scope — catching the exact pattern of NousResearch#80343.
@alt-glitch alt-glitch added type/test Test coverage or test infrastructure comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P3 Low — cosmetic, nice to have duplicate This issue or pull request already exists labels Aug 6, 2026
@alt-glitch

Copy link
Copy Markdown

This was generated by AI during triage.

Duplicate of #80343: both repair the same module-scope developer .env load during pytest collection. #80343 is the earlier, narrower implementation; please consolidate any additional regression coverage there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint duplicate This issue or pull request already exists P3 Low — cosmetic, nice to have type/test Test coverage or test infrastructure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants