fix(logging): stop pytest runs from writing to the live agent.log - #57119
fix(logging): stop pytest runs from writing to the live agent.log#57119lEWFkRAD wants to merge 3 commits into
Conversation
teknium1
left a comment
There was a problem hiding this comment.
Thanks for isolating the import-time HERMES_HOME path; the current defect is still present at agent/agent_init.py:706.
Problems
- The new helper in
tests/run_agent/test_logging_home_isolation.py:25only scans root handlers. Current main routes real file handlers through the asyncQueueListener;hermes_logging.rotating_file_handlers()is the supported accessor (hermes_logging.py:694-700). The proposed assertion will therefore not seeagent.log. - Resolving a per-test live home at
agent/agent_init.py:706needs handler lifecycle work.tests/conftest.py:355-361changesHERMES_HOMEevery test, whilesetup_logging()registers per-path handlers before its initialized guard (hermes_logging.py:320-365) and only deduplicates identical paths (hermes_logging.py:740-746). That can retain queued handlers for prior test homes.
Suggested changes
- Make the new test use
rotating_file_handlers()and reset queued logging state in its fixture/teardown. - Ensure changing
HERMES_HOMEdoes not accumulate stale logging handlers, then cover that invariant.
Automated hermes-sweeper review.
| from pathlib import Path | ||
| from unittest.mock import patch | ||
|
|
||
| import run_agent |
There was a problem hiding this comment.
Current main keeps rotating file handlers on the QueueListener, not the root logger (hermes_logging.py:694-700). Please inspect hermes_logging.rotating_file_handlers() here and reset queued-handler state in teardown; otherwise this helper cannot see the agent.log handler and the test leaks listener/file-handler state.
run_agent._hermes_home is frozen at import time. Under pytest, test modules are imported during collection — before the per-test fixture redirects HERMES_HOME — so agent_init's setup_logging(hermes_home=...) bound root-logger file handlers to the REAL Hermes home (%LOCALAPPDATA%\hermes\logs\agent.log on native Windows). Every mock-provider test then appended records to the live install's log, which read like real X.ai/Copilot API calls. * agent_init: call setup_logging() with no override so the log dir is resolved from the environment at call time (get_hermes_home()). * tests/conftest.py: export a sandbox HERMES_HOME at conftest import time, before pytest imports any test module, as defense-in-depth for other import-time get_hermes_home() consumers (cli._hermes_home, dotenv loading). * Regression tests for both (each fails with its fix reverted). Fixes NousResearch#57118 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
11e971d to
249db87
Compare
|
Thanks @lEWFkRAD — closing as resolved on main: PR #74553 landed a session-level HERMES_HOME sandbox at conftest import time (via #71271), which closes the import-time-frozen log path this PR targeted, and tests/test_log_isolation.py now pins the invariant. Your diagnosis of the class (also filed as #57118) was right — credited there too. |
What does this PR do?
Stops
pytest tests/from appending mock-provider log records to the real user'sagent.log(%LOCALAPPDATA%\hermes\logs\agent.logon native Windows,~/.hermes/logs/agent.logon POSIX).run_agent._hermes_homeis resolved once at import time. Under pytest, test modules are imported during collection — before the per-test_hermetic_environmentfixture redirectsHERMES_HOME— soagent_init'ssetup_logging(hermes_home=_ra()._hermes_home)attached root-logger file handlers to the real install'slogs/agent.log. Every mock-provider test in the process then wrote to the live log, producing entries likeprovider=xai-oauth base_url=https://api.x.ai/v1 model=grok-4.3 ... msg='Say OK'that look exactly like the agent silently calling real provider APIs.The fix is at the callsite (per the guidance already in the
tests/conftest.pydocstring) plus defense-in-depth in the conftest itself:agent_initnow callssetup_logging()with nohermes_home=override — the default resolvesget_hermes_home()at call time, which reads the env var live. Behavior for real CLI/gateway runs is unchanged (env doesn't change between import and agent init there).tests/conftest.pyexports a sandboxHERMES_HOMEat conftest import time, before pytest imports any test module, so every other import-timeget_hermes_home()consumer (cli._hermes_home, dotenv loading) freezes a sandbox path instead of the real install.Related Issue
Fixes #57118
Type of Change
Changes Made
agent/agent_init.py— drop thehermes_home=_ra()._hermes_homeoverride on thesetup_logging()call; resolve the log directory from the live environment instead of run_agent's import-time frozen path.tests/conftest.py— setHERMES_HOMEto a session tempdir at module import time (unconditionally, so aHERMES_HOMEinherited from a developer shell or Docker deployment can't leak into tests either). The existing per-test fixture still re-points it per test.tests/run_agent/test_logging_home_isolation.py(new) — two regression tests:test_agent_logging_ignores_import_time_frozen_home: simulates a stalerun_agent._hermes_homeand assertsAIAgent.__init__attachesagent.logunder the liveHERMES_HOME, with nothing under the stale path. Fails if theagent_initfix is reverted.test_conftest_sandboxes_hermes_home_at_import_time: assertsrun_agent._hermes_homedid not freeze the platform-default (real) home. Fails if the conftest sandbox is reverted.How to Test
<hermes-home>/logs/agent.log.main, runpython -m pytest tests/run_agent/test_run_agent_codex_responses.py -q—grep 'api.x.ai' <hermes-home>/logs/agent.logshows new mock records (13 new lines in my repro, Windows 11 native).test_run_agent_codex_responses.py+test_auxiliary_client_xai_oauth_recovery.py, live-log signature count unchanged).python -m pytest tests/run_agent/test_logging_home_isolation.py -q— both regression tests pass; each fails with its half of the fix reverted.Checklist
Code
fix(scope):,feat(scope):, etc.)tests/cli/conftest.pyreload pollution but not this)pytest tests/ -qand all tests pass (per-file runs of the affected areas:tests/run_agent/test_run_agent.py414 passed,tests/run_agent/test_run_agent_codex_responses.py+ xai recovery 93 passed,tests/cron/test_cron_profile_isolation.py,tests/cron/test_cron_no_agent.py,tests/cli/test_cli_approval_ui.py,tests/agent/test_auxiliary_client.pyall green;tests/test_hermes_logging.py/tests/test_hermes_constants.pyhave pre-existing Windows-native chmod/symlink failures identical on unmodifiedmain)Documentation & Housekeeping
docs/, docstrings) — comments at both changed sites explain the import-time-freeze trapcli-config.yaml.exampleif I added/changed config keys — N/ACONTRIBUTING.mdorAGENTS.mdif I changed architecture or workflows — N/Apytest tests/wrote to the real~/.hermes/logs/agent.logthe same way); fix uses onlytempfile/env, no platform branchesScreenshots / Logs
Pollution on
main(live%LOCALAPPDATA%\hermes\logs\agent.logafter running one test file):After this branch: same run adds zero records to the live log.