test(hermetic): gate on unrestored sys.modules purges (+2 leakers)

[Kyzcreig]

Codifies the bug class behind the tests/agent pollution work as a durable gate — so it fails at the leaking test instead of mysteriously downstream.

Companion to #538 (leaker #1) and #540 (leaker #2). Land #540 first — the gate fires on test_verification_stop_caching.py until #540's fix is in.

The class

A test wants a fresh import (a patched module, a new HERMES_HOME), so it deletes run_agent / agent.* / tools.* / hermes_* from sys.modules and re-imports — and never puts them back. Every later importer then receives a brand-new module object, so any subsequent test that captured a reference to (or monkeypatched) one of those modules silently operates on an orphaned copy.

The failures land far from the cause and read as unrelated bugs, which is why each instance cost a full per-victim bisect to find.

Four instances found in one day:

file	cost
test_empty_tool_name_loop_dampening.py (#538)	120 → 74 suite failures
test_verification_stop_caching.py (#540)	~20 more
test_kanban_per_profile_cap.py (here)	leaked hermes_cli._subprocess_compat
test_save_url_image.py (here)	found BY the gate on its first full-suite run

That last one is the argument for the gate: it caught a leaker nobody had bisected for.

The gate

tests/sys_modules_leak_gate.py — an autouse fixture that fails a test which removes a pre-existing watched module without restoring it. The failure names the test, lists the leaked modules, and carries the fix recipe (including the parent-attribute restore). Opt out with @pytest.mark.allow_sys_modules_purge.

Deliberately narrow:

• Adding modules is normal (imports happen); only removing pre-existing ones is the hazard.
• plugins. is excluded. Plugin-discovery tests purge plugins.model_providers.* on purpose to force a re-scan — that is the behaviour under test. Gating them produced ~34 false positives on tests working as designed (measured, then excluded — not assumed).
• Third-party prefixes (botocore, urllib3) are out of scope: lazy-import churn would be pure noise.

Proven to FIRE

19 unit tests, both directions: fires on an unrestored purge, names the test, carries the recipe, truncates a large leak; stays silent on a restored purge, added modules, an untouched run, unwatched modules, and under the opt-out marker. Prefix matching asserted narrow (agentic_unrelated, my_agent must NOT match).

Then replayed the real leaker's exact purge: 41 modules reported, correct message.

An earlier version of that probe said "does not fire" — because the probe itself had loaded no watched modules, so nothing could be removed. Worth stating plainly: the first negative was the probe's bug, not the gate's, and it was only caught by asking why.

Note

I also fixed test_verification_stop_caching.py independently, then found #540 fixing the same file with a more thorough approach (a context manager that also detaches/restores parent attributes, deepest-first). Dropped mine rather than ship a competing duplicate.

[Kyzcreig]

⏳ Blocked on #540 — merge that first.

CI here is red exactly as designed: the gate is catching test_verification_stop_caching.py leaking 51-59 modules, which is the leaker #540 fixes.

Failed: tests/agent/test_verification_stop_caching.py::test_json_log_drops_only_nudge_keeps_candidate
        removed 59 module(s) from sys.modules and did not restore them

That's the gate working, not a defect in it. Once #540 lands I'll rebase and CI should go green.

Verified independently on #540's branch: tests/agent/ = 4919 passed, 0 failed (vs 120 failed on main).

[Kyzcreig]

Rebased onto fork/main (1677b9d, which includes #540).

The slice 9/12 red was inherited, not caused by this PR: it was tests/agent/test_verification_stop_caching.py leaking 59 and 51 unrestored sys.modules entries — the exact bug class #540 fixed. This branch was based on a340b8a, which predates #540, so it was gating on a leak that main had already repaired.

Verified locally after the rebase:

scripts/run_tests.sh tests/agent/test_verification_stop_caching.py \ tests/agent/test_empty_tool_name_loop_dampening.py \ tests/agent/test_save_url_image.py tests/test_sys_modules_leak_gate.py -q
=> 4 files, 33 tests passed, 0 failed

[Kyzcreig]

✅ Rebased on #540 — green now.

Full tests/agent/ sweep on the rebased branch, gate armed:

4944 passed, 18 skipped, 0 failed   (7m21s)

The gate is silent because the leaker it was catching is fixed by #540. Combined with the two additional leakers fixed here (test_kanban_per_profile_cap.py, test_save_url_image.py), the suite is fully clean.

Journey: 120 failed → 0, from four fixtures of one bug class.

[Kyzcreig]

FleetReview

Confidence: 2/5

Findings

• P2 agent/context_engine.py:514 — Uncaught Persistence
• P2 tests/agent/test_save_url_image.py:105 — Incomplete Restoration
• P2 tests/hermes_cli/test_kanban_per_profile_cap.py:42 — Module Leak
• P1 hermes_state.py:4801 — Calibration Lost
• P1 tests/agent/test_skew_model_keyed_calibration.py:1 — PR silently reverts the shipped (provider, model)-keyed skew calibration feature and drops its table
• P0 tests/sys_modules_leak_gate.py:108 — Autouse sys.modules leak gate red-lights pre-existing tests that were not fixed in this PR
• P1 tests/sys_modules_leak_gate.py:121 — Replacement Missed
• P1 tests/sys_modules_leak_gate.py:68 — Reimports Undetected
• P2 tests/sys_modules_leak_gate.py:64 — Overbroad Prefix
• P1 agent/context_compressor.py:1715 — Model-keyed durable skew calibration removed; fresh sessions never inherit priors
• P0 tests/conftest.py:46 — Enabling sys.modules leak gate will fail CI on known unrestored purges
• P0 tests/agent/test_verification_stop_caching.py:22 — Gate ships red: test_verification_stop_caching.py still has the unrestored purge the gate's own docstring says was fixed

---

FleetReview provenance · models: B=gpt-5.6-sol, C=claude-code-opus-4-8, F=gpt-5.6-sol, G=grok-4.5 · cost: $13.65 · duration: 22m 28s · rounds: 1 · files examined: 5