test: round-3 stability closeout — 12 salvaged PRs, order-dependence root fixes, guard precision - #74576
Merged
Merged
Conversation
(cherry picked from commit 1ac105e)
…the marker and repair paths Guard the .lazy-refresh-incomplete marker writer (update_cmd), launch-time recovery (main.py), and _early_recovery repair paths behind a two-condition check: running under pytest AND the target is this live checkout. Sandboxed tmp_path tests still exercise the real code paths. Salvaged from PR #72002 by @fcavalcantirj. Fixes #72000. Co-authored-by: fcavalcantirj <felipe.cavalcanti.rj@gmail.com>
… bounds Replace OS-scheduler-dependent elapsed-time ceilings with deterministic concurrency proofs in three tests: - test_context_refs_concurrent: asyncio.Barrier rendezvous — all 3 URL fetches must be in flight simultaneously before any returns. - test_memory_boundary_commit: positive non-blocking witness — the provider call list must still be empty when the async commit returns. - test_mem0_v3 slow-prefetch: threading.Event park/release — prefetch must return while the backend search is still parked. The tests/tools/test_mcp_tool.py hunk from the original PR is dropped: main no longer carries the 'elapsed < 2.5' assertion it targeted (superseded by a delay-relative bound). Salvaged from #71913.
Cross-file leaks made tests/tui_gateway + tests/test_tui_gateway_server.py fail when run in one process (issue #57068): - conftest: _hermetic_environment now re-pins hermes_state.DEFAULT_DB_PATH to the fake home so no test ever touches the developer's real state.db - conftest: new autouse _reset_tui_gateway_server_state snapshots/restores _methods, cfg cache, db handle and _real_stdout, and tears down leftover sessions via the production _close_session_by_id(..., end_reason= "test_cleanup") boundary - per-file fixtures scope patch.dict(sys.modules) to the import only - drop reload()-based teardowns that duplicated atexit hooks Conflict resolution vs main: kept main's mod._live_transports.clear() in the test_protocol.py server fixture alongside the snapshot/restore logic. Combined run now 792 passed / 0 failed. Salvaged from PR #57066 by @lEWFkRAD. Fixes #57068.
Registers the two genuinely-unregistered markers (no_isolate, ssh) in pyproject.toml, silencing PytestUnknownMarkWarning for tests/tools/test_mcp_* and tests/tools/test_file_sync_perf.py. Dropped hunk: the live_system_guard_bypass line from the original PR — that marker is already registered dynamically via pytest_configure/addinivalue_line in tests/conftest.py, so the ini entry would be a duplicate. Salvaged from #71403.
Contract tests for tools/fal_common.py: queue-URL normalization (trailing slash / whitespace / empty-raises), _extract_http_status response-vs-exc precedence and non-int rejection, the _ManagedFalSyncClient RuntimeError guards against fal_client private API drift, and submit()'s queue-URL + POST/json/timeout wiring. Trimmed on landing: test_non_string_coerced_to_string (pins an implementation accident, not a contract) per the coverage-padding policy in AGENTS.md. Salvaged from #52166.
Argv-list commands now scan only argv[0] against _PROCESS_KILLERS, ending
false positives like ['cat', '.../skill'] ('skill' is a real util-linux
binary name). Wrapper executables (sh/bash/env/nohup/timeout/sudo/xargs/…)
keep full-token scanning so ['bash', '-c', 'pkill …'] and
['env', …, 'pkill', …] stay blocked; string commands are unchanged.
Adds tests/test_live_system_guard.py pinning both directions.
Salvaged from PR #43299 by @eazye19.
Co-authored-by: Tony (eazye19) <support@captureclient.net>
…me _hermes_home freeze Sibling files (test_billing_rpc etc.) import tui_gateway.server at collection time, freezing module-level _hermes_home = get_hermes_home() (server.py:54) to the developer's real home before conftest isolation runs. _load_cfg() then reads the REAL ~/.hermes/config.yaml — e.g. a local MoA preset — instead of what _write_moa_config wrote. The server fixture now monkeypatches _hermes_home to the isolated HERMES_HOME and resets the mtime-keyed cfg cache (_cfg_cache/_cfg_mtime/ _cfg_path); monkeypatch restores originals on teardown. Complementary to the #57066 autouse teardown, which only restores the cfg cache to its pre-test value and never re-points _hermes_home. Combined tests/tui_gateway + tests/test_tui_gateway_server.py: 792 passed. Salvaged from PR #63981 by @lEWFkRAD.
Two Windows fixes for the canonical test runner, salvaged from #66496: - scripts/run_tests.sh: probe the native Windows venv layout (Scripts/activate → Scripts/python.exe) alongside bin/activate, adapted to main's pytest-import-guarded loop with SKIPPED_VENVS reporting. Without it a python -m venv / uv venv on Git Bash/MSYS is never found and the runner refuses to start. - scripts/run_tests_parallel.py: _make_stdio_glyph_safe() reconfigures stdout/stderr to UTF-8 (errors=replace fallback) so the ✓/✗ progress glyphs cannot crash a cp1252 console when the runner is invoked directly (run_tests.sh's PYTHONUTF8=1 only covers the wrapped path). No-op on UTF-8 stdio. Ships 3 OS-independent cp1252 tests plus encoding=utf-8 in the runner-subprocess assertions. Dropped from the original PR: the USERPROFILE/HOMEDRIVE/HOMEPATH/ SYSTEMROOT env-forwarding hunk — main's WIN_ENV loop already forwards a superset (#67385/#70813).
Autouse fixture also resets approval_module._YOLO_MODE_FROZEN so a HERMES_YOLO_MODE=1 host env can't poison every case (the one startup-frozen test still patches it back explicitly). Adds the darwin 'ps -o stat=' zombie branch to _is_alive_like_dispatcher, mirroring production hermes_cli/kanban_db.py — a no-op on Linux. Salvaged from PR #34069 by @sunwz1115. Co-authored-by: sunwz1115 <192549904+sunwz1115@users.noreply.github.com>
tests/test_zz_guard_probe_tmp.py was a throwaway verification probe that was swept into the PR #43299 salvage commit by mistake (and one of its shell=True cases fails by design of the probe). The durable regression coverage lives in tests/test_live_system_guard.py.
Two tests made real HTTP calls to homeassistant.local:8123 — on a LAN with an actual Home Assistant instance they could turn on real lights, and otherwise burned ~10s in network timeouts. Replace with AsyncMock at _async_call_service and assert the exact production call signature (domain, service, entity_id, data). 35 pass in ~0.2s. Salvaged from PR #72634 by @jeeves-assistant. Co-authored-by: Jeeves Assistant <jeevesassistant00@gmail.com>
The file-local telegram mock in test_dm_topics.py installed unconditionally (no __file__ guard), registered a separate string-valued telegram.constants module, and force-popped the adapter — poisoning the session for any later telegram test in the same process (assert 'MARKDOWN_V2' in "'MarkdownV2'"). Fix at the source: - conftest: _FakeEnumMember(str) with PTB-faithful str()==value and repr()==<ChatType.X: 'x'>, satisfying both repr assertions and the adapter's str(chat.type) normalization; the same object is bound to mod.ParseMode and mod.constants.ParseMode. - test_dm_topics.py: delete the divergent local mock installer; import the shared conftest one. - release.py: mailmap entry for the author. Verified: the 5-failure cluster repro (dm_topics + slash_confirm + approval_buttons + model_picker + network_reconnect + telegram_format in one process) goes 83/83 green (3x); full tests/gateway single-process run drops 10 -> 5 failed, the remainder being pre-existing discord order-dep failures out of scope here. Salvaged from #68873. Credit to @liuhao1024 for the earliest root-cause diagnosis of this str-enum mock class in PR #33875, two months earlier. Fixes the telegram-mock order-dependent flake cluster.
Contributor
૮ >ﻌ< ა ci reviewran on 422008e ℹ️ InfoDesktop E2E visual evidence · View test artifacts · View job1 visual diff. inline evidence upload failed. Failed to upload diff-665a0833239e-onboarding-overlay-diff.png with gh image (exit code 1): Error uploading /home/runner/work/_temp/e2e-evidence/diff-665a0833239e-onboarding-overlay-diff.png: step 0 (get upload token): uploadToken not found on repo page — do you have write access to NousResearch/hermes-agent? (or, if NousResearch enforces SAML SSO, authorize at https://github.com/orgs/NousResearch/sso) |
…no vercel dist The tests fake the vercel SDK entirely via sys.modules, but _ensure_vercel_sdk checks installed DISTRIBUTION metadata through tools.lazy_deps.ensure(): on CI (no vercel dist + lazy installs disabled) it raised FeatureUnavailable→ImportError before the fake SDK was ever reached — 16 failures on slice 6, green on dev boxes that happen to have vercel==0.7.2 installed. Failure mechanism reproduced locally by forcing _is_satisfied False; fixture patch verified to close it while the real-dist path stays green (16/16).
This was referenced Jul 30, 2026
Closed
This was referenced Jul 30, 2026
Closed
This was referenced Jul 30, 2026
This was referenced Jul 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
12 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Lands the long tail of the test-stability backlog: 12 more external PRs salvaged (10 contributors, authorship preserved) covering order-dependence root fixes, guard precision, and hermeticity — full suite green (22,817 passed / 0 failed / 0 flaky). With this, every open test-stability PR and issue in the repo is either merged or closed with credit (2 macOS-only PRs remain open pending Mac verification).
Salvaged (16 commits)
_FakeEnumMemberin shared conftest + removal of dm_topics' unconditional divergent mock (the poison source). Credits @liuhao1024 (fix(tests): use str-enum for mock ParseMode/ChatType to fix order-dependent failures #33875) as earliest diagnoser of the class. tests/gateway telegram failures 5→0_methodsstubs/sys.modules mocks, DEFAULT_DB_PATH re-pin, collection-time_hermes_homefreeze pin. Combined tui suite 792 passed / 0 failed. Closes tui_gateway tests are order-dependent across files (leaked _methods stubs, sys.modules mocks, import-time DEFAULT_DB_PATH hits the real state.db) #57068.lazy-refresh-incompletemarker writer + launch-time recovery/repair paths; also un-tracks the accidentally-committed marker (+ .gitignore entry)bash -c,env, …) full-scan exemption; innocentcat …/skillno longer blocked, every real kill shape still caught (probe-verified)_YOLO_MODE_FROZENhermeticity + darwinps-based zombie liveness (Linux no-op)Scripts/venv probe + glyph-safe stdio + cp1252 tests (env-forwarding hunk dropped — already covered)no_isolate/sshpytest markersHousekeeping: LEGACY_AUTHOR_MAP kept frozen (mapping rerouted to contributors/emails/), temp guard probe removed.
Validation
Infographic