fix(tests): eliminate all baseline flaky/broken tests (cluster A: shadow imports, unmocked network, quadratic redact regexes) - #74330
Merged
Conversation
…nmocked network in compressor tests, stale-SDK feishu pin guard, quadratic redact regexes - Remove tests/-shadowing sys.path.insert(dirname/'..') from 11 test files: it prepended the tests/ dir itself to sys.path, so 'import agent' / 'import hermes_cli' resolved to the test packages and collection died with ModuleNotFoundError depending on import order (2 files failed in every full-suite run; 9 more were latent). - Patch call_llm in 5 context-compressor tests that called compress() unmocked: each burned ~50s attempting live LLM traffic through the relay before falling back (572s file — the slowest in the suite, and flaky under the 300s per-file timeout). File now runs in ~5s. - agent/redact.py: fix two catastrophically-backtracking regexes hit by the compressor's redaction pass on large payloads — _STRICT_URL_USERINFO_RE anchors on the mandatory '//' (optional-scheme prefix backtracked O(n^2): ~55s on a 320KB payload, now sub-ms; output-equivalence fuzz-verified on 20k random strings), and the _CFG_DOTTED_RE/_CFG_ANCHORED_RE subs gain an exact linear keyword pre-gate so secret-free text skips the quadratic pattern entirely. - tests/gateway/test_feishu.py: version-guard the extra_ua_tags SDK signature check; the repo pins lark-oapi==1.6.8 but stale local installs (1.5.3) fail the assertion — skip below the pin. - tests/tools/test_managed_browserbase_and_modal.py: stub agent.redact + agent.credential_persistence in the fake agent package (empty __path__ blocks all real agent.* imports added since the fake was written). - tests/gateway/test_startup_restart_race.py: raise wait_for timeouts 2s -> 30s; 2s wall-clock on a loaded 40-worker box flaked in the baseline run (passes instantly when the box is quiet).
Contributor
૮ >ﻌ< ა ci reviewran on 3dd8059 ℹ️ InfoDesktop E2E visual evidence · View test artifacts · View job1 visual diff. inline evidence upload failed. Failed to upload diff-665a0833239e-onboarding-overlay-diff.png with gh image (exit code 1): Error uploading /home/runner/work/_temp/e2e-evidence/diff-665a0833239e-onboarding-overlay-diff.png: step 0 (get upload token): uploadToken not found on repo page — do you have write access to NousResearch/hermes-agent? (or, if NousResearch enforces SAML SSO, authorize at https://github.com/orgs/NousResearch/sso) |
alt-glitch
added
type/perf
This was referenced Jul 30, 2026
19 tasks
14 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every flaky/broken test in the full-suite baseline run is fixed at the root cause — the suite now passes 2437 files / 50,530 tests clean, and the slowest file dropped from 572s to ~5s.
Baseline evidence (full
scripts/run_tests.shrun on this box): 4 test failures across 4 files, 2 collection errors, and 1 FLAKY retry — all reproduced, traced, and fixed below. No test was deleted or weakened; two genuinely-quadratic production regexes inagent/redact.pywere fixed as the underlying class.Changes
sys.path.insert(0, dirname(__file__)/".."): it prependstests/itself, soimport agent/import hermes_cliresolve to the test packages (they have__init__.py) and collection dies withModuleNotFoundErrordepending on import order. 2 files failed every run; 9 more were the same latent class (fixed all sites).compress()without patchingcall_llm, silently attempting live LLM traffic (~50s connect/retry each). Patched to the fast failure path. File: 572s → ~5s, and no longer trips the 300s per-file timeout (the baseline FLAKY)._STRICT_URL_USERINFO_RE: optional-scheme prefix backtracked O(n²) — ~55s on a 320KB payload. Now anchors on the mandatory//(scheme was outside the replacement either way). Output-equivalence fuzz-verified on 20k random strings + targeted URL forms._CFG_DOTTED_RE/_CFG_ANCHORED_REsubs: added an exact linear keyword pre-gate (_CFG_SECRET_WORD_RE.search) — text without any secret keyword can never match, so secret-free blobs skip the quadratic pattern entirely.extra_ua_tagsSDK signature check now skips below the repo pin (lark-oapi==1.6.8); stale lazy-installed 1.5.3 on dev boxes failed a check that only measures the environment, not the product.agentpackage (empty__path__) blocks every realagent.*import added since it was written; stubbedagent.redact+agent.credential_persistence.wait_fortimeouts 2s → 30s (loaded 40-worker box flaked; skill rule: loose wall-clock bounds).Validation
Infographic