fix(gateway): O(n²) watcher recovery and plugin/bundle error logging improvements - #32708
Closed
ErnestHysa wants to merge 1 commit into
Closed
fix(gateway): O(n²) watcher recovery and plugin/bundle error logging improvements#32708ErnestHysa wants to merge 1 commit into
ErnestHysa wants to merge 1 commit into
Conversation
…her recovery N43 — Silent plugin/bundle errors: - Plugin command dispatch: logger.debug() -> logger.warning() - Bundle dispatch: logger.debug() -> logger.warning() Plugin/auth failures are no longer invisible to operators. N42 — O(n^2) pending_watchers recovery: - Both recovery loops (startup + per-message) used while+pop(0) which is O(n) per pop - Replaced with enumerate() over the list + periodic asyncio.sleep(0) yield points - Clears the list after iteration instead of per-pop - Batch size of 100 balances throughput vs event-loop responsiveness
alt-glitch
added
type/perf
ErnestHysa
changed the title
kshitijk4poor
added a commit
to kshitijk4poor/hermes-agent
that referenced
this pull request
May 31, 2026
…align test fixtures + AUTHOR_MAP Self-review follow-up on top of the salvaged perf fixes: - gateway/run.py (both watcher-drain sites): the salvaged O(n^2) fix (NousResearch#32708) replaced `while pending_watchers: pop(0)` with iterate-then- `watchers.clear()`, but `watchers` aliased the registry's live list. A watcher appended by a concurrent session during the `await asyncio.sleep(0)` yield would be cleared without ever being scheduled. Detach the batch atomically (`pending_watchers = []`) before iterating. - gateway/platforms/bluebubbles.py: normalize the salvaged _guid_cache LRU (NousResearch#30523) to match feishu/codebase precedent — module-level `_GUID_CACHE_SIZE` constant, `while len > cap`, and drop the redundant post-insert `move_to_end` (a fresh insert is already most-recent). - gateway/platforms/feishu.py: drop the same redundant post-insert `move_to_end` from the salvaged _message_text_cache LRU (NousResearch#23706). - scripts/release.py: add AUTHOR_MAP entries for the salvaged commits' authors (amathxbt NousResearch#22155, ErnestHysa NousResearch#32636/NousResearch#32708) so the contributor audit passes when these commits land on main. - tests/tools/test_tool_output_limits.py: autouse fixture resets the new module-level limits cache between tests. - tests/gateway/test_feishu.py: hand-built adapter fixture seeded _message_text_cache as a plain dict; it's now an OrderedDict, so the fixture type had to match.
kshitijk4poor
added a commit
that referenced
this pull request
May 31, 2026
…align test fixtures + AUTHOR_MAP Self-review follow-up on top of the salvaged perf fixes: - gateway/run.py (both watcher-drain sites): the salvaged O(n^2) fix (#32708) replaced `while pending_watchers: pop(0)` with iterate-then- `watchers.clear()`, but `watchers` aliased the registry's live list. A watcher appended by a concurrent session during the `await asyncio.sleep(0)` yield would be cleared without ever being scheduled. Detach the batch atomically (`pending_watchers = []`) before iterating. - gateway/platforms/bluebubbles.py: normalize the salvaged _guid_cache LRU (#30523) to match feishu/codebase precedent — module-level `_GUID_CACHE_SIZE` constant, `while len > cap`, and drop the redundant post-insert `move_to_end` (a fresh insert is already most-recent). - gateway/platforms/feishu.py: drop the same redundant post-insert `move_to_end` from the salvaged _message_text_cache LRU (#23706). - scripts/release.py: add AUTHOR_MAP entries for the salvaged commits' authors (amathxbt #22155, ErnestHysa #32636/#32708) so the contributor audit passes when these commits land on main. - tests/tools/test_tool_output_limits.py: autouse fixture resets the new module-level limits cache between tests. - tests/gateway/test_feishu.py: hand-built adapter fixture seeded _message_text_cache as a plain dict; it's now an OrderedDict, so the fixture type had to match.
KKT-OPT
pushed a commit
to KKT-OPT/hermes-agent
that referenced
this pull request
May 31, 2026
…align test fixtures + AUTHOR_MAP Self-review follow-up on top of the salvaged perf fixes: - gateway/run.py (both watcher-drain sites): the salvaged O(n^2) fix (NousResearch#32708) replaced `while pending_watchers: pop(0)` with iterate-then- `watchers.clear()`, but `watchers` aliased the registry's live list. A watcher appended by a concurrent session during the `await asyncio.sleep(0)` yield would be cleared without ever being scheduled. Detach the batch atomically (`pending_watchers = []`) before iterating. - gateway/platforms/bluebubbles.py: normalize the salvaged _guid_cache LRU (NousResearch#30523) to match feishu/codebase precedent — module-level `_GUID_CACHE_SIZE` constant, `while len > cap`, and drop the redundant post-insert `move_to_end` (a fresh insert is already most-recent). - gateway/platforms/feishu.py: drop the same redundant post-insert `move_to_end` from the salvaged _message_text_cache LRU (NousResearch#23706). - scripts/release.py: add AUTHOR_MAP entries for the salvaged commits' authors (amathxbt NousResearch#22155, ErnestHysa NousResearch#32636/NousResearch#32708) so the contributor audit passes when these commits land on main. - tests/tools/test_tool_output_limits.py: autouse fixture resets the new module-level limits cache between tests. - tests/gateway/test_feishu.py: hand-built adapter fixture seeded _message_text_cache as a plain dict; it's now an OrderedDict, so the fixture type had to match.
alt-glitch
pushed a commit
that referenced
this pull request
Jun 14, 2026
…align test fixtures + AUTHOR_MAP Self-review follow-up on top of the salvaged perf fixes: - gateway/run.py (both watcher-drain sites): the salvaged O(n^2) fix (#32708) replaced `while pending_watchers: pop(0)` with iterate-then- `watchers.clear()`, but `watchers` aliased the registry's live list. A watcher appended by a concurrent session during the `await asyncio.sleep(0)` yield would be cleared without ever being scheduled. Detach the batch atomically (`pending_watchers = []`) before iterating. - gateway/platforms/bluebubbles.py: normalize the salvaged _guid_cache LRU (#30523) to match feishu/codebase precedent — module-level `_GUID_CACHE_SIZE` constant, `while len > cap`, and drop the redundant post-insert `move_to_end` (a fresh insert is already most-recent). - gateway/platforms/feishu.py: drop the same redundant post-insert `move_to_end` from the salvaged _message_text_cache LRU (#23706). - scripts/release.py: add AUTHOR_MAP entries for the salvaged commits' authors (amathxbt #22155, ErnestHysa #32636/#32708) so the contributor audit passes when these commits land on main. - tests/tools/test_tool_output_limits.py: autouse fixture resets the new module-level limits cache between tests. - tests/gateway/test_feishu.py: hand-built adapter fixture seeded _message_text_cache as a plain dict; it's now an OrderedDict, so the fixture type had to match.
T02200059
pushed a commit
to T02200059/hermes-agent
that referenced
this pull request
Jun 18, 2026
…align test fixtures + AUTHOR_MAP Self-review follow-up on top of the salvaged perf fixes: - gateway/run.py (both watcher-drain sites): the salvaged O(n^2) fix (NousResearch#32708) replaced `while pending_watchers: pop(0)` with iterate-then- `watchers.clear()`, but `watchers` aliased the registry's live list. A watcher appended by a concurrent session during the `await asyncio.sleep(0)` yield would be cleared without ever being scheduled. Detach the batch atomically (`pending_watchers = []`) before iterating. - gateway/platforms/bluebubbles.py: normalize the salvaged _guid_cache LRU (NousResearch#30523) to match feishu/codebase precedent — module-level `_GUID_CACHE_SIZE` constant, `while len > cap`, and drop the redundant post-insert `move_to_end` (a fresh insert is already most-recent). - gateway/platforms/feishu.py: drop the same redundant post-insert `move_to_end` from the salvaged _message_text_cache LRU (NousResearch#23706). - scripts/release.py: add AUTHOR_MAP entries for the salvaged commits' authors (amathxbt NousResearch#22155, ErnestHysa NousResearch#32636/NousResearch#32708) so the contributor audit passes when these commits land on main. - tests/tools/test_tool_output_limits.py: autouse fixture resets the new module-level limits cache between tests. - tests/gateway/test_feishu.py: hand-built adapter fixture seeded _message_text_cache as a plain dict; it's now an OrderedDict, so the fixture type had to match.
xyshanren
pushed a commit
to xyshanren/hermes-agent-cn
that referenced
this pull request
Jun 25, 2026
…align test fixtures + AUTHOR_MAP Self-review follow-up on top of the salvaged perf fixes: - gateway/run.py (both watcher-drain sites): the salvaged O(n^2) fix (NousResearch#32708) replaced `while pending_watchers: pop(0)` with iterate-then- `watchers.clear()`, but `watchers` aliased the registry's live list. A watcher appended by a concurrent session during the `await asyncio.sleep(0)` yield would be cleared without ever being scheduled. Detach the batch atomically (`pending_watchers = []`) before iterating. - gateway/platforms/bluebubbles.py: normalize the salvaged _guid_cache LRU (NousResearch#30523) to match feishu/codebase precedent — module-level `_GUID_CACHE_SIZE` constant, `while len > cap`, and drop the redundant post-insert `move_to_end` (a fresh insert is already most-recent). - gateway/platforms/feishu.py: drop the same redundant post-insert `move_to_end` from the salvaged _message_text_cache LRU (NousResearch#23706). - scripts/release.py: add AUTHOR_MAP entries for the salvaged commits' authors (amathxbt NousResearch#22155, ErnestHysa NousResearch#32636/NousResearch#32708) so the contributor audit passes when these commits land on main. - tests/tools/test_tool_output_limits.py: autouse fixture resets the new module-level limits cache between tests. - tests/gateway/test_feishu.py: hand-built adapter fixture seeded _message_text_cache as a plain dict; it's now an OrderedDict, so the fixture type had to match.
waefrebeorn
pushed a commit
to waefrebeorn/slermes
that referenced
this pull request
Jul 2, 2026
…align test fixtures + AUTHOR_MAP Self-review follow-up on top of the salvaged perf fixes: - gateway/run.py (both watcher-drain sites): the salvaged O(n^2) fix (NousResearch#32708) replaced `while pending_watchers: pop(0)` with iterate-then- `watchers.clear()`, but `watchers` aliased the registry's live list. A watcher appended by a concurrent session during the `await asyncio.sleep(0)` yield would be cleared without ever being scheduled. Detach the batch atomically (`pending_watchers = []`) before iterating. - gateway/platforms/bluebubbles.py: normalize the salvaged _guid_cache LRU (NousResearch#30523) to match feishu/codebase precedent — module-level `_GUID_CACHE_SIZE` constant, `while len > cap`, and drop the redundant post-insert `move_to_end` (a fresh insert is already most-recent). - gateway/platforms/feishu.py: drop the same redundant post-insert `move_to_end` from the salvaged _message_text_cache LRU (NousResearch#23706). - scripts/release.py: add AUTHOR_MAP entries for the salvaged commits' authors (amathxbt NousResearch#22155, ErnestHysa NousResearch#32636/NousResearch#32708) so the contributor audit passes when these commits land on main. - tests/tools/test_tool_output_limits.py: autouse fixture resets the new module-level limits cache between tests. - tests/gateway/test_feishu.py: hand-built adapter fixture seeded _message_text_cache as a plain dict; it's now an OrderedDict, so the fixture type had to match.
Collaborator
|
Thanks for this fix! This landed on main as your commit 0036c72 — both O(n²) |
santhreal
pushed a commit
to santhreal/hermes-agent
that referenced
this pull request
Jul 13, 2026
…align test fixtures + AUTHOR_MAP Self-review follow-up on top of the salvaged perf fixes: - gateway/run.py (both watcher-drain sites): the salvaged O(n^2) fix (NousResearch#32708) replaced `while pending_watchers: pop(0)` with iterate-then- `watchers.clear()`, but `watchers` aliased the registry's live list. A watcher appended by a concurrent session during the `await asyncio.sleep(0)` yield would be cleared without ever being scheduled. Detach the batch atomically (`pending_watchers = []`) before iterating. - gateway/platforms/bluebubbles.py: normalize the salvaged _guid_cache LRU (NousResearch#30523) to match feishu/codebase precedent — module-level `_GUID_CACHE_SIZE` constant, `while len > cap`, and drop the redundant post-insert `move_to_end` (a fresh insert is already most-recent). - gateway/platforms/feishu.py: drop the same redundant post-insert `move_to_end` from the salvaged _message_text_cache LRU (NousResearch#23706). - scripts/release.py: add AUTHOR_MAP entries for the salvaged commits' authors (amathxbt NousResearch#22155, ErnestHysa NousResearch#32636/NousResearch#32708) so the contributor audit passes when these commits land on main. - tests/tools/test_tool_output_limits.py: autouse fixture resets the new module-level limits cache between tests. - tests/gateway/test_feishu.py: hand-built adapter fixture seeded _message_text_cache as a plain dict; it's now an OrderedDict, so the fixture type had to match.
donbowman
pushed a commit
to donbowman/hermes-agent
that referenced
this pull request
Jul 13, 2026
…align test fixtures + AUTHOR_MAP Self-review follow-up on top of the salvaged perf fixes: - gateway/run.py (both watcher-drain sites): the salvaged O(n^2) fix (NousResearch#32708) replaced `while pending_watchers: pop(0)` with iterate-then- `watchers.clear()`, but `watchers` aliased the registry's live list. A watcher appended by a concurrent session during the `await asyncio.sleep(0)` yield would be cleared without ever being scheduled. Detach the batch atomically (`pending_watchers = []`) before iterating. - gateway/platforms/bluebubbles.py: normalize the salvaged _guid_cache LRU (NousResearch#30523) to match feishu/codebase precedent — module-level `_GUID_CACHE_SIZE` constant, `while len > cap`, and drop the redundant post-insert `move_to_end` (a fresh insert is already most-recent). - gateway/platforms/feishu.py: drop the same redundant post-insert `move_to_end` from the salvaged _message_text_cache LRU (NousResearch#23706). - scripts/release.py: add AUTHOR_MAP entries for the salvaged commits' authors (amathxbt NousResearch#22155, ErnestHysa NousResearch#32636/NousResearch#32708) so the contributor audit passes when these commits land on main. - tests/tools/test_tool_output_limits.py: autouse fixture resets the new module-level limits cache between tests. - tests/gateway/test_feishu.py: hand-built adapter fixture seeded _message_text_cache as a plain dict; it's now an OrderedDict, so the fixture type had to match.
Gravezzz
pushed a commit
to Gravezzz/hermes-agent
that referenced
this pull request
Jul 21, 2026
…align test fixtures + AUTHOR_MAP Self-review follow-up on top of the salvaged perf fixes: - gateway/run.py (both watcher-drain sites): the salvaged O(n^2) fix (NousResearch#32708) replaced `while pending_watchers: pop(0)` with iterate-then- `watchers.clear()`, but `watchers` aliased the registry's live list. A watcher appended by a concurrent session during the `await asyncio.sleep(0)` yield would be cleared without ever being scheduled. Detach the batch atomically (`pending_watchers = []`) before iterating. - gateway/platforms/bluebubbles.py: normalize the salvaged _guid_cache LRU (NousResearch#30523) to match feishu/codebase precedent — module-level `_GUID_CACHE_SIZE` constant, `while len > cap`, and drop the redundant post-insert `move_to_end` (a fresh insert is already most-recent). - gateway/platforms/feishu.py: drop the same redundant post-insert `move_to_end` from the salvaged _message_text_cache LRU (NousResearch#23706). - scripts/release.py: add AUTHOR_MAP entries for the salvaged commits' authors (amathxbt NousResearch#22155, ErnestHysa NousResearch#32636/NousResearch#32708) so the contributor audit passes when these commits land on main. - tests/tools/test_tool_output_limits.py: autouse fixture resets the new module-level limits cache between tests. - tests/gateway/test_feishu.py: hand-built adapter fixture seeded _message_text_cache as a plain dict; it's now an OrderedDict, so the fixture type had to match.
19 tasks
leewenjie
pushed a commit
to leewenjie/hermes-agent
that referenced
this pull request
Aug 7, 2026
…align test fixtures + AUTHOR_MAP Self-review follow-up on top of the salvaged perf fixes: - gateway/run.py (both watcher-drain sites): the salvaged O(n^2) fix (NousResearch#32708) replaced `while pending_watchers: pop(0)` with iterate-then- `watchers.clear()`, but `watchers` aliased the registry's live list. A watcher appended by a concurrent session during the `await asyncio.sleep(0)` yield would be cleared without ever being scheduled. Detach the batch atomically (`pending_watchers = []`) before iterating. - gateway/platforms/bluebubbles.py: normalize the salvaged _guid_cache LRU (NousResearch#30523) to match feishu/codebase precedent — module-level `_GUID_CACHE_SIZE` constant, `while len > cap`, and drop the redundant post-insert `move_to_end` (a fresh insert is already most-recent). - gateway/platforms/feishu.py: drop the same redundant post-insert `move_to_end` from the salvaged _message_text_cache LRU (NousResearch#23706). - scripts/release.py: add AUTHOR_MAP entries for the salvaged commits' authors (amathxbt NousResearch#22155, ErnestHysa NousResearch#32636/NousResearch#32708) so the contributor audit passes when these commits land on main. - tests/tools/test_tool_output_limits.py: autouse fixture resets the new module-level limits cache between tests. - tests/gateway/test_feishu.py: hand-built adapter fixture seeded _message_text_cache as a plain dict; it's now an OrderedDict, so the fixture type had to match.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes N42 (O(n²) pending_watchers recovery) and N43 (silent plugin/bundle errors) in gateway/run.py.
N42 — O(n²) pending_watchers recovery (Medium — DoS)
Both recovery loops (startup at ~line 4350 and per-message at ~line 8852) used
while list: watcher = list.pop(0)which is O(n) per pop — O(n²) total. Under large crash-recovery scenarios (thousands of watchers), the event loop would block for extended periods.Fix: Replaced with
enumerate()iteration + periodicasyncio.sleep(0)yield points every 100 items, thenlist.clear(). Also converts the second occurrence that was missing thelogger.infomessage.N43 — Silent plugin/bundle errors (Medium — Security/Reliability)
Plugin command dispatch and bundle dispatch both used
logger.debug()which is below the default INFO threshold — invisible in normal operation. Auth failures and config errors were silently swallowed.Fix: Changed both to
logger.warning()so operators are alerted when plugin commands fail.