fix(gateway): bound adapter disconnect in fatal-error handler - #54041
fix(gateway): bound adapter disconnect in fatal-error handler#54041srojk34 wants to merge 1 commit into
Conversation
_handle_adapter_fatal_error() called raw adapter.disconnect() with no timeout. A half-dead adapter (e.g. a wedged Feishu/Lark WebSocket) could block the handler indefinitely, preventing the platform from being popped from the adapter map and queued for reconnection. Replace with the existing _safe_adapter_disconnect() helper, which wraps the call in HERMES_GATEWAY_ADAPTER_DISCONNECT_TIMEOUT — the same bounded-teardown pattern applied to _stop_impl() in NousResearch#14128.
tonydwb
left a comment
There was a problem hiding this comment.
Code Review Summary
Verdict: Approved
Bounds the adapter disconnect in the fatal-error handler to prevent hangs. When _handle_adapter_fatal_error calls adapter.disconnect(), a wedged disconnect (e.g. WebSocket hang) could block the entire handler indefinitely. The fix wraps the disconnect in a timeout via _safe_adapter_disconnect. Test verifies a wedged adapter completes within the timeout and is properly cleaned up.
Reviewed by Hermes Agent
teknium1
left a comment
There was a problem hiding this comment.
Thanks for identifying the unbounded fatal-error teardown. The defect remains on current main: gateway/run.py:4038 still awaits adapter.disconnect() directly, while _safe_adapter_disconnect() already bounds that operation at gateway/run.py:3347-3359.
Problems
- The PR predates commit
a682091044955167c9a728f9641ff279c96a73a7, which intentionally changed the handler to remove the adapter before awaiting teardown (gateway/run.py:4030-4038) so concurrent fatal notifications cannot disconnect it twice. Do not restore the older try/finally ordering from the PR base.
Suggested changes
- Preserve the current pre-await
pop()and delivery-router update, and replace only the raw await atgateway/run.py:4038with_safe_adapter_disconnect(adapter, adapter.platform). - Carry over the wedged-disconnect regression test to verify the handler reaches reconnection queuing after the timeout.
This is an automated hermes-sweeper review.
| @@ -3633,7 +3633,7 @@ async def _handle_adapter_fatal_error(self, adapter: BasePlatformAdapter) -> Non | |||
| existing = self.adapters.get(adapter.platform) | |||
There was a problem hiding this comment.
When salvaging this change, retain current main's later ordering that removes the adapter before the awaited teardown (gateway/run.py:4036-4038, commit a682091044955167c9a728f9641ff279c96a73a7). Replace only the current raw await with this helper; restoring the PR base's surrounding try/finally ordering would reopen the concurrent-fatal-notification race.
Summary
_handle_adapter_fatal_error()called rawadapter.disconnect()with no timeout — a half-dead adapter (e.g. a wedged Feishu/Lark WebSocket) blocked the handler indefinitely, preventing the platform from being popped and queued for reconnection_safe_adapter_disconnect(), which wraps the call inHERMES_GATEWAY_ADAPTER_DISCONNECT_TIMEOUT— the same bounded-teardown pattern applied to_stop_impl()in fix(gateway): Gateway shutdown hangs causing 'PID file race lost' on restart #14128Test plan
test_fatal_error_bounds_wedged_disconnect— wedgeddisconnect()times out, adapter is removed, platform is queued for reconnectionTestRuntimeDisconnectQueuing+test_bounded_adapter_teardowntests pass