Conversation
…conds Two coupled problems in _ConcurrentToolAuthorizationGate: 1. self._serialization_lock is unbounded — a wedged pre_tool_block plugin or dead approval client blocks every other worker forever. 2. excluded_seconds() grows 1:1 with wall clock, so the batch-deadline loop adds it on every poll and remaining never decreases — the deadline never fires and the turn hangs indefinitely. Fix: add a 120s timeout to _serialization_lock.acquire() (proceed in degraded mode on timeout) and cap excluded_seconds() at the same bound. Closes NousResearch#79719
|
Thanks for the fix — bounding the serialization lock and capping the exclusion were the right two levers for #79719. We ended up fixing it via #80297, which restructures the exclusion instead of capping it: a flat cap on total Closing in favor of #80297. Thanks again! |
Summary
Two coupled problems in
_ConcurrentToolAuthorizationGate(agent/tool_executor.py):self._serialization_lockis an unbounded blocking acquire. If the worker holding it wedges (hungpre_tool_blockplugin, dead approval client), every other worker needing authorization blocks behind it forever.excluded_seconds()grows 1:1 with wall clock. The batch-deadline loop adds it to the deadline on every poll (tool_executor.py:1168,1190), soremainingis constant and the deadline never fires — the turn hangs indefinitely.Fix
_serialization_lock.acquire(). On timeout, log a warning and proceed in degraded mode (run the callback without serialization) rather than hanging forever.excluded_seconds()at_MAX_EXCLUDED_SECONDS(120s) so the batch deadline can eventually fire.Tests
test_excluded_seconds_capped— verifies the cap is enforcedtest_lock_timeout_proceeds— verifies a second caller proceeds when the lock is held too longtest_normal_operation_unaffected— verifies normal non-wedged calls work as beforeCloses #79719