Skip to content

fix(agent): bound authorization gate lock acquire and cap excluded_seconds (#79719) - #80134

Closed
RelaxJonh wants to merge 1 commit into
NousResearch:mainfrom
RelaxJonh:fix/auth-gate-timeout-79719
Closed

RelaxJonh wants to merge 1 commit into
NousResearch:mainfrom
RelaxJonh:fix/auth-gate-timeout-79719

Conversation

@RelaxJonh

Copy link
Copy Markdown
Contributor

Summary

Two coupled problems in _ConcurrentToolAuthorizationGate (agent/tool_executor.py):

  1. self._serialization_lock is an unbounded blocking acquire. If the worker holding it wedges (hung pre_tool_block plugin, dead approval client), every other worker needing authorization blocks behind it forever.

  2. excluded_seconds() grows 1:1 with wall clock. The batch-deadline loop adds it to the deadline on every poll (tool_executor.py:1168,1190), so remaining is constant and the deadline never fires — the turn hangs indefinitely.

Fix

  • Add a 120s timeout to _serialization_lock.acquire(). On timeout, log a warning and proceed in degraded mode (run the callback without serialization) rather than hanging forever.
  • Cap excluded_seconds() at _MAX_EXCLUDED_SECONDS (120s) so the batch deadline can eventually fire.

Tests

  • test_excluded_seconds_capped — verifies the cap is enforced
  • test_lock_timeout_proceeds — verifies a second caller proceeds when the lock is held too long
  • test_normal_operation_unaffected — verifies normal non-wedged calls work as before

Closes #79719

…conds

Two coupled problems in _ConcurrentToolAuthorizationGate:

1. self._serialization_lock is unbounded — a wedged pre_tool_block
   plugin or dead approval client blocks every other worker forever.

2. excluded_seconds() grows 1:1 with wall clock, so the batch-deadline
   loop adds it on every poll and remaining never decreases — the
   deadline never fires and the turn hangs indefinitely.

Fix: add a 120s timeout to _serialization_lock.acquire() (proceed in
degraded mode on timeout) and cap excluded_seconds() at the same bound.

Closes NousResearch#79719
@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P1 High — major feature broken, no workaround labels Aug 6, 2026
@kshitijk4poor

Copy link
Copy Markdown
Contributor

Thanks for the fix — bounding the serialization lock and capping the exclusion were the right two levers for #79719.

We ended up fixing it via #80297, which restructures the exclusion instead of capping it: a flat cap on total excluded_seconds (as here) would also truncate legitimate approval waits — two real 90s approvals in one batch would exhaust a 120s budget, and a single legit approval can run to approvals.timeout (300s default), past the cap. #80297 measures the exclusion at the source of the human wait (tools/approval.py marks the CLI prompt and gateway approval poll loop as explicit windows), so a wedged plugin contributes zero exclusion while genuine approvals are excluded in full, and the lock bound is derived from approvals.timeout rather than fixed. You're credited in the PR body.

Closing in favor of #80297. Thanks again!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P1 High — major feature broken, no workaround type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Wedged authorization gate hangs the turn and defeats the concurrent batch deadline entirely

3 participants