Skip to content

fix(security): circuit breaker for tirith crashes to prevent agent hangs (#41400) - #52997

Merged
kshitijk4poor merged 1 commit into
NousResearch:mainfrom
kshitijk4poor:salvage/41400-tirith-circuit-breaker
Jun 26, 2026
Merged

fix(security): circuit breaker for tirith crashes to prevent agent hangs (#41400)#52997
kshitijk4poor merged 1 commit into
NousResearch:mainfrom
kshitijk4poor:salvage/41400-tirith-circuit-breaker

Conversation

@kshitijk4poor

Copy link
Copy Markdown
Collaborator

Summary

Tirith security scanner crashes (SIGSEGV, FileNotFoundError, timeout) no longer cause 20-minute agent hangs — after 3 consecutive failures, a circuit breaker disables tirith for the rest of the process.

Root cause (#41400)

When tirith_path points to a non-existent or broken binary, every tool invocation triggers a crash (exit code -11/SIGSEGV, FileNotFoundError, etc.). With fail_open: true, the tool proceeds anyway, and the agent retries the same tool call — creating a 20-minute hang where no responses are delivered. 2,704+ crashes were logged over 7 days, all hidden behind fail_open.

Changes

  • tools/tirith_security.py: circuit breaker pattern — tracks consecutive failures across all three failure paths (OSError, TimeoutExpired, unexpected exit code including signals). After 3 consecutive failures (_CRASH_LIMIT), sets _circuit_open = True and short-circuits all subsequent calls with {"action": "allow", "summary": "tirith disabled (circuit breaker)"}. Successful execution resets the counter.
  • tests/tools/test_tirith_security.py: existing tests updated to account for circuit breaker behavior after 3 failures.

Validation

Before After
SIGSEGV crash (fail_open=true) crash every call, agent retries forever 3 crashes → circuit opens → no more spawns
FileNotFoundError same hang pattern circuit breaker opens after 3
Timeout same hang pattern circuit breaker opens after 3
Successful execution n/a resets crash counter
  • scripts/run_tests.sh tests/tools/test_tirith_security.py — 95/95 passed.
  • E2E: verified circuit breaker opens after 3 SIGSEGV/FileNotFoundError/Timeout failures, short-circuits subsequent calls, resets on success, and doesn't trigger when tirith is disabled.

Salvaged from #41446 by @kyssta-exe. Chosen over #41423 (@iamlukethedev) because it's simpler (66 lines vs 262), uses module-level globals without a lock (matching the existing _warn_once pattern in the same file), and doesn't add a separate test file with placeholder assertions. #41423 uses a thread lock and has a placeholder test (assert result is not None or result is None).

Closes #41400

@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P1 High — major feature broken, no workaround sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data labels Jun 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P1 High — major feature broken, no workaround sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Security scanner (Tirith) SIGSEGV crash causes 20-minute agent hang with no response delivery

3 participants