fix(#41400): add circuit breaker to prevent tirith retry loops

[iamlukethedev]

Summary

Fixes critical issue where misconfigured or unavailable Tirith binary causes infinite retry loops, leading to:

• 2,704+ crashes over 7 days (all hidden behind fail_open: true)
• 20+ minute agent hangs with no response delivery to users
• User messages queued indefinitely without responses

Root Cause

When tirith_path points to a non-existent binary:

1. check_command_security() calls subprocess.run() with bad path
2. Process returns exit code -11 (SIGSEGV)
3. fail_open: true silently allows the tool to proceed
4. Model retries the same tool call
5. Infinite loop blocks all responses

Solution: Circuit Breaker

Add circuit breaker pattern to prevent retry loops:

• Track consecutive failures (OSError, TimeoutExpired, unexpected exit codes)
• After 3 consecutive failures, disable Tirith for the session
• Circuit breaker is thread-safe (mutex-protected)
• Reset counter on success to allow recovery if Tirith becomes available

Changes

tools/tirith_security.py:

• Added module-level state: _consecutive_failures, _circuit_breaker_disabled, _circuit_breaker_lock
• Circuit breaker check at start of check_command_security()
• Failure tracking in all three failure paths (OSError, TimeoutExpired, unexpected exit)
• Counter reset on success

test_tirith_security.py:

• Updated fixture to reset circuit breaker state between tests
• Updated tests to account for circuit breaker behavior after 3 failures

tests/tools/test_tirith_startup_validation.py:

• New test suite with 8 regression tests covering:
  • Non-existent path with fail_open
  • SIGSEGV handling
  • PermissionError/FileNotFoundError handling
  • Circuit breaker activation

Testing

✅ 100 Tirith-related tests pass
✅ 8 new regression tests for circuit breaker
✅ All existing tests updated and passing
✅ Manual verification of circuit breaker activation

Fixes #41400

[tonydwb]

Code Review Summary

Reviewed PR #41423: fix(#41400): add circuit breaker to prevent tirith retry loops

✅ Approve

• Adds a circuit-breaker pattern to stop retry amplification in tirith calls, which matches the reported failure mode.
• Guardrails appear to target the exact retry entrypoint.

---

Reviewed in batch on 2026-06-08

[kshitijk4poor]

Merged via #52997. Your circuit breaker approach was considered but #41446 by @kyssta-exe was chosen as the base — simpler (66 lines vs 262), no placeholder tests, matches the existing lock-free pattern. Thanks for the contribution!