fix(security): circuit breaker for tirith crashes to prevent agent hangs (#41400) - #52997
Merged
kshitijk4poor merged 1 commit intoJun 26, 2026
Conversation
kshitijk4poor
force-pushed
the
salvage/41400-tirith-circuit-breaker
branch
from
June 26, 2026 09:39
1bdcfda to
689ecf4
Compare
This was referenced Jun 26, 2026
18 tasks
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Tirith security scanner crashes (SIGSEGV, FileNotFoundError, timeout) no longer cause 20-minute agent hangs — after 3 consecutive failures, a circuit breaker disables tirith for the rest of the process.
Root cause (#41400)
When
tirith_pathpoints to a non-existent or broken binary, every tool invocation triggers a crash (exit code -11/SIGSEGV, FileNotFoundError, etc.). Withfail_open: true, the tool proceeds anyway, and the agent retries the same tool call — creating a 20-minute hang where no responses are delivered. 2,704+ crashes were logged over 7 days, all hidden behindfail_open.Changes
tools/tirith_security.py: circuit breaker pattern — tracks consecutive failures across all three failure paths (OSError, TimeoutExpired, unexpected exit code including signals). After 3 consecutive failures (_CRASH_LIMIT), sets_circuit_open = Trueand short-circuits all subsequent calls with{"action": "allow", "summary": "tirith disabled (circuit breaker)"}. Successful execution resets the counter.tests/tools/test_tirith_security.py: existing tests updated to account for circuit breaker behavior after 3 failures.Validation
scripts/run_tests.sh tests/tools/test_tirith_security.py— 95/95 passed.Salvaged from #41446 by @kyssta-exe. Chosen over #41423 (@iamlukethedev) because it's simpler (66 lines vs 262), uses module-level globals without a lock (matching the existing
_warn_oncepattern in the same file), and doesn't add a separate test file with placeholder assertions. #41423 uses a thread lock and has a placeholder test (assert result is not None or result is None).Closes #41400