fix: add retry circuit breaker and backoff cap to prevent infinite retry loops#17668
Closed
dawidbednarczyk wants to merge 1 commit intoanomalyco:devfrom
Closed
fix: add retry circuit breaker and backoff cap to prevent infinite retry loops#17668dawidbednarczyk wants to merge 1 commit intoanomalyco:devfrom
dawidbednarczyk wants to merge 1 commit intoanomalyco:devfrom
Conversation
…try loops When API errors trigger retries with response headers present but no retry-after header, the exponential backoff grows without bound (observed 202s+ delays in production). Combined with the while(true) loop in processor.ts having no exit condition, this causes sessions to hang indefinitely burning CPU and tokens. Changes: - Add RETRY_MAX_ATTEMPTS (10) to cap total retry count - Add RETRY_MAX_DELAY_WITH_HEADERS (60s) to cap backoff when headers are present but missing retry-after - Add circuit breaker in processor.ts that breaks the retry loop after max attempts, publishes error event, and sets session to idle Validated against production logs showing 11 retries over 542 seconds with AI_APICallError: Could not relay message upstream. Relates to anomalyco#17648
Contributor
|
Thanks for your contribution! This PR doesn't have a linked issue. All PRs must reference an existing issue. Please:
See CONTRIBUTING.md for details. |
Author
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue for this PR
Relates to #17648
Type of change
What does this PR do?
When API errors trigger retries and response headers are present but don't contain
retry-after,delay()inretry.tscomputes exponential backoff with no upper bound (line 54). The no-headers path is capped at 30s, but the with-headers fallback path isn't — it just returnsRETRY_INITIAL_DELAY * 2^(attempt-1)raw.Meanwhile
processor.tshas awhile(true)loop with no exit condition on retries — it incrementsattempt, sleeps the unbounded delay, andcontinues forever.I hit this in production: 11 consecutive
AI_APICallError: Could not relay message upstreamerrors over 9 minutes, with delays escalating to 202 seconds between attempts. Process never recovered, had to be killed manually.Three changes:
RETRY_MAX_DELAY_WITH_HEADERS = 60_000— caps the with-headers fallback path at 60s (matching the spirit of the existing 30s no-headers cap)RETRY_MAX_ATTEMPTS = 10— exported constant for max retry countprocessor.ts— afterattempt > RETRY_MAX_ATTEMPTS, logs the failure, publishes error event, sets session idle, and breaks the loopThe fix is minimal and doesn't change behavior for successful retries or retries that respect
retry-afterheaders.How did you verify your code works?
tsc --noEmitpasses cleanChecklist