fix: add retry circuit breaker and backoff cap to prevent infinite retry loops#17790
Open
dawidbednarczyk wants to merge 1 commit intoanomalyco:devfrom
Open
fix: add retry circuit breaker and backoff cap to prevent infinite retry loops#17790dawidbednarczyk wants to merge 1 commit intoanomalyco:devfrom
dawidbednarczyk wants to merge 1 commit intoanomalyco:devfrom
Conversation
…try loops When API errors trigger retries with response headers present but no retry-after header, the exponential backoff grows without bound (observed 202s+ delays in production). Combined with the while(true) loop in processor.ts having no exit condition, this causes sessions to hang indefinitely burning CPU and tokens. Changes: - Add RETRY_MAX_ATTEMPTS (10) to cap total retry count - Add RETRY_MAX_DELAY_WITH_HEADERS (60s) to cap backoff when headers are present but missing retry-after - Add circuit breaker in processor.ts that breaks the retry loop after max attempts, publishes error event, and sets session to idle Validated against production logs showing 11 retries over 542 seconds with AI_APICallError: Could not relay message upstream. Fixes anomalyco#17648
github-actions
bot
added
needs:compliance
Contributor
|
Thanks for updating your PR! It now meets our contributing guidelines. 👍 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue for this PR
Closes #17648
Type of change
What does this PR do?
This fixes an infinite retry loop in the session processor.
Today
processor.tsretries inside awhile (true)loop with no exit condition. At the same time,retry.tscaps backoff only when response headers are missing. If headers are present butretry-afteris missing, the fallback path returns raw exponential backoff with no cap.In practice this means transient upstream API failures can leave a session stuck retrying for hours. In my logs this showed up as repeated
AI_APICallError: Could not relay message upstreamfailures with delays growing to 202 seconds between attempts, and the process never recovered without manual intervention.This PR makes three small changes:
RETRY_MAX_ATTEMPTS = 10RETRY_MAX_DELAY_WITH_HEADERS = 60_000processor.tsafter max attempts, publishes the error event, and returns the session to idleThis is a minimal fix. It does not change behavior for successful retries or for responses that already provide a valid
retry-afterheader.This is also the same underlying problem described in #12234 and #17169.
How did you verify your code works?
devpulled in the recent Windows CI fix (fix(ci): workaround by using hoisted Bun linker on Windows #17751)bun turbo typecheckpasses locally across all packagesScreenshots / recordings
Not a UI change.
Checklist