Skip to content

fix(server): resume Claude turns after gateway quota recovers - #27000

Closed
Tsopic wants to merge 3 commits into
pingdotgg:mainfrom
Tsopic:codex/quota-recovery-pr-20261029
Closed

Tsopic wants to merge 3 commits into
pingdotgg:mainfrom
Tsopic:codex/quota-recovery-pr-20261029

Conversation

@Tsopic

@Tsopic Tsopic commented Oct 29, 2026

Copy link
Copy Markdown

Claude-compatible gateways can return 503 claude_quota_exhausted or a structured quota error without a known reset time. T3 currently treats these as generic failures or leaves them without a deadline, so the existing automatic-resume workflow never continues them when capacity returns.

Recognize the stable gateway quota codes on 429/503, including a result preceded by an assistant rate-limit frame. Keep a real quota reset separate from a persisted retry deadline. When the gateway cannot report a reset or retry hint, use bounded exponential backoff with deterministic jitter. Persist the attempt count in provider-thread metadata and clear it after success.

Reuse the existing durable recovery worker and settings. A per-thread opt-in follows automatic quota continuations through another failed attempt, while Stop, cancellation, pending requests, replacement and manual runs retain their boundaries. Web/desktop and mobile distinguish retries from confirmed resets. No recovery setting or product default changes; this repairs an existing recovery path for an obvious rate-limit failure.

Validation: the integrated branch passed vp run check:changed origin/main, including 602 tests, affected-package typechecks, formatting, lint and React Compiler checks. Regression tests failed before the fixes. Coverage includes SQL recovery candidates, duplicate delivery, cancellation/Stop races, per-thread opt-in with the global setting off, a later manual run, and a fresh adapter restoring serialized retry metadata through another quota failure and eventual success. Interactive client screenshots and live provider calls were not performed.

Written with GPT-6.1 Sol through the Codex harness in T3 Code; independently reviewed with GPT-6 Astra.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review 🔄 Running since 2026-10-29T14:52:41.453959Z e04a1e4 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions github-actions Bot added vouch:unvouched PR author is not yet trusted in the VOUCHED list. size:L 100-499 changed lines (additions + deletions). labels Oct 29, 2026
@macroscopeapp

macroscopeapp Bot commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved at e04a1e4

Macroscope's review found this PR approvable — Claude gateway quota failures now support scheduled retries, with web and mobile distinguishing retry times from confirmed quota resets. Recovery remains opt-in, with backoff and existing Stop, cancellation and stale-request safeguards.

You can add or adjust custom eligibility rules. Learn more.

@coderabbitai

coderabbitai Bot commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between 37fbefd and e04a1e4.

Files selected for processing: 17
  • apps/mobile/src/features/threads/UsageLimitRecoveryCard.tsx
  • apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.test.ts
  • apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.ts
  • apps/server/src/orchestration-v2/Adapters/claudeGatewayQuota.test.ts
  • apps/server/src/orchestration-v2/Adapters/claudeGatewayQuota.ts
  • apps/server/src/orchestration-v2/Orchestrator.ts
  • apps/server/src/orchestration-v2/ProjectionStore.test.ts
  • apps/server/src/orchestration-v2/ProjectionStore.ts
  • apps/server/src/orchestration-v2/UsageLimitRecoveryWorker.ts
  • apps/server/src/orchestration-v2/runtimeLayer.test.ts
  • apps/web/src/components/ChatView.tsx
  • apps/web/src/components/chat/UsageLimitRecoveryBanner.test.tsx
  • apps/web/src/components/chat/UsageLimitRecoveryBanner.tsx
  • packages/client-runtime/src/state/models.ts
  • packages/contracts/src/orchestrationV2.ts
  • packages/provider-core/src/server/failure.ts
  • packages/shared/src/orchestrationV2ThreadError.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 8 remain after this review.

Walkthrough

Claude quota failures can now include a retry deadline when no usable reset time is available. Orchestration uses that deadline for recovery scheduling, and web and mobile interfaces display retry timing and corresponding actions.

Changes: Usage-limit retry recovery

Layer / File(s) Summary
Classify Claude quota failures
packages/contracts/src/orchestrationV2.ts, packages/provider-core/src/server/failure.ts, apps/server/src/orchestration-v2/Adapters/claudeGatewayQuota.ts, apps/server/src/orchestration-v2/Adapters/ClaudeAdapterV2.ts, apps/server/src/orchestration-v2/Adapters/*test.ts
The Claude adapter recognizes eligible quota errors and returns retryable usage-limit failures with reset or retry timing. Provider-thread metadata tracks quota retry attempts, and tests cover classification and backoff.
Propagate retry deadlines through projections
packages/shared/src/orchestrationV2ThreadError.ts, apps/server/src/orchestration-v2/ProjectionStore.ts, apps/server/src/orchestration-v2/ProjectionStore.test.ts, packages/client-runtime/src/state/models.ts
Retry timestamps flow into shell and runtime summaries. Recovery candidate selection uses a retry deadline when the failure is retryable and has no reset time.
Schedule and continue retry recovery
apps/server/src/orchestration-v2/Orchestrator.ts, apps/server/src/orchestration-v2/UsageLimitRecoveryWorker.ts, apps/server/src/orchestration-v2/runtimeLayer.test.ts
Recovery validation and scheduling use reset or retry timing. Continuations can carry matching auto-resume recovery state to a new run. Runtime tests cover retry deadlines and recovery scenarios.
Display retry timing and recovery actions
apps/web/src/components/ChatView.tsx, apps/web/src/components/chat/UsageLimitRecoveryBanner.tsx, apps/web/src/components/chat/UsageLimitRecoveryBanner.test.tsx, apps/mobile/src/features/threads/UsageLimitRecoveryCard.tsx
The web banner and mobile card use retry timing when reset timing is absent. Their descriptions and actions distinguish retry times from known reset times.

Priority: Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant ClaudeAdapterV2
  participant claudeGatewayQuotaFailure
  participant ProjectionStore
  participant UsageLimitRecoveryWorker
  participant UsageLimitRecoveryBanner
  ClaudeAdapterV2->>claudeGatewayQuotaFailure: classify quota API error
  claudeGatewayQuotaFailure-->>ClaudeAdapterV2: return retryable failure with retryAt
  ClaudeAdapterV2->>ProjectionStore: provide usage-limit failure
  ProjectionStore->>UsageLimitRecoveryWorker: expose retry-time recovery candidate
  UsageLimitRecoveryWorker->>UsageLimitRecoveryBanner: provide shell retry timing
Loading

Suggested labels: bugfix

Suggested reviewers: juliusmarminge, t3dotgg, fixfon

Merge Risk: Minimal · up to e04a1

This change lets Claude gateway quota failures without a known reset time resume automatically through the existing recovery worker. Web and mobile now show the retry time and use retry-specific action labels. No merge-blocking risk was identified in the supplied changes. Interactive client behavior and live provider calls were not exercised.

Pre-merge checks: Passed 4
Check name Status Explanation
Title check Passed The title clearly identifies the main change: resuming Claude turns after gateway quota recovery. It is concise, specific, and uses a conventional commit format.
Description check Passed The description covers the problem, implementation, scope boundaries, verification results, limitations, and agent disclosure. It does not use the template headings and does not include the requested …
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Description check

Explanation

The description covers the problem, implementation, scope boundaries, verification results, limitations, and agent disclosure. It does not use the template headings and does not include the requested UI screenshots, but it remains sufficiently complete and directly relevant.

Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@t3-code-bot

t3-code-bot Bot commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

closing at @StiensWout's request under Provide evidence for the changed behavior.

Changes the visible web/mobile quota-recovery states to distinguish estimated retry deadlines from confirmed resets, but supplies no before/after screenshots or interaction recording. The PR explicitly says interactive client screenshots were not performed. Its extensive backend recovery tests do not demonstrate those changed client states.

Add clear before/after captures of the affected web/mobile recovery states, with a short recording if needed to demonstrate retry timing or interaction. Retain the existing focused backend proof.

thanks for trying to help improve t3 code. if you think this closure was a mistake, please open a fresh pr addressing the requirement above, and link back to this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:L 100-499 changed lines (additions + deletions). vouch:unvouched PR author is not yet trusted in the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant