fix(responses): enforce shared send budgets across retries and recovery

docs-site/src/content/docs/fr/reference/configuration/providers.md

@@ -154,6 +154,8 @@ sauvegarde dont le contenu diffère, puis réécrit en identifiants sans préfix
 | `unsafeAllowNativeLocalExec?` | `boolean` | Ancien booléen de Cursor, équivalent à `nativeLocalExec: "on"` uniquement lorsque le champ plus récent n'est pas défini. |
 | `nativeLocalExec?` | `"off" \| "codex-sandbox" \| "on"` | Politique d'exécution locale de Cursor. `off` est la valeur par défaut ; actuellement, `codex-sandbox` échoue de manière sûre comme `off`. |
 
+Pour les requêtes Responses traduites utilisant un pool de plusieurs clés, chaque invocation du fournisseur routé partage au maximum `N - 1` rotations de clés API entre la récupération initiale et les requêtes de continuation jusqu’à la fin de la réponse, où `N` est la taille du pool avant le premier envoi. L’expiration d’un délai de refroidissement ou l’agrandissement ultérieur du pool ne renouvelle pas cette limite. D’autres budgets d’envoi peuvent arrêter les tentatives plus tôt. Si la rotation est refusée, le dernier 429 enregistre toujours le délai de refroidissement de la clé en échec, mais aucune clé de remplacement n’est sélectionnée et la réponse suit le traitement d’erreur existant.
+
 Les fournisseurs à clé API peuvent détenir une clé littérale ou une référence à une variable d'environnement. Les fournisseurs OAuth utilisent le
 magasin d'identifiants alimenté par `ocx login` ; le comportement de lancement de Claude Code avec abonnement est
 configuré sous [`claudeCode.authMode`](/fr/reference/configuration/server/#claude-code-claudecode).

docs-site/src/content/docs/ja/reference/configuration/providers.md

@@ -146,6 +146,8 @@ account を削除しても mapping は保持され、同じ id を再追加す
 | `unsafeAllowNativeLocalExec?` | `boolean` |カーソルのレガシー ブール値。新しいフィールドが設定されていない場合のみ、`nativeLocalExec: "on"` と同等です。 |
 | `nativeLocalExec?` | `"off" \| "codex-sandbox" \| "on"` |カーソルのローカル実行ポリシー。 `off` がデフォルトです。 `codex-sandbox` は現在、`off` と同様にフェールクローズされます。 |
 
+複数キーのプールを使用する変換済み Responses リクエストでは、ルーティング先プロバイダーの呼び出しごとに、初期リカバリーと応答完了のための後続リクエストで API キーの切り替え上限 `N - 1` 回を共有し、`N` は最初の送信前のプールサイズに固定されます。クールダウンの満了や後からのプール拡張で、この上限が補充されることはありません。他の送信予算によって、再試行がさらに早く停止する場合もあります。切り替えが拒否された場合も、最後の 429 に対する失敗したキーのクールダウンは記録されますが、代わりのキーは選択されず、応答には既存のエラー処理が適用されます。
+
 API キープロバイダーは、リテラルキーまたは環境参照を保持する場合があります。 OAuth プロバイダーは、`ocx login` によって設定された資格情報ストアを使用します。サブスクリプションに基づくクロード コードの起動動作は、[`claudeCode.authMode`](/reference/configuration/server/#claude-code) で構成されます。
 
 ## プロバイダーによるアウトバウンドの安全性診断

docs-site/src/content/docs/ko/reference/configuration/providers.md

@@ -146,6 +146,8 @@ managed map을 활성화하면 privacy-safe selector를 만들고, 이후 계정
 | `unsafeAllowNativeLocalExec?` | `boolean` | Cursor 레거시 불리언입니다. 더 새로운 필드가 설정되지 않았을 때만 `nativeLocalExec: "on"`과 같습니다. |
 | `nativeLocalExec?` | `"off" \| "codex-sandbox" \| "on"` | Cursor 로컬 실행 정책입니다. 기본값은 `off`입니다. `codex-sandbox`는 현재 `off`처럼 실패를 닫습니다. |
 
+여러 키가 있는 풀을 사용하는 변환된 Responses 요청에서는 라우팅된 프로바이더 호출마다 최초 복구와 응답 완료를 위한 후속 요청이 최대 `N - 1`회의 API 키 회전 한도를 공유하며, `N`은 첫 전송 전의 풀 크기로 고정합니다. 쿨다운이 만료되거나 이후 풀이 커져도 이 한도는 충전되지 않습니다. 다른 전송 예산에 따라 재시도가 더 일찍 멈출 수 있습니다. 회전이 거부되면 마지막 429에 따른 실패 키의 쿨다운은 기록하지만 대체 키를 선택하지 않으며, 응답은 기존 오류 처리 방식으로 마무리합니다.
+
 API 키 공급자는 리터럴 키나 환경 참조를 둘 수 있습니다. OAuth 공급자는 `ocx login`으로 채워지는 자격 증명 저장소를 사용합니다. 구독 기반 Claude Code 실행 동작은 [`claudeCode.authMode`](/reference/configuration/server/#claude-code)에서 설정합니다.
 
 ## 공급자 진단용 외부 요청 안전성

docs-site/src/content/docs/reference/configuration/providers.md

@@ -224,6 +224,8 @@ Providers can expose a built-in shorthand, such as `agy` for `google-antigravity
 | `unsafeAllowNativeLocalExec?` | `boolean` | Cursor legacy boolean, equivalent to `nativeLocalExec: "on"` only when the newer field is unset. |
 | `nativeLocalExec?` | `"off" \| "codex-sandbox" \| "on"` | Cursor local-exec policy. `off` is default; `codex-sandbox` currently fails closed like `off`. |
 
+For translated Responses requests using a multi-key pool, each routed provider invocation shares a maximum of `N - 1` API-key rotations between its initial recovery and terminal continuations, where `N` is the pool size before the first send. Cooldown expiry or later pool growth does not replenish this allowance. Other send budgets may stop retries sooner. Once rotation is refused, the last 429 still records the failed key's cooldown, but no replacement key is selected and the response follows the existing error handling.
+
 With `webSearchBridge` enabled, a search continuation stays bound to the API-key selection that
 served the first request. Changing the selected key, its reference or resolved value, authentication
 mode, or base URL during search or provider pacing ends the turn with a bridge error before another

docs-site/src/content/docs/ru/reference/configuration/providers.md

@@ -159,6 +159,8 @@ cross-route credential fallback не существует. Строки API GPT-
 | `unsafeAllowNativeLocalExec?` | `boolean` | Legacy boolean Cursor, эквивалентен `nativeLocalExec: "on"` только если новое поле не задано. |
 | `nativeLocalExec?` | `"off" \| "codex-sandbox" \| "on"` | Политика local-exec для Cursor. `off` — дефолт; `codex-sandbox` сейчас ведёт себя fail-closed как `off`. |
 
+Для преобразованных запросов Responses с пулом из нескольких API-ключей каждый вызов выбранного маршрутизацией провайдера использует общий предел в `N - 1` переключений API-ключей для первоначального восстановления и последующих запросов, завершающих ответ; `N` фиксируется как размер пула до первой отправки. Истечение периода ожидания или последующее расширение пула не восстанавливает этот лимит. Другие бюджеты отправки могут остановить повторы раньше. Если переключение запрещено, для последнего 429 всё равно записывается период ожидания отказавшего ключа, но другой ключ не выбирается, а ответ обрабатывается существующим механизмом обработки ошибок.
+
 Провайдеры с API-key могут хранить literal key или environment-reference. OAuth-провайдеры
 используют credential store, заполняемый через `ocx login`; поведение subscription-backed launcher'а
 Claude Code настраивается через

docs-site/src/content/docs/tr/reference/configuration/providers.md

@@ -160,6 +160,8 @@ alanlı seçilmiş kimlikleri yalın kimliklere yeniden yazar.
 | `unsafeAllowNativeLocalExec?` | `boolean` | Cursor eski boolean değeri, yalnızca daha yeni alan ayarlanmadığında `nativeLocalExec: "on"` değerine eşdeğerdir. |
 | `nativeLocalExec?` | `"off" \| "codex-sandbox" \| "on"` | Cursor yerel yürütme politikası. `off` varsayılandır; `codex-sandbox` şu anda `off` gibi kapalı olarak başarısız olur. |
 
+Birden çok anahtar içeren havuz kullanan dönüştürülmüş Responses isteklerinde, yönlendirilen sağlayıcının her çağrısı ilk kurtarma ile yanıtı tamamlayan devam istekleri arasında en fazla `N - 1` API anahtarı değişimini paylaşır; `N`, ilk gönderimden önceki havuz boyutuna sabitlenir. Bekleme süresinin dolması veya havuzun sonradan büyümesi bu hakkı yenilemez. Diğer gönderim bütçeleri yeniden denemeleri daha erken durdurabilir. Anahtar değişimi reddedildiğinde son 429 için başarısız anahtarın bekleme süresi yine kaydedilir, ancak yerine başka bir anahtar seçilmez ve yanıt mevcut hata işleme yolunu izler.
+
 API anahtarı sağlayıcıları değişmez bir anahtar veya bir ortam referansı
 tutabilir. OAuth sağlayıcıları `ocx login` tarafından doldurulan kimlik bilgisi
 deposunu kullanır; abonelik destekli Claude Code başlatma davranışı

docs-site/src/content/docs/zh-cn/reference/configuration/providers.md

@@ -146,6 +146,8 @@ selector，而不是分配一个新名称。
 | `unsafeAllowNativeLocalExec?` | `boolean` | Cursor 旧布尔值；仅当更新字段未设置时，等同于 `nativeLocalExec: "on"`。 |
 | `nativeLocalExec?` | `"off" \| "codex-sandbox" \| "on"` | Cursor 本地执行策略。`off` 是默认值；`codex-sandbox` 目前会像 `off` 一样失败关闭。 |
 
+对于使用多密钥池的转换后 Responses 请求，每次路由到提供商的调用，其初始恢复和用于完成响应的后续请求共同使用最多 `N - 1` 次 API 密钥轮换额度，其中 `N` 固定为首次发送前的密钥池大小。冷却期结束或随后扩大密钥池都不会补充此额度。其他发送预算可能让重试更早停止。轮换被拒绝时，仍会根据最后一个 429 记录失败密钥的冷却期，但不会选择替代密钥，响应继续按现有错误处理方式处理。
+
 API key 提供者可以持有字面量 key，或环境引用。OAuth 提供者使用由 `ocx login` 填充的凭据存储；基于订阅的 Claude Code 启动行为在 [`claudeCode.authMode`](/reference/configuration/server/#claude-code) 下配置。
 
 ## 提供者诊断出站安全性

docs-site/src/content/docs/zh-tw/reference/configuration/providers.md

@@ -116,6 +116,8 @@ ocx models provider openrouter on
 | `unsafeAllowNativeLocalExec?` | `boolean` | Cursor 舊版布林值，僅在較新欄位未設定時等同於 `nativeLocalExec: "on"`。 |
 | `nativeLocalExec?` | `"off" \| "codex-sandbox" \| "on"` | Cursor 本機執行政策。`off` 為預設；`codex-sandbox` 目前像 `off` 般 fail closed。 |
 
+對於使用多金鑰集區的轉換後 Responses 要求，每次路由至提供者的呼叫，其初始復原和用於完成回應的後續要求共同使用最多 `N - 1` 次 API 金鑰輪替額度，其中 `N` 固定為首次傳送前的金鑰池大小。冷卻期結束或之後擴大金鑰池都不會補充此額度。其他傳送預算可能讓重試更早停止。輪替遭拒時，仍會根據最後一個 429 記錄失敗金鑰的冷卻期，但不會選取替代金鑰，回應繼續依現有錯誤處理方式處理。
+
 API-key 供應商可持有字面值金鑰或環境參考。OAuth 供應商使用由 `ocx login` 填入的憑證存放；訂閱支援的 Claude Code 啟動行為在 [`claudeCode.authMode`](/zh-tw/reference/configuration/server/#claude-code) 下設定。
 
 ## 供應商診斷對外安全

src/providers/key-failover.ts

@@ -363,6 +363,7 @@ function rotateKeyAfterFailure(
   now = Date.now(),
   attemptedKey?: string,
   attemptedSelection?: ProviderApiKeySelection,
+  allowRotation = true,
 ): OcxProviderConfig | null {
   const provider = config.providers[providerName];
   if (!provider) return null;
@@ -382,6 +383,10 @@ function rotateKeyAfterFailure(
       ? pool.find(entry => entry.id === attemptedSelection.entryId && entry.key === failedKey)
       : pool.find(entry => entry.key === failedKey);
 
+    // A spent request still records the failed key, but must not select or persist an
+    // unattempted replacement. Keep the fresh identity check and the changed:false path.
+    if (!allowRotation) return { changed: false, value: { failedId: failedEntry?.id } };
+
     if (freshProvider.apiKey !== failedKey) {
       const activeEntry = pool.find(entry => entry.key === freshProvider.apiKey);
       if (activeEntry && !isKeyInCooldown(providerName, activeEntry.id, now)) {
@@ -410,6 +415,7 @@ function rotateKeyAfterFailure(
   }, attemptedSelection);
   if (outcome.status === "unavailable") return null;
   if (outcome.status === "superseded") {
+    if (!allowRotation) return null;
     // A newer manual selection (including A→B→A) owns subsequent dispatch. Reusing the
     // same failed key here would loop forever; preserve its original failure instead.
     return outcome.provider.apiKey !== failedKey ? structuredClone(outcome.provider) : null;
@@ -425,6 +431,7 @@ function rotateKeyAfterFailure(
     keyCooldowns.set(cooldownKey(providerName, outcome.value.failedId), { cooldownUntil: now + cooldownMs });
     sweepExpiredOnWrite(now);
   }
+  if (!allowRotation) return null;
   if ("exhaustedCount" in outcome.value) {
     console.warn(`[key-failover] ${providerName}: all ${outcome.value.exhaustedCount} keys in cooldown after ${failureStatus}; returning the upstream status to the client`);
     return null;
@@ -448,8 +455,9 @@ export function rotateKeyOn429(
   now = Date.now(),
   attemptedKey?: string,
   attemptedSelection?: ProviderApiKeySelection,
+  allowRotation = true,
 ): OcxProviderConfig | null {
-  return rotateKeyAfterFailure(config, providerName, 429, retryAfterHeader, now, attemptedKey, attemptedSelection);
+  return rotateKeyAfterFailure(config, providerName, 429, retryAfterHeader, now, attemptedKey, attemptedSelection, allowRotation);
 }
 
 /**
@@ -486,6 +494,8 @@ interface RotateProviderTransportOptions {
   attemptedKey?: string;
   attemptedSelection?: ProviderApiKeySelection;
   promptCacheKey?: string;
+  /** False records a proven 429 cooldown without changing the selected key or returning a retry. */
+  allowRotation?: boolean;
 }
 
 /**
@@ -507,6 +517,7 @@ export function rotateProviderTransportOn429(
     options.now,
     options.attemptedKey,
     options.attemptedSelection ?? routedProvider._apiKeyAttempt,
+    options.allowRotation,
   );
   if (!rotated) return null;
   return applyRotatedTransport(providerName, routedProvider, rotated, options.promptCacheKey);
@@ -517,7 +528,7 @@ export function rotateProviderTransportOn401(
   config: OcxConfig,
   providerName: string,
   routedProvider: OcxProviderTransport,
-  options: Omit<RotateProviderTransportOptions, "retryAfter"> = {},
+  options: Omit<RotateProviderTransportOptions, "retryAfter" | "allowRotation"> = {},
 ): OcxProviderTransport | null {
   const rotated = rotateKeyOn401(config, providerName, options.now, options.attemptedKey,
     options.attemptedSelection ?? routedProvider._apiKeyAttempt);

src/server/responses/core.ts

@@ -7644,6 +7644,34 @@ async function handleResponsesInner(
     : 300_000;
   activeAdapter = adapter;
 
+  // Bound 429 rotations independently of cooldown expiry. Capture the pool before the first
+  // send; a later provider refresh cannot enlarge this invocation's allowance. The initial
+  // recovery and terminal continuations share it; it is a failover count, not a distinct-key set.
+  const maxKeyPoolFailovers = Math.max(0, (route.provider.apiKeyPool?.length ?? 0) - 1);
+  let keyPoolFailovers = 0;
+  const keyPool429RetryAllowed = (continuation: boolean): boolean => {
+    if (keyPoolFailovers >= maxKeyPoolFailovers) return false;
+    // Adapter-owned sends retain their existing base-only admission (for example Kiro).
+    if (activeAdapter.fetchResponse) {
+      return !adapterSendBudget
+        || adapterSendBudget.remainingBaseSends(adapterSendBudget.policy.baseSendAllowance) > 0;
+    }
+    const policy = transientRetryPolicyFor(route.provider);
+    // Reset-only transports do not opt into the shared transient policy; the rotation cap
+    // still bounds them without granting a new retry policy or changing their reset limit.
+    if (!policy) return true;
+    if (!Number.isInteger(policy.attempts) || policy.attempts <= 0) return false;
+    if (remainingTransientSendBudget(policy.attempts) > 0) return true;
+    // Continuations currently draw base sends only. The initial recovery can use the existing
+    // auth-recovery reserve; checking this decision does not consume a permit or add allowance.
+    if (continuation || !isRequestExecutionBudget(sendBudget)) return false;
+    return sendBudget.reserveDispatch({
+      sendClass: "auth-recovery",
+      targetKey: `${route.providerName}|${route.modelId}|key-429`,
+      countedExternally: true,
+    }).allowed;
+  };
+
   // One immutable, body-safe outbound request per same-target sequence (URL, serialized body,
   // auth headers, generated compat headers). Same-target 429 replays reuse it verbatim; the
   // builder runs again only after a key/account/adapter rotation, an oauth refresh, or an
@@ -8093,8 +8121,10 @@ async function handleResponsesInner(
           now: Date.now(),
           attemptedKey: route.provider.apiKey,
           promptCacheKey: parsed.options.promptCacheKey,
+          allowRotation: keyPool429RetryAllowed(false),
         });
         if (!rotated) break;
+        keyPoolFailovers += 1;
         // Release the failed response's socket before retrying; unread bodies otherwise linger
         // until runtime cleanup (one per rotated key under a rate-limit storm).
         try { void upstreamResponse.body?.cancel().catch(() => {}); } catch { /* already consumed/closed */ }
@@ -8516,6 +8546,7 @@ async function handleResponsesInner(
         response.status === 429
         && rateLimitPolicy !== null
         && rateLimitRetries < rateLimitPolicy.attempts
+        && !sendBudgetExhausted()
       ) {
         rateLimitRetries += 1;
         // Release unread body + heartbeat-fed wait via the shared same-target helper.
@@ -8562,8 +8593,10 @@ async function handleResponsesInner(
           now: Date.now(),
           attemptedKey: route.provider.apiKey,
           promptCacheKey: nextParsed.options.promptCacheKey,
+          allowRotation: keyPool429RetryAllowed(true),
         });
         if (rotated) {
+          keyPoolFailovers += 1;
           try { void response.body?.cancel().catch(() => {}); } catch { /* already closed */ }
           route.provider = rotated;
           invalidateSameTargetRequest();

structure/adapters/registry.md

@@ -1,7 +1,7 @@
 # Adapter Registry Authority
 
 The configuration-only [plaintext V2 contract](../subagents.md#plaintext-v2-agent-messages)
-is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged.
+is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Generic Responses API-key failover follows the [bounded rotation contract](../transports/responses.md#bounded-api-key-429-rotation).
 
 Shared parsing and streaming follow the [request-copy](../transports/byte-accounting.md#request-copy-accounting) and [stream-buffer accounting](../transports/byte-accounting.md#stream-buffer-accounting) contracts.
 

structure/catalog.md

@@ -1,7 +1,7 @@
 # Model Catalog
 
 The configuration-only [plaintext V2 contract](subagents.md#plaintext-v2-agent-messages)
-is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged.
+is scoped to canonical ChatGPT Responses forwarding; other source-area behavior described here is unchanged. Generic Responses API-key failover follows the [bounded rotation contract](transports/responses.md#bounded-api-key-429-rotation).
 
 Shared parsing and streaming follow the [request-copy](transports/byte-accounting.md#request-copy-accounting) and [stream-buffer accounting](transports/byte-accounting.md#stream-buffer-accounting) contracts.