diff --git a/docs-site/src/content/docs/fr/reference/configuration/providers.md b/docs-site/src/content/docs/fr/reference/configuration/providers.md index af0b2a50512..9ca5519edf6 100644 --- a/docs-site/src/content/docs/fr/reference/configuration/providers.md +++ b/docs-site/src/content/docs/fr/reference/configuration/providers.md @@ -142,7 +142,7 @@ sauvegarde dont le contenu diffère, puis réécrit en identifiants sans préfix | `responsesItemIdRepair?` | `{ message?: string[]; reasoning?: string[]; repairMissingTerminalIds?: boolean; repairInvalidIds?: boolean }` | Réparation SSE en aval désactivée par défaut pour les identifiants d'espace réservé exacts, les identifiants de terminal manquants et (avec `repairInvalidIds`) les identifiants message/reasoning manquant du préfixe canonique `msg_`/`rs_`. Les identifiants d’appel de fonction ne sont jamais réécrits. Le DeepSeek intégré active les deux derniers par défaut. | | `responsesSnapshotRepair?` | `boolean` | Réparation côté client désactivée par défaut pour les instantanés du cycle de vie des réponses clairsemés dans SSE et JSON. Remplit les métadonnées d'état canonique, de sortie et d'outil manquantes tandis que l'inspection brute et la persistance restent inchangées. | | `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | Fournisseurs à clé API uniquement (`authMode: "key"`). Nouvelle tentative facultative sur la même cible après un 429 : lorsque `retryOn429` est absent, la fonctionnalité est désactivée ; la présence d'un objet l'active, sauf avec `enabled: false`. Après un 429, le proxy attend selon `Retry-After` reçu en amont ou selon l'intervalle fixe, puis relit la requête à l'identique avec la même clé avant tout basculement de clé. Ce comportement couvre la boucle principale de récupération d'un tour textuel, le protocole de transfert Responses, le pont d'images et de vidéos, le service auxiliaire de recherche Web et les continuations du terminal. Seules les réponses HTTP 429 reçues avant le début de la diffusion peuvent être relues ; les transports `runTurn` personnalisés ne font pas partie de la boucle de nouvelle tentative HTTP. `attempts` compte les relectures avec la même clé après le premier 429, soit `attempts` + 1 envois au total, et constitue un budget commun à toute la requête, partagé entre la boucle principale de récupération, la continuation de la garde du terminal et les nouvelles tentatives du pont. L'épuisement de `attempts` arrête uniquement les relectures supplémentaires avec la même clé : le basculement normal de clé ou la gestion de l'erreur finale s'applique ensuite selon les cibles disponibles. Sur le protocole de transfert authentifié par clé, aucun basculement n'est possible ; le 429 final est donc renvoyé sans modification. Codex ne retente jamais lui-même une requête après un 429 : cette option constitue ainsi la seule protection pour les fournisseurs à clé unique. Valeurs par défaut : `enabled: true`, `attempts: 3`, `intervalMs: 5000`, `maxIntervalMs: 60000` (chaque attente est plafonnée à `maxIntervalMs`, lui-même plafonné à 600000), `respectRetryAfter: true`. | -| `transientRetryOn5xx?` | `{ enabled?: boolean; attempts?: number }` | Fournisseurs `openai-chat` et `openai-responses` authentifiés par clé uniquement. Les fournisseurs `authMode: "forward"` (le pool de comptes ChatGPT) ne lisent jamais cette option et conservent l'échelle par défaut. Nouvelle tentative facultative pour les états transitoires reçus en amont avant le début de la diffusion (500, 502, 503, 504, 520, 521, 522) : l'absence de l'option la désactive ; la présence d'un objet l'active, sauf avec `enabled: false`. Ce comportement couvre la requête Responses initiale, la continuation de la garde du terminal, le point de terminaison natif `/v1/chat/completions` et les réémissions liées à la récupération après un 429 ou à la récupération de compte. `attempts` représente le nombre TOTAL d'envois en amont autorisés pour une requête, premier envoi compris (de 1 à 10, valeur par défaut : 3). Il constitue un budget commun à la requête, partagé avec la récupération après une réinitialisation de connexion ; ainsi, `3` signifie qu'au plus trois requêtes réelles atteignent le fournisseur. Les attentes utilisent une temporisation exponentielle à base fixe de 400 ms, plafonnée à 5 s, et respectent `Retry-After`. Cette option est distincte de `retryOn429`, qui traite la limitation de débit ; les échecs en cours de diffusion ne sont jamais relus. | +| `transientRetryOn5xx?` | `{ enabled?: boolean; attempts?: number }` | Fournisseurs `openai-chat` et `openai-responses` authentifiés par clé uniquement. Les fournisseurs `authMode: "forward"` (le pool de comptes ChatGPT) ne lisent jamais cette option et conservent l'échelle par défaut. Nouvelle tentative facultative pour les états transitoires reçus en amont avant le début de la diffusion (500, 502, 503, 504, 520, 521, 522) : l'absence de l'option la désactive ; la présence d'un objet l'active, sauf avec `enabled: false`. Ce comportement couvre la requête Responses initiale, la continuation de la garde du terminal, le point de terminaison natif `/v1/chat/completions` et les réémissions liées à la récupération après un 429 ou à la récupération de compte. `attempts` représente le nombre TOTAL d'envois en amont autorisés pour une requête, premier envoi compris (de 1 à 10, valeur par défaut : 3). Il constitue un budget commun à la requête, partagé avec la récupération après une réinitialisation de connexion ; ainsi, `3` signifie qu'au plus trois requêtes réelles atteignent le fournisseur. Les attentes utilisent une temporisation exponentielle à base fixe de 400 ms, plafonnée à 5 s, et respectent `Retry-After`. Cette option est distincte de `retryOn429`, qui traite la limitation de débit ; les échecs en cours de diffusion ne sont jamais relus. Chaque cible Combo compte son premier envoi prépayé dans son propre total configuré : `attempts: 1` autorise un envoi, sans rembourser la réservation ni élargir le plafond partagé. Un refus local ou une annulation avant l'envoi libère la réservation inutilisée. | | `retryOnReset?` | `{ enabled?: boolean; replacements?: number }` | Fournisseurs `openai-responses` natifs, `authMode: "forward"` compris, ainsi que les requêtes Responses traduites envoyées par la voie générique avant les en-têtes reçus en amont. Remplacement facultatif d'un envoi qui a échoué alors que l'appelant n'avait rien observé : l'absence de l'option la désactive ; la présence d'un objet l'active, sauf avec `enabled: false`. Couvre les deux étapes ambiguës — une connexion rompue avant tout en-tête de réponse, et un corps SSE rompu après l'en-tête alors qu'il ne portait que des événements de contrôle. Une WebSocket canonique ChatGPT en amont qui s'est fermée ou a échoué après l'envoi de sa trame de création, avant tout événement Responses, est couverte de la même façon ; son remplacement est envoyé en HTTP. Seule une requête autonome est remplacée : `store: false`, `input` complet, ni `previous_response_id`, ni `conversation`, ni `stream_id`, et uniquement des outils exécutés par le client. `replacements` est le nombre d'envois de remplacement qu'UNE requête logique peut effectuer, toutes étapes et tous enfants de combo confondus (de 1 à 2, valeur par défaut : 1). Ce n'est ni un nombre de tentatives par étape ni un budget d'envoi : un remplacement doit toujours tenir dans l'allocation d'envois dont l'étape disposait déjà. Une requête qui a déjà émis une sortie ou un appel d'outil n'est jamais remplacée, quelle que soit cette valeur. L'inférence de remplacement peut tout de même être facturée si l'origine avait déjà démarré la première, d'où la désactivation par défaut. La voie générique de traduction prend en charge le remplacement après une rupture avant les en-têtes pour les envois initiaux et reconstruits, avec la même autorisation et le même budget d’envois. Les transports gérés par les adaptateurs et les erreurs des flux traduits après les en-têtes reçus en amont sont exclus. | | `autoToolChoiceOnlyModels?` | `string[]` | Modèles dont `tool_choice` accepte uniquement `auto` ou `none` ; les choix forcés sont dévalorisés. | | `preserveReasoningContentModels?` | `string[]` | Modèles nécessitant un assistant préalable `reasoning_content` dans l'historique des discussions. Les enregistrements depuis le tableau de bord conservent la liste enregistrée, y compris `[]`. `PATCH /api/providers?name=` accepte un tableau ou `null` pour l'effacer. Une sauvegarde qui déplace le fournisseur vers un autre adaptateur, une autre URL de base ou un autre mode d'authentification ne la conserve pas (voir la section ci-dessous). | diff --git a/docs-site/src/content/docs/ja/reference/configuration/providers.md b/docs-site/src/content/docs/ja/reference/configuration/providers.md index fe4649c729b..7e9d88bfc1e 100644 --- a/docs-site/src/content/docs/ja/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ja/reference/configuration/providers.md @@ -134,7 +134,7 @@ account を削除しても mapping は保持され、同じ id を再追加す | `responsesItemIdRepair?` | `{ message?: string[]; reasoning?: string[]; repairMissingTerminalIds?: boolean; repairInvalidIds?: boolean }` |正確なプレースホルダー ID、欠落している端末 ID、および（`repairInvalidIds` で）正規の `msg_`/`rs_` 接頭辞を欠く message/reasoning ID に対するダウンストリーム SSE 修復はデフォルトで無効になっています。関数呼び出し ID は決して書き換えられません。組み込み DeepSeek は最後の 2 つをデフォルトで有効にします。 | | `responsesSnapshotRepair?` | `boolean` | デフォルトで無効のクライアント向け修復です。SSE と JSON の Responses ライフサイクルで欠落した status、output、ツールメタデータを補完し、raw 検査と永続化は変更しません。 | | `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | API-key プロバイダーのみ(`authMode: "key"`)。オプトインの同一ターゲット 429 リトライ: `retryOn429` が無ければ無効で、オブジェクトがあれば `enabled: false` でない限り有効になります。429 時に待機(上流の `Retry-After` または固定間隔)してから、キー フェイルオーバーの前に同一キーで同一リクエストを再送します — メインのテキストターン回復ループ、Responses passthrough、画像/動画ブリッジ、web-search サイドカー、ターミナル継続要求をすべてカバーします。再送の対象はプリストリームの HTTP 429 応答のみで、カスタム `runTurn` トランスポートは HTTP リトライループの対象外です。`attempts` は最初の 429 以降の同一キー再送回数(合計送信数 = `attempts` + 1)で、メインの回復ループ・ターミナルガード継続・ブリッジ再試行で共有されるリクエスト単位の予算です。`attempts` を使い切っても同一キーでの再送が止まるだけで、通常のキー フェイルオーバーまたは最終エラー処理が利用可能なターゲットに応じて続きます — キー認証の passthrough ワイヤにはフェイルオーバーがないため、使い切った 429 はそのまま返ります。Codex 自体は 429 をリトライしないため、単一キーのプロバイダーでは唯一の防御です。デフォルト: `enabled: true`、`attempts: 3`、`intervalMs: 5000`、`maxIntervalMs: 60000`(1回の待機は `maxIntervalMs` で上限、その上限は 600000)、`respectRetryAfter: true`。 | -| `transientRetryOn5xx?` | `{ enabled?: boolean; attempts?: number }` | キー認証の `openai-chat` および `openai-responses` プロバイダーのみ。`authMode: "forward"` のプロバイダー（ChatGPT アカウントプール）はこのオプションを読まず、既定の再試行段数を維持します。ストリーム開始前に上流から返される一時的なステータス（500、502、503、504、520、521、522）に対するオプトインの再試行です。設定がなければ無効で、オブジェクトを指定すると `enabled: false` でない限り有効になります。最初の Responses リクエスト、ターミナルガード継続、ネイティブの `/v1/chat/completions`、および 429／アカウント回復時の再取得が対象です。`attempts` は最初の送信を含め、1 回のリクエストで許可される上流への送信総数です（1～10、デフォルトは 3）。接続リセット回復と共有するリクエスト単位の単一予算であるため、`3` を指定した場合、プロバイダーに到達する実リクエストは最大 3 回です。待機には 400 ms を基準とする固定式の指数バックオフを使用し、上限は 5 秒で、`Retry-After` に従います。レート制限を扱う `retryOn429` とは別の機能であり、ストリーム開始後の失敗は再送されません。 | +| `transientRetryOn5xx?` | `{ enabled?: boolean; attempts?: number }` | キー認証の `openai-chat` および `openai-responses` プロバイダーのみ。`authMode: "forward"` のプロバイダー（ChatGPT アカウントプール）はこのオプションを読まず、既定の再試行段数を維持します。ストリーム開始前に上流から返される一時的なステータス（500、502、503、504、520、521、522）に対するオプトインの再試行です。設定がなければ無効で、オブジェクトを指定すると `enabled: false` でない限り有効になります。最初の Responses リクエスト、ターミナルガード継続、ネイティブの `/v1/chat/completions`、および 429／アカウント回復時の再取得が対象です。`attempts` は最初の送信を含め、1 回のリクエストで許可される上流への送信総数です（1～10、デフォルトは 3）。接続リセット回復と共有するリクエスト単位の単一予算であるため、`3` を指定した場合、プロバイダーに到達する実リクエストは最大 3 回です。待機には 400 ms を基準とする固定式の指数バックオフを使用し、上限は 5 秒で、`Retry-After` に従います。レート制限を扱う `retryOn429` とは別の機能であり、ストリーム開始後の失敗は再送されません。 各 Combo ターゲットでは、事前予約した初回送信もそのターゲットの設定総数に含まれます。`attempts: 1` は 1 回だけ送信し、予約を返却して余裕を作ったり、共有上限を広げたりしません。送信前のローカル拒否やキャンセルでは未使用の予約を解放します。 | | `retryOnReset?` | `{ enabled?: boolean; replacements?: number }` | ネイティブ `openai-responses` プロバイダー（`authMode: "forward"` を含む）と、上流ヘッダー受信前の汎用 Responses 変換経路に適用されます。呼び出し側が何も観測しないまま失敗した送信を、オプトインで置き換えます。設定がなければ無効で、オブジェクトを指定すると `enabled: false` でない限り有効になります。レスポンスヘッダーが届く前に接続が切れた場合と、ヘッダー後に SSE 本文が制御イベントだけを運んだまま切れた場合の両方が対象です。canonical ChatGPT upstream WebSocket で create フレームの送信後、Responses イベントが届く前にソケットが閉じたかエラーになった場合も同じく対象で、その置き換えは HTTP で送信します。置き換えるのは自己完結したリクエストだけで、`store: false`、完全な `input`、`previous_response_id` / `conversation` / `stream_id` がないこと、クライアントが実行するツールのみ、が条件です。`replacements` は、すべてのレッグとすべてのコンボ子リクエストを合わせて 1 つの論理リクエストが行える置き換え送信の回数です（1..2、デフォルトは 1）。レッグ単位の再試行回数でも送信予算でもないため、置き換え送信もそのレッグがすでに持つ送信許容量に収まる必要があります。すでに出力やツール呼び出しを送ったリクエストは、この値に関わらず置き換えません。元の送信がすでに開始されていた場合は置き換えた推論も課金される可能性があるため、既定では無効です。 汎用変換経路では、初回送信と再構築後の送信のどちらも、同じ許可と送信予算でヘッダー受信前の接続リセットを置き換えます。アダプター独自のトランスポートと、上流ヘッダー受信後の変換ストリーム障害は対象外です。 | | `autoToolChoiceOnlyModels?` | `string[]` | `tool_choice` が `auto` または `none` のみを受け入れるモデル。強制的な選択は格下げされます。 | | `preserveReasoningContentModels?` | `string[]` |チャット履歴に以前のアシスタント `reasoning_content` が必要なモデル。ダッシュボードから保存しても、保存済みのリスト（`[]` を含む）は保持されます。`PATCH /api/providers?name=` は配列、または消去するための `null` を受け付けます。 アダプター、ベース URL、または認証モードを変えて別の宛先に移す保存では保持されません（下の節を参照）。 | diff --git a/docs-site/src/content/docs/ko/reference/configuration/providers.md b/docs-site/src/content/docs/ko/reference/configuration/providers.md index b7f21a0ecb2..1827674e5e5 100644 --- a/docs-site/src/content/docs/ko/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ko/reference/configuration/providers.md @@ -134,7 +134,7 @@ managed map을 활성화하면 privacy-safe selector를 만들고, 이후 계정 | `responsesItemIdRepair?` | `{ message?: string[]; reasoning?: string[]; repairMissingTerminalIds?: boolean; repairInvalidIds?: boolean }` | 기본값이 꺼진 downstream SSE 복구입니다. 정확한 자리표시자 id, 누락된 종료 id, 그리고(`repairInvalidIds`) 정규 `msg_`/`rs_` 접두사가 없는 message/reasoning id를 복구합니다. function-call id는 다시 쓰지 않습니다. 내장 DeepSeek은 마지막 두 가지를 기본으로 켭니다. | | `responsesSnapshotRepair?` | `boolean` | 기본값이 꺼진 클라이언트용 복구입니다. SSE와 JSON의 Responses 수명 주기에서 누락된 status, output, 도구 메타데이터를 채우며 raw 검사와 영속화는 변경하지 않습니다. | | `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | API-key 프로바이더 전용(`authMode: "key"`). 동일 대상 429 재시도: `retryOn429`가 없으면 기능이 꺼져 있고, 객체가 있으면 `enabled: false`가 아닌 한 활성화됩니다. 429 시 대기(업스트림 `Retry-After` 또는 고정 간격) 후 키 장애 조치 전에 동일 키로 동일 요청을 재전송합니다 — 일반 텍스트 턴 복구 루프, Responses passthrough, 이미지/비디오 브리지, web-search 사이드카, 터미널 연속 요청을 모두 포함합니다. 재전송 대상은 프리스트림 HTTP 429 응답뿐이며, 커스텀 `runTurn` 전송은 HTTP 재시도 루프에서 제외됩니다. `attempts`는 첫 429 이후의 동일 키 재전송 횟수(총 전송 = `attempts` + 1)이며, 메인 복구 루프·터미널 가드 연속 요청·브리지 재시도가 공유하는 요청 단위 예산입니다. `attempts`를 모두 소진해도 동일 키 재전송만 중단되며, 이후에는 일반 키 장애 조치 또는 최종 오류 처리가 사용 가능한 대상에 따라 진행됩니다 — 키 인증 passthrough 와이어에는 장애 조치가 없으므로 소진된 429가 그대로 반환됩니다. Codex 자체는 429를 재시도하지 않으므로 단일 키 프로바이더의 유일한 방어선입니다. 기본값: `enabled: true`, `attempts: 3`, `intervalMs: 5000`, `maxIntervalMs: 60000`(단일 대기는 `maxIntervalMs`로 상한, 그 자체는 600000으로 상한), `respectRetryAfter: true`. | -| `transientRetryOn5xx?` | `{ enabled?: boolean; attempts?: number }` | 키 인증 `openai-chat` 및 `openai-responses` 프로바이더 전용입니다. `authMode: "forward"` 프로바이더(ChatGPT 계정 풀)는 이 옵션을 읽지 않고 기본 재시도 단계를 유지합니다. 스트림 시작 전의 일시적인 업스트림 상태(500, 502, 503, 504, 520, 521, 522)를 선택적으로 재시도합니다. 이 옵션이 없으면 꺼져 있고, 객체가 있으면 `enabled: false`가 아닌 한 활성화됩니다. 최초 Responses 요청, 터미널 가드 연속 요청, 네이티브 `/v1/chat/completions`, 429/계정 복구 재조회를 포함합니다. `attempts`는 최초 전송을 포함하여 요청 하나에 허용되는 업스트림 전송의 총횟수(1..10, 기본값 3)입니다. 연결 재설정 복구와 요청 단위 예산 하나를 공유하므로 `3`이면 실제로 프로바이더에 도달하는 요청은 최대 세 번입니다. 대기에는 400ms로 고정된 지수 백오프를 사용하고 상한은 5초이며 `Retry-After`를 따릅니다. 속도 제한을 처리하는 `retryOn429`와는 별개이며, 스트림 도중의 실패는 절대 재전송하지 않습니다. | +| `transientRetryOn5xx?` | `{ enabled?: boolean; attempts?: number }` | 키 인증 `openai-chat` 및 `openai-responses` 프로바이더 전용입니다. `authMode: "forward"` 프로바이더(ChatGPT 계정 풀)는 이 옵션을 읽지 않고 기본 재시도 단계를 유지합니다. 스트림 시작 전의 일시적인 업스트림 상태(500, 502, 503, 504, 520, 521, 522)를 선택적으로 재시도합니다. 이 옵션이 없으면 꺼져 있고, 객체가 있으면 `enabled: false`가 아닌 한 활성화됩니다. 최초 Responses 요청, 터미널 가드 연속 요청, 네이티브 `/v1/chat/completions`, 429/계정 복구 재조회를 포함합니다. `attempts`는 최초 전송을 포함하여 요청 하나에 허용되는 업스트림 전송의 총횟수(1..10, 기본값 3)입니다. 연결 재설정 복구와 요청 단위 예산 하나를 공유하므로 `3`이면 실제로 프로바이더에 도달하는 요청은 최대 세 번입니다. 대기에는 400ms로 고정된 지수 백오프를 사용하고 상한은 5초이며 `Retry-After`를 따릅니다. 속도 제한을 처리하는 `retryOn429`와는 별개이며, 스트림 도중의 실패는 절대 재전송하지 않습니다. 각 Combo 대상은 선예약한 첫 전송을 해당 대상의 설정 총횟수에 포함합니다. `attempts: 1`은 한 번만 전송하며, 예약을 환불해 여유를 만들거나 공유 상한을 늘리지 않습니다. 전송 전 로컬 거부나 취소는 미사용 예약을 해제합니다. | | `retryOnReset?` | `{ enabled?: boolean; replacements?: number }` | 네이티브 `openai-responses` 프로바이더(`authMode: "forward"` 포함)와 업스트림 헤더 수신 전의 일반 Responses 변환 전송 경로에 적용됩니다. 호출자가 아무것도 관측하지 못한 채 실패한 전송을 선택적으로 대체합니다. 이 옵션이 없으면 꺼져 있고, 객체가 있으면 `enabled: false`가 아닌 한 활성화됩니다. 응답 헤더가 오기 전에 연결이 끊어진 경우와, 헤더 이후 SSE 본문이 제어 이벤트만 실은 채 끊어진 경우를 모두 다룹니다. canonical ChatGPT 업스트림 WebSocket에서 create 프레임을 보낸 뒤 Responses 이벤트가 오기 전에 소켓이 닫히거나 오류가 난 경우도 같은 방식으로 다루며, 이때 대체 전송은 HTTP로 보냅니다. 자체 완결된 요청만 대체합니다. `store: false`, 완전한 `input`, `previous_response_id`·`conversation`·`stream_id` 없음, 클라이언트가 실행하는 도구만 해당합니다. `replacements`는 모든 구간과 모든 콤보 자식을 합쳐 논리 요청 하나가 만들 수 있는 대체 전송 횟수입니다(1..2, 기본값 1). 구간별 재시도 횟수도 전송 예산도 아니므로, 대체 전송도 해당 구간이 이미 가진 전송 허용량 안에 들어가야 합니다. 이미 출력이나 도구 호출을 내보낸 요청은 이 값과 무관하게 대체하지 않습니다. 원본 전송이 이미 시작됐다면 대체한 추론도 과금될 수 있어서 기본값은 꺼짐입니다. 일반 변환 전송 경로에서는 최초 전송과 재구축한 전송 모두 같은 대체 권한과 전송 예산으로 헤더 수신 전 연결 리셋을 대체합니다. 어댑터가 자체 관리하는 전송 경로와 업스트림 헤더 수신 후 변환 스트림 장애는 제외됩니다. | | `autoToolChoiceOnlyModels?` | `string[]` | `tool_choice`가 `auto` 또는 `none`만 받는 모델입니다. 강제 선택은 낮은 수준으로 바뀝니다. | | `preserveReasoningContentModels?` | `string[]` | chat 기록에서 이전 assistant `reasoning_content`가 필요한 모델입니다. 대시보드에서 저장해도 저장된 목록(`[]` 포함)은 유지됩니다. `PATCH /api/providers?name=`는 배열 또는 지우기 위한 `null`을 받습니다. 어댑터, 기본 URL, 인증 모드를 바꿔 다른 목적지로 옮기는 저장에서는 유지되지 않습니다(아래 절 참고). | diff --git a/docs-site/src/content/docs/reference/configuration/providers.md b/docs-site/src/content/docs/reference/configuration/providers.md index 82594b33438..d3b1c834708 100644 --- a/docs-site/src/content/docs/reference/configuration/providers.md +++ b/docs-site/src/content/docs/reference/configuration/providers.md @@ -287,7 +287,7 @@ Providers can expose a built-in shorthand, such as `agy` for `google-antigravity | `responsesSnapshotRepair?` | `boolean` | Disabled-by-default client-facing repair for sparse Responses lifecycle snapshots in SSE and JSON. Fills missing canonical status, output, and tool metadata while raw inspection and persistence remain unchanged. | | `webSearchBridge?` | `{ enabled?: boolean; backend?: "ollama" \| "openai" \| "anthropic" \| "xai" \| "gemini" \| "exa"; maxSearches?: number; timeoutMs?: number; endpoint?: string }` | Key-auth `openai-responses` passthrough providers only. Off by default. Codex always declares the hosted `web_search` tool, and the passthrough relays it on the assumption the destination executes it. A gateway that does not run hosted search answers with a `function_call` named `web_search` that nothing runs, and the undeclared-tool guard ends the turn. With `enabled: true` and an explicit `backend` OpenCodex intercepts that call, runs the search itself, feeds the result back to the same upstream, and shows Codex a hosted `web_search_call` cell. Never armed for `authMode: "forward"` (ChatGPT already searches) or for a provider that executes hosted search upstream. `backend` is required; there is no implicit default and a missing credential for the named backend leaves the bridge disarmed rather than falling through to another paid search. `ollama` reuses this provider's own API key on `POST /api/web_search`, so the origin must be `https://ollama.com` unless the operator names `endpoint` explicitly. `openai` / `anthropic` / `xai` / `gemini` / `exa` reuse the matching sidecar executor and that executor's own credential (`webSearchSidecar.exaApiKey` for Exa). The search model comes from `webSearchSidecar.model` only when `webSearchSidecar.backend` resolves to the same backend this bridge names; otherwise the bridge runs that backend's own default, because a model chosen for one vendor is rejected by another. An unset `webSearchSidecar.backend` resolves to `openai`, so an unset-backend model reaches an `openai` bridge and no other. There is no per-provider bridge model override. Streaming turns only. A turn that mixes `web_search` with another client tool call still fails closed rather than dropping the client's call. Assistant text such as XML-like `` prose is not executed. Defaults: `maxSearches: 3` (1..10), `timeoutMs: 60000` (1000..600000). | | `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | API-key providers only (`authMode: "key"`). Opt-in same-target 429 retry: when `retryOn429` is absent the feature is off, except that key-auth OpenCode Go and Command Code (canonical endpoints) fall back to a patient policy (6 replays, 10 s interval, 60 s cap); object presence enables it unless `enabled: false`, which also turns that fallback off. On 429 the proxy waits (upstream `Retry-After` or the fixed interval) and replays the identical request on the same key before any key failover — across the main text-turn recovery loop, the Responses passthrough wire, the image/video bridge, the web-search sidecar, and terminal continuations. Only pre-stream HTTP 429 responses are eligible for replay; custom `runTurn` transports are outside the HTTP retry loop. `attempts` counts same-key replays after the first 429 (total sends = `attempts` + 1) and is one request-wide budget shared by the main recovery loop, the terminal-guard continuation, and bridge retries. Exhausting `attempts` only stops further same-key replays: normal key failover or final-error handling then applies per the available targets — on the key-auth passthrough wire there is no failover, so the exhausted 429 surfaces as-is. Codex itself never retries 429, so this is the only defense for single-key providers. Defaults: `enabled: true`, `attempts: 3`, `intervalMs: 5000`, `maxIntervalMs: 60000` (any single wait is capped at `maxIntervalMs`, itself capped at 600000), `respectRetryAfter: true`. | -| `transientRetryOn5xx?` | `{ enabled?: boolean; attempts?: number }` | Key-auth `openai-chat` and `openai-responses` providers only. `authMode: "forward"` providers (the ChatGPT account pool) never read this option and keep the default ladder. Opt-in retry for pre-stream transient upstream statuses (500, 502, 503, 504, 520, 521, 522): absent means off, object presence enables it unless `enabled: false`. Covers the initial Responses request, the Responses passthrough lane and each of its recovery legs (OAuth-401 replay, same-target 429 replay, validated rebuild), the terminal-guard continuation, and native `/v1/chat/completions`. `attempts` is the TOTAL number of upstream sends allowed for one request including the first (1..10, default 3) — it is one budget shared with connection-reset recovery, so `3` means at most three real requests reach the provider. On the Responses passthrough lane the configured value is additionally intersected with the request-wide send allowance, so a value below that allowance narrows the ladder exactly while a value above it does not raise the bound. Waits use a fixed 400 ms exponential backoff capped at 5 s and honor `Retry-After`. Separate from `retryOn429`, which handles rate limiting; mid-stream failures are never replayed. | +| `transientRetryOn5xx?` | `{ enabled?: boolean; attempts?: number }` | Key-auth `openai-chat` and `openai-responses` providers only. `authMode: "forward"` providers (the ChatGPT account pool) never read this option and keep the default ladder. Opt-in retry for pre-stream transient upstream statuses (500, 502, 503, 504, 520, 521, 522): absent means off, object presence enables it unless `enabled: false`. Covers the initial Responses request, the Responses passthrough lane and each of its recovery legs (OAuth-401 replay, same-target 429 replay, validated rebuild), the terminal-guard continuation, and native `/v1/chat/completions`. `attempts` is the TOTAL number of upstream sends allowed for one request including the first (1..10, default 3) — it is one budget shared with connection-reset recovery, so `3` means at most three real requests reach the provider. On the Responses passthrough lane the configured value is additionally intersected with the request-wide send allowance, so a value below that allowance narrows the ladder exactly while a value above it does not raise the bound. Each Combo target includes its prepaid first send in that target's configured total; `attempts: 1` sends once, without refunding the reservation or widening the shared Combo ceiling. A local refusal or cancellation before dispatch releases an unsent booking. Waits use a fixed 400 ms exponential backoff capped at 5 s and honor `Retry-After`. Separate from `retryOn429`, which handles rate limiting; mid-stream failures are never replayed. | | `retryOnReset?` | `{ enabled?: boolean; replacements?: number }` | Native `openai-responses` providers, including `authMode: "forward"`, plus generic translated Responses dispatch before upstream headers. Opt-in replacement of a send that failed while the caller had observed nothing: absent means off, object presence enables it unless `enabled: false`. Covers both ambiguous stages — a connection that died before any response header, and an SSE body that died after the header while carrying only control events. A canonical ChatGPT upstream WebSocket that closed or errored after its create frame left, before any Responses event, is covered the same way, and its replacement is sent over HTTP. Only a self-contained request is ever replaced: `store: false`, complete `input`, no `previous_response_id`, `conversation` or `stream_id`, and only client-executed tools. `replacements` is the number of replacement sends ONE logical request may make across every leg and every combo child (1..2, default 1) — not a per-leg retry count and not a send budget, so a replacement still has to fit inside the send allowance the leg already had. A request that already emitted output or a tool call is never replaced, whatever this is set to. The replacement inference may still be billed if the origin had already started the first one, which is why this is off by default. Generic translated dispatch supports pre-header reset replacement for initial and rebuilt sends under the same grant and send budget. Adapter-owned transports and translated stream failures after upstream headers are excluded. | | `autoToolChoiceOnlyModels?` | `string[]` | Models whose `tool_choice` accepts only `auto` or `none`; forced choices are downgraded. | | `preserveReasoningContentModels?` | `string[]` | Models requiring prior assistant `reasoning_content` in chat history. A provider save that keeps the destination keeps the stored list, including `[]`; see [What a provider save keeps](#what-a-provider-save-keeps). `PATCH /api/providers?name=` accepts an array or `null` to clear it. | diff --git a/docs-site/src/content/docs/ru/reference/configuration/providers.md b/docs-site/src/content/docs/ru/reference/configuration/providers.md index 83501986f90..0b1551ab73e 100644 --- a/docs-site/src/content/docs/ru/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ru/reference/configuration/providers.md @@ -147,7 +147,7 @@ cross-route credential fallback не существует. Строки API GPT- | `responsesItemIdRepair?` | `{ message?: string[]; reasoning?: string[]; repairMissingTerminalIds?: boolean; repairInvalidIds?: boolean }` | По умолчанию выключенная downstream SSE-repair для exact placeholder-id, отсутствующих terminal-id и (с `repairInvalidIds`) message/reasoning id без канонического префикса `msg_`/`rs_`. Function-call id никогда не переписываются. Встроенный DeepSeek включает последние два по умолчанию. | | `responsesSnapshotRepair?` | `boolean` | По умолчанию выключенная клиентская repair для неполных lifecycle snapshot'ов Responses в SSE и JSON. Добавляет отсутствующие status, output и tool metadata, не меняя raw inspection и persistence. | | `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | Только для провайдеров с API-ключом (`authMode: "key"`). Опциональный повтор при 429 на том же таргете: если `retryOn429` отсутствует, функция выключена; наличие объекта включает её, если только `enabled: false`. При 429: ожидание (`Retry-After` апстрима или фиксированный интервал) и повтор идентичного запроса на том же ключе до любого фейловера ключей — покрывает основной цикл восстановления текстовых ходов, passthrough-канал Responses, мост изображений/видео, sidecar web-search и терминальные продолжения. Повтор допустим только для HTTP 429, полученных до начала потока; пользовательские транспорты `runTurn` не входят в цикл HTTP-повторов. `attempts` — это число повторов на том же ключе после первого 429 (всего отправок = `attempts` + 1) и единый бюджет на запрос, общий для основного цикла восстановления, терминального продолжения и повторов моста. Исчерпание `attempts` лишь останавливает дальнейшие повторы на том же ключе; далее применяется обычный фейловер ключей или финальная обработка ошибки в зависимости от доступных таргетов — на passthrough-канале с ключевой аутентификацией фейловера нет, поэтому исчерпанный 429 возвращается как есть. Codex сам никогда не повторяет 429, поэтому это единственная защита для провайдеров с одним ключом. По умолчанию: `enabled: true`, `attempts: 3`, `intervalMs: 5000`, `maxIntervalMs: 60000` (любое ожидание ограничено `maxIntervalMs`, который сам ограничен 600000), `respectRetryAfter: true`. | -| `transientRetryOn5xx?` | `{ enabled?: boolean; attempts?: number }` | Только для провайдеров `openai-chat` и `openai-responses` с аутентификацией по ключу. Провайдеры с `authMode: "forward"` (пул аккаунтов ChatGPT) никогда не читают эту настройку и сохраняют число повторов по умолчанию. Опциональный повтор при временных статусах апстрима до начала потока (500, 502, 503, 504, 520, 521, 522): если параметр отсутствует, функция выключена; наличие объекта включает её, если только `enabled: false`. Покрывает исходный запрос `Responses`, продолжение терминального предохранителя, нативный `/v1/chat/completions`, а также повторные запросы при восстановлении после 429 или ошибки учётной записи. `attempts` — ОБЩЕЕ число разрешённых отправок в апстрим для одного запроса, включая первую (1..10, по умолчанию 3). Это единый бюджет на запрос, общий с восстановлением после сброса соединения, поэтому `3` означает, что до провайдера дойдут не более трёх реальных запросов. Ожидание использует экспоненциальную задержку с фиксированной начальной величиной 400 мс, ограниченную 5 с, и учитывает `Retry-After`. Параметр не связан с `retryOn429`, который обрабатывает ограничение частоты запросов; сбои после начала потока никогда не воспроизводятся. | +| `transientRetryOn5xx?` | `{ enabled?: boolean; attempts?: number }` | Только для провайдеров `openai-chat` и `openai-responses` с аутентификацией по ключу. Провайдеры с `authMode: "forward"` (пул аккаунтов ChatGPT) никогда не читают эту настройку и сохраняют число повторов по умолчанию. Опциональный повтор при временных статусах апстрима до начала потока (500, 502, 503, 504, 520, 521, 522): если параметр отсутствует, функция выключена; наличие объекта включает её, если только `enabled: false`. Покрывает исходный запрос `Responses`, продолжение терминального предохранителя, нативный `/v1/chat/completions`, а также повторные запросы при восстановлении после 429 или ошибки учётной записи. `attempts` — ОБЩЕЕ число разрешённых отправок в апстрим для одного запроса, включая первую (1..10, по умолчанию 3). Это единый бюджет на запрос, общий с восстановлением после сброса соединения, поэтому `3` означает, что до провайдера дойдут не более трёх реальных запросов. Ожидание использует экспоненциальную задержку с фиксированной начальной величиной 400 мс, ограниченную 5 с, и учитывает `Retry-After`. Параметр не связан с `retryOn429`, который обрабатывает ограничение частоты запросов; сбои после начала потока никогда не воспроизводятся. Для каждой цели Combo первая заранее зарезервированная отправка входит в настроенный итог этой цели: `attempts: 1` допускает одну отправку, без возврата резерва ради дополнительного лимита и без расширения общего потолка. Локальный отказ или отмена до отправки освобождает неиспользованный резерв. | | `retryOnReset?` | `{ enabled?: boolean; replacements?: number }` | Для нативных провайдеров `openai-responses`, включая `authMode: "forward"`, а также для общего пути отправки переведённых запросов Responses до получения заголовков от upstream. Необязательная замена отправки, которая завершилась неудачей, когда вызывающая сторона ещё ничего не наблюдала: если параметр отсутствует, функция выключена; наличие объекта включает её, если только `enabled: false`. Покрывает обе неоднозначные стадии — обрыв соединения до любого заголовка ответа и обрыв тела SSE после заголовка, когда оно несло только управляющие события. Так же покрывается upstream WebSocket canonical ChatGPT, который закрылся или завершился ошибкой после отправки кадра create и до любого события Responses; замена в этом случае отправляется по HTTP. Заменяется только самодостаточный запрос: `store: false`, полный `input`, отсутствие `previous_response_id`, `conversation` и `stream_id`, и только инструменты, исполняемые клиентом. `replacements` — число замещающих отправок, которые ОДИН логический запрос может сделать по всем участкам и всем дочерним запросам комбо (1..2, по умолчанию 1). Это не число повторов на участок и не бюджет отправок, поэтому замена всё равно должна поместиться в уже имеющийся у участка лимит отправок. Запрос, который уже выдал вывод или вызов инструмента, не заменяется никогда, каким бы ни было это значение. Замещающий вывод модели всё равно может быть оплачен, если источник уже начал первый, поэтому параметр выключен по умолчанию. Общий путь перевода поддерживает замену при сбросе соединения до заголовков для первоначальных и заново сформированных отправок, используя то же разрешение и бюджет отправок. Транспорты, управляемые самими адаптерами, и сбои переведённого потока после получения заголовков от upstream исключены. | | `autoToolChoiceOnlyModels?` | `string[]` | Модели, у которых `tool_choice` принимает только `auto` или `none`; forced choice понижается. | | `preserveReasoningContentModels?` | `string[]` | Модели, которым нужен предыдущий assistant `reasoning_content` в chat history. Сохранение в дашборде не меняет записанный список, в том числе `[]`. `PATCH /api/providers?name=` принимает массив или `null`, чтобы его очистить. Сохранение, которое переносит провайдера на другой адаптер, базовый URL или режим аутентификации, список не сохраняет (см. раздел ниже). | diff --git a/docs-site/src/content/docs/tr/reference/configuration/providers.md b/docs-site/src/content/docs/tr/reference/configuration/providers.md index 156cd41eac5..8343a9433e9 100644 --- a/docs-site/src/content/docs/tr/reference/configuration/providers.md +++ b/docs-site/src/content/docs/tr/reference/configuration/providers.md @@ -148,7 +148,7 @@ alanlı seçilmiş kimlikleri yalın kimliklere yeniden yazar. | `responsesItemIdRepair?` | `{ message?: string[]; reasoning?: string[]; repairMissingTerminalIds?: boolean; repairInvalidIds?: boolean }` | Tam yer tutucu kimlikleri, eksik terminal kimlikleri ve (`repairInvalidIds` ile) kurallı `msg_`/`rs_` öneki eksik olan mesaj/akıl yürütme kimlikleri için varsayılan olarak devre dışı bırakılmış aşağı akış SSE onarımı. Fonksiyon çağrısı kimlikleri asla yeniden yazılmaz. Yerleşik DeepSeek son ikisini varsayılan olarak etkinleştirir. | | `responsesSnapshotRepair?` | `boolean` | SSE ve JSON'daki seyrek Responses yaşam döngüsü anlık görüntüleri için varsayılan olarak devre dışı bırakılmış istemciye yönelik onarım. Ham inceleme ve kalıcılık değişmeden kalırken eksik kurallı durumu, çıktıyı ve araç meta verilerini doldurur. | | `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | Yalnızca API anahtarı sağlayıcıları (`authMode: "key"`). İsteğe bağlı aynı hedef 429 yeniden denemesi: `retryOn429` olmadığında özellik kapalıdır; nesnenin varlığı `enabled: false` olmadığı sürece özelliği etkinleştirir. 429'da proxy bekler (yukarı akış `Retry-After` veya sabit aralık) ve herhangi bir anahtar yük devretmesinden önce aynı istek üzerinde aynı anahtarla aynı isteği yeniden oynatır — ana metin turu kurtarma döngüsü, Responses doğrudan geçiş hattı, görsel/video köprüsü, web araması sidecar'ı ve terminal devamları genelinde. Yalnızca akış öncesi HTTP 429 yanıtları yeniden oynatma için uygundur; özel `runTurn` aktarımları HTTP yeniden deneme döngüsünün dışındadır. `attempts`, ilk 429'dan sonraki aynı anahtar yeniden oynatmalarını sayar (toplam gönderim = `attempts` + 1) ve ana kurtarma döngüsü, terminal koruma devamı ve köprü yeniden denemeleri tarafından paylaşılan tek bir istek genelinde bütçedir. `attempts`'ı tüketmek yalnızca daha fazla aynı anahtar yeniden oynatmasını durdurur: normal anahtar yük devretmesi veya nihai hata işleme daha sonra kullanılabilir hedeflere göre geçerli olur — anahtar kimlik doğrulamalı doğrudan geçiş hattında yük devretme yoktur, bu nedenle tükenen 429 olduğu gibi görünür. Codex'in kendisi 429'u asla yeniden denemez, bu nedenle tek anahtarlı sağlayıcılar için tek savunma budur. Varsayılanlar: `enabled: true`, `attempts: 3`, `intervalMs: 5000`, `maxIntervalMs: 60000` (tek bir bekleme `maxIntervalMs` ile sınırlandırılır, kendisi de 600000 ile sınırlandırılır), `respectRetryAfter: true`. | -| `transientRetryOn5xx?` | `{ enabled?: boolean; attempts?: number }` | Yalnızca anahtarla kimlik doğrulanan `openai-chat` ve `openai-responses` sağlayıcıları. `authMode: "forward"` sağlayıcıları (ChatGPT hesap havuzu) bu seçeneği hiç okumaz ve varsayılan merdiveni korur. Akış öncesi geçici yukarı akış durumları (500, 502, 503, 504, 520, 521, 522) için isteğe bağlı yeniden deneme: seçenek belirtilmezse kapalıdır; nesnenin varlığı, `enabled: false` olmadığı sürece özelliği etkinleştirir. İlk Responses isteğini, terminal koruma devamını, yerel `/v1/chat/completions` isteklerini ve 429/hesap kurtarma yeniden getirmelerini kapsar. `attempts`, bir istek için ilk gönderim dahil izin verilen yukarı akış gönderimlerinin TOPLAM sayısıdır (1..10, varsayılan 3) — bağlantı sıfırlama kurtarmasıyla paylaşılan, istek kapsamlı tek bütçedir; dolayısıyla `3`, sağlayıcıya en fazla üç gerçek isteğin ulaşması anlamına gelir. Beklemelerde 400 ms'lik sabit üstel geri çekilme uygulanır, süre 5 sn ile sınırlandırılır ve `Retry-After` dikkate alınır. Hız sınırlamasını işleyen `retryOn429` seçeneğinden ayrıdır; akış ortası hataları hiçbir zaman yeniden oynatılmaz. | +| `transientRetryOn5xx?` | `{ enabled?: boolean; attempts?: number }` | Yalnızca anahtarla kimlik doğrulanan `openai-chat` ve `openai-responses` sağlayıcıları. `authMode: "forward"` sağlayıcıları (ChatGPT hesap havuzu) bu seçeneği hiç okumaz ve varsayılan merdiveni korur. Akış öncesi geçici yukarı akış durumları (500, 502, 503, 504, 520, 521, 522) için isteğe bağlı yeniden deneme: seçenek belirtilmezse kapalıdır; nesnenin varlığı, `enabled: false` olmadığı sürece özelliği etkinleştirir. İlk Responses isteğini, terminal koruma devamını, yerel `/v1/chat/completions` isteklerini ve 429/hesap kurtarma yeniden getirmelerini kapsar. `attempts`, bir istek için ilk gönderim dahil izin verilen yukarı akış gönderimlerinin TOPLAM sayısıdır (1..10, varsayılan 3) — bağlantı sıfırlama kurtarmasıyla paylaşılan, istek kapsamlı tek bütçedir; dolayısıyla `3`, sağlayıcıya en fazla üç gerçek isteğin ulaşması anlamına gelir. Beklemelerde 400 ms'lik sabit üstel geri çekilme uygulanır, süre 5 sn ile sınırlandırılır ve `Retry-After` dikkate alınır. Hız sınırlamasını işleyen `retryOn429` seçeneğinden ayrıdır; akış ortası hataları hiçbir zaman yeniden oynatılmaz. Her Combo hedefinde önceden ayrılmış ilk gönderim, o hedefin yapılandırılmış toplamına dahildir: `attempts: 1` tek gönderim yapar; rezervasyonu geri vererek ek hak yaratmaz ve ortak tavanı artırmaz. Gönderim öncesi yerel ret veya iptal, kullanılmamış rezervasyonu serbest bırakır. | | `retryOnReset?` | `{ enabled?: boolean; replacements?: number }` | Yerel `openai-responses` sağlayıcıları (`authMode: "forward"` dahil) ve upstream başlıkları gelmeden önceki genel Responses çeviri gönderim yolu için geçerlidir. Çağıranın hiçbir şey gözlemlemediği bir anda başarısız olan gönderimin isteğe bağlı olarak değiştirilmesi: seçenek belirtilmezse kapalıdır; nesnenin varlığı, `enabled: false` olmadığı sürece özelliği etkinleştirir. İki belirsiz aşamayı da kapsar: yanıt başlığı gelmeden kopan bağlantı ve başlıktan sonra yalnızca denetim olayları taşırken kopan SSE gövdesi. Kanonik ChatGPT upstream WebSocket'i create çerçevesi gönderildikten sonra, herhangi bir Responses olayından önce kapanır veya hata verirse aynı şekilde kapsanır ve değiştirme gönderimi HTTP üzerinden yapılır. Yalnızca kendi kendine yeten bir istek değiştirilir: `store: false`, eksiksiz `input`, `previous_response_id`, `conversation` veya `stream_id` bulunmaması ve yalnızca istemcinin yürüttüğü araçlar. `replacements`, BİR mantıksal isteğin tüm bacaklar ve tüm combo alt istekleri boyunca yapabileceği değiştirme gönderimi sayısıdır (1..2, varsayılan 1). Bacak başına yeniden deneme sayısı da gönderim bütçesi de değildir; bu yüzden bir değiştirme gönderimi, ilgili bacağın hâlihazırda sahip olduğu gönderim payına sığmak zorundadır. Halihazırda çıktı veya araç çağrısı üretmiş bir istek, bu değer ne olursa olsun asla değiştirilmez. Kaynak ilk çıkarımı zaten başlatmışsa değiştirilen çıkarım yine ücretlendirilebilir; bu nedenle varsayılan olarak kapalıdır. Genel çeviri gönderim yolu, ilk ve yeniden oluşturulan gönderimlerde başlık öncesi bağlantı sıfırlamasından sonra aynı izin ve gönderim bütçesiyle değiştirme destekler. Adaptörlerin kendi yönettiği taşıma yolları ve upstream başlıklarından sonraki çevrilmiş akış hataları kapsam dışıdır. | | `autoToolChoiceOnlyModels?` | `string[]` | `tool_choice`'u yalnızca `auto` veya `none` kabul eden modeller; zorunlu seçimlerin derecesi düşürülür. | | `preserveReasoningContentModels?` | `string[]` | Sohbet geçmişinde önceki asistan `reasoning_content`'ini gerektiren modeller. Dashboard üzerinden yapılan kayıtlar, `[]` dahil saklanan listeyi korur. `PATCH /api/providers?name=`, bir dizi veya temizlemek için `null` kabul eder. Sağlayıcıyı başka bir bağdaştırıcıya, temel URL'ye veya kimlik doğrulama moduna taşıyan bir kayıt listeyi korumaz (aşağıdaki bölüme bakın). | diff --git a/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md b/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md index 9dfedddeb7c..c653d5d158e 100644 --- a/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md +++ b/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md @@ -135,7 +135,7 @@ selector，而不是分配一个新名称。 | `responsesItemIdRepair?` | `{ message?: string[]; reasoning?: string[]; repairMissingTerminalIds?: boolean; repairInvalidIds?: boolean }` | 默认关闭的下游 SSE 修复，用于精确占位 id、缺失的终止 id，以及（`repairInvalidIds`）缺少规范 `msg_`/`rs_` 前缀的 message/reasoning id。function-call id 永远不会被重写。内置 DeepSeek 默认启用后两项。 | | `responsesSnapshotRepair?` | `boolean` | 默认关闭的客户端修复，用于补全 SSE 与 JSON 中稀疏 Responses 生命周期快照缺失的 status、output 和工具元数据；原始检查与持久化保持不变。 | | `retryOn429?` | `{ enabled?: boolean; attempts?: number; intervalMs?: number; maxIntervalMs?: number; respectRetryAfter?: boolean }` | 仅限 API-key 提供商（`authMode: "key"`）。可选的同目标 429 重试：未配置 `retryOn429` 时功能关闭；对象存在即启用，除非 `enabled: false`。收到 429 时等待（上游 `Retry-After` 或固定间隔）后在相同 key 上重放完全相同请求，再进入任何 key 故障转移——覆盖主文本恢复循环、Responses passthrough、图像/视频桥、web-search 侧车与终结续接。重放仅适用于流开始前的 HTTP 429 响应；自定义 `runTurn` 传输不在 HTTP 重试循环范围内。`attempts` 是首个 429 之后的同 key 重放次数（总发送次数 = `attempts` + 1），是主恢复循环、终结守卫续接与桥接重试共享的按请求统一预算；`attempts` 耗尽只会停止进一步的同 key 重放：随后按可用目标进行正常的 key 故障转移或最终错误处理——key 认证的 passthrough 线路上没有故障转移，因此耗尽的 429 会原样透出。Codex 自身从不重试 429，因此这是单 key 提供商唯一的防线。默认值：`enabled: true`、`attempts: 3`、`intervalMs: 5000`、`maxIntervalMs: 60000`（单次等待以 `maxIntervalMs` 为上限，其本身上限 600000）、`respectRetryAfter: true`。 | -| `transientRetryOn5xx?` | `{ enabled?: boolean; attempts?: number }` | 仅限使用 key 认证的 `openai-chat` 与 `openai-responses` 提供商。`authMode: "forward"` 的提供商（ChatGPT 账号池）从不读取此选项，保持默认重试次数。可选的流开始前上游瞬态状态码（500、502、503、504、520、521、522）重试：未配置时关闭；对象存在即启用，除非 `enabled: false`。覆盖初始 Responses 请求、终结守卫续接、原生 `/v1/chat/completions`，以及 429/账户恢复重新获取。`attempts` 是单个请求允许向上游发送的总次数，包含首次发送（1..10，默认 3）；它是与连接重置恢复共享的按请求预算，因此 `3` 表示最多只有三个实际请求到达提供商。等待采用固定 400 毫秒的指数退避，上限为 5 秒，并遵循 `Retry-After`。此选项独立于处理速率限制的 `retryOn429`；流开始后的故障绝不会重放。 | +| `transientRetryOn5xx?` | `{ enabled?: boolean; attempts?: number }` | 仅限使用 key 认证的 `openai-chat` 与 `openai-responses` 提供商。`authMode: "forward"` 的提供商（ChatGPT 账号池）从不读取此选项，保持默认重试次数。可选的流开始前上游瞬态状态码（500、502、503、504、520、521、522）重试：未配置时关闭；对象存在即启用，除非 `enabled: false`。覆盖初始 Responses 请求、终结守卫续接、原生 `/v1/chat/completions`，以及 429/账户恢复重新获取。`attempts` 是单个请求允许向上游发送的总次数，包含首次发送（1..10，默认 3）；它是与连接重置恢复共享的按请求预算，因此 `3` 表示最多只有三个实际请求到达提供商。等待采用固定 400 毫秒的指数退避，上限为 5 秒，并遵循 `Retry-After`。此选项独立于处理速率限制的 `retryOn429`；流开始后的故障绝不会重放。 每个 Combo 目标都把预先预约的首次发送计入该目标配置的总次数；`attempts: 1` 只发送一次，不会退还预约来增加额度，也不会提高共享上限。发送前的本地拒绝或取消会释放未使用的预约。 | | `retryOnReset?` | `{ enabled?: boolean; replacements?: number }` | 适用于原生 `openai-responses` 提供商（包含 `authMode: "forward"`），以及收到上游响应头之前的通用 Responses 转换发送路径。可选地替换一次在调用方尚未观察到任何内容时就失败的发送：未配置时关闭；对象存在即启用，除非 `enabled: false`。涵盖两个不确定阶段——响应头到达前连接断开，以及响应头之后 SSE 正文只承载控制事件时断开。canonical ChatGPT 上游 WebSocket 在 create 帧发出之后、任何 Responses 事件到达之前关闭或出错时，也按同样方式处理，其替换发送走 HTTP。只有自包含的请求才会被替换：`store: false`、完整的 `input`、没有 `previous_response_id`／`conversation`／`stream_id`，且只使用由客户端执行的工具。`replacements` 是单个逻辑请求在所有环节和所有组合子请求中可以进行的替换发送次数（1..2，默认 1）；它既不是按环节的重试次数，也不是发送预算，因此替换发送仍必须落在该环节已有的发送额度之内。已经产生输出或工具调用的请求，无论此值为何都不会被替换。如果上游已经开始了第一次推理，被替换的推理仍可能计费，因此该选项默认关闭。 通用转换发送路径的首次发送和重建后的发送都支持响应头之前连接重置后的替换，使用同一授权和发送预算。适配器自行管理的传输，以及收到上游响应头之后的转换流故障，均不在此选项范围内。 | | `autoToolChoiceOnlyModels?` | `string[]` | `tool_choice` 只接受 `auto` 或 `none` 的模型；强制选择会被降级。 | | `preserveReasoningContentModels?` | `string[]` | 需要在聊天历史中保留先前 assistant `reasoning_content` 的模型。从仪表板保存时会保留已存储的列表（包括 `[]`）。`PATCH /api/providers?name=` 接受数组，或传入 `null` 清除该字段。将提供方改到其他适配器、base URL 或认证模式的保存不会保留该列表（见下文）。 | diff --git a/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md b/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md index 29f1287ebc7..ef32fed2ed1 100644 --- a/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md +++ b/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md @@ -106,7 +106,7 @@ ocx models provider openrouter on | `foldDeveloperRoleToSystem?` | `boolean` | 記錄某個 `openai-chat` 目的地是否接受 `developer` 角色。`foldDeveloperRoleToSystem` 未設定時以 `system` 傳送，`true` 時以 `system` 傳送，`false` 時以 `developer` 傳送。未設定表示尚未記錄該目的地的情況；`true` 記錄上游拒絕該角色；`false` 記錄其接受該角色。無論何者，訊息都保留在對話中的原有位置，只有角色改變。拒絕該角色的目的地會回應 `400 role 'developer' is not allowed`，該回合根本無法開始，這就是未記錄狀態預設摺疊的原因。 | | `parallelToolCalls?` | `boolean` | 切換平行工具呼叫。OpenAI Chat 預設開啟；非 chat adapter 僅在明確 `true` 時廣告。 | | `responsesItemIdRepair?` | `{ message?: string[]; reasoning?: string[]; repairMissingTerminalIds?: boolean }` | 預設停用的下游 SSE 修復，用於精確佔位 id 與缺失的終端 id。Function-call id 永不被重寫。 | -| `transientRetryOn5xx?` | `{ enabled?: boolean; attempts?: number }` | 僅限使用金鑰認證的 `openai-chat` 與 `openai-responses` 供應商。`authMode: "forward"` 的供應商（ChatGPT 帳號池）從不讀取此選項，維持預設重試次數。選擇性重試串流開始前的暫時性上游狀態（500、502、503、504、520、521、522）：未設定時停用；只要有此物件即啟用，除非 `enabled: false`。涵蓋初始 `Responses` 請求、終止防護續接、原生 `/v1/chat/completions`，以及 429／帳號復原的重新擷取。`attempts` 是單一請求允許傳送至上游的總次數，包含第一次（1..10，預設 3）；這是與連線重設復原共用的單一請求範圍預算，因此 `3` 表示最多只有三個實際請求會送達供應商。等待採固定 400 毫秒、上限 5 秒的指數退避，並遵循 `Retry-After`。此機制獨立於處理速率限制的 `retryOn429`；串流中的失敗絕不重播。 | +| `transientRetryOn5xx?` | `{ enabled?: boolean; attempts?: number }` | 僅限使用金鑰認證的 `openai-chat` 與 `openai-responses` 供應商。`authMode: "forward"` 的供應商（ChatGPT 帳號池）從不讀取此選項，維持預設重試次數。選擇性重試串流開始前的暫時性上游狀態（500、502、503、504、520、521、522）：未設定時停用；只要有此物件即啟用，除非 `enabled: false`。涵蓋初始 `Responses` 請求、終止防護續接、原生 `/v1/chat/completions`，以及 429／帳號復原的重新擷取。`attempts` 是單一請求允許傳送至上游的總次數，包含第一次（1..10，預設 3）；這是與連線重設復原共用的單一請求範圍預算，因此 `3` 表示最多只有三個實際請求會送達供應商。等待採固定 400 毫秒、上限 5 秒的指數退避，並遵循 `Retry-After`。此機制獨立於處理速率限制的 `retryOn429`；串流中的失敗絕不重播。 每個 Combo 目標都把預先預約的首次傳送計入該目標設定的總次數；`attempts: 1` 只傳送一次，不會退還預約來增加額度，也不會提高共用上限。傳送前的本機拒絕或取消會釋放未使用的預約。 | | `retryOnReset?` | `{ enabled?: boolean; replacements?: number }` | 適用於原生 `openai-responses` 供應商（包含 `authMode: "forward"`），以及收到上游回應標頭之前的通用 Responses 轉換傳送路徑。可選擇性地替換一次在呼叫端尚未觀察到任何內容時就失敗的傳送：未設定時停用；只要有此物件即啟用，除非 `enabled: false`。涵蓋兩個不確定階段——回應標頭抵達前連線中斷，以及標頭之後 SSE 內文只載有控制事件時中斷。canonical ChatGPT upstream WebSocket 在 create 訊框送出之後、任何 Responses 事件抵達之前關閉或出錯時，也以相同方式處理，其替換傳送改走 HTTP。只有自我完備的請求才會被替換：`store: false`、完整的 `input`、沒有 `previous_response_id`／`conversation`／`stream_id`，且僅使用由用戶端執行的工具。`replacements` 是單一邏輯請求在所有環節與所有組合子請求中可進行的替換傳送次數（1..2，預設 1）；它既不是各環節的重試次數，也不是傳送預算，因此替換傳送仍必須落在該環節既有的傳送額度之內。已經產生輸出或工具呼叫的請求，無論此值為何都不會被替換。若上游已經開始第一次推論，被替換的推論仍可能計費，因此此選項預設停用。 通用轉換傳送路徑的首次傳送和重建後的傳送都支援標頭之前連線重設後的替換，使用同一授權和傳送預算。配接器自行管理的傳輸，以及收到上游回應標頭之後的轉換串流故障，均不在此選項範圍內。 | | `autoToolChoiceOnlyModels?` | `string[]` | 其 `tool_choice` 僅接受 `auto` 或 `none` 的模型；強制選擇被降級。 | | `preserveReasoningContentModels?` | `string[]` | 需要在 chat 歷史中保留先前 assistant `reasoning_content` 的模型。從儀表板儲存時會保留已儲存的清單（包括 `[]`）。`PATCH /api/providers?name=` 接受陣列，或傳入 `null` 清除該欄位。將供應商改到其他轉接器、base URL 或驗證模式的儲存不會保留該清單（見下文）。 | diff --git a/scripts/test-layout/layout.json b/scripts/test-layout/layout.json index 5f226f55b41..3ec9103292c 100644 --- a/scripts/test-layout/layout.json +++ b/scripts/test-layout/layout.json @@ -310,6 +310,7 @@ "protocol-direct-encoders-messages.test.ts": "responses", "chat-native-combo.test.ts": "responses", "messages-native-eligibility.test.ts": "responses", + "responses-first-send-reservation.test.ts": "responses", "responses-first-send-spend-policy.test.ts": "responses", "responses-dispatch-receipt.test.ts": "responses", "responses-hosted-send-hop.test.ts": "responses", "responses-hosted-send-producer.test.ts": "responses", "responses-hosted-send-reservation.test.ts": "responses", "responses-hosted-send-round-parity.test.ts": "responses", "responses-hosted-send-unsent-hop.test.ts": "responses", "responses-hosted-send-ws.test.ts": "responses", "messages-native-bridge-policy.test.ts": "responses", "messages-native-oauth-eligibility.test.ts": "responses", "chat-inbound-reasoning-none.test.ts": "responses", diff --git a/src/images/loop.ts b/src/images/loop.ts index 9da3e4ca2cf..33df568ebaa 100644 --- a/src/images/loop.ts +++ b/src/images/loop.ts @@ -300,6 +300,11 @@ export interface ImageBridgeDeps { onRequestBuilt?: (request: AdapterRequest) => void; abortSignal?: AbortSignal; onFirstOutput?: () => void; + /** Transfer prepaid cleanup before asynchronous inference can outlive Response construction. */ + onProducerStart?: () => void; + onProducerEnd?: () => void; + /** Release a prepaid recovery that did not dispatch during this fetch iteration. */ + onIterationEnd?: () => void; /** Max image-generation rounds before forcing a final answer. Defaults to 3; clamped to [0, 10]. */ maxRounds?: number; /** Connect / response-header budget for non-runTurn iterations. */ @@ -309,7 +314,7 @@ export interface ImageBridgeDeps { /** Provider-specific fetch (e.g. xAI transport wrapper). Falls back to global fetch. */ fetchImpl?: typeof globalThis.fetch; /** Bind physical dispatch to this iteration's built request; pacing remains owned by the loop. */ - fetchForRequest?: (request: AdapterRequest, parsed: OcxParsedRequest) => typeof globalThis.fetch; + fetchForRequest?: (request: AdapterRequest, parsed: OcxParsedRequest, adapter: ProviderAdapter) => typeof globalThis.fetch; /** Reserve the routed provider's next request-start slot before each adapter dispatch. */ waitForRequestSlot?: (signal?: AbortSignal) => Promise; /** Raw adapter usage at the terminal event, pre wire-normalization (see bridgeToResponsesSSE onUsage). */ @@ -372,6 +377,7 @@ export async function runWithImageBridge(deps: ImageBridgeDeps): Promise { if (!a) return b; @@ -514,19 +520,25 @@ export async function runWithImageBridge(deps: ImageBridgeDeps): Promise { + activeProducers++; + let producer: Promise; + try { + producer = adapter.runTurn(iterParsed, { + ...deps.incomingMeta, + headers: deps.forwardHeaders ? new Headers(deps.forwardHeaders) : new Headers(), + ...(deps.incomingMeta.providerName ? { providerName: deps.incomingMeta.providerName } : {}), + abortSignal: signal, + translatorBudget, + pacingSlot, + }, emit); + } catch (err) { producer = Promise.reject(err); } + void producer.then(closeOnAbort).catch(err => { if (accepting) { collectionError = err; if (isTranslatorBudgetExceededError(err)) internalAbort.abort(err); } closeOnAbort(); - }); + }).finally(() => { activeProducers--; deps.onProducerEnd?.(); }); idle.reset(); for await (const event of queue.stream()) { if (timedOut) break; @@ -602,7 +614,7 @@ export async function runWithImageBridge(deps: ImageBridgeDeps): Promise { if (!sendBudget) return Number.POSITIVE_INFINITY; const remaining = sendBudget.remainingBaseSends(sharedSendCap); - return physicalSends === 0 ? Math.max(1, remaining) : remaining; + return Math.min(sharedSendCap, remaining + (!initialSendStarted && execution.comboDispatchPermit ? 1 : 0)); }; const remainingTransientSends = (): number => Math.min( requestTransientPolicy @@ -404,6 +406,21 @@ export async function runNativeChatAttempt( claimAmbiguousResend, ); + // Non-enforced accounting for one physical send. Enforced spend is owned by the reporter. + const comboReceipt = Boolean(sendBudget && execution.comboDispatchPermit); + const settleNonEnforcedSend = (url: string): void => { + if (sendBudget) { + // Backstop for sends the helper cannot see coming (a reset replay). The first + // report settles the combo's booking; each later one is charged and booked. + if (physicalSends > 0 && sendBudget.remainingBaseSends(sharedSendCap) <= 0) { + throw new SendBudgetExhaustedError(safeHostLabel(url)); + } + physicalSends += 1; + reportDispatchSends(sendBudget, 1, execution.comboDispatchPermit); + initialSendStarted = true; + } else if (!spendTracker?.charge()) throw new NativeChatSpendRefusal(); + }; + const send = async ( request: AdapterRequest, recovery?: "rate-limit-429" | "key-429" | UpstreamSendRecovery, @@ -433,6 +450,9 @@ export async function runNativeChatAttempt( providerFetch(activeProvider, undefined, { providerName: route.providerName, modelId: route.modelId, + onPhysicalDispatch: comboReceipt + ? () => { if (!physicalBudget.spendEnforced) settleNonEnforcedSend(request.url); } + : undefined, dispatchOverride: async (_input, init, execute) => { if (!providerApiKeySelectionIsCurrent(config, route.providerName, activeProvider)) { const current = resolveCurrentProviderApiKeyTransport(config, route.providerName, activeProvider); @@ -466,15 +486,11 @@ export async function runNativeChatAttempt( if ((logCtx.spendTracker as { refusals?: number } | undefined)?.refusals) throw new NativeChatSpendRefusal(); throw new SendBudgetExhaustedError(); } - if (!physicalBudget.spendEnforced && sendBudget) { - // Backstop for sends the helper cannot see coming (a reset replay). The first - // report settles the combo's booking; each later one is charged and booked. - if (physicalSends > 0 && sendBudget.remainingBaseSends(sharedSendCap) <= 0) { - throw new SendBudgetExhaustedError(safeHostLabel(request.url)); - } - physicalSends += 1; - reportDispatchSends(sendBudget, 1, execution.comboDispatchPermit); - } else if (!physicalBudget.spendEnforced && !spendTracker?.charge()) throw new NativeChatSpendRefusal(); + // Enforced spend started this send at `beforeSend`. A Combo child's non-enforced + // send is booked by the physical-dispatch receipt instead, so a local refusal + // before the wire refunds the booking rather than charging it. + if (physicalBudget.spendEnforced) initialSendStarted = true; + else if (!comboReceipt) settleNonEnforcedSend(request.url); noteProviderAttemptSend(logCtx, route.providerName, activeProvider, logCtx.usageLogInputTokens, transportRecovery ?? recovery); // A reselected provider transport is still a physical send: the connection policy // and manual-redirect ownership wrap the selected implementation (#4992). diff --git a/src/server/responses/adapter-dispatch.ts b/src/server/responses/adapter-dispatch.ts index 4be989dd1d7..4a7cf9490a1 100644 --- a/src/server/responses/adapter-dispatch.ts +++ b/src/server/responses/adapter-dispatch.ts @@ -170,6 +170,9 @@ export async function prepareAdapterExchange( | "workflowRootId" | "claimAmbiguousResend" | "sendsUsed" + | "targetSendsUsed" + | "initialSendAllowance" + | "noteInitialDispatch" >, ) { const { options, config, logCtx, req } = requestContext; @@ -374,24 +377,33 @@ export async function prepareAdapterExchange( const compactPrepaid = options.compactionRecoveryAttempted ? sendBudgetState.pendingHopPermit : undefined; if (compactPrepaid) sendBudgetState.pendingHopPermit = undefined; let compactPrepaidUsed = false; + // A combo-owned or compaction-prepaid send is settled by the physical-dispatch receipt + // when spend enforcement is off. Enforced spend and direct requests keep the shared + // physical-send reporter, which already owns the permit lifecycle there. + // Stable per request: a Combo or compaction permit comes from `reserveDispatch`, which starts + // (and freezes) the spend policy first, so `spendEnforced` cannot change before dispatch. + const receiptMode = !sendBudgetState.adapterSendBudget?.spendEnforced + && Boolean(options.comboInitialSend || compactPrepaid); // Combo and emergency compaction admission already book this target's first send. // Its configured initial ceiling is target-local; the shared remainder below still // accounts for earlier targets without deducting their sends from this target twice. const initialSendCap = transientPolicy || resetPolicy + || (options.comboAttempt && route.provider.adapter === "google") ? transientSendCapFor(transientPolicy?.attempts, (options.comboAttempt || compactPrepaid) ? 0 : sendBudgetState.sendsUsed) : 1; const fetchWithRetryPolicy = (route.provider.adapter === "google" || transientPolicy) ? fetchWithTransientRetry : fetchWithResetRetry; - upstreamResponse = await fetchWithRetryPolicy( + try { upstreamResponse = await fetchWithRetryPolicy( recovery => { // Unconfigured generic sends may omit the counting reporter. Capture policy // independently so a later recovery still belongs to this request's start. sendBudgetState.adapterSendBudget?.startRequest?.({ poolId: logCtx.spendPoolId ?? route.providerName, identityId: logCtx.accountLogLabel, }); - if (compactPrepaid && !compactPrepaidUsed) { + // Receipt mode books the physical send at executor admission instead of here. + if (!receiptMode && compactPrepaid && !compactPrepaidUsed) { if (!compactPrepaid.use()) throw new SendBudgetExhaustedError(safeHostLabel(builtInitialRequest.url)); compactPrepaidUsed = true; } @@ -405,6 +417,14 @@ export async function prepareAdapterExchange( dispatchOverride: oauthDispatch(builtInitialRequest), providerName: route.providerName, modelId: route.modelId, + onPhysicalDispatch: () => { + // Preserve the unconfigured direct-adapter accounting contract; Combo admission + // and explicit retry/compaction budgets require a receipt for their booked send. + if (receiptMode) { + sendBudgetState.noteInitialDispatch(compactPrepaid && !compactPrepaidUsed ? compactPrepaid : undefined); + compactPrepaidUsed = true; + } + }, beforeDispatch: isCanonicalOpenAiForwardProvider(route.provider) ? createCodexAuthDispatchGuard(admissionState.authCtx, options.codexAuthPolicy ?? config, route.modelId, options.admission, options.visionDescribeTerminal === true) : undefined, })); @@ -413,22 +433,24 @@ export async function prepareAdapterExchange( abortSignal: upstream.signal, label: safeHostLabel(builtInitialRequest.url), claimAmbiguousResend: claimPreHeaderResend, - ...(sendBudgetState.adapterSendBudget?.spendEnforced || options.comboDispatchPermit || transientPolicy || resetPolicy || compactPrepaid + ...(sendBudgetState.adapterSendBudget?.spendEnforced || options.comboDispatchPermit || transientPolicy || resetPolicy || compactPrepaid || options.comboAttempt ? { // A pending compaction permit already booked this leg's first physical send. - ...(transientPolicy || resetPolicy || compactPrepaid + ...(transientPolicy || resetPolicy || compactPrepaid || options.comboAttempt ? { attempts: Math.min(initialSendCap, - remainingTransientSendBudget(initialSendCap) + (compactPrepaid ? 1 : 0)), + sendBudgetState.initialSendAllowance(initialSendCap) + (compactPrepaid ? 1 : 0)), } : {}), - // Keep the exact combo or compaction receipt on the helper callback. The reporter - // also settles the request and workflow counters for reset-only retries. - onSendsConsumed: transientSendReporter(compactPrepaid ?? options.comboDispatchPermit), + // Reporter path: direct requests and enforced spend. Receipt-mode sends are + // settled by `onPhysicalDispatch`, so a reporter here would count them twice. + ...(receiptMode + ? {} + : { onSendsConsumed: transientSendReporter(compactPrepaid ?? options.comboDispatchPermit) }), } : {}), }, - ); + ); } finally { compactPrepaid?.release(); } } } catch (err) { cleanupUpstreamAbort(); @@ -609,8 +631,7 @@ export async function prepareAdapterExchange( const helperCountsSends = sendBudgetState.adapterSendBudget?.spendEnforced === true || refetchTransientPolicy !== null || resetReplayPolicyFor(route.provider) !== null; const prepaid = sendBudgetState.pendingHopPermit; const configuredTotal = refetchTransientPolicy?.attempts; - const refetchCap = transientSendCapFor(configuredTotal, - sendBudgetState.sendsUsed - (prepaid ? 1 : 0)); + const refetchCap = transientSendCapFor(configuredTotal, sendBudgetState.targetSendsUsed); // An exact total includes a booked hop, even when it consumed the last base slot. // Keep that funded send while forbidding the final reserve from widening the total. const prepaidLastSlot = helperCountsSends && configuredTotal !== undefined && prepaid diff --git a/src/server/responses/codex-ws-exchange.ts b/src/server/responses/codex-ws-exchange.ts index 897883efa11..ddb587a1a4d 100644 --- a/src/server/responses/codex-ws-exchange.ts +++ b/src/server/responses/codex-ws-exchange.ts @@ -23,6 +23,7 @@ interface ExchangeOptions { sseFallback: typeof globalThis.fetch; onQuota?: CodexWsQuotaObserver; beforeDispatch?: (headers: Headers) => void; + onPhysicalDispatch?: () => void; /** Bun version string the caller gated on; stamped onto the stage record. */ bunVersion?: string; } @@ -411,6 +412,17 @@ export function codexWsExchange(options: ExchangeOptions): Promise { failStream(error); return; } + try { options.onPhysicalDispatch?.(); } catch (error) { + // Admission refused before the frame left. This is a pre-open rejection, + // not a transport failure and not an authorization to replay over SSE. + settledPreOpen = true; + sent = false; + terminal = true; + cleanup(); + session.dispose(); + reject(error); + return; + } try { ws.send(frameText); sentAt = Date.now(); diff --git a/src/server/responses/core-combo.ts b/src/server/responses/core-combo.ts index 846c3b1688a..225e0c512ee 100644 --- a/src/server/responses/core-combo.ts +++ b/src/server/responses/core-combo.ts @@ -717,30 +717,32 @@ export async function executeComboResponses( const targetRoute = routeConcreteModel(config, `${pick.target.provider}/${pick.target.model}`); // The inherited spend tracker observes this parent log, before the child has its own label. logCtx.spendPoolId = targetRoute.providerName; - // The first target seeds the ledger's target identity and charges nothing; every later one - // is a real transition, refused once the declared hops, the alternate-target ledger or the - // request total are spent. `countedExternally` is required: the child charges its own - // physical sends, and charging here as well would halve the cap without saying so. + // Derive the target's allowance before booking its initial send; the booking occupies it. + const targetSendBudget = comboSendScope + ? comboTargetSendBudget(comboSendScope, combo.targets.length - 1 - comboTargetsDispatched) + : options.sendBudget; + // Every target prepays one send. Only its child may consume that booking; later targets + // remain bounded by the shared total, transition ledger and per-target holdback. + // `countedExternally` is required: the child charges its own physical sends against this + // booking through the exact permit, and charging here as well would halve the cap. const hopDecision = comboSendScope?.reserveDispatch({ sendClass: firstComboTarget ? "initial" : "combo-failover", targetKey: `${pick.target.provider}/${pick.target.model}`, countedExternally: true, }); - if (hopDecision && hopDecision.allowed) hopDecision.permit.use(); - else if (hopDecision && firstComboTarget) { + if (hopDecision && !hopDecision.allowed && firstComboTarget) { // A refused initial reservation authorizes no child send and has no upstream failure to return. return formatErrorResponse(429, SEND_BUDGET_EXHAUSTED_CODE, "request send budget exhausted before combo dispatch"); } - else if (hopDecision) { + else if (hopDecision && !hopDecision.allowed) { // Out of budget is not this target's failure. The established exhaustion contract is to // return the last real upstream answer with its status, headers and any quota body // intact rather than to mint a synthetic error, and a later target only exists because // an earlier one already recorded one. return exhaustedFailure(); } - const targetSendBudget = comboSendScope - ? comboTargetSendBudget(comboSendScope, combo.targets.length - 1 - comboTargetsDispatched) - : options.sendBudget; + const initialSend = hopDecision?.allowed ? { permit: hopDecision.permit, producerOwned: false } : undefined; + try { comboTargetsDispatched += 1; const childLog: RequestLogContext = { model: pick.target.model, @@ -890,6 +892,7 @@ export async function executeComboResponses( // parent arrived with. sendBudget: targetSendBudget, comboAttempt: true, + comboInitialSend: initialSend, comboDispatchPermit: hopDecision?.allowed ? hopDecision.permit : undefined, comboReplaySnapshot, deferCodexResetDerivedCooldown, @@ -1185,6 +1188,11 @@ export async function executeComboResponses( // Waiting or recovery may have observed cancellation after the check above. if (options.abortSignal?.aborted) return clientCancelledResponse(); } + } finally { + // External receipts make release a no-op after real work. runTurn owns its asynchronous + // producer; core transfers cleanup there before returning a streaming response. + if (!initialSend?.producerOwned) initialSend?.permit.release(); + } } const failure = exhaustedFailure(); if ( diff --git a/src/server/responses/core-options.ts b/src/server/responses/core-options.ts index 51c43ec7a0c..732641097f8 100644 --- a/src/server/responses/core-options.ts +++ b/src/server/responses/core-options.ts @@ -158,6 +158,8 @@ export interface HandleResponsesOptions { callerDirectAuth?: CallerDirectAuth | null; /** Internal recursion guard; callers outside this module must not set it. */ comboAttempt?: boolean; + /** Child-owned prepaid initial send; capacity remains charged until dispatch or unsent release. */ + comboInitialSend?: { permit: SingleUseDispatchPermit; producerOwned?: boolean }; /** Exact externally booked combo hop, used for its child's spend preflight and send reports. */ comboDispatchPermit?: SingleUseDispatchPermit; /** Internal handoff: this combo was selected by shadow-call interception. */ diff --git a/src/server/responses/core.ts b/src/server/responses/core.ts index 68862d971d5..1468a43f689 100644 --- a/src/server/responses/core.ts +++ b/src/server/responses/core.ts @@ -29,8 +29,6 @@ import { releaseUpstreamHostAdmission } from "../../codex/upstream-host-health"; import { releaseCodexAuthContextProbeLease } from "../../codex/auth-context"; import { runWithCompactionRecovery } from "./compaction-recovery"; -/** Public Responses entry and compatibility exports. Implementations live with their owners. */ - /** * Route one `/v1/responses` request through the adapter pipeline: recovery loop, passthrough * wire, image/web-search bridges, and the terminal-guard continuation. @@ -73,6 +71,8 @@ export async function handleResponses( release(); if (ownsBudget) translatorBudget.dispose(); throw error; + } finally { + if (!options.comboInitialSend?.producerOwned) options.comboInitialSend?.permit.release(); } } diff --git a/src/server/responses/fetch-helpers.ts b/src/server/responses/fetch-helpers.ts index 610b7c2cac0..b7d21e7756d 100644 --- a/src/server/responses/fetch-helpers.ts +++ b/src/server/responses/fetch-helpers.ts @@ -43,6 +43,10 @@ const pacingWebsocketDowngradeWarned = new Set(); */ const EGRESS_DECIDED = Symbol.for("opencodex.provider-egress.decided"); const UPSTREAM_REWRITTEN = Symbol.for("opencodex.plugins.upstream-rewritten"); +// Survives queued credential rebuilds and executor replacement without importing budget owners. +const PHYSICAL_DISPATCH = Symbol.for("opencodex.physical-dispatch"); +// One WS attempt and its HTTP fallback are one logical physical-send receipt. +const WS_FALLBACK_RECEIPT = Symbol.for("opencodex.ws-fallback-receipt"); /** * Announce once, per provider, that an explicit egress route moved this provider off the @@ -200,6 +204,7 @@ export function sendWithConnectionPolicy( const largeCodexBody = typeof body === "string" && /^https:\/\/chatgpt\.com\/backend-api\/codex\/responses(?:\/compact)?$/.test(target) && Buffer.byteLength(body, "utf8") >= 1024 * 1024; + (init as Record void) | undefined> | undefined)?.[PHYSICAL_DISPATCH]?.(); return physicalFetch(input, { ...init, ...(largeCodexBody ? { body: Buffer.from(body, "utf8") } : {}), @@ -233,10 +238,13 @@ export interface ProviderFetchOptions { onCodexWsQuota?: CodexWsQuotaObserver; /** Synchronous admission at actual credential dispatch, after pacing/backoff. */ beforeDispatch?: (headers: Headers) => void; + /** One-shot receipt after final local HTTP admission, immediately before executor entry. */ + onPhysicalDispatch?: () => void; /** Revalidate/rebuild a queued request at its physical send boundary, after pacing. */ dispatchOverride?: (input: Parameters[0], init: RequestInit, execute: typeof globalThis.fetch) => Promise; } +/** Compose provider pacing and transport admission; receipts run only at executor entry. */ export function providerFetch( provider: OcxProviderConfig, runtime: BunRuntimeGateInput = currentBunRuntimeIdentity(), @@ -301,7 +309,14 @@ export function providerFetch( options.beforeDispatch?.(new Headers(init?.headers ?? (input instanceof Request ? input.headers : undefined))); // No proxy option is attached here: a `dispatchOverride` may rebuild this request against // a different destination, so the route is decided at the physical send instead. - const dispatchInit = { ...withUpstreamHttpVersion(input, init, provider), timeout: 0 }; + const sharedReceipt = (init as Record void) | undefined> | undefined)?.[WS_FALLBACK_RECEIPT]; + let notified = false; + const dispatchInit = { ...withUpstreamHttpVersion(input, init, provider), timeout: 0, + ...(options.onPhysicalDispatch ? { [PHYSICAL_DISPATCH]: () => { + if (sharedReceipt) sharedReceipt(); + else if (!notified) { options.onPhysicalDispatch!(); notified = true; } + } } : {}), + }; return options.dispatchOverride ? options.dispatchOverride(input, dispatchInit, dispatch) : dispatch(input, dispatchInit); @@ -332,8 +347,16 @@ export function providerFetch( // used, protocol pin included: a WS turn that falls back is serving the // request over HTTP, and dropping the provider's `upstreamHttpVersion` // there would silently negotiate a transport the operator ruled out. - return codexWsUpstreamFetch(input, init, httpFetch, runtime, options.onCodexWsQuota, options.beforeDispatch, options.nativeControl, - async () => { (await waitForPacing(init.signal ?? undefined))?.release(); }); + let notified = false; + const notifyOnce = (): void => { + if (notified) return; + notified = true; + options.onPhysicalDispatch?.(); + }; + const fallback = ((url: Parameters[0], fallbackInit?: RequestInit) => + httpFetch(url, { ...fallbackInit, [WS_FALLBACK_RECEIPT]: notifyOnce } as RequestInit)) as typeof globalThis.fetch; + return codexWsUpstreamFetch(input, init, fallback, runtime, options.onCodexWsQuota, options.beforeDispatch, options.nativeControl, + async () => { (await waitForPacing(init.signal ?? undefined))?.release(); }, notifyOnce); } return httpFetch(input, init); }; diff --git a/src/server/responses/passthrough-dispatch.ts b/src/server/responses/passthrough-dispatch.ts index e61ea985550..a2b423b0a0e 100644 --- a/src/server/responses/passthrough-dispatch.ts +++ b/src/server/responses/passthrough-dispatch.ts @@ -225,6 +225,10 @@ export async function preparePassthroughExchange( | "pendingHopPermit" | "workflowRootId" | "sendsUsed" + | "targetSendsUsed" + | "initialSendAllowance" + | "noteInitialDispatch" + | "adapterSendBudget" >, ) { const { config, logCtx, options, req } = requestContext; @@ -753,7 +757,7 @@ export async function preparePassthroughExchange( const transientSendPolicy = () => transientRetryPolicyFor(route.provider); const transientSendAttempts = (): number => transientSendCapFor( transientSendPolicy()?.attempts, - sendBudgetState.sendsUsed, + sendBudgetState.targetSendsUsed, ); const configuredTransientSendBudgetExhausted = (): boolean => transientSendPolicy() !== null && transientSendAttempts() === 0; @@ -968,6 +972,11 @@ export async function preparePassthroughExchange( }; const initialBodyRefusal = refuseOversizedOutboundBody(request); if (initialBodyRefusal) return initialBodyRefusal; + // A combo-owned first send is settled by the physical-dispatch receipt unless spend + // enforcement is on, where the shared reporter already owns the permit lifecycle. + // Stable per request: the Combo booking (`reserveDispatch`) already started and froze the spend + // policy, so `spendEnforced` cannot change before dispatch even though no reporter starts it here. + const receiptMode = !sendBudgetState.adapterSendBudget?.spendEnforced && Boolean(options.comboInitialSend); try { // Transient-5xx pre-stream retry (devlog/_plan/260716_claudecode_hardening/010): // the ChatGPT backend emits transient 502/520s that an immediate retry absorbs. @@ -994,6 +1003,7 @@ export async function preparePassthroughExchange( dispatchOverride: oauthDispatch(request), providerName: route.providerName, modelId: route.modelId, + onPhysicalDispatch: receiptMode ? sendBudgetState.noteInitialDispatch : undefined, onCodexWsQuota: codexWsQuotaObserver(admissionState.authCtx, route.provider, route.modelId), beforeDispatch: isCanonicalOpenAiForwardProvider(route.provider) ? createCodexAuthDispatchGuard(admissionState.authCtx, options.codexAuthPolicy ?? config, route.modelId, options.admission, options.visionDescribeTerminal === true) : undefined, @@ -1004,7 +1014,10 @@ export async function preparePassthroughExchange( .then(adoptObservedResponse); }, { abortSignal: upstream.signal, label: safeHostLabel(request.url), - attempts: remainingTransientSendBudget(transientSendAttempts()), onSendsConsumed: transientSendReporter(), + attempts: sendBudgetState.initialSendAllowance(transientSendAttempts()), + // Reporter path: direct requests and enforced spend. Receipt-mode sends are settled + // by `onPhysicalDispatch`; a reporter as well would count them twice. + ...(receiptMode ? {} : { onSendsConsumed: transientSendReporter() }), claimAmbiguousResend: claimPreHeaderResend, }, ); diff --git a/src/server/responses/request-send-budget.ts b/src/server/responses/request-send-budget.ts index da331ff3291..27f1c5ac18e 100644 --- a/src/server/responses/request-send-budget.ts +++ b/src/server/responses/request-send-budget.ts @@ -9,7 +9,7 @@ import { import { poolContinuityRefusalReason, workflowRefusalResponse } from "../workflow-refusal"; import type { AttemptRecoveryKind, AttemptRecoveryWithheld } from "../../usage/log"; import { noteAttemptRecoveryWithheld, noteAttemptSend } from "../request-log"; -import { TRANSIENT_RETRY_MAX_ATTEMPTS } from "../../lib/upstream-retry"; +import { TRANSIENT_RETRY_MAX_ATTEMPTS, SendBudgetExhaustedError } from "../../lib/upstream-retry"; import type { DispatchDecision, DispatchIntent, @@ -58,10 +58,24 @@ export function createResponsesSendBudget( // sends once per child seven hundred times, so every send charged to the request is charged // to the root as well (#4546). const workflowRootId = req.headers.get("x-codex-parent-thread-id")?.trim() || undefined; - const inheritedPermit = options.compactionRecoveryPermit ?? options.comboDispatchPermit; + // A child that owns its Combo booking settles that exact permit through the reporter, even + // when its caller handed ownership over only as `comboInitialSend`. + const inheritedPermit = options.compactionRecoveryPermit ?? options.comboDispatchPermit + ?? options.comboInitialSend?.permit; let pendingHopPermit: SingleUseDispatchPermit | undefined = options.compactionRecoveryPermit; + let initialPermit = options.comboInitialSend?.permit; + // Count this owner's physical sends, not later bookings made by other shared-ledger owners. + // Direct callers retain sends reported before this owner was constructed. + let targetSendsUsed = initialPermit ? 0 : Math.max(0, + sendBudget.used - (options.compactionRecoveryPermit ? 1 : 0)); + /** Attribute reported physical sends to this owner; the prepaid booking is spent once reported. */ + const noteTargetSends = (charged: number): void => { + targetSendsUsed += charged; + if (charged > 0) initialPermit = undefined; + }; const recordTransientSends = (used: number, permit?: SingleUseDispatchPermit): void => { const charged = Math.max(0, used); + noteTargetSends(charged); reportDispatchSends(sendBudget, charged, permit); options.onCompactionRecoverySendsReported?.(charged); chargeWorkflowSends(workflowRootId, charged); @@ -71,6 +85,7 @@ export function createResponsesSendBudget( const transientSendReporter = (permit = pendingHopPermit ?? inheritedPermit) => createPhysicalSendReporter(sendBudget, () => ({ poolId: logCtx.spendPoolId ?? logCtx.provider, identityId: logCtx.accountLogLabel }), permit, charged => { + noteTargetSends(Math.max(0, charged)); options.onCompactionRecoverySendsReported?.(charged); chargeWorkflowSends(workflowRootId, charged); }); @@ -136,6 +151,7 @@ export function createResponsesSendBudget( // Ordinal 1 is skipped because the caller normally records it before dispatch. An adapter // that reports every send asks for it to be counted here instead, so that the first send is // logged where it actually happens rather than before admission could still refuse it. + targetSendsUsed += 1; if (send.ordinal <= 1 && options.includeFirst !== true) return; noteAttemptSend(logCtx.activeAttempt, inputTokens, send.recovery); }; @@ -191,21 +207,29 @@ export function createResponsesSendBudget( * The hop hands its reservation down through `pendingHopPermit`, the same seam the * passthrough ladder already uses, and this view spends it on the adapter's FIRST * reservation. Every later send in that ladder is a new physical send and is charged - * normally. The refundable view below keeps an unused hop open until executor invocation; - * its adapter release hands back an abandoned replay. The ordinary view retains early - * settlement for existing callers. + * normally. Claiming preserves the refundable single-use booking until physical dispatch; + * an abandoned replay releases only that exact booking. */ - const claimHopPermit = (): SingleUseDispatchPermit | undefined => { - const permit = pendingHopPermit; - pendingHopPermit = undefined; - return permit; + const permits = { + /** Transfer the initial booking once to the adapter dispatch owner. */ + claimInitialPermit: () => { + const permit = initialPermit; + initialPermit = undefined; + return permit; + }, + /** Transfer a pending recovery booking once, without creating a new allowance. */ + claimHopPermit: () => { + const permit = pendingHopPermit; + pendingHopPermit = undefined; + return permit; + }, }; - const adapterDispatchBudget: RequestExecutionBudget | undefined = adapterSendBudget === undefined - ? undefined : adapterDispatchBudgetView(adapterSendBudget, { claimHopPermit }); - // Opt in only where the adapter's executor is the confirmation boundary. Ordinary callers - // keep the existing early settlement contract, including already-settled permit fallback. - const refundableAdapterDispatchBudget: RequestExecutionBudget | undefined = adapterSendBudget === undefined - ? undefined : adapterDispatchBudgetView(adapterSendBudget, { claimHopPermit }, true); + const adapterDispatchBudget = adapterSendBudget === undefined + ? undefined : adapterDispatchBudgetView(adapterSendBudget, permits); + // Preserve the existing opted-in view for direct callers, including its admitted replacement + // of an already-settled hop. Exact-owned Combo and sidecar handoffs use the ordinary view. + const refundableAdapterDispatchBudget = adapterSendBudget === undefined + ? undefined : adapterDispatchBudgetView(adapterSendBudget, permits, true); /** * How many sends a recovery leg may make, and the permit that authorises the last one. * @@ -292,12 +316,34 @@ export function createResponsesSendBudget( noteTransientSends, transientSendReporter, remainingTransientSendBudget, + /** Only this child's prepaid send, not every pending external booking, is spendable here. */ + initialSendAllowance: (cap: number): number => Math.min(cap, + remainingTransientSendBudget(cap) + (initialPermit ? 1 : 0)), + /** At final executor admission, settle only the owning booking or reserve this send. */ + noteInitialDispatch: (prepaid = initialPermit): void => { + if (prepaid) { + if (!prepaid.assumeCharge()) throw new SendBudgetExhaustedError(); + if (prepaid === initialPermit) initialPermit = undefined; + } else if (adapterSendBudget && (options.comboInitialSend || options.compactionRecoveryPermit)) { + const decision = adapterSendBudget.reserveDispatch({ sendClass: "transient", + targetKey: adapterSendBudget.lastTargetKey ?? `${logCtx.provider}/${logCtx.model}` }); + if (!decision.allowed || !decision.permit.use()) throw new SendBudgetExhaustedError(); + } else { + // Legacy direct callers can prebook without handing over a permit; keep their reporter. + noteTransientSends(1); + return; + } + targetSendsUsed += 1; + options.onCompactionRecoverySendsReported?.(1); + chargeWorkflowSends(workflowRootId, 1); + }, + /** Physical sends reported by this owner; unrelated shared reservations cannot consume it. */ + get targetSendsUsed(): number { return targetSendsUsed; }, /** - * Physical sends this logical request has already made. + * Charged sends, including open prepaid reservations, across this logical request. * - * A live getter, not a snapshot: it is read once per dispatch leg to resolve a configured - * ladder, and a value frozen at construction would answer for a request that had sent - * nothing. + * A live getter, not a completed-send snapshot. Initial ladders use the child's own + * prepaid allowance and target-local count; shared admission keeps every booking spent. */ get sendsUsed(): number { return sendBudget.used; }, adapterSendBudget, @@ -335,8 +381,11 @@ export type ResponsesSendBudget = Exclude SingleUseDispatchPermit | undefined }, - refundableHop = false, + hop: { + claimInitialPermit: () => SingleUseDispatchPermit | undefined; + claimHopPermit: () => SingleUseDispatchPermit | undefined; + }, + legacyHopReplacement = false, ): RequestExecutionBudget { return { get physicalStarted() { return budget.physicalStarted; }, @@ -360,10 +409,22 @@ function adapterDispatchBudgetView( claimAmbiguousResend: (limit: number): boolean => budget.claimAmbiguousResend?.(limit) === true, get ambiguousResendSpent(): boolean { return budget.ambiguousResendSpent === true; }, reserveDispatch(intent: DispatchIntent): DispatchDecision { - const refundable = refundableHop || budget.spendEnforced === true; // A dispatch whose upstream state is unknown is refused on its own merits. A hop that // already paid does not make an unsafe replay safe, so that check stays with the budget. if (intent.replaySafe !== false) { + // The Combo reserves a logical provider/model identity. The child's first actual + // endpoint is not a recovery transition; retain the original refundable permit. + const initialPermit = hop.claimInitialPermit(); + if (initialPermit) { + budget.bindPrepaidTarget?.(intent.targetKey); + return { allowed: true, permit: { + sendClass: initialPermit.sendClass, + execute: run => initialPermit.execute ? initialPermit.execute(run) : run(), + use: () => initialPermit.assumeCharge(), + assumeCharge: () => initialPermit.assumeCharge(), + release: () => initialPermit.release(), + } }; + } const hopPermit = hop.claimHopPermit(); if (hopPermit && intent.targetKey !== budget.lastTargetKey) { // The rotated credential can select a different regional endpoint. Replace the @@ -372,35 +433,37 @@ function adapterDispatchBudgetView( hopPermit.release(); return budget.reserveDispatch({ ...intent, sendClass: hopPermit.sendClass }); } - if (hopPermit !== undefined && (refundable || hopPermit.assumeCharge())) { - let settled = false; - let executionPermit = hopPermit; - const confirm = (): boolean => { - if (settled) return false; - settled = true; - // Admission can still reject this replay. Close its external booking only when - // the adapter invokes its executor, leaving release() able to refund it until then. - if (!refundable || hopPermit.assumeCharge()) return true; - // Preserve the old fallback for a permit another leg already settled. - const fresh = budget.reserveDispatch(intent); - if (!fresh.allowed) return false; - executionPermit = fresh.permit; - return fresh.permit.assumeCharge(); - }; - return { - allowed: true, - permit: { + // Reservation precedes pacing and beforeDispatch. Transfer the exact open booking; + // only physical consumption closes external reporting and makes it non-refundable. + // The underlying permit enforces single use, including settlement by another owner. + if (hopPermit !== undefined) { + if (legacyHopReplacement) { + let settled = false; + let executionPermit = hopPermit; + const confirm = (): boolean => { + if (settled) return false; + settled = true; + if (hopPermit.assumeCharge()) return true; + // Preserve the old fallback for a permit another leg already settled. + const fresh = budget.reserveDispatch(intent); + if (!fresh.allowed) return false; + executionPermit = fresh.permit; + return fresh.permit.assumeCharge(); + }; + return { allowed: true, permit: { sendClass: hopPermit.sendClass, execute: run => executionPermit.execute ? executionPermit.execute(run) : run(), - use: confirm, - assumeCharge: confirm, - release: (): void => { - if (!refundable || settled) return; - settled = true; - hopPermit.release(); - }, - }, - }; + use: confirm, assumeCharge: confirm, + release: () => { if (!settled) { settled = true; hopPermit.release(); } }, + } }; + } + return { allowed: true, permit: { + sendClass: hopPermit.sendClass, + execute: run => hopPermit.execute ? hopPermit.execute(run) : run(), + use: () => hopPermit.assumeCharge(), + assumeCharge: () => hopPermit.assumeCharge(), + release: () => hopPermit.release(), + } }; } } return budget.reserveDispatch(intent); diff --git a/src/server/responses/run-turn-execution.ts b/src/server/responses/run-turn-execution.ts index bfe13fba794..54553d55088 100644 --- a/src/server/responses/run-turn-execution.ts +++ b/src/server/responses/run-turn-execution.ts @@ -235,6 +235,7 @@ export async function executeResponsesRunTurn( preacquiredSlot?: ProviderRequestSlot, ): Promise => { const attemptSeq = ++runTurnAttemptSeq; + if (attemptSeq === 1 && options.comboInitialSend) options.comboInitialSend.producerOwned = true; let pacingSlot = preacquiredSlot; const emit = (event: AdapterEvent) => { options.onCompactionRecoveryAdapterEvent?.(event); @@ -339,6 +340,7 @@ export async function executeResponsesRunTurn( message: err instanceof Error ? err.message : String(err), }); } finally { + if (attemptSeq === 1) options.comboInitialSend?.permit.release(); devinProducers.delete(producerAbort); cleanupProducerAbort?.(); releaseProviderRequestSlot(pacingSlot); diff --git a/src/server/responses/sidecar-execution.ts b/src/server/responses/sidecar-execution.ts index 30da31ceb71..831b298994f 100644 --- a/src/server/responses/sidecar-execution.ts +++ b/src/server/responses/sidecar-execution.ts @@ -48,6 +48,7 @@ import { normalizeLogConversationId } from "../request-log-conversation"; import { rememberResponseState } from "../../responses/state"; import { trackStreamLifetime } from "../lifecycle"; import { isAntigravityRawValidationRefusal } from "./antigravity-validation-refusal"; +import { createSidecarSendBudget } from "./sidecar-send-budget"; /** One responsibility of the Responses request pipeline; state owners are explicit. */ export async function executeResponsesSidecars( @@ -90,7 +91,7 @@ export async function executeResponsesSidecars( | "notifyResponseComplete" | "cancelResponseCompletion" >, - sendBudgetState: Pick, + sendBudgetState: Pick, ) { const { config, options, logCtx } = requestContext; const antigravityPoolActivated = requestState.route.providerName === "google-antigravity" @@ -114,6 +115,7 @@ export async function executeResponsesSidecars( } = requestState; const { routedCompaction, openAiSidecar } = sidecarState; const { reserveCredentialHop } = sendBudgetState; + const sidecarBudget = createSidecarSendBudget(options, sendBudgetState, () => logCtx.usageLogInputTokens); const { commitReasoningReplayServingRoute, continuationStateForResponse, @@ -264,7 +266,8 @@ export async function executeResponsesSidecars( return null; } recoveryKind = "oauth-account-403"; - sendBudgetState.pendingHopPermit = hop.permit; + if (!options.comboInitialSend) sendBudgetState.pendingHopPermit = hop.permit; + else sidecarBudget.ownCredentialHop(hop.permit); } const rotated = !originalResponse || originalResponse.status === 429 ? rotateProviderTransportOn429(config, route.providerName, route.provider, { @@ -323,8 +326,9 @@ export async function executeResponsesSidecars( return null; } recoveryKind = "oauth-account-429"; - if (canRunWebSearch && route.providerName === "google-antigravity") sendBudgetState.pendingHopPermit = hop.permit; - else hop.permit?.use(); + if (!options.comboInitialSend && canRunWebSearch && route.providerName === "google-antigravity") + sendBudgetState.pendingHopPermit = hop.permit; + else sidecarBudget.ownCredentialHop(hop.permit); } else if ( // Anthropic's pool is excluded from generic failover, so without this arm a 429 inside a // web-search or image-bridge turn was terminal even with the pool fully enabled -- while @@ -365,7 +369,7 @@ export async function executeResponsesSidecars( return null; } recoveryKind = "anthropic-oauth-429"; - hop.permit?.use(); + sidecarBudget.ownCredentialHop(hop.permit); } else { // No key pool, no generic OAuth roster, no Anthropic pool could produce a replacement // credential. The 429 is terminal for this sidecar turn. @@ -451,11 +455,14 @@ export async function executeResponsesSidecars( const imageProviderFetch = providerFetch( route.provider, options.codexWsRuntimeIdentity, - { providerName: route.providerName, modelId: route.modelId }, + { providerName: route.providerName, modelId: route.modelId, ...sidecarBudget.fetchOptions(transportState.adapter) }, ); const imgResponse = await runWithImageBridge({ parsed, adapter: transportState.adapter, - incomingMeta: { headers: requestState.selectedForwardHeaders, providerName: route.providerName, abortSignal: options.abortSignal, translatorBudget }, + incomingMeta: { headers: requestState.selectedForwardHeaders, providerName: route.providerName, abortSignal: options.abortSignal, translatorBudget, ...(options.comboInitialSend ? { providerFetch: imageProviderFetch } : {}), ...sidecarBudget.incomingMeta }, + onProducerStart: sidecarBudget.takeProducerOwnership, + onProducerEnd: sidecarBudget.release, + onIterationEnd: sidecarBudget.releaseUnsentHop, ...(imgPlan ? { plan: imgPlan } : {}), ...(vidPlan ? { videoPlan: vidPlan } : {}), forwardHeaders: requestState.selectedForwardHeaders, @@ -471,10 +478,10 @@ export async function executeResponsesSidecars( stallTimeoutSec: config.stallTimeoutSec, waitForRequestSlot: imageProviderFetch.waitForPacing, fetchImpl: imageProviderFetch.unpacedFetch ?? imageProviderFetch, - fetchForRequest: (request, iterParsed) => { + fetchForRequest: (request, iterParsed, adapter) => { const fetch = providerFetch(route.provider, options.codexWsRuntimeIdentity, { dispatchOverride: oauthDispatch(request, iterParsed), - providerName: route.providerName, modelId: route.modelId, + providerName: route.providerName, modelId: route.modelId, ...sidecarBudget.fetchOptions(adapter), }); return fetch.unpacedFetch ?? fetch; }, @@ -541,12 +548,14 @@ export async function executeResponsesSidecars( providerFetch(route.provider, options.codexWsRuntimeIdentity, { providerName: route.providerName, modelId: route.modelId, + ...sidecarBudget.fetchOptions(transportState.adapter), })(input, init)) as typeof globalThis.fetch; const wsResponse = await runWithWebSearch({ parsed, adapter: transportState.adapter, - fetchForRequest: (request, iterParsed) => providerFetch(route.provider, options.codexWsRuntimeIdentity, { + onIterationEnd: sidecarBudget.releaseUnsentHop, + fetchForRequest: (request, iterParsed, adapter) => providerFetch(route.provider, options.codexWsRuntimeIdentity, { dispatchOverride: oauthDispatch(request, iterParsed), - providerName: route.providerName, modelId: route.modelId, + providerName: route.providerName, modelId: route.modelId, ...sidecarBudget.fetchOptions(adapter), }), incomingMeta: { headers: requestState.selectedForwardHeaders, @@ -554,11 +563,11 @@ export async function executeResponsesSidecars( abortSignal: options.abortSignal, translatorBudget, providerFetch: routedProviderFetch, - ...(route.providerName === "google-antigravity" ? { + ...(!options.comboInitialSend && route.providerName === "google-antigravity" ? { sendBudget: sendBudgetState.refundableAdapterDispatchBudget, onPhysicalSend: (send: { ordinal: number; recovery?: AttemptRecoveryKind }) => sendBudgetState.noteAdapterPhysicalSend(logCtx.usageLogInputTokens, send), - } : {}), + } : sidecarBudget.incomingMeta), }, backend: wsPlan.backend, forwardProvider: wsPlan.forwardSidecar?.provider, @@ -591,7 +600,7 @@ export async function executeResponsesSidecars( stallTimeoutSec: wsPlan.stallTimeoutSec, streamRoutedModelOutput: wsPlan.streamRoutedModelOutput, on429: rotateSidecarProviderOn429, - ...(route.providerName === "google-antigravity" ? { onIterationEnd: () => { + ...(!options.comboInitialSend && route.providerName === "google-antigravity" ? { onIterationEnd: () => { // The next Antigravity adapter dispatch claims this permit. If request construction, // admission or cancellation stopped it first, return the unused send allowance. const permit = sendBudgetState.pendingHopPermit; diff --git a/src/server/responses/sidecar-send-budget.ts b/src/server/responses/sidecar-send-budget.ts new file mode 100644 index 00000000000..8d0402a22a1 --- /dev/null +++ b/src/server/responses/sidecar-send-budget.ts @@ -0,0 +1,58 @@ +import type { IncomingMeta, ProviderAdapter } from "../../adapters/base"; +import type { HandleResponsesOptions } from "./core-options"; +import type { ResponsesSendBudget } from "./request-send-budget"; +import type { ProviderFetchOptions } from "./fetch-helpers"; +import type { SingleUseDispatchPermit } from "../../lib/request-execution-budget"; + +/** Hosted inference consumes the same child-owned booking as ordinary dispatch. */ +export function createSidecarSendBudget( + options: HandleResponsesOptions, + budget: Pick, + inputTokens: () => number | undefined, +) { + const initial = options.comboInitialSend; + let hopPermit: SingleUseDispatchPermit | undefined; + let producerActive = false; + /** Detach this owner’s hop before refunding its still-unused reservation. */ + const releaseHop = (): void => { + const permit = hopPermit; + hopPermit = undefined; + if (budget.pendingHopPermit === permit) budget.pendingHopPermit = undefined; + permit?.release(); + }; + return { + /** Retain a Combo hop until dispatch; direct requests keep eager settlement. */ + ownCredentialHop(permit?: SingleUseDispatchPermit): void { + if (!initial) { permit?.use(); return; } // Preserve direct callers' reporting contract. + releaseHop(); + hopPermit = permit; + budget.pendingHopPermit = permit; + }, + // Adapter-owned transports settle through the live dispatch view, not an HTTP receipt too. + incomingMeta: initial ? { + comboAttempt: options.comboAttempt === true, + sendBudget: budget.adapterDispatchBudget, + onPhysicalSend: send => budget.noteAdapterPhysicalSend(inputTokens(), send), + onRecoveryWithheld: budget.noteAdapterRecoveryWithheld, + } satisfies Partial : {}, + /** Attach HTTP receipts only when the adapter does not own physical dispatch. */ + fetchOptions(adapter: ProviderAdapter): Pick { + return initial && !adapter.fetchResponse && !adapter.runTurn + ? { onPhysicalDispatch: () => { + const prepaid = budget.pendingHopPermit; + budget.pendingHopPermit = undefined; + budget.noteInitialDispatch(prepaid); + } } : {}; + }, + /** Transfer cleanup to the asynchronous producer before it can outlive the response. */ + takeProducerOwnership(): void { + if (initial) { initial.producerOwned = true; producerActive = true; } + }, + /** Release unused bookings only after the owning producer has settled. */ + release(): void { + initial?.permit.release(); producerActive = false; releaseHop(); + }, + /** An iteration may finish before runTurn; defer its refund while that producer is active. */ + releaseUnsentHop(): void { if (!producerActive) releaseHop(); }, + }; +} diff --git a/src/server/responses/ws-upstream.ts b/src/server/responses/ws-upstream.ts index 4363a39d24d..b51f66abc1a 100644 --- a/src/server/responses/ws-upstream.ts +++ b/src/server/responses/ws-upstream.ts @@ -148,6 +148,7 @@ export function shouldUseCodexWsUpstream( } } +/** Select the bounded WS lane or HTTP fallback, forwarding receipts at physical dispatch. */ export function codexWsUpstreamFetch( url: string, init: RequestInit, @@ -157,6 +158,7 @@ export function codexWsUpstreamFetch( beforeDispatch?: (headers: Headers) => void, nativeControl?: NativeResponseControl, beforeContinuation?: () => Promise, + onPhysicalDispatch?: () => void, ): Promise { const prepared = prepareCodexWsRequest(url, init); if (!prepared) return sseFallback(url, prepareCodexHttpInit(url, init)); @@ -229,7 +231,7 @@ export function codexWsUpstreamFetch( return codexWsExchange({ session, url, init, prepared, sseFallback, onQuota, beforeDispatch, nativeControl: control, - beforeContinuation, + beforeContinuation, onPhysicalDispatch, bunVersion: typeof runtime === "string" ? runtime : runtime.version, }); } diff --git a/src/web-search/loop.ts b/src/web-search/loop.ts index 05d45fe2aa8..83164a004d8 100644 --- a/src/web-search/loop.ts +++ b/src/web-search/loop.ts @@ -323,7 +323,7 @@ export interface WebSearchLoopDeps { /** Observe the exact adapter request selected for each routed-model iteration. */ onRequestBuilt?: (request: AdapterRequest) => void; /** Request-scoped executor retains the core's selection binding across loop retries. */ - fetchForRequest?: (request: AdapterRequest, parsed: OcxParsedRequest) => typeof globalThis.fetch; + fetchForRequest?: (request: AdapterRequest, parsed: OcxParsedRequest, adapter: ProviderAdapter) => typeof globalThis.fetch; /** Called before each routed-model dispatch in the loop, for attempt telemetry. Same-target 429 replays pass the `rate-limit-429` recovery kind. */ onAttemptSend?: (recovery?: AttemptRecoveryKind) => void; /** @@ -494,7 +494,7 @@ export async function runWithWebSearch(deps: WebSearchLoopDeps): Promise { const configured = { adapter: "openai-chat", baseUrl: "https://adapter-fixture.invalid", requestPacing: { enabled: true, maxConcurrentRequests: 1 }, - fetch: Object.assign(async () => new Response("ok"), { preconnect() {} }) as typeof fetch, + fetch: Object.assign(async () => new Response("ok"), { preconnect() {} }) as unknown as typeof fetch, } as OcxProviderConfig & { fetch: typeof fetch }; const executor = providerFetch(configured, undefined, { providerName: "physical", modelId: "a" }); const controller = new AbortController(); @@ -51,7 +51,7 @@ describe("adapter physical inference admission", () => { const send = createAdapterPhysicalSend({ sendBudget: scope, onPhysicalSend: event => ordinals.push(event.ordinal) }, Object.assign(async () => { sends += 1; return new Response("ok"); }, { waitForPacing: async () => { pacingSlots += 1; }, - }) as typeof fetch); + }) as unknown as typeof fetch); await send({ url, dispatch: executor => executor(url) }); await expect(send({ url, sendClass: "repair", beforeDispatch: () => { waits += 1; }, dispatch: executor => executor(url) })).rejects.toBeInstanceOf(SendBudgetExhaustedError); @@ -71,7 +71,7 @@ describe("adapter physical inference admission", () => { const failure = new Error(`fixture ${phase} refusal`); const executor = Object.assign(async () => { sends += 1; return new Response("unexpected"); }, { waitForPacing: async () => { if (phase === "pacing") throw failure; }, - }) as typeof fetch; + }) as unknown as typeof fetch; const send = createAdapterPhysicalSend({ sendBudget: parent, abortSignal: controller.signal }, executor); // A reserve-funded class still gets a real permit once the base allowance is spent; the // refusal paths below never reach its dispatch, so the reservation must be handed back. @@ -87,14 +87,14 @@ describe("adapter physical inference admission", () => { expect(sends).toBe(0); }); - test.each(["pacing", "backoff", "abort", "adapter"] as const)("a settled hop charge stays charged when the %s leg never dispatches", async phase => { + test.each(["pacing", "backoff", "abort", "adapter"] as const)("an open hop refunds when the %s leg never dispatches", async phase => { const { parent, scope, dispose } = prepaid(); let sends = 0; const controller = new AbortController(); const failure = new Error(`fixture ${phase} refusal`); const executor = Object.assign(async () => { sends += 1; return new Response("unexpected"); }, { waitForPacing: async () => { if (phase === "pacing") throw failure; }, - }) as typeof fetch; + }) as unknown as typeof fetch; try { const send = createAdapterPhysicalSend({ sendBudget: scope, abortSignal: controller.signal }, executor); await expect(send({ url, beforeDispatch: () => { @@ -104,10 +104,9 @@ describe("adapter physical inference admission", () => { if (phase === "adapter") throw failure; return physical(url); } })).rejects.toBe(failure); - // The hop's reservation was the charge and the dispatch view settled it at admission; - // the adapter's release has nothing left to refund. - expect(parent.used).toBe(4); - expect(parent.reserveSpent).toBe(true); + // Claiming the hop is not dispatch. Adapter cleanup refunds its exact open booking. + expect(parent.used).toBe(3); + expect(parent.reserveSpent).toBe(false); expect(sends).toBe(0); } finally { dispose(); } }); @@ -119,7 +118,7 @@ describe("adapter physical inference admission", () => { const failure = new Error(`fixture ${phase} refusal`); const executor = Object.assign(async () => { sends += 1; return new Response("unexpected"); }, { waitForPacing: async () => { if (phase === "pacing") throw failure; }, - }) as typeof fetch; + }) as unknown as typeof fetch; try { const send = createAdapterPhysicalSend({ sendBudget: scope, abortSignal: controller.signal }, executor); await expect(send({ url, beforeDispatch: () => { @@ -141,7 +140,7 @@ describe("adapter physical inference admission", () => { budget.used = 4; let prepared = false, sends = 0; const send = createAdapterPhysicalSend({ sendBudget: budget }, - (async () => { sends += 1; return new Response("unexpected"); }) as typeof fetch); + (async () => { sends += 1; return new Response("unexpected"); }) as unknown as typeof fetch); await expect(send({ url, beforeDispatch: () => { prepared = true; }, dispatch: physical => physical(url) })).rejects.toBeInstanceOf(SendBudgetExhaustedError); expect(prepared).toBe(false); diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index e7953e2947a..11f1a21d97f 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -226,6 +226,7 @@ "protocol-direct-encoders-local-upstream.test.ts": "responses", "protocol-direct-encoders-messages.test.ts": "responses", "chat-native-combo.test.ts": "responses", "messages-native-eligibility.test.ts": "responses", "messages-native-bridge-policy.test.ts": "responses", + "responses-first-send-reservation.test.ts": "responses", "responses-first-send-spend-policy.test.ts": "responses", "responses-dispatch-receipt.test.ts": "responses", "responses-hosted-send-hop.test.ts": "responses", "responses-hosted-send-producer.test.ts": "responses", "responses-hosted-send-reservation.test.ts": "responses", "responses-hosted-send-round-parity.test.ts": "responses", "responses-hosted-send-unsent-hop.test.ts": "responses", "responses-hosted-send-ws.test.ts": "responses", "messages-native-oauth-eligibility.test.ts": "responses", "chat-inbound-reasoning-none.test.ts": "responses", "chat-native-decline-reason.test.ts": "responses", "chat-inbound-reasoning-replay.test.ts": "responses", "chat-inline-document-bytes.test.ts": "responses", "chat-json-sse-fallback.test.ts": "responses", diff --git a/tests/helpers/hosted-send-fixture.ts b/tests/helpers/hosted-send-fixture.ts new file mode 100644 index 00000000000..040997aea9e --- /dev/null +++ b/tests/helpers/hosted-send-fixture.ts @@ -0,0 +1,94 @@ +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import type { OcxConfig, OcxProviderConfig } from "../../src/types"; + +// providerFetch supports executor injection; persisted provider configuration does not expose it. +type SyntheticConfig = Omit & { providers: Record }; +import type { RequestLogContext } from "../../src/server/request-log"; +import { handleResponses } from "../../src/server/responses/core"; +import { createRequestExecutionBudget, type RequestExecutionBudgetPolicy } from "../../src/lib/request-execution-budget"; +import { createRequestSpendTracker } from "../../src/server/responses/request-spend"; +import { sharedSpendLedger, resetSharedSpendLedgerForTest } from "../../src/lib/spend-reservation-ledger"; +import { spendLedgerOwnerSnapshot } from "../../src/lib/spend-ledger-owner"; +import { expect } from "bun:test"; +import { acquireOwnedSpendHome } from "./owned-spend-home"; +import { installIsolatedCodexHome } from "./isolated-codex-home"; +import { removeTreeWithRetry } from "./remove-tree"; +import { clearComboSelectionState, clearComboTargetCooldowns } from "../../src/combos"; +import { closeRequestHistoryIndex } from "../../src/routing/history/indexer"; +import { clearResponseStateForTests, flushResponseState } from "../../src/responses/state"; +import { chatStream } from "./combo-failover-upstream"; + +export type HostedKind = "search" | "image" | "video"; +/** Isolate hosted inference and restore all state before asserting owner release. */ +export async function hostedSendFixture(kind: HostedKind, strategy: "failover" | "jev", run: (f: ReturnType) => Promise, policy?: RequestExecutionBudgetPolicy) { + const prior = process.env.OPENCODEX_HOME; + const home = mkdtempSync(join(tmpdir(), "hosted-send-")); + process.env.OPENCODEX_HOME = home; + const codex = installIsolatedCodexHome("hosted-send-codex-"); + const release = acquireOwnedSpendHome(); + try { + clearComboSelectionState(); clearComboTargetCooldowns(); + await run(createFixture(kind, strategy, policy)); + } finally { + let ownership: ReturnType["ownership"] | undefined; + try { + release(); ownership = spendLedgerOwnerSnapshot().ownership; + closeRequestHistoryIndex(); await flushResponseState(); clearResponseStateForTests(); + } finally { + clearComboSelectionState(); clearComboTargetCooldowns(); codex.restore(); + if (prior === undefined) delete process.env.OPENCODEX_HOME; else process.env.OPENCODEX_HOME = prior; + removeTreeWithRetry(home); + } + expect(ownership).toBe("unheld"); + } +} +/** Build synthetic Combo dispatch with observable physical sends and durable spend settlement. */ +function createFixture(kind: HostedKind, strategy: "failover" | "jev", policy?: RequestExecutionBudgetPolicy) { + let inference = 0, judge = 0, charges = 0, refunds = 0; + const bodies: Record[] = []; + const config: SyntheticConfig = { port: 0, defaultProvider: "a", providers: { + a: { adapter: "openai-chat", baseUrl: "https://synthetic.invalid/v1", apiKey: "synthetic-key", authMode: "key", liveModels: false, + models: ["m"], reasoningEfforts: ["low", "high"], transientRetryOn5xx: { attempts: 1 }, + fetch: (async (_input: unknown, init?: RequestInit) => { + inference++; bodies.push(JSON.parse(String(init?.body))); return chatStream("hosted route final answer"); + }) as unknown as typeof fetch }, + jev: { adapter: "jev-decision", baseUrl: "https://api.typesafe.ai/v1/systemone", apiKey: "synthetic-judge-key", liveModels: false, + fetch: (async () => { judge++; return Response.json({ answers: { route: { choice: "a/m:low" } } }); }) as unknown as typeof fetch }, + xai: { adapter: "openai-chat", baseUrl: "https://api.x.ai/v1", apiKey: "synthetic-unused-key", authMode: "key", liveModels: false, + fetch: (async () => { throw new Error("Synthetic test must not execute a media service"); }) as unknown as typeof fetch }, + }, combos: { auto: { strategy, targets: [{ provider: "a", model: "m" }] } }, + ...(kind === "search" ? { webSearchSidecar: { backend: "exa" as const, exaApiKey: "synthetic-unused-key" } } + : { images: kind === "image" ? { bridgeEnabled: true } : { videoBridgeEnabled: true } }) }; + const log: RequestLogContext = { model: "", provider: "a", spendInputEstimateTokens: 2, spendOutputCeilingTokens: 1 }; + const tracker = createRequestSpendTracker(log, "hosted-send-root"); + const budget = createRequestExecutionBudget(policy, undefined, { + charge: options => { charges++; return tracker.charge(options); }, + refund: () => { refunds++; tracker.refund(); }, + }); + const request = (signal?: AbortSignal) => new Request("http://localhost/v1/responses", { method: "POST", signal, + headers: { "content-type": "application/json" }, body: JSON.stringify({ model: "combo/auto", input: "Synthetic assignment", stream: true, + ...(kind === "video" ? {} : { tools: [{ type: kind === "search" ? "web_search" : "image_generation" }] }) }) }); + return { config, budget, log, bodies, request, + async dispatch(signal?: AbortSignal) { const req = request(signal); return handleResponses(req, config, log, { sendBudget: budget, abortSignal: req.signal }); }, + async dispatchChild(signal?: AbortSignal) { + const booking = budget.reserveDispatch({ sendClass: "initial", targetKey: "a/m", countedExternally: true }); + if (!booking.allowed) throw new Error("fixture initial booking denied"); + const req = request(signal); + const body = await req.json() as Record; + body.model = "a/m"; + const child = new Request(req.url, { method: "POST", headers: req.headers, body: JSON.stringify(body), signal }); + return handleResponses(child, config, log, { sendBudget: budget, abortSignal: child.signal, + comboAttempt: true, comboInitialSend: { permit: booking.permit } }); + }, + settle() { + tracker.settle({ inputTokens: 2, outputTokens: 1 }); + const snapshot = sharedSpendLedger().snapshot("root", "hosted-send-root"); + resetSharedSpendLedgerForTest(); // Prove settlement by replaying the durable journal, not only memory. + expect(sharedSpendLedger().snapshot("root", "hosted-send-root")).toEqual(snapshot); + return snapshot; + }, + get inference() { return inference; }, get judge() { return judge; }, get charges() { return charges; }, get refunds() { return refunds; }, + }; +} diff --git a/tests/helpers/responses-core-source.ts b/tests/helpers/responses-core-source.ts index 42052227e2e..968b98890db 100644 --- a/tests/helpers/responses-core-source.ts +++ b/tests/helpers/responses-core-source.ts @@ -52,6 +52,7 @@ export const RESPONSES_CORE_MODULES = [ "terminal-error-redaction.ts", "non-replayable-error.ts", "sidecar-execution.ts", + "sidecar-send-budget.ts", "completion-policy.ts", "run-turn-execution.ts", "adapter-dispatch.ts", diff --git a/tests/responses/responses-dispatch-receipt.test.ts b/tests/responses/responses-dispatch-receipt.test.ts new file mode 100644 index 00000000000..53cf5bc62c5 --- /dev/null +++ b/tests/responses/responses-dispatch-receipt.test.ts @@ -0,0 +1,126 @@ +import { afterEach, expect, test } from "bun:test"; +import { providerFetch, sendWithConnectionPolicy } from "../../src/server/responses/fetch-helpers"; +import { codexWsExchange } from "../../src/server/responses/codex-ws-exchange"; +import { CodexWsSession } from "../../src/server/responses/codex-ws-session"; +import { prepareCodexWsRequest } from "../../src/server/responses/codex-ws-request"; +import { createRequestExecutionBudget } from "../../src/lib/request-execution-budget"; +import { streamingInit } from "../helpers/ws-upstream-fixtures"; + +const realWebSocket = globalThis.WebSocket; +afterEach(() => { globalThis.WebSocket = realWebSocket; }); + +for (const nested of [false, true]) test(`HTTP receipt follows rebuilt executor and fires once (${nested})`, async () => { + let receipts = 0, sends = 0; + const physical = (async (input) => { + sends++; + expect(String(input)).toBe("https://fixture.invalid/rebuilt"); + expect(receipts).toBe(1); + return new Response("fixture"); + }) as typeof fetch; + const fetcher = providerFetch(Object.assign({ adapter: "openai-responses", baseUrl: "https://fixture.invalid" }, { fetch: physical }), "1.3.14", { + onPhysicalDispatch: () => { receipts++; }, + dispatchOverride: async (_input, init, execute) => sendWithConnectionPolicy( + nested ? execute : physical, "https://fixture.invalid/rebuilt", { ...init }), + }); + expect((await fetcher("https://fixture.invalid/original", { method: "POST", body: "{}" })).status).toBe(200); + expect(sends).toBe(1); expect(receipts).toBe(1); +}); + +test("HTTP local dispatch refusal leaves receipt and physical executor untouched", async () => { + let receipts = 0, sends = 0; + const fetcher = providerFetch(Object.assign({ adapter: "openai-responses", baseUrl: "https://fixture.invalid" }, { fetch: (async () => { + sends++; return new Response("unexpected"); + }) as unknown as typeof fetch }), "1.3.14", { + onPhysicalDispatch: () => { receipts++; }, + dispatchOverride: async () => { throw new Error("fixture local refusal"); }, + }); + await expect(fetcher("https://fixture.invalid/refused", { method: "POST", body: "{}" })).rejects.toThrow("fixture local refusal"); + expect(receipts).toBe(0); expect(sends).toBe(0); +}); + +class FakeSocket { + static latest: FakeSocket; + static onSend: ((data: string) => void) | undefined; + listeners = new Map void>>(); + sent: string[] = []; + constructor() { FakeSocket.latest = this; queueMicrotask(() => this.emit("open", {})); } + addEventListener(name: string, fn: (event: unknown) => void) { + if (!this.listeners.has(name)) this.listeners.set(name, new Set()); + this.listeners.get(name)!.add(fn); + } + removeEventListener(name: string, fn: (event: unknown) => void) { this.listeners.get(name)?.delete(fn); } + emit(name: string, event: unknown) { for (const fn of this.listeners.get(name) ?? []) fn(event); } + send(data: string) { FakeSocket.onSend?.(data); this.sent.push(data); } + close() {} +} + +afterEach(() => { FakeSocket.onSend = undefined; }); + +test("WebSocket receipt refusal before send rejects without fallback", async () => { + globalThis.WebSocket = FakeSocket as unknown as typeof WebSocket; + const url = "https://chatgpt.com/backend-api/codex/responses", init = streamingInit(); + const prepared = prepareCodexWsRequest(url, init)!; + const session = new CodexWsSession("wss://chatgpt.com/backend-api/codex/responses", prepared.headers, false); + let fallbacks = 0, receipts = 0; + try { + expect(session.reserve()).toBe(true); + await expect(codexWsExchange({ session, url, init, prepared, + sseFallback: (async () => { fallbacks++; return new Response("unexpected"); }) as unknown as typeof fetch, + onPhysicalDispatch: () => { receipts++; throw new Error("fixture receipt failure"); }, + })).rejects.toThrow("fixture receipt failure"); + expect(FakeSocket.latest.sent).toHaveLength(0); expect(receipts).toBe(1); expect(fallbacks).toBe(0); + expect([...FakeSocket.latest.listeners.values()].every(set => set.size === 0)).toBe(true); + } finally { session.dispose(); } +}); + +const wsUrl = "https://chatgpt.com/backend-api/codex/responses"; +const wsProvider = (physical: typeof fetch) => Object.assign({ adapter: "openai-responses", baseUrl: wsUrl }, { fetch: physical }); + +test("providerFetch WS send failure shares one receipt with HTTP fallback", async () => { + globalThis.WebSocket = FakeSocket as unknown as typeof WebSocket; + let receipts = 0, httpSends = 0, receiptAtSend = -1; + FakeSocket.onSend = () => { receiptAtSend = receipts; expect(receipts).toBe(1); throw new Error("fixture socket send failure"); }; + const fetcher = providerFetch(wsProvider((async () => { + httpSends++; + expect(receipts).toBe(1); + return new Response("fallback"); + }) as typeof fetch), "1.4.0", { onPhysicalDispatch: () => { receipts++; } }); + expect(await (await fetcher(wsUrl, streamingInit())).text()).toBe("fallback"); + expect(httpSends).toBe(1); expect(receipts).toBe(1); expect(receiptAtSend).toBe(1); +}); + +test("normal WebSocket frame receives exactly one receipt", async () => { + globalThis.WebSocket = FakeSocket as unknown as typeof WebSocket; + let receipts = 0, httpSends = 0; + FakeSocket.onSend = () => { + expect(receipts).toBe(1); + queueMicrotask(() => FakeSocket.latest.emit("message", { data: JSON.stringify({ type: "response.completed", response: { id: "r1", status: "completed" } }) })); + }; + const fetcher = providerFetch(wsProvider((async () => { httpSends++; return new Response("unexpected"); }) as typeof fetch), "1.4.0", { + onPhysicalDispatch: () => { receipts++; }, + }); + expect(await (await fetcher(wsUrl, streamingInit())).text()).toContain("response.completed"); + expect(receipts).toBe(1); expect(httpSends).toBe(0); +}); + +test("D7: accepted WS receipt remains charged if send and local HTTP admission both fail", async () => { + globalThis.WebSocket = FakeSocket as unknown as typeof WebSocket; + let receipts = 0, refunds = 0, httpSends = 0, overrides = 0; + const budget = createRequestExecutionBudget(undefined, undefined, { + charge: () => true, refund: () => { refunds++; }, + }); + const booking = budget.reserveDispatch({ sendClass: "initial", targetKey: "chatgpt/codex", countedExternally: true }); + if (!booking.allowed) throw new Error("fixture booking denied"); + FakeSocket.onSend = () => { expect(receipts).toBe(1); throw new Error("fixture socket send failure"); }; + const fetcher = providerFetch(wsProvider((async () => { + httpSends++; + return new Response("unexpected"); + }) as typeof fetch), "1.4.0", { + onPhysicalDispatch: () => { receipts++; expect(booking.permit.use()).toBe(true); }, + dispatchOverride: async () => { overrides++; throw new Error("fixture local HTTP refusal"); }, + }); + await expect(fetcher(wsUrl, streamingInit())).rejects.toThrow("fixture local HTTP refusal"); + booking.permit.release(); + expect(overrides).toBe(1); expect(httpSends).toBe(0); expect(receipts).toBe(1); + expect(refunds).toBe(0); expect(budget.used).toBe(1); +}); diff --git a/tests/responses/responses-first-send-reservation.test.ts b/tests/responses/responses-first-send-reservation.test.ts new file mode 100644 index 00000000000..d31dde87a7c --- /dev/null +++ b/tests/responses/responses-first-send-reservation.test.ts @@ -0,0 +1,418 @@ +import { afterEach, beforeEach, expect, spyOn, test } from "bun:test"; +import { ADAPTER_REGISTRY } from "../../src/adapters/registry"; +import { createNativeChatComboSource } from "../../src/server/chat-native"; +import { createProtocolEnvelope } from "../../src/protocols/envelope"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { handleResponses } from "../../src/server/responses/core"; +import { clearComboSelectionState, clearComboTargetCooldowns } from "../../src/combos"; +import { closeRequestHistoryIndex } from "../../src/routing/history/indexer"; +import { clearResponseStateForTests, flushResponseState } from "../../src/responses/state"; +import { clearKeyCooldowns } from "../../src/providers/key-failover"; +import { resetProviderRequestPacingForTest } from "../../src/providers/request-pacing"; +import { getActiveTurnCount } from "../../src/server/lifecycle"; +import { createRequestExecutionBudget, deriveRequestExecutionBudget, type RequestSendObserver } from "../../src/lib/request-execution-budget"; +import { createResponsesSendBudget, transientSendCapFor } from "../../src/server/responses/request-send-budget"; +import { createTranslatorBudget } from "../../src/lib/translator-budget"; +import { SendBudgetExhaustedError } from "../../src/lib/upstream-retry"; +import { sharedSpendLedger } from "../../src/lib/spend-reservation-ledger"; +import { comboExecutionBudgetPolicy } from "../../src/server/responses/core-combo"; +import type { RequestLogContext } from "../../src/server/request-log"; +import type { OcxConfig, OcxProviderConfig } from "../../src/types"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; +import { installIsolatedCodexHome, type IsolatedCodexHome } from "../helpers/isolated-codex-home"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; +import { responsesSuccess, chatSuccess } from "../helpers/combo-failover-upstream"; + +let home: string, prior: string | undefined, codex: IsolatedCodexHome, release: () => void; +const servers: Array> = []; +beforeEach(() => { + prior = process.env.OPENCODEX_HOME; + home = mkdtempSync(join(tmpdir(), "first-send-reservation-")); + process.env.OPENCODEX_HOME = home; + codex = installIsolatedCodexHome("first-send-codex-"); + release = acquireOwnedSpendHome(); + clearComboSelectionState(); clearComboTargetCooldowns(); clearKeyCooldowns(); +}); +afterEach(async () => { + release(); + for (const server of servers.splice(0)) await server.stop(true); + closeRequestHistoryIndex(); await flushResponseState(); clearResponseStateForTests(); + clearComboSelectionState(); clearComboTargetCooldowns(); clearKeyCooldowns(); + resetProviderRequestPacingForTest(); + if (prior === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = prior; + codex.restore(); removeTreeWithRetry(home); +}); + +/** Capture physical inference bodies on a loopback server with a fixed response status. */ +function upstream(status = 200) { + const bodies: Record[] = []; + const server = Bun.serve({ hostname: "127.0.0.1", port: 0, async fetch(req) { + bodies.push(await req.json() as Record); + return status === 200 ? Response.json(responsesSuccess("fixture answer", "m")) + : Response.json({ error: { message: "fixture unavailable" } }, { status }); + } }); + servers.push(server); + return { bodies, baseUrl: new URL("/v1", server.url).href }; +} +/** Configure the synthetic Responses provider with an optional transient-send ceiling. */ +function provider(baseUrl: string, attempts: number | undefined = 1): OcxProviderConfig { + return { adapter: "openai-responses", baseUrl, allowPrivateNetwork: true, + apiKey: "fixture-inference-key", authMode: "key", liveModels: false, models: ["m"], + reasoningEfforts: ["low", "high"], ...(attempts === undefined ? {} : { transientRetryOn5xx: { attempts } }) }; +} +/** Observe shared charges and refunds independently of synthetic judge calls. */ +function fixture(strategy: "jev" | "failover" | "direct", status = 200, attempts: number | undefined = 1) { + const inference = upstream(status); + let judgeCalls = 0; + const config: OcxConfig = { port: 0, defaultProvider: "a", providers: { a: provider(inference.baseUrl, attempts) }, + ...(strategy === "direct" ? {} : { combos: { auto: { strategy, targets: [{ provider: "a", model: "m" }] } } }) }; + if (strategy === "jev") config.providers.jev = Object.assign({ adapter: "jev-decision", liveModels: false, + baseUrl: "https://api.typesafe.ai/v1/systemone", apiKey: "fixture-service-key" }, { + fetch: (async () => { judgeCalls++; return Response.json({ answers: { route: { choice: "a/m:low" } } }); }) as unknown as typeof fetch }); + let charges = 0, refunds = 0; + const observer: RequestSendObserver = { charge: () => { charges++; return true; }, refund: () => { refunds++; } }; + const budget = createRequestExecutionBudget(undefined, undefined, observer); + return { config, inference, budget, get judgeCalls() { return judgeCalls; }, get charges() { return charges; }, get refunds() { return refunds; } }; +} +/** Dispatch and consume one synthetic request through the real Responses handler. */ +async function send(f: ReturnType, signal?: AbortSignal) { + const log: RequestLogContext = { model: "", provider: "" }; + const req = new Request("http://localhost/v1/responses", { method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: f.config.combos ? "combo/auto" : "a/m", input: "Synthetic reservation task.", stream: false, + reasoning: { effort: "high" } }), ...(signal ? { signal } : {}) }); + const response = await handleResponses(req, f.config, log, { sendBudget: f.budget, abortSignal: req.signal }); + const text = await response.text(); + return { response, log, text }; +} + +for (const strategy of ["jev", "failover", "direct"] as const) { + test(`${strategy}: one-send policy reaches real Responses upstream exactly once`, async () => { + const f = fixture(strategy); + const { response, log } = await send(f); + expect(response.status).toBe(200); expect(f.inference.bodies).toHaveLength(1); + expect(f.budget.used).toBe(1); expect(f.charges).toBe(1); expect(f.refunds).toBe(0); + expect(f.judgeCalls).toBe(strategy === "jev" ? 1 : 0); + if (strategy === "jev") { + expect(log.jevDecision?.selected.effort).toBe("low"); + expect((f.inference.bodies[0]!.reasoning as Record).effort).toBe("low"); + } + }); + test(`${strategy}: configured one-send 5xx cannot buy a retry or final recovery`, async () => { + const f = fixture(strategy, 503); + const { response, text } = await send(f); + expect(response.status).toBe(503); expect(text).toContain("fixture unavailable"); + expect(f.inference.bodies).toHaveLength(1); expect(f.budget.used).toBe(1); + expect(f.charges).toBe(1); expect(f.refunds).toBe(0); + }); + test(`${strategy}: three-send 5xx ladder charges each physical send once`, async () => { + const f = fixture(strategy, 503, 3); + expect((await send(f)).response.status).toBe(503); + expect(f.inference.bodies).toHaveLength(3); expect(f.budget.used).toBe(3); + expect(f.charges).toBe(3); expect(f.refunds).toBe(0); + }, 10_000); +} + +for (const attempts of [1, 3, 10]) { + test(`ordinary Combo: later target keeps its configured allowance under shared ceiling (${attempts})`, async () => { + const f = fixture("failover", 503, attempts); + const second = upstream(503), third = upstream(503); + f.config.providers.b = provider(second.baseUrl, attempts); f.config.providers.c = provider(third.baseUrl, attempts); + f.config.combos!.auto!.targets.push({ provider: "b", model: "m" }, { provider: "c", model: "m" }); + const { response, log } = await send(f); + expect(response.status).toBe(503); + const counts = [f.inference.bodies.length, second.bodies.length, third.bodies.length]; + expect(counts.every(count => count >= 1 && count <= attempts)).toBe(true); + const sent = counts.reduce((sum, count) => sum + count, 0); + expect(sent).toBeLessThanOrEqual(comboExecutionBudgetPolicy(3).maxTotalModelSends); + expect(f.budget.used).toBe(sent); expect(f.charges).toBe(sent); expect(f.refunds).toBe(0); + expect(log.attempts!.map(attempt => attempt.sendCount)).toEqual(counts); + }, 15_000); +} + +test("unconfigured Google Combo ladder respects prepaid allowance and later-target holdback", async () => { + const f = fixture("failover", 503); + const second = upstream(503), third = upstream(503); + for (const [name, baseUrl] of [["a", f.inference.baseUrl], ["b", second.baseUrl], ["c", third.baseUrl]] as const) { + const row = provider(baseUrl); delete row.transientRetryOn5xx; row.adapter = "google"; + f.config.providers[name] = row; + } + f.config.combos!.auto!.targets.push({ provider: "b", model: "m" }, { provider: "c", model: "m" }); + expect((await send(f)).response.status).toBe(503); + const counts = [f.inference.bodies.length, second.bodies.length, third.bodies.length]; + expect(counts).toEqual([3, 2, 1]); + expect(f.budget.used).toBe(comboExecutionBudgetPolicy(3).maxTotalModelSends); + expect(f.charges).toBe(6); expect(f.refunds).toBe(0); +}, 10_000); + +test("later translated Combo target retains configured recovery total", async () => { + const f = fixture("failover", 503, 1); + let sent = 0; + f.config.providers.b = Object.assign({ ...provider("https://fixture.invalid", 2), adapter: "openai-chat", + retryOn429: { attempts: 1, intervalMs: 1, maxIntervalMs: 1 } }, { fetch: (async () => { + sent++; + return sent === 1 ? Response.json({ error: { message: "fixture limited" } }, { status: 429 }) + : chatSuccess("fixture recovery", "m"); + }) as unknown as typeof fetch }); + f.config.combos!.auto!.targets.push({ provider: "b", model: "m" }); + expect((await send(f)).response.status).toBe(200); + expect(sent).toBe(2); expect(f.budget.used).toBe(3); expect(f.charges).toBe(3); +}); + +for (const attempts of [1, 2]) for (const refused of [false, true]) test(`native Chat receipt owns pending booking (attempts=${attempts}, refused=${refused})`, async () => { + const f = fixture("failover", 200, attempts); + let sends = 0; + f.config.providers.a = Object.assign({ ...f.config.providers.a!, adapter: "openai-chat", + ...(refused ? { proxy: "direct" as const } : {}) }, { + fetch: (async () => { + sends++; + return attempts === 2 && sends === 1 ? Response.json({ error: { message: "fixture transient" } }, { status: 503 }) + : chatSuccess("fixture native", "m"); + }) as unknown as typeof fetch }); + f.config.protocols = { rollout: { nativeChatCombos: true } }; + const other = f.budget.reserveDispatch({ sendClass: "initial", targetKey: "unrelated", countedExternally: true }); + if (!other.allowed) throw new Error("fixture refused"); + const translatorBudget = createTranslatorBudget(); + const body = { model: "combo/auto", messages: [{ role: "user", content: "fixture" }], stream: false }; + const req = new Request("http://localhost/v1/responses", { method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "combo/auto", input: "fixture", stream: false }) }); + const protocolSource = createNativeChatComboSource({ req, config: f.config, + envelope: createProtocolEnvelope({ inbound: "chat", body, translatorBudget }), + requestedModel: "combo/auto", requestedStream: false, translatorBudget }); + try { + const response = await handleResponses(req, f.config, { model: "", provider: "" }, { + sendBudget: f.budget, translatorBudget, protocolSource }); + expect(response.ok).toBe(!refused); await response.text(); expect(sends).toBe(refused ? 0 : attempts); + expect(f.budget.used).toBe(refused ? 1 : attempts + 1); expect(f.charges).toBe(refused ? 2 : attempts + 1); + expect(f.refunds).toBe(refused ? 1 : 0); + other.permit.release(); expect(f.budget.used).toBe(refused ? 0 : attempts); expect(f.refunds).toBe(refused ? 2 : 1); + } finally { other.permit.release(); translatorBudget.dispose(); } +}); + +test("pre-dispatch body refusal refunds prepaid reservation and observer charge", async () => { + const f = fixture("failover"); f.config.maxUpstreamBodyBytes = 1; + expect((await send(f)).response.status).toBe(413); + expect(f.inference.bodies).toHaveLength(0); expect(f.budget.used).toBe(0); + expect(f.charges).toBe(1); expect(f.refunds).toBe(1); +}); + +test("missing credential refusal sends nothing and refunds reservation", async () => { + const f = fixture("failover"); + delete f.config.providers.a!.apiKey; + const { response } = await send(f); + expect(response.ok).toBe(false); expect(f.inference.bodies).toHaveLength(0); + expect(f.budget.used).toBe(0); expect(f.charges).toBe(f.refunds); +}); + +test("caller abort during queued dispatch refunds unsent booking", async () => { + const f = fixture("failover"); + const controller = new AbortController(); + f.config.providers.a!.requestPacing = { enabled: true, minIntervalMs: 1000 }; + // Prime pacing with a separate logical request; this turn owns only its unsent booking. + expect((await send({ ...f, budget: createRequestExecutionBudget() })).response.status).toBe(200); + const before = f.budget.used; + const timer = setTimeout(() => controller.abort(), 40); + try { + expect((await send(f, controller.signal)).response.status).toBe(499); + expect(f.inference.bodies).toHaveLength(1); expect(f.budget.used).toBe(before); + expect(f.charges).toBe(1); expect(f.refunds).toBe(1); + } finally { clearTimeout(timer); } +}); + +test("detached Responses model judge keeps its own one-send booking and lease", async () => { + const f = fixture("jev"); + delete f.config.providers.jev; + const judge = upstream(); + let judgeSends = 0; + // A model judge needs the choice in its output text rather than in the service envelope. + f.config.providers.judge = Object.assign(provider(judge.baseUrl), { fetch: (async () => { + judgeSends++; + expect(getActiveTurnCount()).toBe(turns + 1); + return Response.json(responsesSuccess('{"choice":"a/m:low"}', "m")); + }) as unknown as typeof fetch }); + f.config.combos!.auto!.decisionModel = "judge/m"; + const turns = getActiveTurnCount(); + const { response, log } = await send(f); + expect(response.status).toBe(200); expect(f.inference.bodies).toHaveLength(1); + expect(log.jevDecision).toMatchObject({ backend: "model", gate: "apply" }); + expect(f.budget.used).toBe(1); expect(f.charges).toBe(1); expect(judgeSends).toBe(1); + expect(sharedSpendLedger().snapshot("pool", "judge")).toMatchObject({ reserved: 0, settled: 3, unresolved: 0 }); + expect(getActiveTurnCount()).toBe(turns); +}); + +for (const abort of [false, true]) test(`translated Chat Combo consumes or refunds prepaid initial send (${abort})`, async () => { + const f = fixture("failover"); + let sent = 0; + f.config.providers.a = Object.assign({ ...f.config.providers.a!, adapter: "openai-chat", + requestPacing: { enabled: true, minIntervalMs: 1000 } }, { + fetch: (async () => { sent++; return chatSuccess("fixture translated answer", "m"); }) as unknown as typeof fetch }); + if (!abort) { + expect((await send(f)).response.status).toBe(200); + expect(sent).toBe(1); expect(f.budget.used).toBe(1); expect(f.charges).toBe(1); expect(f.refunds).toBe(0); + return; + } + expect((await send({ ...f, budget: createRequestExecutionBudget() })).response.status).toBe(200); + const controller = new AbortController(), timer = setTimeout(() => controller.abort(), 40); + try { + expect((await send(f, controller.signal)).response.status).toBe(499); + expect(sent).toBe(1); expect(f.budget.used).toBe(0); expect(f.charges).toBe(1); expect(f.refunds).toBe(1); + } finally { clearTimeout(timer); } +}); + +test("streaming runTurn producer retains prepaid send after ingress returns", async () => { + const f = fixture("failover"); + f.config.providers.a = { ...f.config.providers.a!, adapter: "openai-chat" }; + let finish!: () => void, sent!: () => void; + const gate = new Promise(resolve => { finish = resolve; }); + const dispatched = new Promise(resolve => { sent = resolve; }); + const adapter = spyOn(ADAPTER_REGISTRY["openai-chat"], "create").mockImplementation(() => ({ + name: "fixture-runturn", reportsPhysicalSends: true, + buildRequest: () => ({ url: f.inference.baseUrl, method: "POST", headers: {}, body: "{}" }), + async *parseStream() { yield { type: "done" as const }; }, + async runTurn(_parsed, incoming, emit) { + emit({ type: "text_delta", text: "fixture preliminary output" }); + await gate; + const booking = incoming.sendBudget!.reserveDispatch({ sendClass: "initial", targetKey: "fixture-endpoint" }); + expect(booking.allowed).toBe(true); if (!booking.allowed) throw new Error("prepaid send lost"); + expect(booking.permit.use()).toBe(true); + incoming.onPhysicalSend?.({ ordinal: 1 }); sent(); + emit({ type: "done" }); + }, + })); + try { + const req = new Request("http://localhost/v1/responses", { method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "combo/auto", input: "fixture", stream: true }) }); + const response = await handleResponses(req, f.config, { model: "", provider: "" }, { sendBudget: f.budget }); + expect(response.status).toBe(200); expect(f.budget.used).toBe(1); expect(f.refunds).toBe(0); + finish(); await dispatched; await response.text(); + expect(f.budget.used).toBe(1); expect(f.charges).toBe(1); expect(f.refunds).toBe(0); + } finally { finish(); adapter.mockRestore(); } +}); + +/** Construct a child-local owner over one externally counted shared reservation. */ +function prepaidOwner() { + let charges = 0, refunds = 0; + const policy = { maxTotalModelSends: 3, baseSendAllowance: 3, finalRecoveryAllowance: 0, + maxAlternateTargetSends: 1, maxTargetTransitions: 1 }; + const budget = createRequestExecutionBudget(policy, undefined, { + charge: () => { charges++; return true; }, refund: () => { refunds++; }, + }); + const target = deriveRequestExecutionBudget(budget, policy); + const booking = budget.reserveDispatch({ sendClass: "initial", targetKey: "a/m", countedExternally: true }); + if (!booking.allowed) throw new Error("fixture reservation refused"); + const translatorBudget = createTranslatorBudget(); + const owner = createResponsesSendBudget({ req: new Request("http://localhost/v1/responses"), + logCtx: { provider: "a", model: "m" }, options: { translatorBudget, sendBudget: target, + comboInitialSend: { permit: booking.permit } } }); + if (owner instanceof Response) throw new Error("unexpected workflow refusal"); + return { budget, target, owner, permit: booking.permit, dispose: () => translatorBudget.dispose(), + get charges() { return charges; }, get refunds() { return refunds; } }; +} + +test("prepaid helper exposes only child's booking, never unrelated pending capacity", () => { + const f = prepaidOwner(); + try { + const other = f.budget.reserveDispatch({ sendClass: "transient", targetKey: "a/m", countedExternally: true }); + expect(other.allowed).toBe(true); + expect(f.budget.used).toBe(2); expect(f.target.remainingBaseSends(10)).toBe(1); + expect(f.owner.initialSendAllowance(10)).toBe(2); + expect(f.owner.initialSendAllowance(1)).toBe(1); + f.owner.noteInitialDispatch(); + expect(f.budget.used).toBe(2); expect(f.charges).toBe(2); expect(f.refunds).toBe(0); + f.permit.release(); expect(f.budget.used).toBe(2); + if (other.allowed) other.permit.release(); + expect(f.budget.used).toBe(1); expect(f.refunds).toBe(1); + expect(transientSendCapFor(1, f.owner.targetSendsUsed)).toBe(0); + } finally { f.dispose(); } +}); + +for (const transport of ["receipt", "adapter", "reporter"] as const) test(`target-local ${transport} sends ignore later bookings by another owner`, () => { + const f = prepaidOwner(); + try { + const other = f.budget.reserveDispatch({ sendClass: "transient", targetKey: "a/m", countedExternally: true }); + if (!other.allowed) throw new Error("fixture refused"); + expect(f.owner.targetSendsUsed).toBe(0); + if (transport === "receipt") f.owner.noteInitialDispatch(); + else if (transport === "reporter") f.owner.noteTransientSends(1); + else { + const first = f.owner.adapterDispatchBudget!.reserveDispatch({ sendClass: "initial", targetKey: "endpoint-one" }); + if (!first.allowed || !first.permit.use()) throw new Error("fixture dispatch denied"); + f.owner.noteAdapterPhysicalSend(undefined, { ordinal: 1 }); + } + expect(f.owner.targetSendsUsed).toBe(1); + expect(transientSendCapFor(2, f.owner.targetSendsUsed)).toBe(1); + expect(f.owner.sendsUsed).toBe(2); // Both owners still occupy shared admission capacity. + if (transport === "receipt") f.owner.noteInitialDispatch(); + else if (transport === "reporter") f.owner.noteTransientSends(1); + else { + const retry = f.owner.adapterDispatchBudget!.reserveDispatch({ sendClass: "transient", targetKey: "endpoint-one" }); + if (!retry.allowed || !retry.permit.use()) throw new Error("fixture retry denied"); + f.owner.noteAdapterPhysicalSend(undefined, { ordinal: 2 }); + } + expect(f.owner.targetSendsUsed).toBe(2); expect(transientSendCapFor(2, f.owner.targetSendsUsed)).toBe(0); + other.permit.release(); + expect(f.owner.targetSendsUsed).toBe(2); + } finally { f.dispose(); } +}); + +test("initial-ladder retry reserves its own send without settling another pending booking", () => { + const f = prepaidOwner(); + try { + const other = f.budget.reserveDispatch({ sendClass: "transient", targetKey: "a/m", countedExternally: true }); + if (!other.allowed) throw new Error("fixture refused"); + expect(f.owner.initialSendAllowance(3)).toBe(2); + f.owner.noteInitialDispatch(); f.owner.noteInitialDispatch(); + expect(f.budget.used).toBe(3); expect(f.charges).toBe(3); + expect(() => f.owner.noteInitialDispatch()).toThrow(SendBudgetExhaustedError); + other.permit.release(); expect(f.budget.used).toBe(2); expect(f.refunds).toBe(1); + } finally { f.dispose(); } +}); + +test("released prepaid permit cannot dispatch or be refunded twice", () => { + const f = prepaidOwner(); + try { + f.permit.release(); f.permit.release(); + expect(() => f.owner.noteInitialDispatch()).toThrow(SendBudgetExhaustedError); + expect(f.budget.used).toBe(0); expect(f.charges).toBe(1); expect(f.refunds).toBe(1); + } finally { f.dispose(); } +}); + +for (const dispatch of [false, true]) test(`adapter claims prepaid send once and preserves endpoint transition bound (${dispatch})`, () => { + const f = prepaidOwner(); + try { + const adapter = f.owner.adapterDispatchBudget!; + const first = adapter.reserveDispatch({ sendClass: "transient", targetKey: "endpoint-one" }); + expect(first.allowed).toBe(true); if (!first.allowed) throw new Error("fixture refused"); + expect(f.budget.used).toBe(1); expect(f.charges).toBe(1); + if (!dispatch) { + first.permit.release(); first.permit.release(); + expect(f.budget.used).toBe(0); expect(f.refunds).toBe(1); + expect(first.permit.use()).toBe(false); return; + } + expect(first.permit.use()).toBe(true); expect(first.permit.use()).toBe(false); + first.permit.release(); expect(f.refunds).toBe(0); + const second = adapter.reserveDispatch({ sendClass: "transient", targetKey: "endpoint-two" }); + expect(second.allowed).toBe(true); if (second.allowed) expect(second.permit.use()).toBe(true); + expect(adapter.targetTransitions).toBe(1); + expect(adapter.reserveDispatch({ sendClass: "transient", targetKey: "endpoint-three" })).toMatchObject({ + allowed: false, reason: "target-transition-exhausted" }); + f.owner.noteTransientSends(1); // No external booking remains to swallow this unrelated send. + expect(f.budget.used).toBe(3); expect(f.charges).toBe(3); + } finally { f.dispose(); } +}); + +test("released initial booking followed by a transient report charges exactly one send", () => { + const f = prepaidOwner(); + try { + f.permit.release(); + expect(f.budget.used).toBe(0); expect(f.charges - f.refunds).toBe(0); + f.owner.noteTransientSends(1); + expect(f.budget.used).toBe(1); expect(f.charges - f.refunds).toBe(1); + expect(f.charges).toBe(2); expect(f.refunds).toBe(1); + expect(f.owner.targetSendsUsed).toBe(1); + f.permit.release(); + expect(f.budget.used).toBe(1); expect(f.refunds).toBe(1); + } finally { f.dispose(); } +}); diff --git a/tests/responses/responses-first-send-spend-policy.test.ts b/tests/responses/responses-first-send-spend-policy.test.ts new file mode 100644 index 00000000000..5c296a7038f --- /dev/null +++ b/tests/responses/responses-first-send-spend-policy.test.ts @@ -0,0 +1,128 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { clearComboSelectionState, clearComboTargetCooldowns } from "../../src/combos"; +import { closeRequestHistoryIndex } from "../../src/routing/history/indexer"; +import { clearResponseStateForTests, flushResponseState } from "../../src/responses/state"; +import { clearKeyCooldowns } from "../../src/providers/key-failover"; +import { resetProviderRequestPacingForTest } from "../../src/providers/request-pacing"; +import { createPhysicalSendReporter, createRequestExecutionBudget } from "../../src/lib/request-execution-budget"; +import { createSpendReservationLedger, DEFAULT_SPEND_RESERVATION_POLICY, type SpendJournal, type SpendReservationPolicy } from "../../src/lib/spend-reservation-ledger"; +import { SendBudgetExhaustedError, fetchWithTransientRetry } from "../../src/lib/upstream-retry"; +import { createTranslatorBudget } from "../../src/lib/translator-budget"; +import { createRequestSpendTracker } from "../../src/server/responses/request-spend"; +import { createResponsesSendBudget } from "../../src/server/responses/request-send-budget"; +import { handleResponses } from "../../src/server/responses/core"; +import type { RequestLogContext } from "../../src/server/request-log"; +import type { OcxConfig } from "../../src/types"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; +import { installIsolatedCodexHome, type IsolatedCodexHome } from "../helpers/isolated-codex-home"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; +import { responsesSuccess } from "../helpers/combo-failover-upstream"; + +/** + * A Combo child decides once per request whether its prepaid first send is settled by the + * physical-dispatch receipt (spend enforcement inactive) or by the shared physical-send + * reporter (enforcement active). These pin what that decision relies on. + */ +let home: string, prior: string | undefined, codex: IsolatedCodexHome, release: () => void; +const servers: Array> = []; +beforeEach(() => { + prior = process.env.OPENCODEX_HOME; + home = mkdtempSync(join(tmpdir(), "first-send-spend-policy-")); + process.env.OPENCODEX_HOME = home; + codex = installIsolatedCodexHome("first-send-spend-policy-codex-"); + release = acquireOwnedSpendHome(); + clearComboSelectionState(); clearComboTargetCooldowns(); clearKeyCooldowns(); +}); +afterEach(async () => { + release(); + for (const server of servers.splice(0)) await server.stop(true); + closeRequestHistoryIndex(); await flushResponseState(); clearResponseStateForTests(); + clearComboSelectionState(); clearComboTargetCooldowns(); clearKeyCooldowns(); + resetProviderRequestPacingForTest(); + if (prior === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = prior; + codex.restore(); removeTreeWithRetry(home); +}); + +const memoryJournal = (): SpendJournal => { + const lines: string[] = []; + return { read: () => [...lines], append: line => { lines.push(line); }, rewrite: next => { lines.splice(0, lines.length, ...next); } }; +}; +const policyWith = (poolMaxTokens: number | undefined): SpendReservationPolicy => ({ + ...DEFAULT_SPEND_RESERVATION_POLICY, pool: poolMaxTokens === undefined ? {} : { maxTokens: poolMaxTokens }, +}); +/** A real tracker over an injected ledger, so enforcement follows the ledger's live policy. */ +function trackerFixture(enforced: boolean) { + const ledger = createSpendReservationLedger({ journal: memoryJournal(), salt: "first-send-policy", policy: policyWith(enforced ? 1_000_000 : undefined) }); + const log: RequestLogContext = { model: "m", provider: "a", spendPoolId: "a", usageLogInputTokens: 10, spendOutputCeilingTokens: 0 }; + const tracker = createRequestSpendTracker(log, undefined, ledger); + const budget = createRequestExecutionBudget(undefined, undefined, tracker); + return { ledger, log, tracker, budget }; +} + +for (const enforced of [false, true]) test(`receipt-versus-reporter choice is fixed by the Combo booking (enforced=${enforced})`, () => { + const f = trackerFixture(enforced); + const translatorBudget = createTranslatorBudget(); + try { + expect(f.tracker.policyStarted).toBe(false); + const booking = f.budget.reserveDispatch({ sendClass: "initial", targetKey: "a/m", countedExternally: true }); + if (!booking.allowed) throw new Error("fixture reservation refused"); + // The booking itself starts the spend policy; a child holding its permit never sees it unstarted. + expect(f.tracker.policyStarted).toBe(true); + const owner = createResponsesSendBudget({ req: new Request("http://localhost/v1/responses"), logCtx: f.log, + options: { translatorBudget, sendBudget: f.budget, comboInitialSend: { permit: booking.permit } } }); + if (owner instanceof Response) throw new Error("unexpected workflow refusal"); + expect(owner.adapterSendBudget?.spendEnforced).toBe(enforced); + // Neither a later ceiling change nor a later account/pool label can flip this request's mode, + // so the once-per-request receiptMode in the dispatch owners cannot disagree with dispatch. + f.ledger.reconfigure(policyWith(enforced ? undefined : 1_000_000)); + f.log.spendPoolId = "other-pool"; f.log.accountLogLabel = "other-account"; + expect(owner.adapterSendBudget?.spendEnforced).toBe(enforced); + expect(f.budget.spendEnforced).toBe(enforced); + booking.permit.release(); + } finally { translatorBudget.dispose(); } +}); + +/** Combo -> Responses passthrough upstream, observing whether the policy had started at the wire. */ +for (const enforced of [false, true]) test(`passthrough Combo send starts spend policy from its booking before the wire (enforced=${enforced})`, async () => { + const f = trackerFixture(enforced); + const startedAtWire: boolean[] = []; + const server = Bun.serve({ hostname: "127.0.0.1", port: 0, async fetch(req) { + await req.json(); + startedAtWire.push(f.tracker.policyStarted); + return Response.json(responsesSuccess("fixture answer", "m")); + } }); + servers.push(server); + const config: OcxConfig = { port: 0, defaultProvider: "a", providers: { a: { + adapter: "openai-responses", baseUrl: new URL("/v1", server.url).href, allowPrivateNetwork: true, + apiKey: "fixture-inference-key", authMode: "key", liveModels: false, models: ["m"], transientRetryOn5xx: { attempts: 1 } } }, + combos: { auto: { strategy: "failover", targets: [{ provider: "a", model: "m" }] } } }; + f.log.model = ""; f.log.provider = ""; + const req = new Request("http://localhost/v1/responses", { method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "combo/auto", input: "Synthetic spend policy task.", stream: false }) }); + expect(f.tracker.policyStarted).toBe(false); + const response = await handleResponses(req, config, f.log, { sendBudget: f.budget, abortSignal: req.signal }); + expect(response.status).toBe(200); await response.text(); + expect(startedAtWire).toEqual([true]); + expect(f.tracker.spendAdmissionPolicy).toBeDefined(); + expect(f.budget.spendEnforced).toBe(enforced); + expect(f.budget.used).toBe(1); +}, 10_000); + +test("enforced spend counts a local refusal after beforeSend as one started send", async () => { + // Same accounting #6763's native Chat path has: `beforeSend` claims the send, and the retry + // helper reports it in `finally` whatever the send did, including a local refusal. + const f = trackerFixture(true); + const booking = f.budget.reserveDispatch({ sendClass: "initial", targetKey: "a/m", countedExternally: true }); + if (!booking.allowed) throw new Error("fixture reservation refused"); + const report = createPhysicalSendReporter(f.budget, () => ({ poolId: "a" }), booking.permit); + await expect(fetchWithTransientRetry(async () => { throw new SendBudgetExhaustedError("fixture"); }, + { attempts: 1, onSendsConsumed: report })).rejects.toBeInstanceOf(SendBudgetExhaustedError); + expect(f.budget.physicalStarted).toBe(1); + expect(f.budget.used).toBe(1); + booking.permit.release(); // The booking was consumed by the started send: nothing to refund. + expect(f.budget.used).toBe(1); +}); diff --git a/tests/responses/responses-hosted-send-hop.test.ts b/tests/responses/responses-hosted-send-hop.test.ts new file mode 100644 index 00000000000..58c7fb3b5e5 --- /dev/null +++ b/tests/responses/responses-hosted-send-hop.test.ts @@ -0,0 +1,54 @@ +import { expect, spyOn, test } from "bun:test"; +import { ADAPTER_REGISTRY } from "../../src/adapters/registry"; +import { createAdapterPhysicalSend } from "../../src/adapters/physical-send"; +import { getActiveTurnCount } from "../../src/server/lifecycle"; +import * as transport from "../../src/server/responses/request-transport"; +import * as oauth from "../../src/oauth/generic-account-failover"; +import { hostedSendFixture } from "../helpers/hosted-send-fixture"; +import { chatStream } from "../helpers/combo-failover-upstream"; + +for (const kind of ["search", "image", "video"] as const) for (const owned of [false, true]) for (const unsent of [false, true]) { + test(`${kind} ${owned ? "adapter" : "generic"} OAuth hop ${unsent ? "unsent cleanup refunds once" : "dispatch consumes its own booking once"}`, async () => hostedSendFixture(kind, "failover", async f => { + delete f.config.providers.a!.transientRetryOn5xx; + let calls = 0, rotated = false; + f.config.providers.a!.fetch = (async () => ++calls === 1 + ? Response.json({ error: { message: "synthetic rate refusal" } }, { status: 429 }) : chatStream("hop final answer")) as unknown as typeof fetch; + // Credential selection is synthetic; dispatch, receipts, shared budget and journal are real. + const prepare = transport.prepareResponsesTransport; + const prepared = spyOn(transport, "prepareResponsesTransport").mockImplementation(async (...args) => { + const state = await prepare(...args); + if (state instanceof Response) return state; + state.genericFailoverAccountId = "synthetic-first"; + state.applyFailoverSnapshot = async snapshot => { rotated = true; return snapshot; }; + return state; + }); + const enabled = spyOn(oauth, "isGenericOAuthFailoverEnabled").mockReturnValue(true); + const rotate = spyOn(oauth, "rotateGenericOAuthAccountOn429").mockReturnValue("synthetic-second"); + const snapshot = spyOn(oauth, "failoverAccountSnapshot").mockResolvedValue({ provider: "a", accountId: "synthetic-second", generation: "synthetic-generation", accessToken: "synthetic-token" }); + const create = ADAPTER_REGISTRY["openai-chat"].create; + const adapter = spyOn(ADAPTER_REGISTRY["openai-chat"], "create").mockImplementation((...args) => { + const base = create(...args); + return { ...base, + buildRequest(...input) { + if (unsent && rotated) throw new Error("synthetic unsent hop build refusal"); + return base.buildRequest(...input); + }, + ...(owned ? { async fetchResponse(request, ctx) { + const send = createAdapterPhysicalSend(ctx); + return send({ url: request.url, dispatch: executor => executor(request.url, { + method: request.method, headers: request.headers, body: request.body, signal: ctx?.abortSignal, + }) }); + } } : {}), + }; + }); + try { + const response = await f.dispatch(); const body = await response.text(); + expect(response.status).toBe(unsent ? 502 : 200); + expect(body).toContain(unsent ? "synthetic unsent hop build refusal" : "hop final answer"); + expect(calls).toBe(unsent ? 1 : 2); expect(f.budget.used).toBe(unsent ? 1 : 2); + expect(f.charges).toBe(2); expect(f.refunds).toBe(unsent ? 1 : 0); + expect(f.settle()).toMatchObject({ settled: 3, reserved: 0, unresolved: unsent ? 0 : 3 }); + expect(getActiveTurnCount()).toBe(0); + } finally { adapter.mockRestore(); snapshot.mockRestore(); rotate.mockRestore(); enabled.mockRestore(); prepared.mockRestore(); } + })); +} diff --git a/tests/responses/responses-hosted-send-producer.test.ts b/tests/responses/responses-hosted-send-producer.test.ts new file mode 100644 index 00000000000..e3da353f26f --- /dev/null +++ b/tests/responses/responses-hosted-send-producer.test.ts @@ -0,0 +1,151 @@ +import { expect, spyOn, test } from "bun:test"; +import { ADAPTER_REGISTRY } from "../../src/adapters/registry"; +import type { IncomingMeta } from "../../src/adapters/base"; +import { createAdapterPhysicalSend } from "../../src/adapters/physical-send"; +import { getActiveTurnCount } from "../../src/server/lifecycle"; +import { hostedSendFixture } from "../helpers/hosted-send-fixture"; +import { existsSync } from "node:fs"; +import * as spendOwner from "../../src/lib/spend-ledger-owner"; + +test("hosted fixture restores both homes and removes directories even when ownership assertion fails", async () => { + const priorHome = process.env.OPENCODEX_HOME, priorCodex = process.env.CODEX_HOME; + let home = "", codex = ""; + const snapshot = spyOn(spendOwner, "spendLedgerOwnerSnapshot").mockReturnValue({ ownership: "held" }); + try { + await expect(hostedSendFixture("image", "failover", async () => { + home = process.env.OPENCODEX_HOME!; codex = process.env.CODEX_HOME!; + })).rejects.toThrow(); + expect(process.env.OPENCODEX_HOME).toBe(priorHome); expect(process.env.CODEX_HOME).toBe(priorCodex); + expect(existsSync(home)).toBe(false); expect(existsSync(codex)).toBe(false); + } finally { snapshot.mockRestore(); } + expect(spendOwner.spendLedgerOwnerSnapshot().ownership).toBe("unheld"); + await hostedSendFixture("image", "failover", async () => { + expect(spendOwner.spendLedgerOwnerSnapshot().ownership).toBe("held"); + }); +}); + +/** Expose a producer gate so tests control dispatch after response construction. */ +function deferred() { let resolve!: () => void; const promise = new Promise(r => { resolve = r; }); return { promise, resolve }; } +/** Fail a stalled synthetic producer and always clear its deadline timer. */ +async function bounded(promise: Promise): Promise { + let timer: ReturnType; + return Promise.race([promise, new Promise((_, reject) => { timer = setTimeout(() => reject(new Error("synthetic producer did not finish")), 2000); })]) + .finally(() => clearTimeout(timer)); +} +/** Exercise adapter-owned physical admission and consume the synthetic response. */ +async function dispatch(incoming: IncomingMeta) { + const send = createAdapterPhysicalSend({ sendBudget: incoming.sendBudget, executor: incoming.providerFetch, + abortSignal: incoming.abortSignal, onPhysicalSend: incoming.onPhysicalSend }); + const response = await send({ url: "https://synthetic.invalid/v1/chat/completions", dispatch: executor => executor( + "https://synthetic.invalid/v1/chat/completions", { method: "POST", body: JSON.stringify({ model: "m", messages: [] }), signal: incoming.abortSignal }) }); + await response.text(); +} +for (const strategy of ["failover", "jev"] as const) test(`search/${strategy} runTurn retains its ordinary producer-owned durable booking`, async () => hostedSendFixture("search", strategy, async f => { + const create = ADAPTER_REGISTRY["openai-chat"].create; + const override = spyOn(ADAPTER_REGISTRY["openai-chat"], "create").mockImplementation((...args) => ({ ...create(...args), reportsPhysicalSends: true, + async runTurn(_parsed, incoming, emit) { + await dispatch(incoming); emit({ type: "text_delta", text: "hosted search producer final answer" }); emit({ type: "done" }); + }, + })); + try { + const response = await f.dispatch(); expect(response.status).toBe(200); expect(await response.text()).toContain("hosted search producer final answer"); + expect(f.inference).toBe(1); expect(f.judge).toBe(strategy === "jev" ? 1 : 0); + expect(f.budget.used).toBe(1); expect(f.charges).toBe(1); expect(f.refunds).toBe(0); + expect(f.settle()).toMatchObject({ settled: 3, reserved: 0, unresolved: 0 }); expect(getActiveTurnCount()).toBe(0); + } finally { override.mockRestore(); } +})); +for (const kind of ["image", "video"] as const) { + for (const mode of ["delayed", "cancel-before", "cancel-after", "unsent-error", "sent-error", "not-dispatched"] as const) { + test(`${kind} asynchronous producer ${mode} owns its booking until dispatch or exit`, async () => hostedSendFixture(kind, "failover", async f => { + const gate = deferred(), entered = deferred(), sent = deferred(), finished = deferred(); + const create = ADAPTER_REGISTRY["openai-chat"].create; + const override = spyOn(ADAPTER_REGISTRY["openai-chat"], "create").mockImplementation((...args) => ({ ...create(...args), reportsPhysicalSends: true, + async runTurn(_parsed, incoming, emit) { + entered.resolve(); + try { + await Promise.race([gate.promise, new Promise(resolve => { + if (incoming.abortSignal?.aborted) resolve(); + else incoming.abortSignal?.addEventListener("abort", () => resolve(), { once: true }); + })]); + if (incoming.abortSignal?.aborted) return; + if (mode === "unsent-error") throw new Error("synthetic unsent producer error"); + if (mode !== "not-dispatched") { await dispatch(incoming); sent.resolve(); } + if (mode === "sent-error") throw new Error("synthetic post-dispatch error"); + if (mode === "cancel-after") await new Promise(resolve => { + if (incoming.abortSignal?.aborted) resolve(); + else incoming.abortSignal?.addEventListener("abort", () => resolve(), { once: true }); + }); + else { emit({ type: "text_delta", text: "hosted producer final answer" }); emit({ type: "done" }); } + } finally { finished.resolve(); } + }, + })); + try { + const response = await f.dispatchChild(); + expect(response.status).toBe(200); + // Response exists while the asynchronous producer has not dispatched inference. + expect(f.budget.used).toBe(1); expect(f.charges).toBe(1); expect(f.refunds).toBe(0); expect(f.inference).toBe(0); + const reading = response.body!.getReader(); + await reading.read(); // response.created starts the bridge without requiring inference. + const pending = reading.read(); + await bounded(entered.promise); + if (mode === "cancel-before") { await reading.cancel(); } + else { + gate.resolve(); + if (mode === "cancel-after") { await bounded(sent.promise); await reading.cancel(); } + else { await bounded(pending); while (!(await reading.read()).done) { /* drain terminal */ } } + } + await bounded(finished.promise); + await Promise.resolve(); await Promise.resolve(); await Promise.resolve(); + const didSend = mode === "delayed" || mode === "cancel-after" || mode === "sent-error"; + expect(f.inference).toBe(didSend ? 1 : 0); expect(f.budget.used).toBe(didSend ? 1 : 0); + expect(f.charges).toBe(1); expect(f.refunds).toBe(didSend ? 0 : 1); + expect(f.settle()).toMatchObject({ settled: didSend ? 3 : 0, reserved: 0, unresolved: 0 }); + expect(getActiveTurnCount()).toBe(0); + } finally { gate.resolve(); override.mockRestore(); } + })); + } + test(`${kind} cancellation before producer startup refunds its booking once`, async () => hostedSendFixture(kind, "failover", async f => { + let entered = 0; + const create = ADAPTER_REGISTRY["openai-chat"].create; + const override = spyOn(ADAPTER_REGISTRY["openai-chat"], "create").mockImplementation((...args) => ({ ...create(...args), + async runTurn() { entered++; }, + })); + try { + const response = await f.dispatchChild(); await response.body!.cancel(); + for (let i = 0; i < 10; i++) await Promise.resolve(); + expect(entered).toBe(0); expect(f.inference).toBe(0); expect(f.budget.used).toBe(0); + expect(f.charges).toBe(1); expect(f.refunds).toBe(1); + expect(f.settle()).toMatchObject({ settled: 0, reserved: 0, unresolved: 0 }); expect(getActiveTurnCount()).toBe(0); + } finally { override.mockRestore(); } + })); + test(`${kind} synchronous producer throw releases its unsent booking once`, async () => hostedSendFixture(kind, "failover", async f => { + const create = ADAPTER_REGISTRY["openai-chat"].create; + const override = spyOn(ADAPTER_REGISTRY["openai-chat"], "create").mockImplementation((...args) => ({ ...create(...args), + runTurn() { throw new Error("synthetic synchronous producer failure"); }, + })); + try { + const response = await f.dispatchChild(); + expect(await response.text()).toContain("synthetic synchronous producer failure"); + expect(f.inference).toBe(0); expect(f.budget.used).toBe(0); expect(f.charges).toBe(1); expect(f.refunds).toBe(1); + expect(f.settle()).toMatchObject({ settled: 0, reserved: 0, unresolved: 0 }); expect(getActiveTurnCount()).toBe(0); + } finally { override.mockRestore(); } + })); + test(`${kind} unsent producer failure refunds before real next-target inference`, async () => hostedSendFixture(kind, "failover", async f => { + f.config.providers.a!.headers = { "x-synthetic-unsent": "first" }; + f.config.providers.b = { ...f.config.providers.a!, headers: {} }; + f.config.combos!.auto!.targets.push({ provider: "b", model: "m" }); + const create = ADAPTER_REGISTRY["openai-chat"].create; + const override = spyOn(ADAPTER_REGISTRY["openai-chat"], "create").mockImplementation((...args) => ({ ...create(...args), reportsPhysicalSends: true, + async runTurn(_parsed, incoming, emit) { + if (args[0].headers?.["x-synthetic-unsent"]) throw new Error("synthetic unsent first target"); + await dispatch(incoming); + emit({ type: "text_delta", text: "next hosted target final answer" }); emit({ type: "done" }); + }, + })); + try { + const response = await f.dispatch(); expect(response.status).toBe(200); expect(await response.text()).toContain("next hosted target final answer"); + expect(f.inference).toBe(1); expect(f.budget.used).toBe(1); expect(f.charges).toBe(2); expect(f.refunds).toBe(1); + expect(f.settle()).toMatchObject({ settled: 3, reserved: 0, unresolved: 0 }); expect(getActiveTurnCount()).toBe(0); + } finally { override.mockRestore(); } + })); +} diff --git a/tests/responses/responses-hosted-send-reservation.test.ts b/tests/responses/responses-hosted-send-reservation.test.ts new file mode 100644 index 00000000000..e03ecd0ca74 --- /dev/null +++ b/tests/responses/responses-hosted-send-reservation.test.ts @@ -0,0 +1,104 @@ +import { expect, spyOn, test } from "bun:test"; +import { ADAPTER_REGISTRY } from "../../src/adapters/registry"; +import { createAdapterPhysicalSend } from "../../src/adapters/physical-send"; +import { getActiveTurnCount } from "../../src/server/lifecycle"; +import { hostedSendFixture } from "../helpers/hosted-send-fixture"; +import { chatStream } from "../helpers/combo-failover-upstream"; +import * as keyFailover from "../../src/providers/key-failover"; + +for (const kind of ["search", "image", "video"] as const) { + for (const strategy of ["failover", "jev"] as const) { + test(`${kind}/${strategy} final HTTP inference retains prepaid durable booking`, async () => hostedSendFixture(kind, strategy, async f => { + const response = await f.dispatch(); + expect(response.status).toBe(200); + expect(await response.text()).toContain("hosted route final answer"); + expect(f.inference).toBe(1); expect(f.judge).toBe(strategy === "jev" ? 1 : 0); + expect(f.budget.used).toBe(1); expect(f.charges).toBe(1); expect(f.refunds).toBe(0); + expect(f.settle()).toMatchObject({ settled: 3, reserved: 0, unresolved: 0 }); + expect(getActiveTurnCount()).toBe(0); + expect(f.bodies[0]?.model).toBe("m"); + expect(f.bodies[0]?.messages).toEqual(expect.arrayContaining([expect.objectContaining({ role: "user", content: "Synthetic assignment" })])); + if (strategy === "jev") expect(f.bodies[0]?.reasoning_effort).toBe("low"); + })); + } + test(`${kind} adapter-owned fetchResponse consumes once, without a duplicate receipt`, async () => hostedSendFixture(kind, "failover", async f => { + const create = ADAPTER_REGISTRY["openai-chat"].create; + const override = spyOn(ADAPTER_REGISTRY["openai-chat"], "create").mockImplementation((...args) => { + const adapter = create(...args); + return { ...adapter, async fetchResponse(request, ctx) { + const send = createAdapterPhysicalSend(ctx); + return send({ url: request.url, dispatch: executor => executor(request.url, { + method: request.method, headers: request.headers, body: request.body, signal: ctx?.abortSignal, + }) }); + } }; + }); + try { + const response = await f.dispatch(); + expect(await response.text()).toContain("hosted route final answer"); + expect(f.inference).toBe(1); expect(f.budget.used).toBe(1); expect(f.charges).toBe(1); expect(f.refunds).toBe(0); + expect(f.settle()).toMatchObject({ settled: 3, reserved: 0, unresolved: 0 }); + expect(getActiveTurnCount()).toBe(0); + } finally { override.mockRestore(); } + })); + test(`${kind} direct hosted inference preserves its existing non-Combo accounting contract`, async () => hostedSendFixture(kind, "failover", async f => { + const req = f.request(); const body = await req.json() as Record; body.model = "a/m"; + const { handleResponses } = await import("../../src/server/responses/core"); + const response = await handleResponses(new Request(req.url, { method: "POST", headers: req.headers, body: JSON.stringify(body) }), f.config, f.log, { sendBudget: f.budget }); + expect(await response.text()).toContain("hosted route final answer"); expect(f.inference).toBe(1); + expect(f.budget.used).toBe(0); expect(f.charges).toBe(0); expect(f.refunds).toBe(0); expect(getActiveTurnCount()).toBe(0); + })); + for (const firstOwned of [false, true]) test(`${kind} rotated ${firstOwned ? "adapter-to-generic" : "generic-to-adapter"} dispatch uses its actual accounting owner`, async () => hostedSendFixture(kind, "failover", async f => { + delete f.config.providers.a!.transientRetryOn5xx; + // Mock credential selection only; both inference dispatches use the real core/loop/executor. + const rotate = spyOn(keyFailover, "rotateProviderTransportOn429").mockImplementation(() => { + const next = { ...f.config.providers.a!, apiKey: "synthetic-second-key" }; + f.config.providers.a = next; return next; + }); + let calls = 0; + f.config.providers.a!.fetch = (async () => ++calls === 1 + ? Response.json({ error: { message: "synthetic key refusal" } }, { status: 429 }) : chatStream("rotated hosted final answer")) as unknown as typeof fetch; + const create = ADAPTER_REGISTRY["openai-chat"].create; + const override = spyOn(ADAPTER_REGISTRY["openai-chat"], "create").mockImplementation((...args) => { + const adapter = create(...args); + const owned = args[0].apiKey === "synthetic-second-key" ? !firstOwned : firstOwned; + return owned ? { ...adapter, async fetchResponse(request, ctx) { + const send = createAdapterPhysicalSend(ctx); + return send({ url: request.url, dispatch: executor => executor(request.url, { + method: request.method, headers: request.headers, body: request.body, signal: ctx?.abortSignal, + }) }); + } } : adapter; + }); + try { + const response = await f.dispatch(); expect(response.status).toBe(200); expect(await response.text()).toContain("rotated hosted final answer"); + expect(calls).toBe(2); expect(f.budget.used).toBe(2); expect(f.charges).toBe(2); expect(f.refunds).toBe(0); + expect(f.settle()).toMatchObject({ settled: 3, reserved: 0, unresolved: 3 }); expect(getActiveTurnCount()).toBe(0); + } finally { override.mockRestore(); rotate.mockRestore(); } + })); + test(`${kind} local build rejection is unsent and refunded once`, async () => hostedSendFixture(kind, "failover", async f => { + const create = ADAPTER_REGISTRY["openai-chat"].create; + const override = spyOn(ADAPTER_REGISTRY["openai-chat"], "create").mockImplementation((...args) => ({ ...create(...args), + buildRequest() { throw new Error("synthetic local build refusal"); }, + })); + try { + const response = await f.dispatch(); await response.text(); + expect(response.status).toBe(502); expect(f.inference).toBe(0); expect(f.budget.used).toBe(0); + expect(f.charges).toBe(1); expect(f.refunds).toBe(1); + expect(f.settle()).toMatchObject({ settled: 0, reserved: 0, unresolved: 0 }); + expect(getActiveTurnCount()).toBe(0); + } finally { override.mockRestore(); } + })); + test(`${kind} sent failure cannot restore capacity for a next target`, async () => hostedSendFixture(kind, "failover", async f => { + // A three-target Combo has a six-send shared ceiling; prior work already occupies four. + f.budget.used = 4; + let first = 0, second = 0, third = 0; + f.config.providers.a!.fetch = (async () => { first++; return Response.json({ error: { message: "synthetic upstream failure" } }, { status: 500 }); }) as unknown as typeof fetch; + f.config.providers.b = { ...f.config.providers.a!, fetch: (async () => { second++; return Response.json({ error: { message: "second synthetic failure" } }, { status: 500 }); }) as unknown as typeof fetch }; + f.config.providers.c = { ...f.config.providers.a!, fetch: (async () => { third++; return chatStream("must not send"); }) as unknown as typeof fetch }; + f.config.combos!.auto!.targets.push({ provider: "b", model: "m" }, { provider: "c", model: "m" }); + const response = await f.dispatch(); await response.text(); + expect(response.status).toBe(500); expect(first).toBe(1); expect(second).toBe(1); expect(third).toBe(0); + expect(f.budget.used).toBe(6); expect(f.charges).toBe(6); expect(f.refunds).toBe(0); + expect(f.settle()).toMatchObject({ settled: 3, reserved: 0, unresolved: 15 }); + expect(getActiveTurnCount()).toBe(0); + })); +} diff --git a/tests/responses/responses-hosted-send-round-parity.test.ts b/tests/responses/responses-hosted-send-round-parity.test.ts new file mode 100644 index 00000000000..bc953bc3466 --- /dev/null +++ b/tests/responses/responses-hosted-send-round-parity.test.ts @@ -0,0 +1,114 @@ +/** + * A hosted web-search turn under transientRetryOn5xx: { attempts: 1 } makes the same model sends + * whether the caller names the provider directly or reaches it as a Combo target. + * + * Round 1 asks for web_search, the bridge runs one (stubbed) Exa search, and round 2 answers. + * Hosted bridge rounds are not transient retries, so neither path counts round 2 against the + * configured attempts. The child-owned prepaid booking must not change that for Combo targets: + * before it existed, the Combo path and the direct path already agreed at two sends, and a regression + * to one send on either path would cut the hosted turn short. + */ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import type { OcxConfig } from "../../src/types"; +import { handleResponses } from "../../src/server/responses/core"; +import { clearComboSelectionState, clearComboTargetCooldowns } from "../../src/combos"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; +import { installIsolatedCodexHome, type IsolatedCodexHome } from "../helpers/isolated-codex-home"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; + +let home: string; +let priorHome: string | undefined; +let codex: IsolatedCodexHome; +let releaseSpend: () => void; +const originalFetch = globalThis.fetch; + +beforeEach(() => { + priorHome = process.env.OPENCODEX_HOME; + home = mkdtempSync(join(tmpdir(), "hosted-round-parity-")); + process.env.OPENCODEX_HOME = home; + codex = installIsolatedCodexHome("hosted-round-parity-codex-"); + releaseSpend = acquireOwnedSpendHome(); + clearComboSelectionState(); clearComboTargetCooldowns(); +}); + +afterEach(() => { + try { + releaseSpend(); + } finally { + globalThis.fetch = originalFetch; + clearComboSelectionState(); clearComboTargetCooldowns(); + codex.restore(); + if (priorHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = priorHome; + removeTreeWithRetry(home); + } +}); + +/** Round 1 calls web_search; every later round answers. */ +function chatRound(round: number): Response { + const chunks = round === 1 + ? [ + { choices: [{ index: 0, delta: { tool_calls: [{ index: 0, id: "call_search", type: "function", function: { name: "web_search", arguments: '{"query":"x"}' } }] }, finish_reason: null }] }, + { choices: [{ index: 0, delta: {}, finish_reason: "tool_calls" }], usage: { prompt_tokens: 2, completion_tokens: 1, total_tokens: 3 } }, + ] + : [ + { choices: [{ index: 0, delta: { content: "synthetic final answer" }, finish_reason: null }] }, + { choices: [{ index: 0, delta: {}, finish_reason: "stop" }], usage: { prompt_tokens: 2, completion_tokens: 1, total_tokens: 3 } }, + ]; + return new Response(chunks.map(chunk => `data: ${JSON.stringify(chunk)}\n\n`).join("") + "data: [DONE]\n\n", { + headers: { "content-type": "text/event-stream" }, + }); +} + +/** The last terminal Responses event in a drained SSE body. */ +function terminalEvent(body: string): string | undefined { + const types = body.split(/\r?\n\r?\n/).map(block => { + const named = block.match(/^event:\s*(.+)$/m)?.[1]; + if (named) return named; + try { return (JSON.parse(block.match(/^data:\s*(.+)$/m)?.[1] ?? "{}") as { type?: string }).type; } catch { return undefined; } + }); + return types.filter(type => type === "response.completed" || type === "response.failed" || type === "response.incomplete" || type === "error").at(-1); +} + +async function hostedSearchTurn(model: "a/m" | "combo/auto") { + let modelSends = 0; + let searches = 0; + globalThis.fetch = (async (input: string | URL | Request) => { + const url = typeof input === "string" ? input : input instanceof URL ? input.href : input.url; + if (!url.includes("api.exa.ai/search")) throw new Error(`unexpected network call: ${url}`); + searches++; + return Response.json({ results: [{ title: "Synthetic result", url: "https://example.test/x", content: "Synthetic result", text: "Synthetic result" }] }); + }) as typeof fetch; + const config = { + port: 0, + defaultProvider: "a", + providers: { + a: { + adapter: "openai-chat", authMode: "key", apiKey: "synthetic-key", baseUrl: "https://synthetic.invalid/v1", + liveModels: false, models: ["m"], transientRetryOn5xx: { attempts: 1 }, + fetch: (async () => chatRound(++modelSends)) as unknown as typeof fetch, + }, + }, + combos: { auto: { strategy: "failover", targets: [{ provider: "a", model: "m" }] } }, + webSearchSidecar: { backend: "exa", exaApiKey: "synthetic-unused-key" }, + } as unknown as OcxConfig; + const request = new Request("http://localhost/v1/responses", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ model, stream: true, input: "Search for x", tools: [{ type: "web_search" }] }), + }); + const response = await handleResponses(request, config, { model: "", provider: "" }); + const terminal = terminalEvent(await response.text()); + return { status: response.status, terminal, modelSends, searches }; +} + +test("a hosted search turn sends both model rounds for a direct request and a Combo target alike", async () => { + const direct = await hostedSearchTurn("a/m"); + clearComboSelectionState(); clearComboTargetCooldowns(); + const combo = await hostedSearchTurn("combo/auto"); + expect(direct).toEqual({ status: 200, terminal: "response.completed", modelSends: 2, searches: 1 }); + expect(combo).toEqual({ status: 200, terminal: "response.completed", modelSends: 2, searches: 1 }); +}); diff --git a/tests/responses/responses-hosted-send-unsent-hop.test.ts b/tests/responses/responses-hosted-send-unsent-hop.test.ts new file mode 100644 index 00000000000..4ac909a3bdc --- /dev/null +++ b/tests/responses/responses-hosted-send-unsent-hop.test.ts @@ -0,0 +1,219 @@ +import { afterEach, beforeEach, expect, spyOn, test } from "bun:test"; +import { ADAPTER_REGISTRY } from "../../src/adapters/registry"; +import { createAdapterPhysicalSend } from "../../src/adapters/physical-send"; +import { getActiveTurnCount } from "../../src/server/lifecycle"; +import * as transport from "../../src/server/responses/request-transport"; +import * as oauth from "../../src/oauth/generic-account-failover"; +import { providerRequestPacingStatus, resetProviderRequestPacingForTest } from "../../src/providers/request-pacing"; +import { hostedSendFixture } from "../helpers/hosted-send-fixture"; +import { budgetOwner } from "../helpers/send-budget-owner"; +import { createRequestExecutionBudget } from "../../src/lib/request-execution-budget"; +import { createResponsesSendBudget } from "../../src/server/responses/request-send-budget"; +import { createSidecarSendBudget } from "../../src/server/responses/sidecar-send-budget"; +import { createTranslatorBudget } from "../../src/lib/translator-budget"; + +let forbiddenNetwork: ReturnType; +beforeEach(() => { + resetProviderRequestPacingForTest(); + forbiddenNetwork = spyOn(globalThis, "fetch").mockImplementation(Object.assign(() => { + throw new Error("Hosted hop regression forbids live network"); + }, { preconnect() {} })); +}); +afterEach(() => { + const calls = forbiddenNetwork.mock.calls.length; + forbiddenNetwork.mockRestore(); + resetProviderRequestPacingForTest(); + expect(calls).toBe(0); +}); + +// Only credential selection is synthetic; the hosted loop, dispatch budget and journal are real. +/** Replace only credential selection while exercising real hosted dispatch and settlement. */ +async function withSyntheticRotation(run: (rotated: () => boolean) => Promise, onRotation = () => {}) { + let rotated = false; + const prepare = transport.prepareResponsesTransport; + const prepared = spyOn(transport, "prepareResponsesTransport").mockImplementation(async (...args) => { + const state = await prepare(...args); + if (state instanceof Response) return state; + state.genericFailoverAccountId = "synthetic-first"; + state.applyFailoverSnapshot = async snapshot => { rotated = true; onRotation(); return snapshot; }; + return state; + }); + const enabled = spyOn(oauth, "isGenericOAuthFailoverEnabled").mockReturnValue(true); + const rotate = spyOn(oauth, "rotateGenericOAuthAccountOn429").mockReturnValue("synthetic-second"); + const snapshot = spyOn(oauth, "failoverAccountSnapshot").mockResolvedValue({ provider: "a", accountId: "synthetic-second", + generation: "synthetic-generation", accessToken: "synthetic-token" }); + try { await run(() => rotated); } + finally { snapshot.mockRestore(); rotate.mockRestore(); enabled.mockRestore(); prepared.mockRestore(); } +} + +test("real Vertex hosted search refunds replay cancelled after reservation while queued for pacing", async () => hostedSendFixture("search", "failover", async f => { + const controller = new AbortController(); + let physical = 0, sawQueuedReservation = false, poll: ReturnType | undefined; + Object.assign(f.config.providers.a!, { adapter: "google", googleMode: "vertex", + requestPacing: { enabled: true, minIntervalMs: 1000 }, + fetch: async () => { physical++; return Response.json({ error: { status: "RESOURCE_EXHAUSTED", message: "daily limit reached" } }, { status: 429 }); } }); + delete f.config.providers.a!.transientRetryOn5xx; + const deadline = setTimeout(() => controller.abort(new Error("synthetic queue observation timeout")), 500); + try { + await withSyntheticRotation(async () => { + const response = await f.dispatch(controller.signal); await response.text(); + expect(response.status).toBe(499); expect(sawQueuedReservation).toBe(true); + expect(physical).toBe(1); expect(f.budget.used).toBe(1); + expect(f.charges).toBe(2); expect(f.refunds).toBe(1); + expect(f.settle()).toMatchObject({ settled: 3, reserved: 0, unresolved: 0 }); + expect(getActiveTurnCount()).toBe(0); + }, () => { + poll = setInterval(() => { + if (providerRequestPacingStatus("a", f.config.providers.a!).queued > 0 && f.charges === 2 && f.budget.used === 2) { + sawQueuedReservation = true; + clearInterval(poll); + controller.abort(new Error("synthetic queued replay cancellation")); + } + }, 1); + }); + } finally { clearInterval(poll); clearTimeout(deadline); } +})); + +for (const kind of ["search", "image", "video"] as const) for (const sent of [false, true]) { + test(`${kind} adapter OAuth replay ${sent ? "cancelled after dispatch stays charged" : "beforeDispatch failure refunds only unsent hop"}`, async () => hostedSendFixture(kind, "failover", async f => { + delete f.config.providers.a!.transientRetryOn5xx; + const controller = new AbortController(); + let physical = 0, replayPrepared = false; + f.config.providers.a!.fetch = (async () => { + physical++; + if (physical === 1) return Response.json({ error: { message: "synthetic quota refusal" } }, { status: 429 }); + controller.abort(new Error("synthetic dispatched replay cancellation")); + throw controller.signal.reason; + }) as unknown as typeof fetch; + await withSyntheticRotation(async rotated => { + const create = ADAPTER_REGISTRY["openai-chat"].create; + const adapter = spyOn(ADAPTER_REGISTRY["openai-chat"], "create").mockImplementation((...args) => ({ + ...create(...args), async fetchResponse(request, ctx) { + const send = createAdapterPhysicalSend(ctx); + return send({ url: request.url, beforeDispatch: () => { + if (!rotated()) return; + replayPrepared = true; + expect(f.budget.used).toBe(2); expect(f.charges).toBe(2); expect(f.refunds).toBe(0); + if (!sent) throw new Error("synthetic abandoned hop before send"); + }, dispatch: executor => executor(request.url, { method: request.method, headers: request.headers, + body: request.body, signal: ctx?.abortSignal }) }); + }, + })); + try { + const response = await f.dispatch(controller.signal); const text = await response.text(); + expect(response.ok).toBe(false); expect(replayPrepared).toBe(true); + if (!sent) { expect(response.status).toBe(502); expect(text).toContain("synthetic abandoned hop before send"); } + expect(physical).toBe(sent ? 2 : 1); expect(f.budget.used).toBe(sent ? 2 : 1); + expect(f.charges).toBe(2); expect(f.refunds).toBe(sent ? 0 : 1); + expect(f.settle()).toMatchObject({ settled: 3, reserved: 0, unresolved: sent ? 3 : 0 }); + expect(getActiveTurnCount()).toBe(0); + } finally { adapter.mockRestore(); } + }); + })); +} + +for (const claimed of [false, true]) for (const sent of [false, true]) test(`sidecar iteration cannot release active producer hop (claimed=${claimed}, sent=${sent})`, async () => { + let refunds = 0, physical = 0; + const budget = createRequestExecutionBudget(undefined, undefined, { charge: () => true, refund: () => { refunds++; } }); + const initial = budget.reserveDispatch({ sendClass: "initial", targetKey: "endpoint-one", countedExternally: true }); + if (!initial.allowed) throw new Error("synthetic initial denied"); + const translatorBudget = createTranslatorBudget(); + const options = { sendBudget: budget, translatorBudget, comboInitialSend: { permit: initial.permit } }; + const owner = createResponsesSendBudget({ req: new Request("http://localhost/v1/responses"), + logCtx: { provider: "a", model: "m" }, options }); + if (owner instanceof Response) throw new Error("synthetic owner refused"); + const sidecar = createSidecarSendBudget(options, owner, () => undefined); + let resume!: () => void, entered!: () => void; + const gate = new Promise(resolve => { resume = resolve; }); + const waiting = new Promise(resolve => { entered = resolve; }); + try { + sidecar.takeProducerOwnership(); + const first = owner.adapterDispatchBudget!.reserveDispatch({ sendClass: "initial", targetKey: "endpoint-one" }); + if (!first.allowed || !first.permit.use()) throw new Error("synthetic initial dispatch denied"); + sidecar.ownCredentialHop(owner.reserveCredentialHop("auth-recovery", "endpoint-one", true).permit); + const send = createAdapterPhysicalSend({ sendBudget: owner.adapterDispatchBudget, + executor: (async () => { physical++; return Response.json({ ok: true }); }) as unknown as typeof fetch }); + const producer = claimed ? send({ url: "endpoint-one", beforeDispatch: async () => { + entered(); await gate; + if (!sent) throw new Error("synthetic abandoned producer"); + }, dispatch: executor => executor("https://synthetic.invalid") }) : undefined; + const result = producer?.then(() => undefined, error => error); + if (claimed) await waiting; + sidecar.releaseUnsentHop(); + expect(budget.used).toBe(2); expect(refunds).toBe(0); + resume(); + if (claimed && sent) expect(await result).toBeUndefined(); + else if (claimed) expect(await result).toBeInstanceOf(Error); + sidecar.release(); sidecar.releaseUnsentHop(); sidecar.release(); + expect(owner.pendingHopPermit).toBeUndefined(); + expect(physical).toBe(claimed && sent ? 1 : 0); + expect(budget.used).toBe(claimed && sent ? 2 : 1); expect(refunds).toBe(claimed && sent ? 0 : 1); + } finally { resume(); sidecar.release(); translatorBudget.dispose(); } +}); + +/** Reserve a recovery hop after a consumed initial send with observable refunds. */ +function hopOwner(maxTargetTransitions = 1) { + let charges = 0, refunds = 0; + const budget = createRequestExecutionBudget({ baseSendAllowance: 1, maxTotalModelSends: 2, + finalRecoveryAllowance: 1, maxAlternateTargetSends: 1, maxTargetTransitions }, undefined, { + charge: () => { charges++; return true; }, refund: () => { refunds++; }, + }); + const { owner, dispose } = budgetOwner(budget); + const first = budget.reserveDispatch({ sendClass: "initial", targetKey: "endpoint-one" }); + if (!first.allowed || !first.permit.use()) throw new Error("synthetic first dispatch denied"); + const hop = owner.reserveCredentialHop("auth-recovery", "diagnostic-hop", true); + if (!hop.permit) throw new Error("synthetic hop denied"); + owner.pendingHopPermit = hop.permit; + return { budget, owner, dispose, hop: hop.permit, get charges() { return charges; }, get refunds() { return refunds; } }; +} + +for (const sent of [false, true]) test(`same-endpoint replay retains exact permit single-use and refund ownership (${sent})`, () => { + const f = hopOwner(); + try { + expect(f.owner.adapterDispatchBudget!.reserveDispatch({ sendClass: "transient", targetKey: "endpoint-one", replaySafe: false })) + .toMatchObject({ allowed: false, reason: "not-replay-safe" }); + expect(f.owner.pendingHopPermit).toBe(f.hop); + const replay = f.owner.adapterDispatchBudget!.reserveDispatch({ sendClass: "transient", targetKey: "endpoint-one" }); + if (!replay.allowed) throw new Error("synthetic replay denied"); + expect(f.owner.pendingHopPermit).toBeUndefined(); expect(f.budget.used).toBe(2); expect(f.charges).toBe(2); + if (sent) expect(replay.permit.use()).toBe(true); + replay.permit.release(); replay.permit.release(); f.hop.release(); + expect(replay.permit.use()).toBe(false); expect(replay.permit.assumeCharge()).toBe(false); + expect(f.hop.use()).toBe(false); expect(f.refunds).toBe(sent ? 0 : 1); expect(f.budget.used).toBe(sent ? 2 : 1); + if (sent) { + // Closing the exact external booking cannot swallow a later independent send report. + f.owner.noteTransientSends(1); expect(f.budget.used).toBe(3); expect(f.charges).toBe(3); + } + } finally { f.dispose(); } +}); + +for (const cap of [0, 1]) test(`rotated endpoint still passes transition admission and refunds unsent replacement (${cap})`, () => { + const f = hopOwner(cap); + try { + const replay = f.owner.adapterDispatchBudget!.reserveDispatch({ sendClass: "transient", targetKey: "endpoint-two" }); + expect(f.hop.use()).toBe(false); expect(f.refunds).toBe(1); + if (cap === 0) { + expect(replay).toMatchObject({ allowed: false, reason: "target-transition-exhausted" }); + expect(f.budget.used).toBe(1); expect(f.charges).toBe(2); + } else { + if (!replay.allowed) throw new Error("synthetic endpoint transition denied"); + expect(f.budget.targetTransitions).toBe(1); expect(f.budget.used).toBe(2); expect(f.charges).toBe(3); + replay.permit.release(); replay.permit.release(); + expect(f.budget.targetTransitions).toBe(0); expect(f.budget.used).toBe(1); expect(f.refunds).toBe(2); + } + } finally { f.dispose(); } +}); + +for (const settled of ["used", "released"] as const) test(`a hop already ${settled} by another owner cannot authorize replacement dispatch`, () => { + const f = hopOwner(); + try { + if (settled === "used") expect(f.hop.assumeCharge()).toBe(true); + else f.hop.release(); + const replay = f.owner.adapterDispatchBudget!.reserveDispatch({ sendClass: "transient", targetKey: "endpoint-one" }); + if (!replay.allowed) throw new Error("synthetic handoff missing"); + expect(replay.permit.use()).toBe(false); expect(replay.permit.assumeCharge()).toBe(false); + replay.permit.release(); + expect(f.charges).toBe(2); expect(f.refunds).toBe(settled === "used" ? 0 : 1); + expect(f.budget.used).toBe(settled === "used" ? 2 : 1); + } finally { f.dispose(); } +}); diff --git a/tests/responses/responses-hosted-send-ws.test.ts b/tests/responses/responses-hosted-send-ws.test.ts new file mode 100644 index 00000000000..e9c4e77042b --- /dev/null +++ b/tests/responses/responses-hosted-send-ws.test.ts @@ -0,0 +1,52 @@ +import { expect, spyOn, test } from "bun:test"; +import { ADAPTER_REGISTRY } from "../../src/adapters/registry"; +import { getActiveTurnCount } from "../../src/server/lifecycle"; +import { hostedSendFixture } from "../helpers/hosted-send-fixture"; + +class HostedSocket { + static frames = 0; + listeners = new Map void>>(); + constructor() { queueMicrotask(() => this.emit("open", {})); } + addEventListener(name: string, fn: (event: unknown) => void) { + if (!this.listeners.has(name)) this.listeners.set(name, new Set()); + this.listeners.get(name)!.add(fn); + } + removeEventListener(name: string, fn: (event: unknown) => void) { this.listeners.get(name)?.delete(fn); } + emit(name: string, event: unknown) { for (const fn of this.listeners.get(name) ?? []) fn(event); } + send(data: string) { + const frame = JSON.parse(data) as Record; + expect(frame.type).toBe("response.create"); HostedSocket.frames++; + queueMicrotask(() => this.emit("message", { data: JSON.stringify({ type: "response.completed", response: { + id: "synthetic-hosted-response", status: "completed", output: [], usage: { input_tokens: 2, output_tokens: 1 }, + } }) })); + } + close() {} +} +for (const kind of ["search", "image", "video"] as const) test(`${kind} successful hosted WebSocket frame retains one durable booking`, async () => hostedSendFixture(kind, "failover", async f => { + const realWebSocket = globalThis.WebSocket; + globalThis.WebSocket = HostedSocket as unknown as typeof WebSocket; + HostedSocket.frames = 0; + const create = ADAPTER_REGISTRY["openai-chat"].create; + const override = spyOn(ADAPTER_REGISTRY["openai-chat"], "create").mockImplementation((...args) => ({ ...create(...args), + buildRequest() { return { url: "https://chatgpt.com/backend-api/codex/responses", method: "POST", headers: { + "content-type": "application/json", authorization: "Bearer synthetic-token", + }, body: JSON.stringify({ model: "m", stream: true, input: "Synthetic assignment" }) }; }, + async *parseStream(response) { + expect(await response.text()).toContain("response.completed"); + yield { type: "text_delta", text: "hosted WebSocket final answer" } as const; + yield { type: "done", usage: { inputTokens: 2, outputTokens: 1 } } as const; + }, + })); + try { + f.config.providers.a!.upstreamWebsocket = true; + // The installed Bun version is not used as evidence for the gated WS policy. + // Pin the bounded-relay runtime through the normal handler option. + const { handleResponses } = await import("../../src/server/responses/core"); + const req = f.request(); + const response = await handleResponses(req, f.config, f.log, { sendBudget: f.budget, abortSignal: req.signal, codexWsRuntimeIdentity: "1.4.0" }); + expect(response.status).toBe(200); expect(await response.text()).toContain("hosted WebSocket final answer"); + expect(HostedSocket.frames).toBe(1); expect(f.inference).toBe(0); + expect(f.budget.used).toBe(1); expect(f.charges).toBe(1); expect(f.refunds).toBe(0); + expect(f.settle()).toMatchObject({ settled: 3, reserved: 0, unresolved: 0 }); expect(getActiveTurnCount()).toBe(0); + } finally { override.mockRestore(); globalThis.WebSocket = realWebSocket; } +})); diff --git a/tests/responses/responses-passthrough-transient-policy.test.ts b/tests/responses/responses-passthrough-transient-policy.test.ts index c2ccf954785..8c50905c87e 100644 --- a/tests/responses/responses-passthrough-transient-policy.test.ts +++ b/tests/responses/responses-passthrough-transient-policy.test.ts @@ -37,7 +37,7 @@ describe("the Responses passthrough lane reads the provider transient policy", ( "const transientSendPolicy = () => transientRetryPolicyFor(route.provider);\n" + "const transientSendAttempts = (): number => transientSendCapFor(\n" + " transientSendPolicy()?.attempts,\n" - + " sendBudgetState.sendsUsed,\n" + + " sendBudgetState.targetSendsUsed,\n" + ");", )); }); @@ -55,7 +55,13 @@ describe("the Responses passthrough lane reads the provider transient policy", ( const budgeted = occurrences(packed, "attempts:remainingTransientSendBudget(transientSendAttempts())"); const allowanceBacked = occurrences(packed, "attempts:allowance.attempts"); - expect(budgeted + allowanceBacked).toBe(sites); + const prepaidInitial = occurrences(packed, "attempts:sendBudgetState.initialSendAllowance(transientSendAttempts())"); + expect(prepaidInitial).toBe(1); + // Combo-owned, non-enforced sends settle at physical dispatch; every other send keeps the + // shared reporter, so exactly one of the two accounts for any given send. + expect(packed).toContain("onPhysicalDispatch:receiptMode?sendBudgetState.noteInitialDispatch:undefined"); + expect(packed).toContain("...(receiptMode?{}:{onSendsConsumed:transientSendReporter()})"); + expect(budgeted + allowanceBacked + prepaidInitial).toBe(sites); // The rebuild leg spends a recovery allowance rather than the base budget directly, so its // cap has to come from the same resolver one level up. diff --git a/tests/server/server-antigravity-web-search-validation.test.ts b/tests/server/server-antigravity-web-search-validation.test.ts index 80d1e8cfff1..a9e23b5c297 100644 --- a/tests/server/server-antigravity-web-search-validation.test.ts +++ b/tests/server/server-antigravity-web-search-validation.test.ts @@ -423,6 +423,50 @@ test.each(["construction", "admission", "admission cancellation"])( }, ); +for (const dispatched of [false, true]) test(`Combo Antigravity 403 sibling hop ${dispatched ? "dispatches" : "stops before dispatch"} with exact settlement`, async () => { + const accounts = await seedAntigravityAccounts(2); + const cfg = webSearchConfig(); + let charges = 0, refunds = 0, physical = 0, refused = 0; + const sendBudget = createRequestExecutionBudget({ + maxTotalModelSends: 2, baseSendAllowance: 2, finalRecoveryAllowance: 0, + maxAlternateTargetSends: 0, maxTargetTransitions: 0, + }, undefined, { charge: () => { charges++; return true; }, refund: () => { refunds++; } }); + // The stopped child keeps Combo's logical key. The dispatched child pins its + // physical endpoint to isolate hop ownership from a target-transition replacement. + const initial = sendBudget.reserveDispatch({ + sendClass: "initial", + targetKey: dispatched ? `${DAILY_API_BASE}/v1internal:streamGenerateContent?alt=sse` : "google-antigravity/gemini-3.8-flash", + countedExternally: true, + }); + if (!initial.allowed) throw new Error("Combo initial booking denied"); + installAntigravityFetchMock(({ auth, project }) => { + physical++; + expect({ auth, project }).toEqual({ auth: accounts[physical - 1]!.auth, project: accounts[physical - 1]!.project }); + return physical === 1 ? new Response(structuredRefusal, { status: 403 }) + : new Response(sseSuccessBody("sibling answered"), { headers: { "content-type": "text/event-stream" } }); + }); + const originalResolve = adapterResolve.resolveAdapter; + const resolveSpy = spyOn(adapterResolve, "resolveAdapter").mockImplementation((provider, ...args) => { + const adapter = originalResolve(provider, ...args); + if (!dispatched && provider.project === accounts[1]!.project) { + adapter.buildRequest = async () => { refused++; throw new Error("synthetic sibling construction refusal"); }; + } + return adapter; + }); + try { + const response = await handleResponses(request(), cfg, route, { + sendBudget, comboAttempt: true, comboInitialSend: { permit: initial.permit }, + }); + await response.text(); + expect(response.status).toBe(dispatched ? 200 : 403); + expect(refused).toBe(dispatched ? 0 : 1); + expect(physical).toBe(dispatched ? 2 : 1); + expect(charges).toBe(2); + expect(refunds).toBe(dispatched ? 0 : 1); + expect(sendBudget.used).toBe(dispatched ? 2 : 1); + } finally { resolveSpy.mockRestore(); } +}); + test("web-search 429 recovery still uses the sibling's identity and charges exactly two physical sends", async () => { const accounts = await seedAntigravityAccounts(2); const cfg = webSearchConfig();