feat(api): adaptive Antigravity resilience, model quarantine and hot timeouts

.github/workflows/selected-upstream-regression.yml

@@ -86,7 +86,14 @@ jobs:
             tests/unit/http-client-abort-guard-empty-response-16347.test.ts \
             tests/unit/release-version-workspace-consistency.test.ts \
             tests/unit/server-owned-tool-loop-flag.test.ts \
-            tests/unit/bug-8020-codex-peek-body-timeout.test.ts
+            tests/unit/bug-8020-codex-peek-body-timeout.test.ts \
+            tests/unit/antigravity-adaptive-health.test.ts \
+            tests/unit/antigravity-timeout-retry.test.ts \
+            tests/unit/antigravity-timing.test.ts \
+            tests/unit/preferred-connection-failover.test.ts \
+            tests/unit/codex-same-account-transport-retry-9708.test.ts \
+            tests/integration/antigravity-resilience-settings.test.ts \
+            tests/integration/antigravity-resilience-routing.test.ts
       - name: UI provider availability (Vitest / jsdom)
         run: ./node_modules/.bin/vitest run tests/unit/ui/resilience-connections.test.tsx --maxWorkers=2
       - name: Typecheck core

docs/audits/2026-10-24-antigravity-resilience/README.md

@@ -0,0 +1,89 @@
+# Antigravity resilience overhaul audit
+
+This branch starts at production commit `44fdae6b4b9d63fa41f62a1941e3ff2f366743f6`.
+It is a code and static review only. The production host was not built, started, or
+used for tests, and no upstream model calls were made.
+
+## Root cause and baseline
+
+The request loop classified every Antigravity 504 as a pre-response timeout in one
+branch, but the generic same-account transport retry remained reachable in other
+paths. Its transport status set included 504, so a confirmed header timeout could
+send the same account again before account rotation. The change requires both
+`errorType=upstream_timeout` and
+`errorCode=ANTIGRAVITY_PRE_RESPONSE_TIMEOUT` for the specialized path, excludes
+confirmed Antigravity timeouts from same-account retry, and keeps 502, 503 and
+socket-reset handling separate.
+
+The owner supplied the baseline for the last 24 hours: about 2,824 Gemini 3.8
+records, 261 HTTP 504, 576 HTTP 499 and 1,874 HTTP 200. Prefix summaries were
+04572930: 303 requests (97 504, 106 499, 93 429, 7 200), 96c44b1c: 2,099
+(100 504, 375 499, 1,612 200), and 42684230: 232 (59 504, 57 499, 116 200).
+The prefixes are not account numbers. The 113 same-account repeats reported by
+the owner were treated as the regression signal. 499 cancellations are excluded
+from health denominators and do not mark accounts; 429 remains quota/rate-limit
+state.
+
+## Implemented behavior
+
+`open-sse/services/antigravityHealth.ts` provides a bounded process-local
+connection + normalized-model rolling window. It requires eight completed samples,
+uses a 50% timeout ratio or three consecutive confirmed timeouts, quarantines for
+30 seconds with exponential backoff capped at five minutes, admits one half-open
+probe, and requires two sequential successes for recovery. 499, quota, auth and
+local rejection outcomes are ignored. Entries and history are bounded, and a
+generation fence prevents a stale concurrent result from undoing a newer state.
+Selection applies health after existing quota, terminal, model and forced-connection
+rules. If every eligible pair is quarantined, one real demand-driven request is
+allowed as an escape probe. The feature is off by default.
+
+The connections API exposes only bounded model, state, counts, ratios, cooldown,
+backoff and inflight fields. It does not expose credentials, upstream text, URLs or
+PII. The dashboard connection detail renders these fields. The API requires the
+existing management authentication policy and access-token write scope.
+
+`open-sse/services/antigravityTiming.ts` records optional phase telemetry with a
+bounded hashed account label and bounded event count. Idle duration, inflight
+count, header latency and first meaningful Gemini delta are measured from actual
+phase boundaries. DNS, connect and TLS phases are explicitly `null` when the
+available fetch API cannot measure them. Timing is disabled by default and emits no
+request body, bearer token, URL or user content. Existing dispatcher keepalive and
+fresh-socket recovery paths remain in control; no synthetic paid probes were added.
+
+Preferred failover defaults remain exported as 5,000 and 12,000 ms for compatibility.
+The runtime settings object adds bounded, persisted values (500..120,000 ms),
+adaptive-health and timing switches. `/api/resilience` validates the patch, requires
+management auth, writes the settings atomically with the existing revision conflict
+guard, and reads settings for each new request. The dashboard exposes the four
+values.
+
+## Tradeoffs and operator knobs
+
+The health tracker is process-local, so a restart clears observations and does not
+create a migration. This avoids persisting ephemeral per-account state or secrets.
+The default feature gate is conservative. Operators can enable adaptive health and
+timing telemetry from Settings → Resilience. Preferred timeout values are clamped to
+500..120,000 ms. Recovery uses real traffic only, so an idle account may remain
+quarantined until the next request reaches its half-open window.
+
+## Verification
+
+Added focused unit coverage for exact timeout classification, forced/pinned behavior,
+bounded transport retry, model normalization, low-sample protection, 499 exclusion,
+rolling expiry, backoff, half-open concurrency, recovery hysteresis, bounded state,
+stream completion and timing lifecycle. Added the two test files to
+`.github/workflows/selected-upstream-regression.yml`.
+
+Only static checks were performed in this worktree: source inspection and
+`git diff --check`. CI has not run yet. This report intentionally does not claim
+typecheck, lint, unit, integration, build, browser or deployment verification.
+
+## Remaining uncertainties and limitations
+
+The owner-provided aggregate counts do not identify Antigravity accounts, and this
+branch cannot establish the Gemini 3.8 account #2 idle-restart diagnosis from
+production. The telemetry will distinguish idle, concurrency, header and first-delta
+phases after CI and an approved deployment. DNS/connect/TLS remain null unless the
+selected dispatcher exposes safe phase hooks. The adaptive tracker is not shared
+between multiple worker processes. No deployment, merge or production setting change
+is included in this PR.

open-sse/executors/antigravity.ts

@@ -1,4 +1,5 @@
 import crypto, { randomUUID } from "crypto";
+import { isAntigravityPreResponseTimeout } from "../services/antigravityTimeout.ts";
 import {
   BaseExecutor,
   mergeUpstreamExtraHeaders,
@@ -565,8 +566,7 @@ export class AntigravityExecutor extends BaseExecutor {
       (status === HTTP_STATUS.RATE_LIMITED ||
         status === HTTP_STATUS.NOT_FOUND ||
         status === HTTP_STATUS.BAD_GATEWAY ||
-        status === HTTP_STATUS.SERVICE_UNAVAILABLE ||
-        status === HTTP_STATUS.GATEWAY_TIMEOUT) &&
+        status === HTTP_STATUS.SERVICE_UNAVAILABLE) &&
       urlIndex + 1 < this.getFallbackCount()
     );
   }
@@ -1197,6 +1197,7 @@ export class AntigravityExecutor extends BaseExecutor {
       try {
         result = await this.executeOnce(input, candidate, physicalSendCounter);
       } catch (error) {
+        if (isAntigravityPreResponseTimeout(error)) throw error;
         const outcome = handleAntigravityFallbackChainError(
           input,
           error,
@@ -1338,13 +1339,14 @@ export class AntigravityExecutor extends BaseExecutor {
         if (signal?.aborted || isAbortError(error)) {
           throw signal?.reason ?? error;
         }
+        if (isAntigravityPreResponseTimeout(error)) throw error;
         lastError = error;
         l.error(
           "TELEMETRY",
-          `[Antigravity] Network/Fetch Error - URL: ${url}, Model: ${model}, Error: ${error instanceof Error ? error.message : String(error)}`
+          "[Antigravity] Network/fetch failure before executor result"
         );
         if (urlIndex + 1 < fallbackCount) {
-          l.debug("RETRY", `Error on ${url}, trying fallback ${urlIndex + 1}`);
+          l.debug("RETRY", `Transport failure, trying endpoint ${urlIndex + 1}`);
           continue;
         }
         throw error;
@@ -1658,6 +1660,7 @@ export class AntigravityExecutor extends BaseExecutor {
         switchAuth: decision.kind === "short_cooldown_switch_auth",
       };
     } catch (error) {
+      if (isAntigravityPreResponseTimeout(error)) throw error;
       if (signal?.aborted || isAbortError(error)) {
         throw signal?.reason ?? error;
       }

open-sse/executors/antigravity/executeAttempt.ts

@@ -27,6 +27,8 @@ import {
   type SsePassthroughResult,
 } from "./streamingPassthrough.ts";
 import type { AntigravityCredentials } from "../antigravity.ts";
+import { isAntigravityPreResponseTimeout } from "../../services/antigravityTimeout.ts";
+import { timeAntigravityFetch } from "../../services/antigravityTiming.ts";
 
 const LONG_RETRY_THRESHOLD_MS = 60_000;
 const CREDITS_EXHAUSTED_TTL_MS = 5 * 60 * 60 * 1000; // 5 hours
@@ -90,22 +92,12 @@ class AntigravityPreResponseTimeoutError extends Error {
   code = ANTIGRAVITY_PRE_RESPONSE_TIMEOUT_CODE;
   status = HTTP_STATUS.GATEWAY_TIMEOUT;
 
-  constructor(timeoutMs: number, url: string) {
-    super(`Antigravity upstream did not return response headers within ${timeoutMs}ms: ${url}`);
+  constructor(timeoutMs: number) {
+    super(`Antigravity upstream did not return response headers within ${timeoutMs}ms`);
     this.name = "TimeoutError";
   }
 }
 
-function getAbortErrorCode(error: unknown): string | null {
-  if (!error || typeof error !== "object") return null;
-  const value = (error as { code?: unknown }).code;
-  return typeof value === "string" ? value : null;
-}
-
-function isAntigravityPreResponseTimeout(error: unknown): boolean {
-  return getAbortErrorCode(error) === ANTIGRAVITY_PRE_RESPONSE_TIMEOUT_CODE;
-}
-
 export function isAbortError(error: unknown): boolean {
   return (
     (error instanceof DOMException && error.name === "AbortError") ||
@@ -123,6 +115,14 @@ export async function fetchAntigravityWithReadinessTimeout(
   url: string,
   init: RequestInit,
   timeoutMs = STREAM_READINESS_TIMEOUT_MS
+): Promise<Response> {
+  return timeAntigravityFetch(() => fetchWithReadinessTimeout(url, init, timeoutMs), init.signal);
+}
+
+async function fetchWithReadinessTimeout(
+  url: string,
+  init: RequestInit,
+  timeoutMs = STREAM_READINESS_TIMEOUT_MS
 ): Promise<Response> {
   const boundedTimeoutMs = Math.max(0, Math.floor(timeoutMs));
   if (boundedTimeoutMs <= 0) {
@@ -131,7 +131,7 @@ export async function fetchAntigravityWithReadinessTimeout(
 
   const timeoutController = new AbortController();
   let timeoutId: ReturnType<typeof setTimeout> | null = setTimeout(() => {
-    timeoutController.abort(new AntigravityPreResponseTimeoutError(boundedTimeoutMs, url));
+    timeoutController.abort(new AntigravityPreResponseTimeoutError(boundedTimeoutMs));
   }, boundedTimeoutMs);
 
   const existingSignal = init.signal instanceof AbortSignal ? init.signal : null;
@@ -142,6 +142,10 @@ export async function fetchAntigravityWithReadinessTimeout(
   try {
     return await fetch(url, { ...init, signal: combinedSignal });
   } catch (error) {
+    // The caller's cancellation wins; never relabel it as an account timeout.
+    if (existingSignal?.aborted && combinedSignal.reason !== timeoutController.signal.reason) {
+      throw existingSignal.reason;
+    }
     if (
       timeoutController.signal.aborted &&
       isAntigravityPreResponseTimeout(timeoutController.signal.reason)
@@ -261,11 +265,6 @@ function serializeAntigravityRequest(
   return applyFingerprint(provider, { ...headers }, serializedBody);
 }
 
-function getRequestTargetModel(body: Record<string, unknown>): string {
-  const target = body.model;
-  return typeof target === "string" && target.length > 0 ? target : "unknown";
-}
-
 /** Sanitize unsupported schema metadata, then apply credits-first injection for one attempt. */
 export function finalizeAntigravityRequestBody(
   transformed: Record<string, unknown>,
@@ -347,7 +346,7 @@ export async function sendAntigravityRequest(
 
   log.debug(
     "TELEMETRY",
-    `[Antigravity] Execute - URL: ${url}, Model: ${model}, Target: ${getRequestTargetModel(transformedBody)}, RetryAttempt: ${retryAttempt}`
+    `[Antigravity] Execute attempt ${Math.min(100, retryAttempt)}`
   );
 
   // Dump outgoing headers (mask Authorization) and envelope shape for debugging.
@@ -361,7 +360,7 @@ export async function sendAntigravityRequest(
   const physicalSendOrdinal = ++physicalSendCounter.value;
   log.debug(
     "TELEMETRY",
-    `[Antigravity] PhysicalSend - RequestId: ${correlationId ?? "none"}, URL: ${url}, Model: ${model}, PhysicalSend: ${physicalSendOrdinal}, RetryAttempt: ${retryAttempt}`
+    `[Antigravity] PhysicalSend: ${physicalSendOrdinal}, RetryAttempt: ${retryAttempt}`
   );
   let response = await fetchAntigravityWithReadinessTimeout(url, {
     method: "POST",
@@ -379,7 +378,7 @@ export async function sendAntigravityRequest(
     const retryPhysicalSendOrdinal = ++physicalSendCounter.value;
     log.debug(
       "TELEMETRY",
-      `[Antigravity] PhysicalSend - RequestId: ${correlationId ?? "none"}, URL: ${url}, Model: ${model}, PhysicalSend: ${retryPhysicalSendOrdinal}, RetryAttempt: ${retryAttempt}, Cause: x-goog-user-project-403`
+      `[Antigravity] PhysicalSend: ${retryPhysicalSendOrdinal}, Cause: x-goog-user-project-403`
     );
     response = await fetchAntigravityWithReadinessTimeout(url, {
       method: "POST",
@@ -394,7 +393,7 @@ export async function sendAntigravityRequest(
   if (!response.ok) {
     log.warn(
       "TELEMETRY",
-      `[Antigravity] Error Response - URL: ${url}, Status: ${response.status}, Model: ${model}`
+      `[Antigravity] Error Response: ${response.status}`
     );
     if (response.status === HTTP_STATUS.NOT_FOUND) {
       // The backend may have shipped/renamed models the synced catalog does not
@@ -450,7 +449,7 @@ export async function tryCreditsRetry(
     const creditsPhysicalSendOrdinal = ++physicalSendCounter.value;
     log.debug(
       "TELEMETRY",
-      `[Antigravity] PhysicalSend - RequestId: ${correlationId ?? "none"}, URL: ${url}, PhysicalSend: ${creditsPhysicalSendOrdinal}, Cause: google-one-ai-credits-retry`
+      `[Antigravity] PhysicalSend: ${creditsPhysicalSendOrdinal}, Cause: google-one-ai-credits-retry`
     );
     const creditsResp = await fetchAntigravityWithReadinessTimeout(url, {
       method: "POST",
@@ -492,6 +491,7 @@ export async function tryCreditsRetry(
     markCreditsExhausted(accountId);
     return null;
   } catch (creditsErr) {
+    if (isAntigravityPreResponseTimeout(creditsErr)) throw creditsErr;
     if (signal?.aborted || isAbortError(creditsErr)) {
       throw signal?.reason ?? creditsErr;
     }

open-sse/executors/antigravity/proFallbackChain.ts

@@ -4,6 +4,7 @@
 // Extracted from antigravity.ts (file-size cap) -- mirrors the existing
 // antigravity/sseCollect.ts submodule pattern.
 import type { ExecuteInput } from "../base.ts";
+import { isAntigravityPreResponseTimeout } from "../../services/antigravityTimeout.ts";
 
 /** Shape of one execute()/executeOnce() result (kept local to avoid importing the class). */
 export type AntigravityExecuteResult = {
@@ -42,7 +43,7 @@ export function handleAntigravityFallbackChainError(
   resolvedUpstreamId: string
 ): AntigravityFallbackChainErrorOutcome {
   // Abort signal (user disconnect) — propagate immediately, do not retry.
-  if (isAntigravityAbortError(input, error)) {
+  if (isAntigravityAbortError(input, error) || isAntigravityPreResponseTimeout(error)) {
     return { action: "throw", error };
   }
   if (i < chain.length - 1) {

open-sse/handlers/chatCore/nonStreamingProviderLeg.ts

@@ -25,6 +25,8 @@ import { extractUsageFromResponse } from "../usageExtractor.ts";
 import { sanitizeUsagePayloadForRequest } from "../../utils/usageTracking.ts";
 import { createErrorResult, formatProviderError } from "../../utils/error.ts";
 import { isLocalStreamLifecycleError } from "@/shared/utils/circuitBreaker";
+import { isAntigravityPreResponseTimeout } from "../../services/antigravityTimeout.ts";
+import { ANTIGRAVITY_PRE_RESPONSE_TIMEOUT_CODE } from "../../config/constants.ts";
 import { unwrapClinepassEnvelope } from "../../utils/clinepassEnvelope.ts";
 import { unwrapClineNonStreamingEnvelope } from "./clineResponseEnvelope.ts";
 import {
@@ -513,7 +515,18 @@ export async function runNonStreamingProviderLeg(
       connectionId,
       model: currentModel,
     });
-    const errorResult = legError(failureStatus, failureMessage, error);
+    // Preserve the PRIVATE exact pre-response timeout marker across the
+    // non-streaming leg. The status-only legError used to discard it, so the
+    // outer router saw an indistinguishable generic 504 and could not score
+    // account+model health or attribute the account switch correctly.
+    const isHeaderTimeout =
+      (provider === "antigravity" || provider === "agy") &&
+      isAntigravityPreResponseTimeout(error);
+    const errorResult = legError(
+      failureStatus, failureMessage, error, null,
+      isHeaderTimeout ? ANTIGRAVITY_PRE_RESPONSE_TIMEOUT_CODE : undefined,
+      isHeaderTimeout ? "upstream_timeout" : undefined
+    );
     return {
       kind: "error",
       result: errorResult as ChatCoreErrorResult,

open-sse/handlers/chatCore/upstreamTimeouts.ts

@@ -1,5 +1,6 @@
 import {
   EXECUTOR_CONTRACT_VIOLATION_CODE,
+  ANTIGRAVITY_PRE_RESPONSE_TIMEOUT_CODE,
   FETCH_TIMEOUT_MS,
   HTTP_STATUS,
 } from "../../config/constants.ts";
@@ -47,6 +48,12 @@ export function createUpstreamStartTimeoutError(
     `Upstream request did not return response headers after ${timeoutMs}ms (${provider}/${model})`
   );
   err.name = "TimeoutError";
+  // The outer per-request header timer is another producer of the SAME
+  // Antigravity pre-response condition as the executor's own readiness
+  // timer. Carry a private marker through the leg, never a raw URL/token.
+  if (provider === "antigravity" || provider === "agy") {
+    (err as Error & { code?: string }).code = ANTIGRAVITY_PRE_RESPONSE_TIMEOUT_CODE;
+  }
   return err;
 }