feat: add configurable upstream AI request timeouts

[steebchen]

Summary

Implements a timeout mechanism for upstream AI API calls to ensure timeouts are caught gracefully before the overall gateway request times out.

• AI_REQUEST_TIMEOUT_MS (default 4 minutes) is configurable via environment variables
• Shorter than GATEWAY_TIMEOUT_MS to allow error handling before client timeout
• Both streaming and non-streaming requests properly handle timeout errors (502 status)
• Comprehensive test coverage with 3 new timeout-specific tests
• Mock server supports timeout simulation for testing

Test plan

• Run pnpm test:unit to verify all 369 unit tests pass including 3 new timeout tests
• Existing tests continue to pass (20 tests in api.spec.ts)
• Timeout tests simulate 5s upstream delay with 2s configured timeout

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added configurable timeout settings: gateway timeout and AI-request timeout environment variables with sensible defaults.

• Improvements

  • Unified timeout/cancellation handling with clearer classification and tailored client notifications.
  • Enhanced logging to include timeout, cancellation, and upstream error context.

• Tests

  • Added timeout-focused tests covering streaming, non‑streaming, and success-under-timeout scenarios; introduced test helpers to simulate upstream delays.

• Documentation

  • Extended environment example with guidance for the new timeout variables.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Walkthrough

Introduces centralized timeout configuration, integrates unified timeout/cancellation signals and upstream-timeout handling into the chat flow, adds tests for timeout scenarios, and extends the mock OpenAI server with message-triggered artificial delays. No behavioral changes outside timeout/cancellation/error paths.

Changes

Cohort / File(s)	Summary
Env & Timeout Config .env.example, apps/gateway/src/lib/timeout-config.ts	Documented GATEWAY_TIMEOUT_MS and AI_REQUEST_TIMEOUT_MS; added a new module providing getters, legacy constants, timeout signal creation, combined abort signal creation, and timeout/cancellation detection helpers.
Chat handler updates apps/gateway/src/chat/chat.ts	Replaced per-request cancellation with createCombinedSignal, added isTimeoutError detection, distinct upstream-timeout error path with logging, SSE error emission, and extended cancellation/error logging; unified fetch signal usage across streaming and non-streaming flows.
Tests & Mocks apps/gateway/src/api.spec.ts, apps/gateway/src/test-utils/mock-openai-server.ts, apps/gateway/src/test-utils/...	Added timeout handling test suite (non-streaming, streaming, success-under-timeout), imported new readAll helper in tests, and added extractTimeoutDelay/delay utilities in mock server to simulate upstream delays; note: tests were duplicated in the spec file in the diff.
Test helpers export change apps/gateway/src/test-utils/test-helpers.js	Export readAll (imported by tests).

Sequence Diagram

sequenceDiagram
    participant Client
    participant Gateway as Gateway<br/>(apps/gateway/src/chat/chat.ts)
    participant TimeoutCfg as Timeout Config<br/>(apps/gateway/src/lib/timeout-config.ts)
    participant Provider as Upstream Provider
    participant Logger as Logger / SSE

    Client->>Gateway: Send chat request
    Gateway->>TimeoutCfg: createCombinedSignal(cancellationController)
    TimeoutCfg->>TimeoutCfg: AbortSignal.timeout(AI_REQUEST_TIMEOUT_MS)
    TimeoutCfg-->>Gateway: fetchSignal
    Gateway->>Provider: fetch(..., { signal: fetchSignal })

    alt Upstream timeout fires
        TimeoutCfg->>Provider: aborts via signal
        Provider-->>Gateway: throws TimeoutError
        Gateway->>Logger: log upstream_timeout with context
        Gateway->>Logger: emit SSE error (upstream_timeout)
        Gateway-->>Client: terminate stream / send error event
    else Request cancelled
        Gateway->>Logger: log cancellation & billing notes
        Gateway->>Logger: emit SSE cancellation event
        Gateway-->>Client: terminate stream (cancellation)
    else Upstream responds in time
        Provider-->>Gateway: response
        Gateway-->>Client: deliver content (stream or final response)
    end

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• feat(tests): parallelize E2E tests #818 — updates test-helpers to export readAll, matching test import changes here.
• fix(gateway): add SSE keepalive pings #1541 — related to streaming lifecycle and SSE keepalive/cleanup changes overlapping chat streaming behavior.
• fix: handle fetch timeouts #1125 — earlier changes to upstream fetch timeout/error handling and SSE/logging in chat.ts.

Suggested labels

auto-merge

Suggested reviewers

• smakosh

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'feat: add configurable upstream AI request timeouts' accurately and concisely summarizes the main change—implementing environment-configurable timeout handling for upstream AI API calls.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In `@apps/gateway/src/chat/chat.ts`:
- Around line 2796-2889: Timeout error handler in the streaming block (the
branch that calls isTimeoutError(error)) sends an "error" SSE but does not send
the final "[DONE]" event; update that handler (the code that calls
stream.writeSSE with event: "error", id: String(eventId++)) to also call
stream.writeSSE a second time sending the completion message/data "[DONE]" (same
format used in other handlers) after the error event so clients receive the
terminal event and the stream is closed consistently.

🧹 Nitpick comments (1)

apps/gateway/src/lib/timeout-config.ts (1)

32-35: Consider removing or deprecating these legacy exports.

These constants are read at module load time, which means they won't reflect runtime environment changes (e.g., in tests). The comment warns against using them, but they could lead to subtle bugs if used inadvertently.