Repository navigation
Conversation
Monthly audit found several docs no longer matching code: Developer role now requires Enterprise, default developer budgets, self-refunds and invoice downloads, credits runway, provider key edit limits, and a stale audit-log action list. On the gateway side: wrong cache-key list, a fabricated error code, missing provider rate-limit headers, incomplete session-id header chains, undocumented n parameter and OpenAI prompt-cache fields, and X-No-Fallback on /v1/messages. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BMTWMmXcbPYq4D6nArXPDU
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Contributor
WalkthroughThis documentation update covers gateway fallback, routing, caching, multiple choices, reasoning errors, audit events, billing, account management, and rate-limit behavior. ChangesGateway feature documentation
Administration and account documentation
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
Syncs apps/docs content to match current gateway/API behavior and dashboard UX discovered during a documentation audit, reducing mismatches between documented and actual features.
Changes:
- Updates billing/team/transactions/provider-keys docs to match current API/UI behavior (roles, budgets, refunds, invoices, edit semantics).
- Updates gateway feature docs for session stickiness, caching semantics, routing retries, and rate-limit behavior/headers.
- Adds a new “Multiple Choices” page documenting OpenAI-compatible
nbehavior and its constraints.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| apps/docs/content/resources/rate-limits.mdx | Clarifies paid vs enterprise-configured limits; documents which rate-limit headers appear and when. |
| apps/docs/content/learn/transactions.mdx | Expands transaction taxonomy and documents refunds + invoice/credit-note downloads. |
| apps/docs/content/learn/team.mdx | Clarifies Developer role plan gating and documents org-wide default developer budget. |
| apps/docs/content/learn/provider-keys.mdx | Corrects what “Edit” can change and what is immutable after creation. |
| apps/docs/content/learn/billing.mdx | Documents credits runway estimate shown in the dashboard. |
| apps/docs/content/features/sessions.mdx | Updates sticky session resolution chain to match implementation. |
| apps/docs/content/features/routing.mdx | Aligns session-id chain; expands same-provider retry behavior details. |
| apps/docs/content/features/reasoning.mdx | Fixes example error payloads (code: null vs fabricated codes). |
| apps/docs/content/features/multiple-choices.mdx | New page documenting n and streaming/provider constraints. |
| apps/docs/content/features/caching/provider-cache-control.mdx | Documents OpenAI prompt-caching fields forwarded by the gateway. |
| apps/docs/content/features/caching/gateway-caching.mdx | Corrects cache-key inputs to match what is actually hashed. |
| apps/docs/content/features/audit-logs.mdx | Expands documented audit actions to cover more real event categories. |
| apps/docs/content/features/anthropic-endpoint.mdx | Documents x-no-fallback behavior for /v1/messages. |
Suppressed comments (1)
apps/docs/content/resources/rate-limits.mdx:53
- The example implies
X-RateLimit-Resetis always present on free-model responses, but the gateway only setsX-RateLimit-Reset(andRetry-After) when the request is actually rate-limited (429). For allowed requests it only setsX-RateLimit-LimitandX-RateLimit-Remaining.
```http
X-RateLimit-Limit: 20
X-RateLimit-Remaining: 19
X-RateLimit-Reset: 1640995200
</details>
---
💡 <a href="/theopenco/llmgateway/new/main?filename=.github/instructions/*.instructions.md" class="Link--inTextBlock" target="_blank" rel="noopener noreferrer">Add Copilot custom instructions</a> for smarter, more guided reviews. <a href="https://docs.github.com/en/copilot/customizing-copilot/adding-repository-custom-instructions-for-github-copilot" class="Link--inTextBlock" target="_blank" rel="noopener noreferrer">Learn how to get started</a>.
| `n` is only forwarded when the resolved model supports it upstream: | ||
|
|
||
| - **OpenAI Chat Completions** models | ||
| - **Google Gemini** models (mapped to Gemini's native `candidateCount` parameter) |
Comment on lines
405
to
+409
| Retries can still happen within the same provider when multiple keys are | ||
| configured and the current key fails with a retryable error. | ||
| configured and the current key fails with a retryable error — including a | ||
| **401/403 auth failure**, since that's often isolated to a single bad | ||
| credential rather than the whole provider. Same-key retries back off | ||
| exponentially (1s, 2s, 4s, ...) up to **2 retries** by default (configurable |
Comment on lines
+40
to
+42
| Eligible transactions — credit top-ups and DevPass/Chat plan payments, within a short window and below a usage threshold — show a **Request Refund** action so the organization owner can self-serve a refund without contacting support. Refunds are processed through Stripe; the transaction and credit balance update once Stripe confirms the refund via webhook. | ||
|
|
||
| Completed payments with an associated Stripe invoice offer a **Download Invoice** action to get a PDF invoice or credit note for that transaction. |
Provider/model rate limits are configured by LLM Gateway staff via the internal admin dashboard (ADMIN_EMAILS-gated), not something organizations configure themselves, and aren't Enterprise-only. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BMTWMmXcbPYq4D6nArXPDU
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 12
🧹 Nitpick comments (1)
apps/docs/content/resources/rate-limits.mdx (1)
59-69: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winDocument both provider rate-limit windows.
X-RateLimit-Limit-ProviderandX-RateLimit-Remaining-Providercurrently report the primary provider limit. Add the RPD headers to this example using the same suffix style as the documented RPM headers.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/docs/content/resources/rate-limits.mdx` around lines 59 - 69, Update the provider/model rate-limit header example to document the daily window alongside RPM: add X-RateLimit-Limit-Provider-RPD and X-RateLimit-Remaining-Provider-RPD using the existing suffix style and representative values.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/docs/content/features/anthropic-endpoint.mdx`:
- Around line 185-187: Update the “Provider Fallback” paragraph to limit
fallback to model-id or session-selected providers. Explicitly state that an
explicit provider/model prefix, or the X-No-Fallback: true header, disables
cross-provider fallback, and preserve the existing reference to the header
documentation.
In `@apps/docs/content/features/caching/gateway-caching.mdx`:
- Around line 68-75: Add tools and tool_choice to the gateway cache payload used
by generateCacheKey(), ensuring different tool schemas or selections produce
distinct cache keys. Update the documented gateway cache contract in
gateway-caching.mdx and add a regression test that verifies requests differing
only in tool contracts do not share cached responses.
In `@apps/docs/content/features/caching/provider-cache-control.mdx`:
- Line 185: Update the prompt_cache_options example in the caching documentation
to use valid JSON, removing the schema-style "implicit" | "explicit" value.
Describe the permitted mode values in prose or provide separate valid payload
examples while preserving the documented default and TTL behavior.
- Line 185: Update the prompt_cache_options documentation to remove the
catalogue-derived “GPT-5.6 and later” cutoff, link to the live model catalogue
or model capability page, and describe availability in terms of models
supporting the prompt-caching capability while preserving the implicit and
explicit mode behavior.
- Line 187: Update the provider forwarding statement in
provider-cache-control.mdx to apply only to prompt_cache_retention and
prompt_cache_options, excluding prompt_cache_key. Keep the existing
documentation and runtime behavior unchanged unless intentionally synchronizing
all affected docs and forwarding logic.
In `@apps/docs/content/features/multiple-choices.mdx`:
- Around line 28-34: Update the “Supported Providers” section in
multiple-choices.mdx to remove the hardcoded OpenAI and Google provider list,
describe support through resolved model mappings, and link readers to the live
model catalogue-generated page for current provider coverage.
- Around line 30-35: Update the rejection statement in the multiple-choices
documentation to apply only when n is greater than 1, matching the model mapping
contract; keep the supported-provider forwarding statements unchanged.
In `@apps/docs/content/features/routing.mdx`:
- Around line 405-411: Update the earlier retry rule in the routing
documentation to apply specifically to cross-provider fallback, not
same-provider key retries. Clarify that 401/403 failures are retried only with
another configured key for the same provider, while cross-provider fallback does
not retry them.
In `@apps/docs/content/learn/billing.mdx`:
- Around line 17-18: Update the Credits page runway description in the billing
documentation to state that it is based on the recent seven-day average spend
and that the page displays “No recent usage” when that average is zero; remove
wording that implies a separate minimum usage-history requirement.
In `@apps/docs/content/learn/team.mdx`:
- Around line 71-74: Update the “Default Developer Budget” section to document
both org-wide defaults: the $500/month recurring budget and the separate cap of
3 active API keys. Replace the vague “usage limit and period usage limit”
wording with explicit recurring and period budget terminology, while preserving
the statement that an individual member budget overrides the default.
In `@apps/docs/content/learn/transactions.mdx`:
- Around line 28-36: Update the transaction type-label mapping used by
TransactionsClient to cover all dev_plan_*, chat_plan_*, and end_user_* values
listed in the transaction table, ensuring both desktop and billing UI display
human-readable labels instead of raw enum values; alternatively, narrow the
transaction query to return only types already supported by that mapping.
In `@apps/docs/content/resources/rate-limits.mdx`:
- Around line 41-43: Update the rate-limit documentation paragraph to
consistently say “provider/model-specific limits” instead of “provider-specific
limits,” and clarify that LLM Gateway staff configure these provider/model caps
through the internal admin dashboard; organizations cannot configure them as
self-service settings, and the caps apply beyond Enterprise plans.
---
Nitpick comments:
In `@apps/docs/content/resources/rate-limits.mdx`:
- Around line 59-69: Update the provider/model rate-limit header example to
document the daily window alongside RPM: add X-RateLimit-Limit-Provider-RPD and
X-RateLimit-Remaining-Provider-RPD using the existing suffix style and
representative values.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 6cd84c7b-ed2c-4e43-b5cb-86cca4cb7685
📒 Files selected for processing (13)
apps/docs/content/features/anthropic-endpoint.mdxapps/docs/content/features/audit-logs.mdxapps/docs/content/features/caching/gateway-caching.mdxapps/docs/content/features/caching/provider-cache-control.mdxapps/docs/content/features/multiple-choices.mdxapps/docs/content/features/reasoning.mdxapps/docs/content/features/routing.mdxapps/docs/content/features/sessions.mdxapps/docs/content/learn/billing.mdxapps/docs/content/learn/provider-keys.mdxapps/docs/content/learn/team.mdxapps/docs/content/learn/transactions.mdxapps/docs/content/resources/rate-limits.mdx
Comment on lines
+185
to
+187
| ### Provider Fallback | ||
|
|
||
| Like `/v1/chat/completions`, this endpoint falls back to the next healthy provider if the pinned one fails. Send `x-no-fallback: true` to disable fallback and see the pinned provider's actual error — see [Disabling Fallback with X-No-Fallback Header](/features/routing#disabling-fallback-with-x-no-fallback-header) for details. |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Scope provider fallback to the routing mode that supports it.
apps/docs/content/features/routing.mdx documents provider-prefixed requests as having no alternate-provider fallback. This paragraph does not distinguish a session pin from an explicit provider/model prefix. State that fallback applies to model-id or session-selected providers. State that an explicit provider prefix or X-No-Fallback: true disables cross-provider fallback.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/docs/content/features/anthropic-endpoint.mdx` around lines 185 - 187,
Update the “Provider Fallback” paragraph to limit fallback to model-id or
session-selected providers. Explicitly state that an explicit provider/model
prefix, or the X-No-Fallback: true header, disables cross-provider fallback, and
preserve the existing reference to the header documentation.
Comment on lines
+68
to
+75
| - Provider and model identifier | ||
| - Messages array (roles and content) | ||
| - Temperature | ||
| - Temperature, Top P, frequency/presence penalty | ||
| - Max tokens | ||
| - Top P | ||
| - Tools/functions | ||
| - Tool choice | ||
| - Response format | ||
| - System prompt | ||
| - Other model-specific parameters | ||
| - Reasoning effort and reasoning max tokens | ||
| - Prompt-cache key, retention, and options | ||
| - `n` (number of choices) and service tier |
Contributor
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Expect both fields in the serialized cache key, or an explicit cache bypass.
rg -n -C 8 \
'cache[_-]?key|cacheKey|tool_choice|tools' \
apps/api apps/gateway \
--glob '*.{ts,tsx}'Repository: theopenco/llmgateway
Length of output: 50377
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '--- cache key generation definitions/usages ---\n'
rg -n "function generateCacheKey|function generateStreamingCacheKey|generateCacheKey|generateStreamingCacheKey|cachePayload|cacheKey" apps/gateway apps/api --glob '*.{ts,tsx}' | head -n 200
printf '\n--- file with cachePayload definition ---\n'
sed -n '5430,5475p' apps/gateway/src/chat/chat.ts
printf '\n--- cache key test files ---\n'
fd -a ".*cache.*spec.*|.*cache.*test.*|.*caching.*spec.*|.*caching.*test.*" apps --glob '*.{ts,tsx}' | sed 's#^\./##'Repository: theopenco/llmgateway
Length of output: 249
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- cache key generation definitions/usages ---'
rg -n "function generateCacheKey|function generateStreamingCacheKey|generateCacheKey|generateStreamingCacheKey|cachePayload|cacheKey" apps/gateway apps/api --glob '*.{ts,tsx}' | head -n 200 || true
echo
echo '--- payload around gateway cachePayload definition ---'
sed -n '5430,5475p' apps/gateway/src/chat/chat.ts
echo
echo '--- cache spec/test files ---'
fd -a '.*cache.*spec.*|.*cache.*test.*|.*caching.*spec.*|.*caching.*test.*' apps --glob '*.{ts,tsx}' | sed 's#^\./##' | head -n 100 || trueRepository: theopenco/llmgateway
Length of output: 3033
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- locate cacheKey module ---'
rg -n "generateCacheKey|generateStreamingCacheKey" apps/gateway/src apps/gateway --glob '*.{ts,tsx}' | head -n 160
echo
echo '--- cache key implementation files ---'
git ls-files 'apps/gateway/src/**/*.ts' 'apps/gateway/src/**/*.tsx' | xargs rg -l "generateCacheKey|generateStreamingCacheKey" | sed 's#^\./##'Repository: theopenco/llmgateway
Length of output: 849
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- apps/gateway/src/chat/chat.ts imports and cache key exports ---'
sed -n '1,150p' apps/gateway/src/chat/chat.ts
echo
echo '--- all cache-related references in gateway chat file ---'
rg -n -C 5 "cache|Cache|Caching|caching|no-cache|x-no-cache|generateCacheKey|generateStreamingCacheKey" apps/gateway/src/chat/chat.ts | head -n 240 || true
echo
echo '--- package/config hints ---'
git ls-files | rg -n '(^|/)cache|cache$|gateway-caching|caching' || true
rg -n "generateCacheKey|generateStreamingCacheKey" . --glob '!node_modules' --glob '!dist' --glob '!build' | head -n 200 || trueRepository: theopenco/llmgateway
Length of output: 15444
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- packages/cache/src/cache.ts ---'
cat -n packages/cache/src/cache.ts
echo
echo '--- deterministic cache key shape probe from source payload mapping ---'
python3 - <<'PY'
payload = {
"provider": "provider-A",
"model": "model-1",
"messages": [{"role": "user", "content": "same"}],
"temperature": 0.5,
"max_tokens": 100,
"top_p": 0.9,
"frequency_penalty": 0,
"presence_penalty": 0,
"response_format": {"type": "json_object"},
"reasoning_effort": "medium",
"reasoning_max_tokens": 256,
"prompt_cache_key": "key",
"prompt_cache_retention": None,
"prompt_cache_options": None,
"n": 1,
"service_tier": None,
}
payload_with_tools_a = {
**payload,
"tools": [{"type": "function", "name": "a"}],
"tool_choice": None,
}
payload_with_tools_b = {
**payload,
"tools": [{"type": "function", "name": "b"}],
"tool_choice": None,
}
# model implementation serializes only this fixed object key order via JSON.stringify(payload)
import json
print(json.dumps([
json.dumps(payload),
json.dumps(payload_with_tools_a),
json.dumps(payload_with_tools_b),
json.dumps(payload_with_tools_a) == json.dumps(payload_with_tools_b),
]))
PYRepository: theopenco/llmgateway
Length of output: 4874
Add tools and tool_choice to the gateway cache contract.
cachePayload does not include tools or tool_choice, while generateCacheKey() hashes JSON.stringify(payload). Requests with the same messages and parameters but different tool schemas or tool selection can replay a cached response from another tool contract. Add both fields to the gateway-level cache key and a regression test for different tool contracts before publishing the documented key contract.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/docs/content/features/caching/gateway-caching.mdx` around lines 68 - 75,
Add tools and tool_choice to the gateway cache payload used by
generateCacheKey(), ensuring different tool schemas or selections produce
distinct cache keys. Update the documented gateway cache contract in
gateway-caching.mdx and add a regression test that verifies requests differing
only in tool contracts do not share cached responses.
|
|
||
| - `prompt_cache_key` — a string used to improve cache routing for requests that share a prompt prefix. It also doubles as a fallback [sticky session](/features/sessions) key if no session header is set. | ||
| - `prompt_cache_retention` — `"in_memory"` or `"24h"`, OpenAI's cache retention policy for eligible models. | ||
| - `prompt_cache_options` — explicit prompt caching controls for GPT-5.6 and later: `{ "mode": "implicit" | "explicit", "ttl": "30m" }`. In `implicit` mode (the default) OpenAI places an automatic cache breakpoint at the latest message. In `explicit` mode, only content parts you mark with a `prompt_cache_breakpoint` object are cached. |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file="apps/docs/content/features/caching/provider-cache-control.mdx"
if [ -f "$file" ]; then
echo "== file excerpt =="
sed -n '160,200p' "$file" | cat -n
else
echo "missing $file"
fi
echo "== nearby references to prompt_cache_options / prompt_cache_breakpoint ========="
rg -n "prompt_cache_options|prompt_cache_breakpoint|GPT-5\.6|implicit|explicit" apps/docs/content -S || true
echo "== JSON syntax probe for the exact object text =="
python3 - <<'PY'
import json, pathlib, re
p=pathlib.Path('apps/docs/content/features/caching/provider-cache-control.mdx')
s=p.read_text()
m=re.search(r'`prompt_cache_options`.+?\[(.*?)\]', s, re.S)
print("has_match", bool(m))
if m:
print("object_text")
print(m.group(0))
try:
json.loads(m.group(0))
print("json_parse=ok")
except Exception as e:
print("json_parse_error", type(e).__name__, e)
PYRepository: theopenco/llmgateway
Length of output: 12345
Use valid JSON for prompt_cache_options.
The object at line 185 includes "implicit" | "explicit", which is schema notation and invalid JSON. Describe the allowed values in prose or show separate valid request payloads.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/docs/content/features/caching/provider-cache-control.mdx` at line 185,
Update the prompt_cache_options example in the caching documentation to use
valid JSON, removing the schema-style "implicit" | "explicit" value. Describe
the permitted mode values in prose or provide separate valid payload examples
while preserving the documented default and TTL behavior.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== target file excerpt =="
sed -n '160,205p' apps/docs/content/features/caching/provider-cache-control.mdx || true
echo
echo "== occurrences of GPT-5.6 and later in target =="
rg -n "GPT-5\.6|later|prompt_cache_breakpoint|prompt_cache_options" apps/docs/content/features/caching/provider-cache-control.mdx || true
echo
echo "== docs catalogue/live-page clues =="
rg -n "catalogue|catalog|gpt-5|GPT-5|model.*capability|capability.*page|implicit|explicit" apps/docs/content -g '*.md' -g '*.mdx' | head -200 || trueRepository: theopenco/llmgateway
Length of output: 40209
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== repository files related to catalog/models/caching =="
git ls-files | rg '(^|/)(catalog|model|provider|caching|capability|gateway|README|.*\.ts$|.*\.js$|.*\.json$)' | head -300
echo
echo "== prompt_cache_options definitions/usages in repo =="
rg -n "prompt_cache_options|prompt_cache_breakpoint|prompt_cache_retention|prompt_cache_key|cache_control|openai" . --glob '!**/node_modules/**' --glob '!**/.git/**' | head -300 || true
echo
echo "== docs catalogue/live link patterns nearby =="
rg -n "llmgateway\.io/models|models\.llmgateway|provider catalog|catalogue|catalog" apps/docs/content/features apps/docs/content/learn apps/docs/content/developers apps/docs/content/guides -g '*.md' -g '*.mdx' | head -200 || true
echo
echo "== read docs models/caching docs relevant links =="
for f in apps/docs/content/learn/models.mdx apps/docs/content/features/models-directory.mdx apps/docs/content/features/caching/index.mdx; do
if [ -f "$f" ]; then
echo "--- $f ---"
sed -n '1,80p' "$f"
fi
doneRepository: theopenco/llmgateway
Length of output: 11320
Replace the GPT-5.6 and later cutoff.
GPT-5.6 and later is a catalogue-derived model constraint. Link to the live model catalogue or model capability page instead, and describe support by capability.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/docs/content/features/caching/provider-cache-control.mdx` at line 185,
Update the prompt_cache_options documentation to remove the catalogue-derived
“GPT-5.6 and later” cutoff, link to the live model catalogue or model capability
page, and describe availability in terms of models supporting the prompt-caching
capability while preserving the implicit and explicit mode behavior.
Source: Coding guidelines
| - `prompt_cache_retention` — `"in_memory"` or `"24h"`, OpenAI's cache retention policy for eligible models. | ||
| - `prompt_cache_options` — explicit prompt caching controls for GPT-5.6 and later: `{ "mode": "implicit" | "explicit", "ttl": "30m" }`. In `implicit` mode (the default) OpenAI places an automatic cache breakpoint at the latest message. In `explicit` mode, only content parts you mark with a `prompt_cache_breakpoint` object are cached. | ||
|
|
||
| These fields are only forwarded to OpenAI models that support them; they have no effect on other providers. |
Contributor
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C 6 \
'prompt_cache_key|prompt_cache_retention|prompt_cache_options' \
apps/api apps/gateway apps/docs \
--glob '*.{ts,tsx,md,mdx}'Repository: theopenco/llmgateway
Length of output: 39968
Narrow the OpenAI-only forwarding statement.
The list includes prompt_cache_key, but apps/docs/content/features/sessions.mdx documents it on OpenAI, Meta, and Azure Responses as well as an Azure chat-completions caveat. Limit this sentence to prompt_cache_retention and prompt_cache_options, or update both docs and runtime forwarding to match.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/docs/content/features/caching/provider-cache-control.mdx` at line 187,
Update the provider forwarding statement in provider-cache-control.mdx to apply
only to prompt_cache_retention and prompt_cache_options, excluding
prompt_cache_key. Keep the existing documentation and runtime behavior unchanged
unless intentionally synchronizing all affected docs and forwarding logic.
Comment on lines
+28
to
+34
| ## Supported Providers | ||
|
|
||
| `n` is only forwarded when the resolved model supports it upstream: | ||
|
|
||
| - **OpenAI Chat Completions** models | ||
| - **Google Gemini** models (mapped to Gemini's native `candidateCount` parameter) | ||
|
|
Contributor
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Replace the hardcoded provider list with a live catalogue link.
This section hardcodes a catalogue-derived provider list. Describe support in terms of resolved model mappings and link to the live model catalogue.
Proposed direction
`n` is only forwarded when the resolved model supports it upstream:
-- **OpenAI Chat Completions** models
-- **Google Gemini** models (mapped to Gemini's native `candidateCount` parameter)
+See the [live model catalogue](/models) for current mappings that support `n`.As per coding guidelines, apps/docs/**/*.{md,mdx} must never hardcode catalogue-derived lists of models or providers; documentation must link to live catalogue-generated pages instead.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| ## Supported Providers | |
| `n` is only forwarded when the resolved model supports it upstream: | |
| - **OpenAI Chat Completions** models | |
| - **Google Gemini** models (mapped to Gemini's native `candidateCount` parameter) | |
| ## Supported Providers | |
| `n` is only forwarded when the resolved model supports it upstream: | |
| See the [live model catalogue](/models) for current mappings that support `n`. |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/docs/content/features/multiple-choices.mdx` around lines 28 - 34, Update
the “Supported Providers” section in multiple-choices.mdx to remove the
hardcoded OpenAI and Google provider list, describe support through resolved
model mappings, and link readers to the live model catalogue-generated page for
current provider coverage.
Source: Coding guidelines
Comment on lines
405
to
+411
| Retries can still happen within the same provider when multiple keys are | ||
| configured and the current key fails with a retryable error. | ||
| configured and the current key fails with a retryable error — including a | ||
| **401/403 auth failure**, since that's often isolated to a single bad | ||
| credential rather than the whole provider. Same-key retries back off | ||
| exponentially (1s, 2s, 4s, ...) up to **2 retries** by default (configurable | ||
| via the `SAME_KEY_MAX_RETRIES` environment variable), unlike cross-provider | ||
| fallback which retries immediately against a different upstream. |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Distinguish same-provider key retries from cross-provider fallback.
Lines 391-394 state that retries do not trigger for 401/403 failures. This paragraph states that same-provider retries can include 401/403. Rewrite the earlier rule as applying to cross-provider fallback, and state that 401/403 retries occur only against another key for the same provider.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/docs/content/features/routing.mdx` around lines 405 - 411, Update the
earlier retry rule in the routing documentation to apply specifically to
cross-provider fallback, not same-provider key retries. Clarify that 401/403
failures are retried only with another configured key for the same provider,
while cross-provider fallback does not retry them.
Comment on lines
+17
to
+18
| Once your organization has enough usage history, the Credits page also shows an estimated **runway** — approximately how many days your current balance will last at your recent average daily spend. | ||
|
|
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Document the actual runway window and no-usage condition.
The Credits page reads avgDailySpend7d and displays No recent usage when that value is zero. The current wording does not state the seven-day window and may imply a separate minimum-history rule. Document the seven-day average and the no-usage behavior.
Suggested wording
-Once your organization has enough usage history, the Credits page also shows an estimated **runway** — approximately how many days your current balance will last at your recent average daily spend.
+When your organization has recent usage, the Credits page also shows an estimated **runway** — approximately how many days your current balance will last based on your average daily spend over the last seven days.🧰 Tools
🪛 LanguageTool
[style] ~17-~17: This is not the usual sequence for adjectives that have no special emphasis.
Context: ... your current balance will last at your recent average daily spend. ## Fees Top-ups are char...
(EN_ADJ_ORDER)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/docs/content/learn/billing.mdx` around lines 17 - 18, Update the Credits
page runway description in the billing documentation to state that it is based
on the recent seven-day average spend and that the page displays “No recent
usage” when that average is zero; remove wording that implies a separate minimum
usage-history requirement.
Comment on lines
+71
to
+74
|
|
||
| ## Default Developer Budget | ||
|
|
||
| Owners and admins on Enterprise can set an org-wide **default developer budget** — a usage limit and period usage limit applied to every Developer who doesn't have their own individual budget set. A member's own budget, when configured via **Manage budget**, always overrides the default. |
Contributor
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
rg -n -C 8 \
'SSO_TEAM_DEFAULT_DEVELOPER_BUDGET|default developer budget|periodUsageLimit|Manage budget' \
packages appsRepository: theopenco/llmgateway
Length of output: 50379
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo "== candidate files =="
git ls-files | rg '(^packages/shared/src/member-budget-limits\.ts$|^packages/db/src/member-budget\.ts$|^apps/docs/content/learn/team\.mdx$|Manage budget|ManageBudget|memberBudget)' || true
echo
echo "== docs section =="
sed -n '55,85p' apps/docs/content/learn/team.mdx | cat -n
echo
echo "== UI/budget mentions excluding docs broad search =="
rg -n -C 4 \
'defaultDeveloperMaxApiKeys|defaultDeveloperUsageLimit|defaultDeveloperPeriodUsageLimit|SSO_TEAM_DEFAULT_DEVELOPER_BUDGET|ManageBudget|Manage budget|Manage Budget|maxApiKeys|periodUsageLimit' \
apps packages \
--glob '!apps/docs/**' \
--glob '!packages/db/src/seed.ts' \
--max-count 200 || true
echo
echo "== focused budget resolver tests =="
if git ls-files | grep -q 'packages/db/src/member-budget.spec.ts'; then
sed -n '1,220p' packages/db/src/member-budget.spec.ts | cat -n
fiRepository: theopenco/llmgateway
Length of output: 50377
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo "== sso provisioning budget handling =="
sed -n '68,130p' apps/api/src/routes/sso.ts | cat -n
sed -n '420,448p' apps/api/src/routes/sso.ts | cat -n
echo
echo "== organization route default developer budget handling =="
sed -n '140,230p' apps/api/src/routes/organization.ts | cat -n
rg -n -C 4 'defaultDeveloper(MaxApiKeys|UsageLimit|PeriodUsageLimit|PeriodUsageDuration)' apps/api/src/routes/organization.ts
echo
echo "== route docs for defaultDeveloperMaxApiKeys =="
rg -n -C 8 'defaultDeveloperMaxApiKeys' apps/docs apps/api/src/routes -g '!apps/docs/content/**' || trueRepository: theopenco/llmgateway
Length of output: 21063
Document the full default developer budget cap.
The SSO default is a $500/month recurring budget plus a separate 3 active API key cap. Include both parts of the org default, and replace “usage limit” with the actual recurring/period budget wording so members do not expect an uncapped lifetime limit.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/docs/content/learn/team.mdx` around lines 71 - 74, Update the “Default
Developer Budget” section to document both org-wide defaults: the $500/month
recurring budget and the separate cap of 3 active API keys. Replace the vague
“usage limit and period usage limit” wording with explicit recurring and period
budget terminology, while preserving the statement that an individual member
budget overrides the default.
Comment on lines
+28
to
+36
| | Type | Description | | ||
| | --------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | | ||
| | **Credit Top-up / Refund / Gift** | A manual/automatic credit purchase, a refund, or credits granted by an admin | | ||
| | **Subscription Start / Cancel / End** | Legacy plan subscription lifecycle events | | ||
| | **Dev Plan Start / Upgrade / Downgrade / Cancel / Resume / End / Renewal** | [DevPass](https://llmgateway.io/devpass) subscription lifecycle events | | ||
| | **Dev Plan Reset Pass** | A purchased, rewarded, or gifted DevPass Reset Pass | | ||
| | **Chat Plan Start / Upgrade / Downgrade / Cancel / Resume / End / Renewal** | Chat plan subscription lifecycle events | | ||
| | **End-User Top-up / Refund / Bonus** | LLM SDK end-user wallet activity, for organizations issuing end-user wallets | | ||
| | **End-User Margin Accrual / Payout** | Margin earned on end-user wallet usage and its payout to the organization | |
Contributor
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
rg -n -C 6 \
'transaction\.type|devpass|dev_plan|chat_plan|end.?user|credit_(topup|refund|gift)|subscription_(start|cancel|end)' \
apps packagesRepository: theopenco/llmgateway
Length of output: 50379
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo "== candidate transaction files =="
git ls-files | rg -n 'transaction|Transaction|TransactionsClient|transactions\.mdx' || true
echo
echo "== docs snippet =="
sed -n '1,80p' apps/docs/content/learn/transactions.mdx
echo
echo "== focused source matches (top 200) =="
rg -n -C 4 \
'getTransactionLabel|formatTransactionLabel|transaction\.type|TransactionType|devPlan|chatPlan|EndUser|credit_(topup|refund|gift)|subscription_(start|cancel|end)|DevPass|Reset Pass' \
apps packages --glob '!**/node_modules/**' --glob '!**/dist/**' --glob '!**/build/**' | head -n 500Repository: theopenco/llmgateway
Length of output: 37245
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo "== dashboard transaction clients =="
wc -l apps/ui/src/components/billing/transactions-client.tsx apps/ui/src/components/dashboard/transactions-client.tsx apps/ui/src/app/dashboard/[orgId]/org/transactions/page.tsx
echo
ast-grep outline apps/ui/src/components/billing/transactions-client.tsx apps/ui/src/components/dashboard/transactions-client.tsx --view expanded || true
echo
echo "== relevant transaction-client snippets =="
sed -n '1,260p' apps/ui/src/components/billing/transactions-client.ts
sed -n '1,260p' apps/ui/src/components/dashboard/transactions-client.tsx
echo
echo "== transaction schema/files =="
git ls-files | rg 'transaction' || true
rg -n -C 3 'createTable\("transaction"|transaction.*Table|enum\s*\(|dev_plan|chat_plan|end_user|credit_{topup,refund,gift}|subscription_{start,cancel,end}' packages apps/db apps/worker -g '!**/migrations/meta/**' | head -n 400Repository: theopenco/llmgateway
Length of output: 1621
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo "== billing transaction-client snippets =="
sed -n '1,330p' apps/ui/src/components/billing/transactions-client.tsx
echo
sed -n '330,620p' apps/ui/src/components/billing/transactions-client.tsx
echo
echo "== dashboard transaction-client =="
sed -n '1,280p' apps/ui/src/components/dashboard/transactions-client.tsx
echo
echo "== transaction schema/files =="
git ls-files | rg 'transaction' || true
rg -n -C 3 'createTable\("transaction"|transaction.*Table|type: .*enum|credit_topup|credit_refund|credit_gift|subscription_start|subscription_cancel|subscription_end' packages apps/db apps/worker -g '!**/migrations/meta/**' | head -n 400Repository: theopenco/llmgateway
Length of output: 37136
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo "== transaction schema =="
sed -n '404,455p' packages/db/src/schema.ts
echo
echo "== transaction backend handlers =="
git ls-files | rg 'transaction' | rg -v '/migrations/meta/'
rg -n -C 4 'transaction|isInvoiceableTransaction|getTransaction|dev_plan_|chat_plan_|end_user|refund|credit_gift' packages apps/worker apps/api -g '!**/migrations/meta/**' | head -n 800
echo
echo "== deterministic label shape for missing type branch =="
python3 - <<'PY'
from pathlib import Path
p = Path('apps/ui/src/components/billing/transactions-client.tsx')
s = p.read_text()
cases = [
'credit_topup', 'credit_refund', 'credit_gift',
'subscription_start', 'subscription_cancel', 'subscription_end',
'dev_plan_start', 'dev_plan_upgrade', 'dev_plan_downgrade',
'dev_plan_renewal', 'dev_plan_reset_pass',
'dev_plan_cancel', 'dev_plan_resume', 'chat_plan_start',
'chat_plan_upgrade', 'chat_plan_downgrade', 'chat_plan_renewal',
'chat_plan_cancel', 'chat_plan_resume', 'chat_plan_end',
'chat_plan_reset_pass',
'end_user_topup', 'end_user_refund', 'end_user_bonus',
'end_user_margin_accrual', 'end_user_margin_payout',
]
print('cases_not_defined_in_switch=', sorted([t for t in cases if f'case "{t}"' not in s]))
PYRepository: theopenco/llmgateway
Length of output: 50376
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
echo "== schema tail for transaction enum =="
sed -n '455,485p' packages/db/src/schema.ts
echo
echo "== UI TransactionCard getTypeLabel complete block (targeted) =="
python3 - <<'PY'
from pathlib import Path
p = Path('apps/ui/src/components/billing/transactions-client.tsx')
text = p.read_text()
for line in range(1, p.stat().st_size):
if text.find('const getTypeLabel = (type: Transaction["type"])') != -1:
start = text.rfind('\n', 0, text.find('const getTypeLabel'))
break
end = text.find('const getStatusColor =', start)
print(text[start:end])
cases = sorted({
'credit_topup', 'credit_refund', 'credit_gift',
'subscription_start', 'subscription_cancel', 'subscription_end',
'dev_plan_start', 'dev_plan_upgrade', 'dev_plan_downgrade',
'dev_plan_cancel', 'dev_plan_pause' or 'dev_plan_resume',
'dev_plan_end', 'dev_plan_renewal',
'dev_plan_reset_pass', 'dev_plan_reset_pass_reward', 'dev_plan_reset_pass_gift',
'chat_plan_start', 'chat_plan_upgrade', 'chat_plan_downgrade',
'chat_plan_cancel', 'chat_plan_pause' or 'chat_plan_resume',
'chat_plan_end', 'chat_plan_renewal',
'end_user_topup', 'end_user_refund', 'end_user_bonus',
'end_user_margin_accrual', 'end_user_margin_payout',
})
PY
echo
echo "== all case labels in TransactionsClient GetTypeLabel =="
rg -n 'case ".*"' apps/ui/src/components/billing/transactions-client.tsxRepository: theopenco/llmgateway
Length of output: 1963
Add UI transaction type labels for the new transaction rows.
Transaction["type"] now includes dev_plan_*, chat_plan_*, and end_user_* values, but TransactionsClient maps only the legacy credit and subscription values and falls back to default: return type. If these rows are returned from the UI transaction API, the Type column can show raw enum values in both desktop and billing UI. Add labels for each supported type, or narrow the transaction list/query so the API only returns types with UI labels.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/docs/content/learn/transactions.mdx` around lines 28 - 36, Update the
transaction type-label mapping used by TransactionsClient to cover all
dev_plan_*, chat_plan_*, and end_user_* values listed in the transaction table,
ensuring both desktop and billing UI display human-readable labels instead of
raw enum values; alternatively, narrow the transaction query to return only
types already supported by that mapping.
Comment on lines
+41
to
+43
| **Paid AI models are not rate limited by default.** You can make as many requests as needed to paid models, subject only to your account's credit balance and any provider-specific limits. | ||
|
|
||
| LLM Gateway may apply its own requests-per-minute and requests-per-day caps per provider or model (globally, or for a specific organization) to protect shared infrastructure. When such a limit is in place, requests against it are enforced regardless of whether the model is free or paid. |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Make provider/model limit ownership explicit.
Line [41] says “provider-specific limits,” but the implementation supports provider and model scopes. Use “provider/model-specific limits” consistently.
Line [43] should state that Gateway staff configure these caps through the internal admin dashboard, that organizations cannot configure them as self-service settings, and that the caps are not limited to Enterprise plans. The current organization scope wording does not communicate these constraints.
Suggested wording
- subject only to your account's credit balance and any provider-specific limits.
+ subject only to your account's credit balance and any provider/model-specific limits.
LLM Gateway may apply its own requests-per-minute and requests-per-day caps per provider or model (globally, or for a specific organization) to protect shared infrastructure.
+Gateway staff configure these limits through the internal admin dashboard. They are not organization self-service settings and are not limited to Enterprise plans.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| **Paid AI models are not rate limited by default.** You can make as many requests as needed to paid models, subject only to your account's credit balance and any provider-specific limits. | |
| LLM Gateway may apply its own requests-per-minute and requests-per-day caps per provider or model (globally, or for a specific organization) to protect shared infrastructure. When such a limit is in place, requests against it are enforced regardless of whether the model is free or paid. | |
| **Paid AI models are not rate limited by default.** You can make as many requests as needed to paid models, subject only to your account's credit balance and any provider/model-specific limits. | |
| LLM Gateway may apply its own requests-per-minute and requests-per-day caps per provider or model (globally, or for a specific organization) to protect shared infrastructure. Gateway staff configure these limits through the internal admin dashboard. They are not organization self-service settings and are not limited to Enterprise plans. When such a limit is in place, requests against it are enforced regardless of whether the model is free or paid. |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/docs/content/resources/rate-limits.mdx` around lines 41 - 43, Update the
rate-limit documentation paragraph to consistently say “provider/model-specific
limits” instead of “provider-specific limits,” and clarify that LLM Gateway
staff configure these provider/model caps through the internal admin dashboard;
organizations cannot configure them as self-service settings, and the caps apply
beyond Enterprise plans.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Monthly documentation audit comparing
apps/gatewayandapps/apiimplementation againstapps/docs. Found several docs that no longer matched the code and fixed them.API-side (
apps/api) discrepancies:learn/team.mdx— the Developer role actually requires the Enterprise plan (project-scoped access is gated); the docs presented it as available on any plan. Added a note plus the required project selection, and documented the undocumented org-wide default developer budget feature.learn/transactions.mdx— the transaction types table listed only 5 of ~27 real types (missing the entire DevPass/Chat plan/end-user-wallet family) and omitted two live features: self-service refunds and PDF invoice/credit-note download.learn/billing.mdx— added the Credits Runway estimate, which is live in the dashboard but undocumented.learn/provider-keys.mdx— corrected the Edit action description: the API key value and base URL can never be changed after creation (onlystatusfor any provider, orname/customModelsOnly/complianceAttestationfor custom providers).features/audit-logs.mdx— the tracked-actions list documented ~19 of 78 real audit log actions, missing entire categories (master keys, custom models, SSO, SCIM, dev/chat plan, self-refunds, auto top-up). Expanded to cover all categories.Gateway-side (
apps/gateway) discrepancies:features/caching/gateway-caching.mdx— the cache-key list claimedtools/tool_choiceare part of the cache key; they are not (confirmed againstapps/gateway/src/chat/chat.ts). Corrected the list to match what's actually hashed. (Flagging for follow-up: this looks like a real cache-key bug — two requests differing only intools/tool_choicecan currently collide and replay each other's response.)features/reasoning.mdx— three example error responses showed a fabricated"code": "model_not_supported"; these errors are actually uncoded 400s (code: null). Fixed all three examples.resources/rate-limits.mdx— documented an org/provider/model-configurable rate-limit layer (Enterprise) that runs regardless of free/paid status, and corrected the "all responses include rate-limit headers" claim (only free-model requests get the plainX-RateLimit-*headers).features/routing.mdx/features/sessions.mdx— the sticky-session-id header resolution chain was incomplete/inconsistent between the two pages (missingx-session-affinity,session_id,session-id). Both now list the full 6-source chain.features/routing.mdx— expanded the same-key retry section: it also retries on 401/403 auth failures (not just network/upstream errors), with exponential backoff details.features/anthropic-endpoint.mdx— documented thatX-No-Fallbackalso works on/v1/messages(previously undocumented for this endpoint).features/caching/provider-cache-control.mdx— documented OpenAI's native prompt-caching fields (prompt_cache_key,prompt_cache_retention,prompt_cache_options), previously unmentioned as a caching feature.features/multiple-choices.mdx— documented thenparameter (multiple completions per request), which had zero doc coverage despite being fully implemented.No catalogue-derived model/provider lists were added, per repo policy.
Test plan
pnpm formatturbo run build --filter=docssucceeds🤖 Generated with Claude Code
https://claude.ai/code/session_01BMTWMmXcbPYq4D6nArXPDU
Generated by Claude Code
Summary by CodeRabbit