diff --git a/docs/cloud/usage-events-and-billing.md b/docs/cloud/usage-events-and-billing.md index 338c2f8a..02aa3fc9 100644 --- a/docs/cloud/usage-events-and-billing.md +++ b/docs/cloud/usage-events-and-billing.md @@ -24,6 +24,15 @@ Those signals support: - budget workflows - billing-oriented control-plane features +## Latency Signals On The Usage Event + +In addition to token counts, status, and cost, each usage event the data plane emits carries: + +- `latency_ms` — total elapsed time for the request, populated on every request path +- `ttft_ms` — time to first token, populated on streaming chat completions when the first chunk carrying generated output arrives (text content **or** a tool-call delta) + +`ttft_ms` is omitted from the wire when it would otherwise be zero — non-streaming, cache-hit, and error paths do not contribute a TTFT value. Use it to break down end-to-end latency into the upstream-warm-up portion versus the streaming portion. + ## Budget Relationship The most important customer-visible effect is that managed deployments can apply real budget decisions on data-plane traffic, which can result in `429` denials when a budget policy is exceeded. diff --git a/docs/configuration/api-keys.md b/docs/configuration/api-keys.md index c51f9acd..ae0fdba2 100644 --- a/docs/configuration/api-keys.md +++ b/docs/configuration/api-keys.md @@ -15,12 +15,17 @@ This resource controls who can call the proxy and which model aliases they can u - `key_hash` - `allowed_models` - optional `rate_limit` +- optional `team_id` +- optional `owner_id` -Think of those fields as three distinct control layers: +Think of those fields as four distinct control layers: - identity: `key_hash` - authorization: `allowed_models` -- policy: `rate_limit` +- inline policy: `rate_limit` +- bucket identity: `team_id` and `owner_id` + +`team_id` and `owner_id` are not access controls in themselves. They are the bucket keys that `team`-scoped and `member`-scoped [`RateLimitPolicy`](rate-limits.md#rate-limit-policy-entities) rows match against. Set them when you want a policy to span all keys belonging to the same team or to the same member. ## Create A Caller Key @@ -84,7 +89,7 @@ Example response shape: ## Rate Limits -The current rate-limit object supports: +The inline rate-limit object on `ApiKey` supports: - `tpm` - `tpd` @@ -92,9 +97,9 @@ The current rate-limit object supports: - `rpd` - `concurrency` -Current enforcement uses the API key's `rate_limit` object. Model-level `rate_limit` exists in the schema, but current hot-path enforcement is keyed off the authenticated API key. +`ApiKey.rate_limit` is one of three layers the proxy enforces. The other two are `Model.rate_limit` (inline on the resolved model) and standalone `RateLimitPolicy` rows. All applicable layers are AND-combined per request. -Use `ApiKey.rate_limit` as the real operator control today. +See [Rate Limits](rate-limits.md) for the full enforcement model and for `team`/`member`-scope policies that match against `team_id` / `owner_id`. ## Budget Boundary @@ -117,9 +122,9 @@ Check `allowed_models` first. That is an authorization failure, not an authentic Make sure the client is using the newly returned plaintext key, not the old one. -### Rate-limit behavior is not matching a model row +### Rate-limit behavior is not matching the layer you configured -That is expected today if you configured only `Model.rate_limit`. Current hot-path enforcement is centered on `ApiKey.rate_limit`. +The proxy combines `ApiKey.rate_limit`, `Model.rate_limit`, and matching `RateLimitPolicy` rows on every request. If a layer looks silent, check whether another layer is the actual gating one — a tighter `ApiKey.rate_limit` will trip before a looser `Model.rate_limit`. See [Rate Limits § Troubleshooting](rate-limits.md#troubleshooting). ## Related Pages diff --git a/docs/configuration/models.md b/docs/configuration/models.md index 1521ab28..82cdb47c 100644 --- a/docs/configuration/models.md +++ b/docs/configuration/models.md @@ -31,10 +31,13 @@ Optional fields include: - `timeout` - `rate_limit` - `cost` +- `cooldown` - `background_model_check` Read those optional fields as metadata and policy hints layered onto the basic alias mapping. +`cooldown` and `background_model_check` are the two runtime-status sources that feed [`GET /admin/v1/models/status`](../reference/admin-api-reference.md#runtime-model-status) and the [routing filter](routing-and-failover.md#runtime-filtering). Both are direct-model-only and rejected on routing models. + Example: ```bash title="Create a direct model" @@ -74,8 +77,45 @@ Current semantics: - only direct models may carry `background_model_check` - routing models reject `background_model_check` -- `ignore_statuses` records the last probe result without marking the model unhealthy +- `ignore_statuses` records the last probe result without marking the model unhealthy; the default ignored set is `[408, 429]` - `stale_after_seconds` is a safety valve for old unhealthy probe state when the checker stops refreshing +- `interval_seconds` has a minimum of `5`; `timeout_seconds`, `max_tokens`, and `stale_after_seconds` have a minimum of `1` + +A failed probe transitions the model to `unhealthy` in the runtime status tracker. A subsequent successful probe clears that state. The routing filter excludes `unhealthy` candidates ahead of `cooldown` candidates. + +### Cooldown + +`cooldown` is the request-path complement to `background_model_check`. Where the background probe sets `unhealthy` from out-of-band probes, `cooldown` sets a short-lived skip window from the failures observed on real traffic. + +```json title="Direct model cooldown" +{ + "cooldown": { + "enabled": true, + "default_seconds": 30, + "max_seconds": 600, + "honor_retry_after": true, + "trigger_statuses": [401, 408, 429, 500, 502, 503, 504], + "trigger_on_timeout": true, + "trigger_on_transport": true + } +} +``` + +All fields are optional. The example shows the *effective* defaults the proxy applies; at the schema level every field is `null` until set, but every accessor falls back to the value shown above. Omitting the `cooldown` block entirely is equivalent to writing the example above verbatim. + +Field semantics: + +- `enabled` (default `true`) — set to `false` to keep the model in rotation no matter what request-path failures look like. +- `default_seconds` (default `30`) — cooldown TTL when the upstream did not return a `Retry-After` header, or when `honor_retry_after` is `false`. Setting this to `0` disables cooldown for the model (alternative to `enabled: false`). +- `max_seconds` (default `600`) — upper bound on the cooldown TTL. Caps a misbehaving upstream that returns an unreasonable `Retry-After` value. +- `honor_retry_after` (default `true`) — when the upstream OpenAI / Anthropic bridge parses a `Retry-After: ` header, the cooldown layer uses that value (clamped by `max_seconds`). +- `trigger_statuses` (default `[401, 408, 429, 500, 502, 503, 504]`) — upstream HTTP status codes that put the target into cooldown. The default set covers auth failures, request timeouts, rate limits, and transient server errors. Caller-mistake classes (`400`, `403`, `422`) are intentionally excluded so a single bad request does not cool down a healthy upstream. +- `trigger_on_timeout` (default `true`) — request-path timeouts trigger cooldown. +- `trigger_on_transport` (default `true`) — transport, decode, and stream-abort errors trigger cooldown. + +Cooldown triggers independently of whether the failure is retryable. A `429`, for example, cools the model down even when the request itself is not retried. + +Once a target enters cooldown, the routing filter prefers other targets within the same routing model. If every candidate is filtered, behavior is governed by [`routing.on_all_filtered`](routing-and-failover.md#all-targets-filtered-policy). ## Routing Models @@ -122,6 +162,7 @@ curl -sS -X POST http://127.0.0.1:3001/admin/v1/models \ - `timeout` is in milliseconds. `0` or omission means no timeout. - `cost` stores pricing metadata used by budget and usage accounting paths. - `background_model_check` drives direct-model runtime unhealthy state and the `/admin/v1/models/status` view. +- `cooldown` drives direct-model request-path cooldown and is also surfaced through `/admin/v1/models/status`. Practical guidance: diff --git a/docs/configuration/rate-limits.md b/docs/configuration/rate-limits.md index e1533fd4..7f3642a6 100644 --- a/docs/configuration/rate-limits.md +++ b/docs/configuration/rate-limits.md @@ -1,14 +1,26 @@ --- title: Rate Limits -description: Configure per-key request, token, and concurrency limits in AISIX AI Gateway. +description: Configure multi-layer per-key, per-model, and policy-based rate limits in AISIX AI Gateway. sidebar_position: 36 --- -AISIX AI Gateway supports rate-limit fields on resources, but current runtime enforcement is centered on the authenticated API key. +AISIX AI Gateway evaluates every LLM request against multiple rate-limit layers. Each layer is independent — the request must pass **all** of them, otherwise the proxy returns `429`. -Use this page to decide where to put your real limits today and what caller-visible behavior to expect when they trigger. +Use this page to decide where each limit belongs and what caller-visible behavior to expect when a layer trips. -## Current Rate-Limit Fields +## Current Rate-Limit Sources + +The proxy applies these layers in order, on every LLM endpoint that goes through the shared quota gate: + +1. **API-key inline limit** — `ApiKey.rate_limit` on the authenticated key. +2. **Model inline limit** — `Model.rate_limit` on the resolved model. +3. **Rate-limit policy entities** — standalone `RateLimitPolicy` rows that match the current request by scope. + +Layers are AND-combined: every layer with a configured limit must have headroom, or the request is rejected before dispatch. + +## Inline Rate-Limit Fields + +`ApiKey.rate_limit` and `Model.rate_limit` share the same shape: - `tpm`: tokens per minute - `tpd`: tokens per day @@ -16,19 +28,15 @@ Use this page to decide where to put your real limits today and what caller-visi - `rpd`: requests per day - `concurrency`: maximum in-flight requests -All fields are optional. Missing fields mean no limit on that dimension. +All fields are optional. A missing field means no limit on that dimension. An empty `rate_limit` object behaves as no limit. In practice, most deployments start with: - `rpm` for request burst control - `concurrency` for in-flight protection -- optional token limits where usage-based control matters - -## Current Enforcement Boundary +- `tpm` or `tpd` where usage-based control matters -Current enforcement uses the API key's `rate_limit` object. - -Example: +Example on an API key: ```json title="ApiKey rate limits" { @@ -42,44 +50,107 @@ Example: } ``` -The shared quota gate now applies rate-limit checks across the current LLM endpoint set, not only `POST /v1/chat/completions`. +## Rate-Limit Policy Entities -That means rate limits are no longer just a chat-completions concern. +`RateLimitPolicy` is a standalone, scope-targeted rate-limit rule stored in etcd under `rate_limit_policies/`. Use it when the limit you want is not naturally attached to a single API key or model — for example, a per-team request quota or a per-member token quota. -## Response Behavior +### Policy Fields + +- `name`: human label (string, required). +- `scope`: which subject the policy targets — one of `api_key`, `model`, `team`, `member` (required). +- `scope_ref`: the resource ID the policy applies to. Interpretation depends on `scope`: + - `api_key` → matches when the authenticated `ApiKey` entry id equals `scope_ref`. + - `model` → matches when the resolved `Model` entry id equals `scope_ref`. + - `team` → matches when the authenticated `ApiKey.team_id` equals `scope_ref`. + - `member` → matches when the authenticated `ApiKey.owner_id` equals `scope_ref`. +- `window`: `second`, `minute`, or `hour` (required). +- `max_requests`: maximum requests allowed in the window (optional). +- `max_tokens`: maximum tokens allowed in the window (optional). + +At least one of `max_requests` or `max_tokens` must be set, or the policy is rejected by validation. + +### Window Mapping + +Policies are normalised to the same internal limit fields used by inline limits: + +| `window` | `max_requests` becomes | `max_tokens` becomes | +| --- | --- | --- | +| `second` | `rpm` (× 60) | `tpm` (× 60) | +| `minute` | `rpm` | `tpm` | +| `hour` | `rpd` (× 24) | `tpd` (× 24) | + +Out-of-enum window values are rejected by the JSON Schema at etcd load — the row never enters the snapshot and is surfaced through the rejection signal. + +### Example Policies -When the request is blocked by rate limiting, the proxy returns `429`. +A team-wide token cap of 1M tokens per minute: -For rate-limit-style rejections that have a retry window, the proxy can also emit `Retry-After`. +```json title="RateLimitPolicy: per-team tokens-per-minute" +{ + "name": "team-acme-tpm", + "scope": "team", + "scope_ref": "team-uuid-acme", + "window": "minute", + "max_tokens": 1000000 +} +``` + +A per-member burst limit: -Successful non-streaming chat responses also include current `x-ratelimit-*` headers based on the post-dispatch limiter state. +```json title="RateLimitPolicy: per-member requests-per-minute" +{ + "name": "member-burst", + "scope": "member", + "scope_ref": "member-uuid-1234", + "window": "minute", + "max_requests": 60 +} +``` -Those headers are useful for debugging and for client-side adaptive throttling. +For `scope = team` or `scope = member` to match, the authenticated `ApiKey` must carry the corresponding `team_id` or `owner_id` field. Set those on the API key resource at create time. -## Important Caveat +### Provisioning -`Model.rate_limit` exists in the current schema and admin surface, but the current enforcement path reads limits from the authenticated API key. +`RateLimitPolicy` rows are loaded directly from etcd into the gateway snapshot. The standalone admin API does not currently expose CRUD routes for them — write rows under `/rate_limit_policies/` through your control-plane projection or directly via `etcdctl` in self-hosted setups. -Document and operate against `ApiKey.rate_limit` as the reliable current control. +The data plane validates each row against the JSON Schema on load: a malformed row is skipped and surfaced through the rejection signal, but does not stop other rows from loading. + +## Response Behavior + +When any layer rejects the request, the proxy returns `429`. For rate-limit-style rejections that have a retry window, the proxy also emits `Retry-After`. + +Successful non-streaming chat responses include `x-ratelimit-*` headers based on the post-dispatch limiter state. Those headers are useful for debugging and for client-side adaptive throttling. ## Operator Guidance -- put caller-facing safety limits on API keys -- use concurrency limits to protect shared upstream capacity -- treat model-level limit fields as schema surface, not as the current primary enforcement tool +- put caller-facing safety limits on `ApiKey.rate_limit` +- use `Model.rate_limit` to protect a specific upstream model alias +- use `RateLimitPolicy` rows when the limit applies to a population that is wider than one key or one model — for example, a whole team +- keep token-based caps proportionate to the burst-control caps; a tight `rpm` with an unlimited `tpm` lets a single long completion still saturate upstream ## Troubleshooting ### A caller sees `429` unexpectedly -Inspect the API key's `rate_limit` object before looking at model rows. +Walk the layers in order: + +1. inspect the `ApiKey.rate_limit` on the authenticated key +2. inspect the resolved `Model.rate_limit` +3. list the `rate_limit_policies` rows that match the key's `team_id` / `owner_id` and the resolved model entry id + +Any one of those can be the gating layer. + +### A team-scope or member-scope policy is not taking effect + +Check the API key. `team` and `member` policies match against `ApiKey.team_id` and `ApiKey.owner_id` respectively. If those fields are missing on the key, the policy will never match. -### Limits appear to work for chat but not other endpoints +### Limits work for chat but appear silent on other endpoints -That should not be assumed. The current quota gate is broader than chat-only behavior. +The shared quota gate runs across the current LLM endpoint set. If you only see limits triggering on chat, the most likely explanation is that the other endpoint isn't seeing enough traffic to hit the cap, not that the gate is chat-only. ## Related Pages - [API Keys](api-keys.md) +- [Models](models.md) - [OpenAI-Compatible API](../integration/openai-compatible-api.md) - [Headers And Error Codes](../reference/headers-and-error-codes.md) diff --git a/docs/configuration/routing-and-failover.md b/docs/configuration/routing-and-failover.md index f6c501f9..339ecf3d 100644 --- a/docs/configuration/routing-and-failover.md +++ b/docs/configuration/routing-and-failover.md @@ -34,7 +34,8 @@ Each strategy answers a different operator question: ], "retries": 1, "max_fallbacks": 1, - "retry_on_429": true + "retry_on_429": true, + "on_all_filtered": "fail" } } ``` @@ -83,22 +84,30 @@ This is an important operational boundary. Routing is not a way to mask bad call ## Runtime Filtering -Before dispatch, routing consults direct-model runtime state. +Before dispatch, routing consults direct-model runtime state and produces the actual attempt list in this order: -Current filtering order is: - -- filter targets currently marked `unhealthy` -- then filter targets currently marked `cooldown` -- if cooldown filtering would empty the candidate set, the proxy falls back to retrying while still excluding `unhealthy` targets +1. partition targets into `healthy`, `cooldown`, and `unhealthy` based on the runtime status tracker +2. if any healthy targets exist, dispatch to those +3. if no healthy targets exist but at least one target is in `cooldown`, dispatch to every target whose runtime status is not `unhealthy` (cooldown candidates are preferred over background-confirmed-unhealthy ones) +4. if every target is filtered out, apply the routing model's [`on_all_filtered`](#all-targets-filtered-policy) policy The runtime state itself is exposed on `GET /admin/v1/models/status`. Source of each state: -- `cooldown` comes from request-path retryable failures on a direct target +- `cooldown` comes from request-path failures on a direct target — see [Models § Cooldown](models.md#cooldown) for the trigger configuration - `unhealthy` comes from direct-model `background_model_check` - routing models themselves are never runtime-filtered and report `not_applicable` +### All-Targets-Filtered Policy + +`routing.on_all_filtered` decides what happens when step 4 of the filter loop is reached — every candidate is excluded by runtime status: + +- `fail` (default) — return `503 all_candidates_unavailable` to the caller with `Retry-After: 30`. Use this when serving a known-broken target is worse than failing fast. +- `original_order` — dispatch to the original target list, in declaration order, ignoring runtime state for this request. Use this when availability matters more than honoring the probe verdict. + +The `Retry-After` value on the `fail` path is a coarse fixed hint. By the time the filter reaches this branch, every candidate is in background-unhealthy state with no live cooldown timer to read. + ## Design Constraints - routing targets refer to other model aliases through `targets[].model` diff --git a/docs/integration/errors-and-retries.md b/docs/integration/errors-and-retries.md index b2804f3b..63fac006 100644 --- a/docs/integration/errors-and-retries.md +++ b/docs/integration/errors-and-retries.md @@ -42,6 +42,7 @@ The proxy emits the following gateway-generated failures. The `error.type` strin | `429` | `rate_limit_exceeded` | Rate-limit rejection (per-key or per-model) | | `429` | `billing_error` (with `code: "budget_exceeded"`) | Budget rejection from the ApiKey's `max_budget_usd` | | `503` | `provider_unavailable` | No provider bridge is registered for the resolved provider | +| `503` | `all_candidates_unavailable` | Every routing candidate was excluded by runtime status (cooldown or background-unhealthy) and the routing model is configured with `on_all_filtered: fail`. The response carries `Retry-After: 30`. See [Routing And Failover § All-Targets-Filtered Policy](../configuration/routing-and-failover.md#all-targets-filtered-policy) | Bridge-level upstream failures inherit their `status` and `error.type` from the upstream provider response (see "Upstream Error Mapping" below). @@ -59,6 +60,8 @@ The `billing_error` row is the one case where `error.code` is set on the wire. T `503 provider_unavailable` is emitted on the direct-dispatch path when no bridge is registered for the resolved provider. On a routing model the same condition is absorbed into the retry/failover loop and surfaces through the per-target runtime state on `GET /admin/v1/models/status` rather than as a top-level `503` to the caller. +`503 all_candidates_unavailable` is the routing-model fail-fast response when every candidate has been removed by the runtime filter. This is distinct from `provider_unavailable`: the bridge is registered and the model is well-configured, but every target is currently in cooldown or has been marked unhealthy by `background_model_check`. + ## Upstream Error Mapping When the upstream returns `4xx`, that client-visible error class is preserved through the proxy mapping. diff --git a/docs/operations/metrics-and-logs.md b/docs/operations/metrics-and-logs.md index 26663b9e..865d6437 100644 --- a/docs/operations/metrics-and-logs.md +++ b/docs/operations/metrics-and-logs.md @@ -30,6 +30,12 @@ These signals answer different questions: - metrics: what is happening over time - usage events: what usage/accounting-oriented event was emitted on supported paths +### Streaming TTFT + +For streaming chat completions, the per-request usage event carries `ttft_ms` — the elapsed milliseconds from request entry to the first upstream chunk that contains generated content (text or tool-call delta). Role-only opening chunks are skipped so the value reflects the time to actual output, not the time to the first SSE frame. + +`ttft_ms` is meaningful only on the streaming path. Non-streaming, cache-hit, and error paths do not surface a TTFT value. + ## Response Headers With Operational Value Current response headers include: diff --git a/docs/overview/core-concepts.md b/docs/overview/core-concepts.md index f26b5965..381a345b 100644 --- a/docs/overview/core-concepts.md +++ b/docs/overview/core-concepts.md @@ -49,11 +49,25 @@ An API key also carries: - `allowed_models` - optional `rate_limit` +- optional `team_id` and `owner_id` + +`team_id` and `owner_id` are bucket identifiers consumed by `team`-scoped and `member`-scoped [`RateLimitPolicy`](#rate-limit-policy) rows. They are not access controls on their own. Managed budget paths may also reference per-key budget state, but `max_budget_usd` is not part of the current verified standalone admin write contract. An empty `allowed_models` list denies access to every model. A wildcard entry `"*"` allows access to every model in scope. +## Rate Limit Policy + +A `RateLimitPolicy` is a standalone rate-limit rule stored in etcd. Each policy targets a single subject through `(scope, scope_ref)`: + +- `api_key` — match by `ApiKey` entry id +- `model` — match by `Model` entry id +- `team` — match by `ApiKey.team_id` +- `member` — match by `ApiKey.owner_id` + +The proxy enforces all matching policies alongside the inline `ApiKey.rate_limit` and `Model.rate_limit` layers. Any layer with a configured limit can reject a request with `429`. See [Rate Limits](../configuration/rate-limits.md) for the full enforcement model. + ## Routing Model A routing model, sometimes called a virtual model, is a model with a `routing` block instead of direct provider fields. diff --git a/docs/overview/feature-matrix.md b/docs/overview/feature-matrix.md index 3dffd06d..17c15af5 100644 --- a/docs/overview/feature-matrix.md +++ b/docs/overview/feature-matrix.md @@ -26,7 +26,7 @@ Use it as a navigation aid, not as a replacement for detailed feature pages. | Standalone admin API | Available | Current admin surface includes models, API keys, provider keys, guardrails, cache policies, observability exporters, health, metrics, OpenAPI, and playground. | | API key allowlist authz | Available | Uses hashed caller keys and model allowlists. | | Per-key budgets | Limited | Live enforcement is currently centered on managed-mode `/dp/budget_check`. Standalone self-hosted mode defaults to allow-all, and the standalone admin write validator does not currently accept `max_budget_usd`. | -| Rate limits and concurrency limits | Available | Current docs should treat these as active gateway behavior. | +| Rate limits and concurrency limits | Available | Three layers are AND-combined per request: `ApiKey.rate_limit`, `Model.rate_limit`, and scope-matched `RateLimitPolicy` rows (`api_key` / `model` / `team` / `member`). | | Routing models and failover | Available | Current model schema supports routing strategies and retry budget behavior. | | Keyword guardrails | Available | Current runtime enforcement is on `POST /v1/chat/completions`; non-chat endpoints do not run the guardrail chain today. | | Bedrock guardrails | Limited | Current code includes feature-gated runtime wiring. Treat it as an advanced capability with deployment and support caveats rather than as a planned-only feature. | diff --git a/docs/reference/admin-api-reference.md b/docs/reference/admin-api-reference.md index 35a80201..290ea3e0 100644 --- a/docs/reference/admin-api-reference.md +++ b/docs/reference/admin-api-reference.md @@ -53,6 +53,10 @@ This auth model is separate from proxy caller API keys. - authenticated operator health route: `/admin/v1/health` - operator convenience route: `/playground/chat/completions` +## Resources Without Admin CRUD + +`RateLimitPolicy` rows are loaded from etcd directly and do not currently have admin API CRUD routes. Provision them through your control-plane projection or, in self-hosted setups, by writing under the etcd `rate_limit_policies/` prefix. See [Rate Limits § Rate-Limit Policy Entities](../configuration/rate-limits.md#rate-limit-policy-entities). + ## Runtime Model Status `GET /admin/v1/models/status` returns one row per model in the current snapshot. diff --git a/docs/reference/glossary.md b/docs/reference/glossary.md index 988e56ea..cd14a4c3 100644 --- a/docs/reference/glossary.md +++ b/docs/reference/glossary.md @@ -12,6 +12,7 @@ sidebar_position: 65 - `Guardrail`: content-policy resource applied on chat paths - `CachePolicy`: dynamic cache-control resource - `ObservabilityExporter`: dynamic OTLP exporter resource +- `RateLimitPolicy`: standalone rate-limit rule stored in etcd, scoped by `api_key`, `model`, `team`, or `member` - `Snapshot`: the in-memory config view used by the proxy hot path - `Watch Supervisor`: the task that keeps the snapshot current from etcd - `Managed DP`: a managed data plane operating under AISIX Cloud control-plane workflows diff --git a/docs/reference/headers-and-error-codes.md b/docs/reference/headers-and-error-codes.md index f669861f..2624f103 100644 --- a/docs/reference/headers-and-error-codes.md +++ b/docs/reference/headers-and-error-codes.md @@ -26,6 +26,7 @@ Current proxy error `type` values include: - `model_not_found` - `invalid_request_error` - `provider_unavailable` +- `all_candidates_unavailable` - `content_filter` - `billing_error` - `rate_limit_exceeded` @@ -40,7 +41,7 @@ These values appear in the proxy's OpenAI-compatible error envelope. - `404` model alias not found - `422` content blocked by policy - `429` rate limit or budget rejection -- `503` provider bridge unavailable +- `503` provider bridge unavailable, or every routing candidate filtered out by runtime status Upstream `5xx` failures generally collapse into `502` through the bridge mapping path, even though `502` is not one of the gateway-originated business-logic classes listed above. diff --git a/docs/reference/resource-schemas.md b/docs/reference/resource-schemas.md index dff7276b..3a838147 100644 --- a/docs/reference/resource-schemas.md +++ b/docs/reference/resource-schemas.md @@ -12,6 +12,7 @@ sidebar_position: 62 - `Guardrail` - `CachePolicy` - `ObservabilityExporter` +- `RateLimitPolicy` - shared `RateLimit` - shared `Routing` @@ -26,6 +27,7 @@ Use this page as the schema map. Use the configuration pages when you need opera - `Guardrail` is discriminated by `kind` with current `keyword` and `bedrock` shapes. - `CachePolicy` currently documents `name`, `enabled`, `backend`, `ttl_seconds`, and `applies_to`. - `ObservabilityExporter` is currently `kind=otlp_http` only. +- `RateLimitPolicy` requires `name`, `scope` (`api_key` / `model` / `team` / `member`), `scope_ref`, and `window` (`second` / `minute` / `hour`); at least one of `max_requests` or `max_tokens` must be set. The standalone admin API does not currently expose CRUD routes for it — rows are written directly under the etcd `rate_limit_policies/` prefix. ## How To Read These Schemas @@ -41,7 +43,7 @@ Not every field or shape present in the schema should be interpreted as equally Examples: - the typed `ApiKey` model and admin OpenAPI mention `max_budget_usd`, but the current standalone admin write validator rejects it and standalone hard-stop budget behavior is not the current documented default -- `Model.rate_limit` exists in schema, but current hot-path enforcement is centered on `ApiKey.rate_limit` +- `Model.rate_limit` and `ApiKey.rate_limit` are both enforced today, alongside scope-matched `RateLimitPolicy` rows. See [Rate Limits](../configuration/rate-limits.md) for the layer order and the AND-combination semantics. - `Model.background_model_check` exists in schema, but it only applies to direct models and its runtime effect is surfaced through `/admin/v1/models/status` - `Guardrail.kind = bedrock` exists in schema, but current generally reliable runtime behavior is strongest on `keyword` @@ -53,3 +55,4 @@ Examples: - [Guardrails](../configuration/guardrails.md) - [Caching](../configuration/caching.md) - [Observability Exporters](../configuration/observability-exporters.md) +- [Rate Limits](../configuration/rate-limits.md)