diff --git a/docs/configuration/models.md b/docs/configuration/models.md index 212ef8b0..a65067c7 100644 --- a/docs/configuration/models.md +++ b/docs/configuration/models.md @@ -156,8 +156,9 @@ curl -sS -X POST http://127.0.0.1:3001/admin/v1/models \ ## Field Notes -- `display_name` is the alias clients send in proxy requests. -- `provider` currently supports `openai`, `anthropic`, `google`, `deepseek`, `cohere`, and `jina`. +- `display_name` is the alias clients send in proxy requests, and the value `response.model` echoes back. It is **not** the upstream model id. +- `model_name` is the upstream model id — the literal string the upstream provider expects in its own `model` field (for example `gpt-4o`, `claude-sonnet-4-5`, an Azure deployment name, or a Bedrock model id). Despite the name, this field holds the upstream id, not a caller alias; the caller alias is `display_name`. +- `provider` is a free-form vendor label, not a closed enum. The value must match the pattern `^[a-z0-9][a-z0-9._-]*$` (lowercase alphanumerics plus `.`, `-`, `_`, and no leading separator) and be at most 64 characters. In AISIX Cloud it is the catalog provider id (for example `openai`, `anthropic`, `deepseek`, `amazon-bedrock`); in the self-hosted gateway it can be any label you choose for a vendor or endpoint (for example `vllm`, `openrouter`, `xai`). Dispatch reads the referenced provider key's `adapter` and `provider`; this field also serves as a metrics and access-log label and gates a few vendor-specific endpoints. See [Adapter protocol families](../reference/adapters.md#how-a-model-resolves-to-a-bridge). - `provider_key_id` must reference an existing `ProviderKey` resource. - `timeout` is in milliseconds. `0` or omission means no timeout. - `cost` stores pricing metadata that AISIX Cloud's cp-api consumes when emitting usage events. The standalone OSS proxy does not consult this field at request time and always emits `cost_usd=0.0`; pricing-aware budget enforcement requires the AISIX Cloud control plane. @@ -221,6 +222,7 @@ That is expected with the current discovery boundary. ## Related Pages - [Provider Keys](provider-keys.md) +- [Adapter protocol families](../reference/adapters.md) — how `provider` and the provider key's `adapter` select an upstream bridge. - [API Keys](api-keys.md) - [Routing And Failover](routing-and-failover.md) - [Configuration Propagation](configuration-propagation.md) diff --git a/docs/configuration/provider-keys.md b/docs/configuration/provider-keys.md index 03bcd54d..38cca9dc 100644 --- a/docs/configuration/provider-keys.md +++ b/docs/configuration/provider-keys.md @@ -19,12 +19,20 @@ Think of a provider key as the upstream credential container, not the client-fac - `display_name` - `secret` - optional `api_base` +- optional `provider` +- optional `adapter` +- optional `telemetry_tags` In practice: - `display_name` is for operator readability - `secret` is the actual upstream credential used at dispatch time - `api_base` is how you override the provider's default endpoint root +- `provider` is the upstream vendor identity (a free-form lowercase label such as `openai`, `anthropic`, `deepseek`, `vllm`); it is the first-tier [dispatch key](../reference/adapters.md#how-a-model-resolves-to-a-bridge) and a metrics label +- `adapter` pins the upstream wire shape to one of `openai`, `anthropic`, `bedrock`, `vertex`, `azure-openai`; it is the second-tier dispatch key that routes Bedrock, Vertex, Azure OpenAI, and long-tail OpenAI-compatible vendors to the right bridge +- `telemetry_tags` carries attribution metadata (`kind` of `catalog` or `byo`, plus optional labels); it is populated by AISIX Cloud and is not required for self-hosted use + +For the complete field reference — every type, validation rule, and the `request`/`response` runtime-config overrides — see the [Provider key schema](../reference/runtime-config-schema.md). Example: @@ -38,6 +46,8 @@ curl -sS -X POST http://127.0.0.1:3001/admin/v1/provider_keys \ -H "Content-Type: application/json" \ -d '{ "display_name": "openai-prod", + "provider": "openai", + "adapter": "openai", "secret": "YOUR_PROVIDER_API_KEY", "api_base": "https://api.openai.com/v1" }' @@ -47,15 +57,17 @@ curl -sS -X POST http://127.0.0.1:3001/admin/v1/provider_keys \ `api_base` overrides the provider's default upstream base URL. Each provider bridge appends a different path at request time, so the canonical form `api_base` should take depends on which `provider` your model selects. -Each provider has its own convention — the four current bridges do **not** share one. Use the table below; do not generalize from one row to another. +Each provider has its own convention — the bridges do **not** share one. Use the table below; do not generalize from one row to another. The first rows are keyed on the vendor `provider`; the last two rows are keyed on the `adapter` family (Bedrock and Azure OpenAI dispatch by adapter, not by a fixed vendor string). For how `provider` and `adapter` select a bridge, see [Adapter protocol families](../reference/adapters.md). -| `provider` | Canonical `api_base` form | Bridge appends | Default if `api_base` is omitted | +| `provider` / `adapter` | Canonical `api_base` form | Bridge appends | Default if `api_base` is omitted | |---|---|---|---| | `openai` | include `/v1` | `/chat/completions`, `/embeddings`, `/completions`, `/images/generations`, `/audio/*` | `https://api.openai.com/v1` | | `deepseek` | bare host (DeepSeek serves OpenAI-compatible paths at the host root) | `/chat/completions` | `https://api.deepseek.com` | | `google` | host plus the OpenAI-compat prefix `/v1beta/openai` | `/chat/completions` | `https://generativelanguage.googleapis.com/v1beta/openai` | | `anthropic` | bare host | `/v1/messages` | `https://api.anthropic.com` | | `google-vertex` | bare host, no path | `/v1/projects//locations//publishers/google/models/:generateContent` (non-streaming) or `:streamGenerateContent?alt=sse` (streaming). `` and `` come from the SA JSON inside `secret`. | `https://-aiplatform.googleapis.com` | +| `bedrock` (adapter) | `api_base` usually **unset** | `/model//converse` or the Anthropic `/invoke` route, SigV4-signed | Region-keyed `bedrock-runtime..amazonaws.com`; the region comes from the `region` field in the credential JSON inside `secret`, not from `api_base`. Set `api_base` only for a private (VPC) Bedrock endpoint. | +| `azure-openai` (adapter) | the resource host `https://.openai.azure.com` (a bare resource name is also accepted) | `/openai/deployments//chat/completions?api-version=` | No default — `api_base` is required and supplies the resource host. A verbatim override host that does not end in `.openai.azure.com` is trusted as-is for a corporate proxy or mock. | The OpenAI and Anthropic conventions match each upstream's official SDK — `openai-python` initialises `base_url = "https://api.openai.com/v1"`, while `anthropic-sdk-python` initialises `base_url = "https://api.anthropic.com"` and appends `/v1/messages` itself. DeepSeek is OpenAI-compatible but exposes `/chat/completions` directly at the host root, and Google's Gemini OpenAI-compatible surface lives under a fixed `/v1beta/openai` prefix that the bridge does not synthesize. The Vertex bridge appends a parameterized URL of the form `/v1/projects//locations//publishers/google/models/:generateContent`, so the canonical `api_base` form for `google-vertex` is the bare host root — operators behind a corporate proxy or air-gapped network point `api_base` at their proxy host, and the bridge tacks on the rest. Note that OAuth token minting still hits `secret.token_uri` (controlled by the SA JSON, not `api_base`); operators behind a fully air-gapped network must additionally point `token_uri` at their internal token endpoint. @@ -118,5 +130,10 @@ That is expected if they all share the same provider key. The shared key is the ## Related Pages - [Models](models.md) +- [Provider key schema](../reference/runtime-config-schema.md) — the complete field reference, including `request`/`response` overrides. +- [Adapter protocol families](../reference/adapters.md) — how `provider` and `adapter` select a bridge. +- [Bring your own endpoint](byo-endpoint.md) — point the `openai` adapter at a private or self-hosted endpoint. +- [OpenAI-compatible vendor upstream](../integration/upstream-openai-compat.md) — onboard a public OpenAI-compatible vendor (DeepSeek, Groq, Mistral). +- [AWS Bedrock upstream](../integration/upstream-bedrock.md), [Google Vertex AI upstream](../integration/upstream-vertex.md), [Azure OpenAI upstream](../integration/upstream-azure-openai.md) — the specialized-family guides. - [OpenAI-Compatible API](../integration/openai-compatible-api.md) - [Configuration Propagation](configuration-propagation.md) diff --git a/docs/index.md b/docs/index.md index b790ad1c..1286e71a 100644 --- a/docs/index.md +++ b/docs/index.md @@ -37,6 +37,14 @@ This documentation set is organized for two primary audiences: - Use [Errors And Retries](integration/errors-and-retries.md) for shared failure handling. - Use the quickstarts to configure a working model and caller key first. +### I want to connect an upstream provider + +- Read [Adapter protocol families](reference/adapters.md) to see which of the five wire shapes your provider uses. +- Onboard a public OpenAI-compatible vendor (DeepSeek, Groq, Mistral) with [OpenAI-compatible vendor upstream](integration/upstream-openai-compat.md). +- Point the gateway at a private or self-hosted endpoint with [Bring your own endpoint](configuration/byo-endpoint.md). +- Connect a specialized provider with [AWS Bedrock](integration/upstream-bedrock.md), [Google Vertex AI](integration/upstream-vertex.md), or [Azure OpenAI](integration/upstream-azure-openai.md). +- Look up the credential resource fields in the [Provider key schema](reference/runtime-config-schema.md). + ### I want to operate the gateway in production - Start with the [Self-Hosted Quickstart](quickstart/self-hosted.md). @@ -59,6 +67,8 @@ This documentation set is organized for two primary audiences: - [AISIX Cloud](cloud/overview.md) - [Operations](operations/production-deployment.md) - [Reference](reference/proxy-api-reference.md) +- [Adapter protocol families](reference/adapters.md) +- [Provider key schema](reference/runtime-config-schema.md) - [Tutorials](tutorials/build-a-virtual-model-with-failover.md) - [Roadmap](roadmap.md) diff --git a/docs/integration/upstream-openai-compat.md b/docs/integration/upstream-openai-compat.md new file mode 100644 index 00000000..a24e38bf --- /dev/null +++ b/docs/integration/upstream-openai-compat.md @@ -0,0 +1,211 @@ +--- +title: OpenAI-compatible vendor upstream +description: Onboard a public OpenAI-compatible vendor such as DeepSeek, Groq, Mistral, Together.ai, Fireworks, or Perplexity to AISIX AI Gateway using the openai adapter. +sidebar_position: 34 +keywords: + - AISIX AI Gateway + - OpenAI-compatible API + - DeepSeek + - Groq + - Mistral + - AI gateway +--- + +This guide shows how to onboard a public OpenAI-compatible vendor — DeepSeek, Groq, Mistral, Together.ai, Fireworks, Perplexity, and similar services — to AISIX AI Gateway. These vendors expose the OpenAI chat-completions wire at a known public host, so they all dispatch through the `openai` [adapter](../reference/adapters.md) family. Callers reach them through the same OpenAI-compatible proxy surface and the same caller API keys as any other model. + +## When to use this + +- Use this when the upstream is a **public** vendor that serves the OpenAI chat-completions API at a documented host (for example `https://api.deepseek.com` or `https://api.groq.com/openai/v1`). +- Use this in the **self-hosted** gateway, where you register the vendor's host and credential yourself through the admin API. +- In **AISIX Cloud**, you usually do not need this guide: the catalog maps these vendors to the `openai` adapter automatically when you select the provider. See [Catalog versus bring-your-own](#self-hosted-versus-cloud) below. + +This page is distinct from two neighbors: + +- [Bring your own endpoint](../configuration/byo-endpoint.md) covers **private or self-hosted** OpenAI-compatible servers (vLLM, SGLang, Ollama, an internal proxy). The mechanics are identical; the difference is that a BYO endpoint is yours and not on a public host. +- [OpenAI-compatible API](openai-compatible-api.md) documents the **client-facing** proxy surface — the API your callers use to reach the gateway. This page is about the **upstream** side: pointing the gateway at a vendor. + +## How it works + +An OpenAI-compatible vendor is configured through two resources, exactly like any other upstream: + +1. A [provider key](../configuration/provider-keys.md) holding the vendor credential (`secret`), its base URL (`api_base`), the vendor identity (`provider`), and the wire shape (`adapter: openai`). +2. A direct [model](../configuration/models.md) that maps a caller-facing alias (`display_name`) to the vendor's model id (`model_name`) and references the provider key. + +Because these are public vendors with a non-`openai` vendor identity, **you must set `api_base`**. The OpenAI-family bridge only falls back to `https://api.openai.com` when the provider key's vendor identity is `openai` (or empty). For any other vendor it refuses to guess a base URL and fails dispatch. Set `api_base` to the vendor's documented host. + +Each vendor's canonical `api_base` form differs. DeepSeek serves the OpenAI-compatible paths at the host root; others use a `/v1` or `/openai/v1` prefix. The gateway tolerates common paste variants but does not synthesize a vendor-specific prefix — paste the form the vendor documents. See [Provider keys § `api_base` behavior](../configuration/provider-keys.md#api_base-behavior) for the full normalization rules. + +```mermaid +sequenceDiagram + autonumber + participant Client + participant Proxy as AISIX proxy (:3000) + participant Hub as Bridge hub + participant Bridge as OpenAI bridge + participant Vendor as OpenAI-compatible vendor + + Client->>Proxy: POST /v1/chat/completions (model = your-alias) + Note over Proxy: resolve alias → Model + ProviderKey + Proxy->>Hub: dispatch by adapter (openai) + Hub->>Bridge: select OpenAI family bridge + Bridge->>Vendor: POST /chat/completions (Bearer secret) + Vendor-->>Bridge: OpenAI-shaped response + Bridge-->>Proxy: normalized chat response + Note over Proxy: restore response.model = your-alias + Proxy-->>Client: OpenAI-shaped JSON +``` + +## Prerequisites + +- A running self-hosted gateway (admin on `:3001`, proxy on `:3000`). See the [Self-Hosted Quickstart](../quickstart/self-hosted.md). +- Your admin key from the bootstrap config. +- A vendor API key and the vendor's documented OpenAI-compatible host. The examples below use DeepSeek (`https://api.deepseek.com`, model id `deepseek-chat`). + +## Configuration + +### Step 1: Create the provider key + +:::warning Production credentials +The standalone gateway stores `secret` as plaintext under the etcd `prefix` from [`config.yaml`](../configuration/bootstrap-config.md). For production, front etcd with encryption-at-rest, restrict etcd network access to the gateway, or use AISIX Cloud's managed [Provider Key Rotation](../cloud/provider-key-rotation.md), where the secret stays in the control plane and only the projected reference reaches the data plane. +::: + +```bash title="Create a DeepSeek provider key" +curl -sS -X POST http://127.0.0.1:3001/admin/v1/provider_keys \ + -H "Authorization: Bearer YOUR_ADMIN_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "display_name": "deepseek-prod", + "provider": "deepseek", + "adapter": "openai", + "secret": "YOUR_PROVIDER_API_KEY", + "api_base": "https://api.deepseek.com" + }' +``` + +Field notes: + +- `provider` is the vendor identity, a free-form lowercase label (`deepseek`, `groq`, `mistral`, `together`, `fireworks`, `perplexity`). It is used for dispatch and metrics. It must not be `openai` unless you genuinely point at OpenAI. +- `adapter` pins the wire shape to `openai` — the only valid value for an OpenAI-compatible vendor. +- `api_base` is required. Use the vendor's documented host: + + | Vendor | Documented `api_base` | + |---|---| + | DeepSeek | `https://api.deepseek.com` (host root) | + | Groq | `https://api.groq.com/openai/v1` | + | Mistral | `https://api.mistral.ai/v1` | + | Together.ai | `https://api.together.xyz/v1` | + | Fireworks | `https://api.fireworks.ai/inference/v1` | + | Perplexity | `https://api.perplexity.ai` | + + Confirm the exact host against the vendor's current API reference before relying on it. + +Capture the returned `id` for the next step. The admin API returns a `ResourceEntry` with an `id` field; the [first-request quickstart](../quickstart/first-model-first-key-first-request.md#step-1-create-a-provider-key) shows a `jq`-capturing one-liner if you want to script it. + +### Step 2: Create the model + +Map a caller-facing alias to the vendor's model id. + +```bash title="Create a model for the vendor" +curl -sS -X POST http://127.0.0.1:3001/admin/v1/models \ + -H "Authorization: Bearer YOUR_ADMIN_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "display_name": "deepseek-chat-prod", + "provider": "deepseek", + "model_name": "deepseek-chat", + "provider_key_id": "YOUR_PROVIDER_KEY_ID" + }' +``` + +- `display_name` is the alias callers send in `model` and the value `response.model` echoes back. +- `model_name` is the vendor's model id — the literal string the vendor expects in its `model` field. +- `provider` on the model is the same vendor label as on the key. +- `cost` is optional. Public vendors are not in the gateway's standalone pricing path, so set a `cost` block if you want per-token budget accounting available to AISIX Cloud or your own usage-event consumer. See [Models § field notes](../configuration/models.md#field-notes). + +### Step 3: Create a caller API key + +The data plane stores `key_hash`, not plaintext. Hash a plaintext caller key, then create the key resource scoped to your new alias. + +```bash title="Hash a plaintext caller key" +printf 'sk-demo-caller' | sha256sum | cut -d' ' -f1 +``` + +```bash title="Create a caller API key" +curl -sS -X POST http://127.0.0.1:3001/admin/v1/apikeys \ + -H "Authorization: Bearer YOUR_ADMIN_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "key_hash": "YOUR_CALLER_KEY_HASH", + "allowed_models": ["deepseek-chat-prod"] + }' +``` + +### Step 4: Send a request + +Admin writes propagate to the proxy asynchronously; allow about a second, or poll `/v1/models` until the alias appears. + +```bash title="Send a chat completion to the vendor" +curl -sS -X POST http://127.0.0.1:3000/v1/chat/completions \ + -H "Authorization: Bearer sk-demo-caller" \ + -H "Content-Type: application/json" \ + -d '{ + "model": "deepseek-chat-prod", + "messages": [ + {"role": "user", "content": "Say hello from DeepSeek."} + ] + }' +``` + +## Self-hosted versus Cloud + +The two modes differ only in where the field values come from: + +- **Self-hosted** — you set `provider`, `adapter: openai`, `api_base`, and `secret` on the provider key yourself, exactly as shown above. The gateway ships no catalog. +- **AISIX Cloud** — the control plane ships a models.dev-driven catalog and maps each catalog provider to its adapter automatically. You select the provider in the dashboard; the adapter (`openai` for these vendors) and the base URL are filled in for you. See [Adapter protocol families § Catalog versus bring-your-own](../reference/adapters.md#catalog-versus-bring-your-own). + +### Non-featured and Community providers + +Within the AISIX Cloud catalog, a subset of providers is **featured** — the ranked set the dashboard surfaces first. A vendor that is in the catalog but not featured (a Community provider) still resolves to the `openai` adapter through the same catalog mapping; it is simply not promoted in the ranked list. Featured status affects discovery and presentation, not dispatch — both run through the same OpenAI bridge. In the self-hosted gateway there is no featured concept; every vendor you register is equal. + +## Verification + +A `200` alone does not prove the gateway reached the vendor and applied the alias contract. Verify the two observable facts that do. + +### The alias is restored on `response.model` + +```bash title="Confirm response.model echoes your alias" +curl -sS -X POST http://127.0.0.1:3000/v1/chat/completions \ + -H "Authorization: Bearer sk-demo-caller" \ + -H "Content-Type: application/json" \ + -d '{"model":"deepseek-chat-prod","messages":[{"role":"user","content":"ping"}]}' \ + | grep -o '"model":"[^"]*"' +``` + +Expected: `"model":"deepseek-chat-prod"` — your caller-facing alias, **not** the upstream `deepseek-chat`. This proves the request resolved through your model and the gateway restored the alias on the way out. If you see the vendor's model id instead, the request did not flow through the gateway's render path. + +### The request actually reached the vendor + +Confirm dispatch targets your configured `api_base` and not a default host. Temporarily point `api_base` at an unreachable host and confirm the gateway returns an upstream error rather than a `200`: + +```bash title="Negative check — unreachable host surfaces an upstream error" +curl -sS -o /dev/null -w "%{http_code}\n" -X POST http://127.0.0.1:3000/v1/chat/completions \ + -H "Authorization: Bearer sk-demo-caller" \ + -H "Content-Type: application/json" \ + -d '{"model":"deepseek-chat-prod","messages":[{"role":"user","content":"ping"}]}' +``` + +With a healthy vendor host, expect `200`. With `api_base` pointing at a dead host, expect a `5xx` upstream error — confirming dispatch uses your `api_base` and not a built-in default. An authentication failure (`401`) instead of a successful response usually means the `secret` is wrong for the vendor. + +## Limitations + +- This path is for vendors that speak the OpenAI chat-completions wire. A vendor with a non-OpenAI wire shape needs a native adapter — see [Adapter protocol families](../reference/adapters.md). +- A missing `api_base` on a non-`openai` vendor fails dispatch with a configuration error. Always set `api_base`. +- Vendor-specific response extensions beyond the OpenAI envelope are not normalized. Reasoning-style fields can be lifted per key via the `response.reasoning_field` override; see [Provider key schema § response overrides](../reference/runtime-config-schema.md#response-overrides). + +## Related pages + +- [Adapter protocol families](../reference/adapters.md) — why an OpenAI-compatible vendor uses the `openai` adapter. +- [Bring your own endpoint](../configuration/byo-endpoint.md) — the same mechanics for a private or self-hosted endpoint. +- [Provider keys](../configuration/provider-keys.md) — the credential resource and the full `api_base` normalization rules. +- [Provider key schema](../reference/runtime-config-schema.md) — the complete field reference. +- [OpenAI-compatible API](openai-compatible-api.md) — the client-facing proxy surface callers use to reach the vendor. diff --git a/docs/reference/provider-compatibility.md b/docs/reference/provider-compatibility.md index ca40ea38..20d48180 100644 --- a/docs/reference/provider-compatibility.md +++ b/docs/reference/provider-compatibility.md @@ -1,38 +1,73 @@ --- -title: Provider Compatibility -description: Reference for current provider coverage and compatibility boundaries in AISIX AI Gateway. +title: Provider compatibility +description: Reference for current adapter-family coverage and compatibility boundaries in AISIX AI Gateway — which wire shape backs each provider and what each family supports. sidebar_position: 64 --- -## Current Provider Enum +This page is the lookup reference for which upstreams AISIX AI Gateway can reach and what each one currently supports. Compatibility is organized around the five [adapter protocol families](adapters.md), not a flat provider list: every upstream — catalog or bring-your-own — resolves to exactly one adapter, and the adapter determines the wire shape and the supported endpoints. -The current provider set is: +## Adapter families -- `openai` -- `anthropic` -- `google` -- `deepseek` -- `cohere` -- `jina` +The gateway encodes requests against a closed set of five adapter families. Vendor identity (`provider`) is a free-form string; the adapter is the closed enum that picks the bridge. -## Compatibility Boundary +| Adapter | Upstream wire shape | Example upstreams | +|---|---|---| +| `openai` | OpenAI chat completions | OpenAI, plus every OpenAI-compatible vendor (DeepSeek, Groq, Mistral, Together.ai, Fireworks, Perplexity, …) and BYO endpoints (vLLM, SGLang, Ollama) | +| `anthropic` | Anthropic Messages | Anthropic (Claude) | +| `bedrock` | AWS Bedrock Runtime (Converse + Anthropic `/invoke`) | Claude, and other Bedrock publishers via Converse | +| `vertex` | Google Vertex AI Gemini | Gemini on Vertex | +| `azure-openai` | Azure OpenAI Service | Azure OpenAI deployments | + +The `openai` family is the broadest: any vendor or self-hosted server that speaks the OpenAI chat-completions wire dispatches through it, differing only in `api_base` and credential. See [OpenAI-compatible vendor upstream](../integration/upstream-openai-compat.md) and [Bring your own endpoint](../configuration/byo-endpoint.md). + +## Coverage matrix + +Support depth varies by adapter family. The matrix below summarizes the current state; each integration guide documents the exact behavior. + +| Capability | `openai` | `anthropic` | `bedrock` | `vertex` | `azure-openai` | +|---|---|---|---|---|---| +| Chat completions | Yes | Yes (Messages) | Yes | Yes (Gemini) | Yes | +| Streaming (SSE) | Yes | Yes | Yes | Yes (Gemini) | Yes | +| Embeddings | Yes (OpenAI / OpenAI-compatible) | No | No | No | No | +| Images, audio, responses | Yes (OpenAI / OpenAI-compatible) | No | No | No | No | +| Rerank | Yes (Cohere / Jina native surface) | No | No | No | No | + +Notes on the matrix: + +- The image, audio, `/v1/responses`, and embeddings endpoints are gated to OpenAI-shaped upstreams. A request that resolves to a non-OpenAI model on those endpoints is rejected rather than mis-dispatched. The gate keys on the literal `provider: "openai"` (plus the OpenAI embeddings/native surfaces), **not** the whole `openai` adapter family — an OpenAI-compatible vendor (for example a DeepSeek model on the `openai` adapter) works on `/v1/chat/completions` but is rejected on `/v1/responses`, images, and audio. +- `/v1/rerank` is served by the Cohere and Jina native rerank surfaces, which bypass the chat bridge; it is keyed on the model's `provider`. +- `/v1/messages` accepts non-Anthropic models through a cross-provider translation path; see [Anthropic Messages](../integration/anthropic-messages.md). + +## Per-family limitations + +- **`openai`** — vendor-specific response extensions beyond the OpenAI envelope are not normalized. Reasoning-style fields can be lifted per key via the `response.reasoning_field` override (see [Provider key schema § response overrides](runtime-config-schema.md#response-overrides)). +- **`anthropic`** — the family speaks the Messages wire; it is not the OpenAI embeddings/images/audio surface. +- **`bedrock`** — Anthropic-on-Bedrock (Claude) models dispatch through the legacy `/invoke` route with an Anthropic Messages body; all other publishers use the unified Converse API. Cross-region inference profile prefixes (`us.`, `eu.`, `apac.`, `global.`, `us-gov.`) are supported. See [AWS Bedrock upstream](../integration/upstream-bedrock.md). +- **`vertex`** — Gemini chat and streaming are wired. **Anthropic-on-Vertex and Llama-on-Vertex are not yet implemented.** See [Google Vertex AI upstream § Limitations](../integration/upstream-vertex.md#limitations) and the [Roadmap](../roadmap.md). +- **`azure-openai`** — chat and streaming are wired for both the `api-key` and the Entra ID (AAD) `client_credentials` auth schemes. See [Azure OpenAI upstream](../integration/upstream-azure-openai.md). + +## Featured versus non-featured catalog providers + +In AISIX Cloud, the catalog distinguishes **featured** providers (the ranked set the dashboard surfaces first) from non-featured (Community) providers. Featured status affects discovery and presentation only — both featured and non-featured providers resolve to one of the five adapters through the same catalog mapping and run through the same bridges. The self-hosted gateway ships no catalog and has no featured concept; you set `provider`, `adapter`, and `api_base` on each provider key yourself. See [Adapter protocol families § Catalog versus bring-your-own](adapters.md#catalog-versus-bring-your-own). + +## Compatibility boundary Provider support is not identical across every endpoint and behavior surface. Current reference point: - the gateway exposes a mixed OpenAI-compatible and Anthropic-style surface -- support depth varies by provider and endpoint family +- support depth varies by adapter family and endpoint This means provider compatibility is not a single yes/no question. The real questions are: - which endpoint family are you using -- which provider backs the resolved model +- which adapter backs the resolved model - whether the path is provider-native or translated -## Practical Reading Guide +## Practical reading guide - start with integration docs for endpoint-family behavior - use the feature matrix for current breadth versus limited support @@ -40,8 +75,9 @@ The real questions are: Use the feature matrix and integration docs as the current contract, and treat broader provider parity as ongoing work. -## Related Pages +## Related pages +- [Adapter protocol families](adapters.md) — the five families and how a model resolves to a bridge. - [Feature Matrix](../overview/feature-matrix.md) - [OpenAI-Compatible API](../integration/openai-compatible-api.md) - [Roadmap](../roadmap.md) diff --git a/docs/reference/runtime-config-schema.md b/docs/reference/runtime-config-schema.md new file mode 100644 index 00000000..68995e38 --- /dev/null +++ b/docs/reference/runtime-config-schema.md @@ -0,0 +1,189 @@ +--- +title: Provider key schema +description: Complete JSON schema reference for the AISIX AI Gateway ProviderKey resource — every field, type, validation rule, the adapter enum, telemetry tags, and the request/response runtime-config overrides. +sidebar_position: 67 +keywords: + - AISIX AI Gateway + - ProviderKey + - schema + - adapter + - runtime config + - AI gateway +--- + +This page is the complete schema reference for the `ProviderKey` resource — the upstream credential container every direct [model](../configuration/models.md) references by `provider_key_id`. It documents every field, its type, whether it is required, the validation rule the admin API enforces, and the closed `adapter` enum. Use it as the lookup companion to the task-oriented [Provider keys](../configuration/provider-keys.md) guide. + +The admin API validates every write against a JSON Schema (Draft 2020-12) before persisting. The top-level object is closed: unknown fields are rejected. + +## Top-level fields + +| Field | Type | Required | Validation | Description | +|---|---|---|---|---| +| `display_name` | string | yes | `minLength: 1`; unique within the gateway | Operator-facing label. Surfaces in the admin list view and dashboard. Duplicate values are rejected with `409`. | +| `secret` | string | yes | `minLength: 1` | Upstream provider credential. Stored as plaintext on the standalone path — see the production warning below. | +| `api_base` | string | no | any string | Override for the upstream base URL. The canonical form depends on the `adapter` — see [Provider keys § `api_base` behavior](../configuration/provider-keys.md#api_base-behavior). | +| `provider` | string | no | free-form string | Vendor identity (for example `openai`, `anthropic`, `deepseek`, `vllm`). First-tier dispatch key. Defaults to an empty string when omitted. | +| `adapter` | string (enum) | no | one of `openai`, `anthropic`, `bedrock`, `vertex`, `azure-openai` | Wire-shape protocol family. Second-tier dispatch key. See [the adapter enum](#the-adapter-enum). | +| `telemetry_tags` | object | no | closed object | Attribution metadata. See [telemetry_tags](#telemetry_tags). | +| `request` | object | no | closed object | Per-key request-shape overrides. See [request overrides](#request-overrides). | +| `response` | object | no | closed object | Per-key response-shape overrides. See [response overrides](#response-overrides). | +| `strip_headers` | array of string | no | array of strings | Inbound headers stripped before forwarding on the `passthrough` endpoint. Defaults to `["authorization", "cookie", "set-cookie", "x-api-key"]` when the field is absent. See [Passthrough](../integration/passthrough.md). | + +:::warning Production credentials +The standalone gateway stores `secret` as plaintext under the etcd `prefix` from [`config.yaml`](../configuration/bootstrap-config.md). Anyone with read access to the etcd keyspace can read the credential. For production, front etcd with encryption-at-rest, restrict etcd network access to the gateway, or use AISIX Cloud's managed [Provider Key Rotation](../cloud/provider-key-rotation.md), where the secret stays in the control plane and only the projected `provider_key_id` reference reaches the data plane. +::: + +:::note `provider` validation differs between resources +On a `ProviderKey`, `provider` is an unconstrained string. On a [`Model`](../configuration/models.md), `provider` is also free-form but additionally enforces `minLength: 1`, `maxLength: 64`, and the pattern `^[a-z0-9][a-z0-9._-]*$`. Neither field is a closed enum — only `adapter` is. +::: + +## The adapter enum + +`adapter` is the one closed enum on the resource. It pins the upstream wire shape the gateway encodes against. It serializes in `kebab-case`, so the Azure family is the wire string `azure-openai`. + +| Value | Upstream wire shape | Integration guide | +|---|---|---| +| `openai` | OpenAI chat completions | [OpenAI-compatible vendor upstream](../integration/upstream-openai-compat.md), [BYO endpoint](../configuration/byo-endpoint.md) | +| `anthropic` | Anthropic Messages | [Anthropic Messages](../integration/anthropic-messages.md) | +| `bedrock` | AWS Bedrock Runtime (Converse + Anthropic `/invoke`) | [AWS Bedrock upstream](../integration/upstream-bedrock.md) | +| `vertex` | Google Vertex AI Gemini | [Google Vertex AI upstream](../integration/upstream-vertex.md) | +| `azure-openai` | Azure OpenAI Service | [Azure OpenAI upstream](../integration/upstream-azure-openai.md) | + +Any string outside this set is rejected at write time. For how `provider` and `adapter` combine at dispatch, see [Adapter protocol families § How a model resolves to a bridge](adapters.md#how-a-model-resolves-to-a-bridge). + +## telemetry_tags + +Attribution metadata carried alongside the key. The object is closed (unknown keys rejected). All fields are optional. + +| Field | Type | Validation | Description | +|---|---|---|---| +| `kind` | string (enum) | one of `catalog`, `byo` | Whether the key is a curated catalog provider or a bring-your-own endpoint. | +| `featured` | boolean | — | Whether the provider is surfaced in the dashboard's featured (ranked) list. Defaults to `false`. | +| `branded_provider` | string or null | — | Branded provider slug for catalog entries (for example `openai`, `deepseek`). Null for BYO. | +| `pk_label` | string or null | — | Operator-defined label for the key (for example `production`, `shared-test`). | +| `byo_label` | string or null | — | Operator-defined label for bring-your-own entries (for example an internal team name). | + +```json title="Catalog telemetry tags" +{ + "telemetry_tags": { + "kind": "catalog", + "featured": true, + "branded_provider": "deepseek", + "pk_label": "production" + } +} +``` + +```json title="Bring-your-own telemetry tags" +{ + "telemetry_tags": { + "kind": "byo", + "branded_provider": null, + "byo_label": "platform-team" + } +} +``` + +## request overrides + +Per-key overrides applied to the outbound request body and headers. The object is closed. Each field maps to a primitive transform the gateway applies before dispatch. + +| Field | Type | Validation | Description | +|---|---|---|---| +| `param_renames` | object (string → string) | values are strings | Top-level body keys named on the left are renamed to the right (for example `max_completion_tokens` → `max_tokens`). | +| `param_constraints` | object | closed; see below | Numeric clamps applied to the request body. | +| `default_headers` | object (string → string) | values are strings | Headers added to the outbound request when the caller did not set them. Reserved auth headers are dropped as defense-in-depth. | +| `default_body_fields` | object | free-form | Top-level body fields added when the caller did not set them (for example `safe_prompt`). | + +`param_constraints` is a closed object with two fields: + +| Field | Type | Description | +|---|---|---| +| `temperature_max` | number | Upper clamp for `temperature`. Values above are clamped down. | +| `temperature_min` | number | Lower clamp for `temperature`. Values below are clamped up. | + +```json title="request overrides" +{ + "request": { + "param_renames": { "max_completion_tokens": "max_tokens" }, + "param_constraints": { "temperature_max": 1.0 }, + "default_headers": { "X-Foo": "bar" }, + "default_body_fields": { "safe_prompt": true } + } +} +``` + +:::note Where overrides are applied +The `request` and `response` blocks are applied at dispatch by the **`openai` and `azure-openai`** bridges (the OpenAI-wire families): request `param_renames` / `param_constraints` / `default_headers` / `default_body_fields` are folded into the outbound call, and response `stream_done_marker` / `content_list_to_string` / `reasoning_field` shape how the upstream reply is interpreted. The `bedrock` and `vertex` bridges build their providers' native request shapes and do **not** apply these blocks today. + +What is not yet shipped is the AISIX Cloud control-plane wiring that auto-populates these blocks from the dashboard, so in AISIX Cloud they are currently empty. A self-hosted operator who sets `request` / `response` directly through the admin API gets them applied on the `openai` / `azure-openai` paths. +::: + +## response overrides + +Per-key overrides describing how the gateway interprets the upstream response. The object is closed. + +| Field | Type | Validation | Description | +|---|---|---|---| +| `stream_done_marker` | string (enum) | one of `required`, `optional`, `none` | The SSE `[DONE]` terminator expectation. `required` — upstream must emit `data: [DONE]`. `optional` — either is acceptable. `none` — upstream is expected to omit it. | +| `content_list_to_string` | boolean | — | When `true`, a `messages[*].content` array of text blocks is flattened to a single string before dispatch (for upstreams that only accept string content). Defaults to `false`. | +| `error_envelope` | string | open string | Error-translation strategy. The control-plane spec uses `openai` (project upstream errors into the OpenAI envelope) or `passthrough` (return the upstream body as-is). Validated as an open string today. | +| `reasoning_field` | string | — | Dotted path to lift an upstream reasoning field (for example `delta.reasoning_content`). | + +```json title="response overrides" +{ + "response": { + "stream_done_marker": "required", + "content_list_to_string": false, + "error_envelope": "openai", + "reasoning_field": "delta.reasoning_content" + } +} +``` + +## Minimal and full examples + +A `ProviderKey` requires only `display_name` and `secret`: + +```json title="Minimal provider key" +{ + "display_name": "openai-prod", + "secret": "YOUR_PROVIDER_API_KEY" +} +``` + +A fully populated catalog provider key: + +```json title="Full provider key" +{ + "display_name": "deepseek-prod", + "secret": "YOUR_PROVIDER_API_KEY", + "api_base": "https://api.deepseek.com/v1", + "provider": "deepseek", + "adapter": "openai", + "telemetry_tags": { + "kind": "catalog", + "featured": true, + "branded_provider": "deepseek", + "pk_label": "production" + }, + "request": { + "param_constraints": { "temperature_max": 1.0 } + }, + "response": { + "stream_done_marker": "required", + "reasoning_field": "delta.reasoning_content" + } +} +``` + +## Backward compatibility + +Every field except `display_name` and `secret` is optional. Provider key payloads written before `provider`, `adapter`, `telemetry_tags`, `request`, `response`, or `strip_headers` existed continue to validate and load: the missing fields fall back to their defaults (`provider` to an empty string, `adapter` to absent, `telemetry_tags` to the all-default object, `strip_headers` to the four-header credential list). + +## Related pages + +- [Provider keys](../configuration/provider-keys.md) — the task-oriented configuration guide, including the `api_base` behavior table and tolerance rules. +- [Adapter protocol families](adapters.md) — how `provider` and `adapter` select a bridge. +- [Models](../configuration/models.md) — the resource that references a provider key by `provider_key_id`. +- [Resource schemas](resource-schemas.md) — schemas for the other admin-managed resources.