fix(llm): honor a client-supplied llm_override endpoint behind an opt-in flag

docs/content/docs/documentation/API.mdx

@@ -715,7 +715,7 @@ OpenAI-compatible text completion endpoint.
 | `websearch` | `bool` | `false` | Augments the RAG context with live web search results. When used with a partition (`openrag-{partition}`), document and web results are combined. When used without a partition (direct LLM mode), web results are the sole context. Requires `WEBSEARCH_API_TOKEN` to be configured. See [web search configuration](/openrag/documentation/env_vars/#web-search-configuration). |
 | `spoken_style_answer` | `bool` | `false` | Generates a succinct spoken-style conversational answer based on the retrieved documents. |
 | `use_map_reduce` | `bool` | `false` | Uses a map-reduce strategy to aggregate information from multiple documents. See [map-reduce configuration](/openrag/documentation/env_vars/#map--reduce-configuration). |
-| `llm_override` | `object` | `null` | Overrides only the downstream LLM model name while still using OpenRAG's configured LLM endpoint and credentials. Accepts: `model` (string). Endpoint URL and API key are server-side configuration and cannot be changed by a client request. |
+| `llm_override` | `object` | `null` | Overrides the downstream LLM for this request. Accepts `model` (string), always honored. Also accepts `base_url` and `api_key`, which are honored **only** when the deployment sets `LLM_OVERRIDE_ALLOW_CUSTOM_ENDPOINT` — otherwise they are ignored and the request goes to the server's configured endpoint. See [custom LLM endpoints](/openrag/documentation/env_vars/#client-supplied-llm-endpoints). |
 | `attachments` | `list[{"id": string}]` | `null` | Scopes RAG retrieval to a specific list of file IDs within the target partition, instead of searching the whole partition. Unknown or unindexed IDs are silently dropped; duplicates are deduplicated. The response's `extra.attachments` reports which IDs were actually used. |
 
 Examples:
@@ -803,6 +803,30 @@ curl -X 'POST' 'http://localhost:8080/v1/chat/completions' \
 }'
 ```
 
+With `LLM_OVERRIDE_ALLOW_CUSTOM_ENDPOINT=true`, the same object may also carry the
+endpoint and its credential, sending the request to a provider of the client's
+choosing instead of the configured one:
+
+```json title="metadata.llm_override with a client-supplied endpoint"
+{
+  "llm_override": {
+    "base_url": "https://api.openai.com/v1",
+    "api_key": "sk-...",
+    "model": "gpt-4o"
+  }
+}
+```
+
+`base_url` must be `https`, must carry no query string, fragment or `..` path
+segment (percent-encoded or not), and is always requested as
+`{base_url}/chat/completions` — `{base_url}/completions` when the request comes
+in on the legacy `/v1/completions` route; anything else is rejected with a
+**400**. The server's own API key is never forwarded — an override without
+`api_key` sends no `Authorization` header at all. When the flag
+is off, `base_url`/`api_key` are ignored (a warning is logged) and only `model`
+applies, which typically surfaces as an "unknown model" error from the configured
+provider.
+
 ```bash title="Scoping retrieval to specific attachments"
 curl -X 'POST' 'http://localhost:8080/v1/chat/completions' \
   -H 'accept: application/json' \

docs/content/docs/documentation/env_vars.md

@@ -289,10 +289,49 @@ These are external services to provide !!!
 | `API_KEY` | str | _(unset)_ | API key for authenticating with the LLM service |
 | `LLM_ENABLE_THINKING` | bool | _(unset)_ | Optional chat-template control for models that support `enable_thinking`; leave unset for Mistral tokenizers, set `false` to suppress Qwen-style reasoning traces |
 | `LLM_SEMAPHORE` | int | 10 | Maximum number of concurrent requests to allow for the LLM service |
+| `LLM_OVERRIDE_ALLOW_CUSTOM_ENDPOINT` | bool | `false` | Honor a client-supplied `base_url`/`api_key` in `metadata.llm_override`. Off by default; read the trade-off below before enabling. |
 | `MAX_LLM_CONTEXT_SIZE` | `int` | `8192` | Fallback maximum token limit for chat/completion requests. At startup, the `/v1/models` endpoint is queried for the model's `max_model_len`; if that query fails this value is used instead. Requests whose total token count (prompt + `max_tokens`) exceeds the limit are rejected with a **413** error. |
 | `MAX_OUTPUT_TOKENS` | `int` | `1024` | Default output-token budget (`max_tokens`) applied to chat completions when the request doesn't set one explicitly. |
 
 
+#### Client-supplied LLM endpoints
+
+A client can always override the **model name** for a single request via
+`metadata.llm_override` (see the [API reference](/openrag/documentation/api/#extra-arguments)).
+`LLM_OVERRIDE_ALLOW_CUSTOM_ENDPOINT=true` additionally honors `base_url` and
+`api_key` from that object, so the request is served by a provider of the
+client's choosing rather than the configured one.
+
+It exists for deployments whose clients already send the full object and would
+otherwise break. Prefer registering a named endpoint under `/model-endpoints`
+and binding it to the partition (`chat_llm`) — same outcome, none of the
+trade-off below.
+
+**What enabling it means.** Any caller who can reach `/v1/chat/completions` can
+make the server issue https POSTs from inside your network. The request shape is
+pinned, which is what bounds the exposure:
+
+- `https` only — plaintext internal services are unreachable.
+- The path is always `{base_url}/chat/completions` (`{base_url}/completions` for
+  the legacy `/v1/completions` route); a query string, fragment or `..` segment
+  — percent-encoded or not — is rejected with a **400**, so the override cannot
+  be aimed at an arbitrary internal path.
+- Redirects are not followed, so a target cannot bounce the server elsewhere.
+- The server's own API key is never forwarded; an override without `api_key`
+  sends no `Authorization` header at all.
+
+What remains reachable is therefore essentially *other https LLM gateways* —
+including an internal one that trusts its network rather than a credential, which
+such a caller could then use without holding its key.
+
+**What it does not change.** It grants no read access a caller does not already
+have: `/search` returns the same partition content directly. What changes is the
+way data leaves — as outbound LLM traffic from the server's egress rather than as
+a user read. That matters against a DLP or approved-subprocessor constraint, not
+against a caller who was never authorized in the first place.
+
+Enable only where every API caller is already trusted with both.
+
 #### VLM Configuration
 
 | Variable | Type | Default | Description |

infra/compose/.env.example

@@ -14,6 +14,9 @@ BASE_URL=
 API_KEY=
 MODEL=
 LLM_SEMAPHORE=10
+# Allow clients to supply their own LLM endpoint through `metadata.llm_override`.
+# LLM_OVERRIDE_ALLOW_CUSTOM_ENDPOINT=true
+
 
 # ── VLM (vision model, used for image understanding) ────────────────────────
 # Can reuse the LLM values above if that model accepts images.

openrag/api/routers/user/chat.py

@@ -29,6 +29,7 @@
 from api.routers.user.source_links import build_document_source_link
 from api.schemas.user.chat import OpenAIChatCompletionRequest, OpenAICompletionRequest
 from core.config import load_config
+from core.config.endpoints import client_llm_override, custom_endpoint_override_enabled
 from core.models.preset import resolve_partition_chat_llm
 from core.utils.exceptions import OpenRAGError
 from core.utils.logging import get_logger
@@ -401,9 +402,18 @@ def _apply_default_max_tokens(
     consistent with the endpoint that serves the request.
 
     An explicit client-supplied value is always honoured.
+
+    Skipped for a client-supplied endpoint: this budget describes the *server's*
+    endpoint, and the client's provider may reject ``max_tokens`` outright (newer
+    OpenAI models want ``max_completion_tokens``). Left unset it drops from the
+    payload; ``validate_tokens_limit`` still falls back to the configured default.
     """
-    if request.max_tokens is None:
-        request.max_tokens = _effective_max_output_tokens(config, partitions)
+    if request.max_tokens is not None:
+        return
+    llm_override = client_llm_override(getattr(request, "metadata", None))
+    if llm_override.get("base_url") and custom_endpoint_override_enabled():
+        return
+    request.max_tokens = _effective_max_output_tokens(config, partitions)
 
 
 def check_tokens_limit(

openrag/api/schemas/user/chat.py

@@ -52,8 +52,14 @@ class OpenAIChatCompletionRequest(BaseModel):
             "llm_override": None,
             "include_all_retrieved_sources": False,
         },
-        description="Extra custom parameters. Supports 'llm_override' object with an optional 'model' to override the downstream model name. The LLM endpoint and credentials are fixed by server configuration and cannot be overridden by the client. "
-        "'include_all_retrieved_sources' (default false) adds the full, unfiltered retrieval set to the response's extra.all_retrieved_sources — off by default since it can be large; opt in only for debugging/evaluation.",
+        description=(
+            "Extra custom parameters. Supports an 'llm_override' object with an optional 'model' "
+            "to override the downstream model name; its 'base_url' and 'api_key' are honored only "
+            "when the deployment sets LLM_OVERRIDE_ALLOW_CUSTOM_ENDPOINT, and ignored otherwise. "
+            "'include_all_retrieved_sources' (default false) adds the full, unfiltered retrieval "
+            "set to the response's extra.all_retrieved_sources — off by default since it can be "
+            "large; opt in only for debugging/evaluation."
+        ),
     )
 
     @model_validator(mode="after")
@@ -102,6 +108,12 @@ class OpenAICompletionRequest(BaseModel):
             "llm_override": None,
             "include_all_retrieved_sources": False,
         },
-        description="Extra custom parameters. Supports 'llm_override' object with an optional 'model' to override the downstream model name. The LLM endpoint and credentials are fixed by server configuration and cannot be overridden by the client. "
-        "'include_all_retrieved_sources' (default false) adds the full, unfiltered retrieval set to the response's extra.all_retrieved_sources — off by default since it can be large; opt in only for debugging/evaluation.",
+        description=(
+            "Extra custom parameters. Supports an 'llm_override' object with an optional 'model' "
+            "to override the downstream model name; its 'base_url' and 'api_key' are honored only "
+            "when the deployment sets LLM_OVERRIDE_ALLOW_CUSTOM_ENDPOINT, and ignored otherwise. "
+            "'include_all_retrieved_sources' (default false) adds the full, unfiltered retrieval "
+            "set to the response's extra.all_retrieved_sources — off by default since it can be "
+            "large; opt in only for debugging/evaluation."
+        ),
     )

openrag/core/config/endpoints.py

@@ -2,10 +2,42 @@
 
 from __future__ import annotations
 
+import os
+from collections.abc import Mapping
+
 from pydantic import Field
 
 from .base import ConfigMixin
 
+LLM_OVERRIDE_ENDPOINT_ENV = "LLM_OVERRIDE_ALLOW_CUSTOM_ENDPOINT"
+
+
+def custom_endpoint_override_enabled() -> bool:
+    """Is a client-supplied ``llm_override.base_url`` honored at all?
+
+    Off by default: enabling it lets any authenticated caller make the server POST
+    to an arbitrary host (SSRF). Only the request *shape* is constrained — see
+    ``VLLMClient._resolve_endpoint_override``.
+
+    Lives here rather than in ``services.inference`` because the API layer reads it
+    too and ``api -> services`` is a forbidden import direction. Read on demand so
+    a test or a reloaded worker sees the current environment.
+    """
+    return os.getenv(LLM_OVERRIDE_ENDPOINT_ENV, "false").strip().lower() == "true"
+
+
+def client_llm_override(metadata: object) -> Mapping[str, object]:
+    """Read ``metadata.llm_override`` as a mapping, or ``{}``.
+
+    Only the *outer* ``metadata`` is schema-validated, so ``{"llm_override":
+    "gpt-4o"}`` is a valid request whose ``.get(...)`` would raise
+    ``AttributeError`` — a 500. A non-mapping override counts as absent.
+    """
+    if not isinstance(metadata, Mapping):
+        return {}
+    override = metadata.get("llm_override")
+    return override if isinstance(override, Mapping) else {}
+
 
 class LLMParamsConfig(ConfigMixin):
     """Shared parameters for LLM/VLM endpoints."""

openrag/services/inference/_circuit_breaker.py

@@ -1,3 +1,4 @@
+from collections.abc import Callable
 from datetime import timedelta
 from functools import wraps
 
@@ -70,10 +71,26 @@ def get_breaker(name: str, fail_max: int = 50, timeout_duration: float = 60.0) -
     return _breakers[name]
 
 
-def with_circuit_breaker(name: str, fail_max: int = 50, timeout_duration: float = 60.0):
+def with_circuit_breaker(
+    name: str,
+    fail_max: int = 50,
+    timeout_duration: float = 60.0,
+    *,
+    skip_if: Callable[..., bool] | None = None,
+):
+    """Guard *fn* with the shared breaker registered under *name*.
+
+    *skip_if* receives the wrapped call's own arguments; returning True runs *fn*
+    outside the breaker entirely. For calls that don't reach the endpoint this
+    breaker describes — folding a second dependency into one health signal makes
+    it wrong in both directions.
+    """
+
     def decorator(fn):
         @wraps(fn)
         async def wrapper(*args, **kwargs):
+            if skip_if is not None and skip_if(*args, **kwargs):
+                return await fn(*args, **kwargs)
             breaker = get_breaker(name, fail_max, timeout_duration)
             try:
                 return await breaker.call_async(fn, *args, **kwargs)