feat(v1.87.0): port PublicReqMiddleware + small UI fixes (Wave 3)

docker/prod_entrypoint.sh

@@ -1,8 +1,15 @@
 #!/bin/sh
 
+# Internal-fork modification: invoke `python -m litellm_extras.entrypoint`
+# instead of the upstream `litellm` console script. The wrapper installs
+# PublicReqMiddleware on the FastAPI app before delegating to the original
+# Click CLI, which is what enforces the X-Public-Req gating in production.
+# Routing through the wrapper at the entrypoint level (rather than via
+# docker-compose overrides) means every deployment of this image picks
+# the middleware up automatically — there is no per-deploy step to forget.
 if [ "$USE_DDTRACE" = "true" ]; then
     export DD_TRACE_OPENAI_ENABLED="False"
-    exec ddtrace-run litellm "$@"
+    exec ddtrace-run python -m litellm_extras.entrypoint "$@"
 else
-    exec litellm "$@"
+    exec python -m litellm_extras.entrypoint "$@"
 fi

e2e/_config/docker-compose.yml

@@ -70,6 +70,11 @@ services:
       # Rendered at proxy-start time by `e2e/tools/proxy` from .env values.
       # See ../tools/proxy `render_config()`. The file is gitignored.
       - ./.litellm.rendered.yaml:/app/config.yaml:ro
+    # NOTE: no `entrypoint:` override here. The fork's
+    # docker/prod_entrypoint.sh already routes through
+    # `python -m litellm_extras.entrypoint`, so PublicReqMiddleware loads
+    # automatically. This keeps e2e behavior aligned with production —
+    # if the wrapper breaks in either path, both surfaces catch it.
     command:
       - --config=/app/config.yaml
       - --port=4000

e2e/cases/18_public_req_middleware.md

@@ -0,0 +1,120 @@
+# Case 18 — `X-Public-Req` middleware: streaming + header / query gating
+
+## Goal
+
+Verify the `PublicReqMiddleware` (installed via
+`litellm_extras.entrypoint`) correctly differentiates external vs internal
+requests **without buffering streaming bodies**.
+
+Two-pronged check:
+
+1. **Streaming integrity.** SSE chunks arrive incrementally; TTFB is
+   small relative to total wall time. This is the regression guard
+   against accidentally switching to `BaseHTTPMiddleware` or otherwise
+   awaiting on body messages — both would collapse TTFB onto end-of-stream.
+2. **Differential behavior.** `X-Public-Req: 1` strips `x-litellm-*`
+   from response headers and blocks `/v1/models?include_metadata=true`.
+   The same requests without the header pass through unchanged.
+
+Functional logic (inbound header strip, case-insensitive matching,
+forbidden-query parsing, non-HTTP scope handling) is covered by
+`tests/test_litellm/test_public_req_middleware.py`. This e2e adds the
+wire-level confirmation.
+
+## Origin
+
+We expose the LiteLLM proxy to external users behind a public Nginx
+ingress. The ingress injects `X-Public-Req: 1` on every forwarded
+request; internal services reach the proxy on a separate Service that
+does not pass through the ingress. The middleware sits in the proxy's
+ASGI stack and applies safeguards conditional on that header.
+
+## Preconditions
+
+- `e2e/tools/proxy status` reports `ready`
+- The proxy image was built **after** `litellm_extras/` was added —
+  run `e2e/tools/proxy rebuild` if unsure
+- `ANTHROPIC_API_KEY` set (one ~50-word streaming completion;
+  cost ≈ $0.002)
+
+## Steps
+
+```bash
+bash e2e/cases/data/18_public_req_middleware.sh
+echo "exit=$?"
+```
+
+The fixture executes six assertions. Each prints `PASS:` or `FAIL:`:
+
+### Streaming integrity (paid)
+
+- **A1** SSE response yields ≥ 3 `data:` chunks
+- **A2** Streaming phase (wall − TTFB) > 200 ms — chunks spread over
+  time rather than dumped at end-of-stream. Provider TTFT variance can
+  push the ratio above 0.5 on short completions, so the absolute
+  duration is more robust than a ratio threshold; a buffering
+  middleware would collapse this to ≤ 5 ms regardless of provider.
+
+### Header gating (paid)
+
+- **A3** Public mode: zero `x-litellm-*` headers in response
+- **A4** Internal mode (no `X-Public-Req`): at least one
+  `x-litellm-*` header in response (control)
+
+### Query gating on `/v1/models` (free)
+
+- **A5** Public mode: `GET /v1/models?include_metadata=true` returns
+  HTTP 200 with the **same** body as `GET /v1/models` (forbidden query
+  silently stripped before reaching the proxy).
+- **A6** Internal mode: same request returns HTTP 200 with a body that
+  **differs** from the bare-models response (metadata expansion still
+  works for internal callers).
+
+### Inbound `x-litellm-*` strip (free — uses `mock_response`)
+
+Two chat completions are made with `mock_response: "pong"` (no upstream
+provider call). Each carries `X-Litellm-Spend-Logs-Metadata` with a
+unique marker. LiteLLM persists honored header values to
+`metadata.spend_logs_metadata` in the spend_logs row, so the marker's
+presence/absence in Postgres is a definitive signal of whether the
+header reached the proxy core.
+
+- **A7** Internal mode (control): marker **must** appear in spend_logs
+  within 15 s (proves the header would otherwise be honored)
+- **A8** Public mode (`X-Public-Req: 1`): marker **must not** appear in
+  any spend_logs row (proves the middleware stripped the header before
+  LiteLLM saw it)
+
+## Expected — GREEN
+
+```
+A1 PASS: 7 SSE chunks received
+A2 PASS: ttfb=8294ms wall=10171ms streaming_phase=1877ms
+A3 PASS: 0 x-litellm-* headers in public response
+A4 PASS: 7 x-litellm-* headers in internal response
+A5 PASS: /v1/models?include_metadata=true returned 200 with query stripped (public)
+A6 PASS: /v1/models?include_metadata=true returned 200 with metadata expanded (internal)
+A7 PASS: internal mode: x-litellm-spend-logs-metadata reached spend_logs (1 row)
+A8 PASS: public mode: inbound x-litellm-spend-logs-metadata stripped (marker absent from spend_logs)
+```
+
+## Failure modes
+
+| Symptom | Likely cause |
+|---|---|
+| A1 fails with 0-1 chunks | Middleware is buffering — switched to `BaseHTTPMiddleware`, or `send_wrapper` is awaiting on body |
+| A2 `streaming_phase ≤ 5ms` | Middleware buffered the body and dumped it on close — same root cause as A1 |
+| A3 fails (still see `x-litellm-*` in public response) | `send_wrapper` not wired, or middleware not installed; check `proxy logs` for `PublicReqMiddleware` |
+| A4 fails (no `x-litellm-*` in internal) | Middleware is stripping for *all* requests; check `_is_public` returns False without the header |
+| A5 fails — public bodies differ between `?include_metadata=true` and bare `/v1/models` | Query strip did not run — `/v1/models` not in `MODELS_PATHS`, middleware not installed, or `parse_qsl`/`urlencode` lost the rewrite |
+| A5 fails — public returns 4xx | Middleware reverted to the old reject-with-400 behavior; revert the strip refactor |
+| A6 fails — internal bodies identical | Middleware running for internal calls — `_is_public` defaulting to True; strip is happening when it shouldn't |
+| A7 fails — internal marker missing from spend_logs | Async spend logger lag or DB schema drift — not a middleware bug. Raise the 15 s poll if reproducible |
+| A8 fails — public marker present in spend_logs | Inbound `x-litellm-*` strip is NOT running. Verify the middleware is installed and that `LITELLM_HEADER_PREFIX` matching is case-insensitive |
+
+## Cross-reference
+
+- `litellm_extras/public_req_middleware.py` — middleware under test
+- `litellm_extras/entrypoint.py` — wrapper that installs the middleware
+- `e2e/_config/docker-compose.yml` — `command:` invokes the wrapper
+- `tests/test_litellm/test_public_req_middleware.py` — functional unit tests

e2e/cases/22_gemini_credential_custom_api_base.md

@@ -0,0 +1,100 @@
+# Case 22 — Gemini provider with custom `api_base`
+
+## Goal
+
+End-to-end validation that the UI change in PR #24 (exposing `api_base`
+on the Google AI Studio credential form) reaches the gemini provider
+runtime correctly. Specifically:
+
+1. A deployment with `litellm_params.model = gemini/...` plus a custom
+   `api_base` pointing at a Gemini-compatible gateway resolves to the
+   gemini provider code path (no ADC, no Vertex auth).
+2. The outbound request hits `{api_base}/models/{model}:generateContent`
+   with an `x-goog-api-key` header — matching what a self-hosted gateway
+   like Raven Router or anispark's ai-router exposes.
+3. The response surface (status, `x-litellm-model-api-base`,
+   `x-litellm-model-group`) reflects the custom api_base, not Google's
+   default `generativelanguage.googleapis.com`.
+
+This case guards the runtime half of PR #24. The UI half is covered by
+`tests/test_litellm/proxy/public_endpoints/test_public_endpoints.py::test_google_ai_studio_provider_fields_expose_api_base`.
+
+## Background
+
+Before this fix, the only LiteLLM UI options for "URL + API key Gemini"
+were:
+
+- **OpenAI credential** mislabeled (admin confusion, wrong logo/group in
+  the dashboard).
+- Raw admin API PATCH of `litellm_params.api_base` (no UI path).
+
+The runtime gemini provider already supported custom `api_base` via
+`litellm/llms/vertex_ai/vertex_llm_base.py:415` —
+`_check_custom_proxy` builds `{api_base}/models/{model}:{endpoint}` and
+attaches `x-goog-api-key: {gemini_api_key}` (line 484). The
+`_ensure_access_token_async` ADC path is skipped because
+`custom_llm_provider == "gemini"` (line 714). The UI just needed to
+expose the `api_base` field — see `provider_create_fields.json` →
+`Google_AI_Studio`.
+
+## Preconditions
+
+- `e2e/.env` has `GEMINI_API_KEY` and `GEMINI_API_BASE` set, pointing at
+  a Gemini-compatible gateway. Examples:
+  - `https://generativelanguage.googleapis.com/v1beta` (canonical Google
+    AI Studio; works with an `AIza...` key)
+  - `https://ai-router-hk.anispark.ai/v1beta` (anispark / Raven Router
+    with their own sk-style key)
+  - any self-hosted proxy that serves
+    `POST /v1beta/models/{model}:generateContent` with
+    `x-goog-api-key` auth.
+- Optionally `MODEL_GEMINI` to override the upstream model id (default
+  `gemini/gemini-3.1-pro-preview`).
+- `e2e/tools/proxy status` reports `ready` against an image built AFTER
+  this PR's branch (`e2e/tools/proxy build` if unsure — the rendered
+  config gains a `gemini-custom-base` deployment when `GEMINI_API_KEY`
+  is set).
+- Without `GEMINI_API_KEY` the case exits 77 (SKIP) — the proxy doesn't
+  render the deployment, so there's nothing to test.
+
+## Steps
+
+```bash
+bash e2e/cases/data/22_gemini_credential_custom_api_base.sh
+echo "exit=$?"
+```
+
+The fixture issues a single chat completion call against the
+`gemini-custom-base` deployment and inspects the response headers:
+
+- **C1** `POST /v1/chat/completions` model=`gemini-custom-base` →
+  expects HTTP 200, body has `choices`, header
+  `x-litellm-model-api-base` equals the value of `GEMINI_API_BASE` from
+  `e2e/.env`, header `x-litellm-model-group == gemini-custom-base`.
+
+## Expected — GREEN
+
+```
+PASS C1: HTTP 200, x-litellm-model-api-base=https://<gateway>/v1beta
+PASS C1: x-litellm-model-group=gemini-custom-base
+PASS C1: body has 'choices' array
+```
+
+## Failure modes
+
+| Symptom | Likely cause |
+|---|---|
+| `SKIP: GEMINI_API_KEY not set` | Add `GEMINI_API_KEY` and `GEMINI_API_BASE` to `e2e/.env`, then `e2e/tools/proxy restart`. |
+| `FAIL C1: HTTP 500 ... DefaultCredentialsError` | The deployment routed to `vertex_ai`/`vertex_ai_beta` instead of `gemini`. Verify `MODEL_GEMINI` does not have `vertex_ai/` prefix and the rendered config shows `model: gemini/...` for `gemini-custom-base`. |
+| `FAIL C1: HTTP 200 but body is HTML (<!doctype html>)` | `api_base` is the gateway's UI root, not the API path. Add `/v1beta` (or the equivalent for your gateway) to `GEMINI_API_BASE`. |
+| `FAIL C1: x-litellm-model-api-base != GEMINI_API_BASE` | LiteLLM is ignoring the custom api_base. Confirm the deployment block in the rendered config has `api_base: os.environ/GEMINI_API_BASE` and not the default Google endpoint. |
+| `FAIL C1: 401 / 403` from upstream | `GEMINI_API_KEY` is wrong for this gateway, or the gateway expects a different auth header (some non-Google gateways want `Authorization: Bearer` instead of `x-goog-api-key`). Confirm with a direct curl to `{api_base}/models/{model}:generateContent`. |
+
+## Cross-reference
+
+- PR #24 (`fix(ui): expose api_base on Google AI Studio credential form`)
+- `litellm/llms/vertex_ai/vertex_llm_base.py:415` —
+  `_check_custom_proxy` URL construction for gemini provider
+- `litellm/llms/vertex_ai/vertex_llm_base.py:714` — gemini provider
+  skipping ADC
+- `tests/test_litellm/proxy/public_endpoints/test_public_endpoints.py::test_google_ai_studio_provider_fields_expose_api_base` — UI field metadata assertion

e2e/cases/README.md

@@ -23,6 +23,7 @@ Humans can execute them too — every step is a concrete shell command.
 | 11 | `11_error_information_message_populated.md` | (none — invalid key) | `spend_logs.metadata.error_information.error_message` non-empty on failure | ✓ |
 | 12 | `12_custom_pricing_must_honor_cache_tokens.md` | (none — direct calc) | `custom_cost_per_token` short-circuit must include cache pricing (Bug #2 root cause) | — |
 | 16 | `16_budget_reset_no_prisma_error.md` | (none — proxy only) | `ResetBudgetJob.reset_budget_windows` background tick must not raise `prisma.errors.MissingRequiredValueError` on `Json?` null-filter. Regression for BerriAI/litellm#26346 | ✓ |
+| 18 | `18_public_req_middleware.md` | Anthropic | `litellm_extras.PublicReqMiddleware` keeps streaming responses incremental, strips `x-litellm-*` under `X-Public-Req: 1`, and rejects sensitive `/v1/models` query params | — |
 
 ## How to invoke