feat(proxy): add PublicReqMiddleware for X-Public-Req gated safeguards

[songkuan-zheng]

Summary

Add an internal ASGI middleware that lets a single LiteLLM proxy serve
both external users (through a public Nginx ingress) and internal
services (direct ClusterIP) from one process. The public ingress
injects X-Public-Req: 1; the middleware keys off that header to apply
two safeguards on external requests while leaving internal traffic
untouched.

When X-Public-Req: 1 is present, the middleware:

• Strips every inbound x-litellm-* request header. Otherwise users
  could inject x-litellm-api-key (override Authorization),
  x-litellm-num-retries (one request → N upstream calls),
  x-litellm-spend-logs-metadata (poison audit logs),
  x-litellm-mock-response (forge responses), etc.
• Silently strips forbidden query parameters from /v1/models and
  /models (include_metadata, fallback_type,
  include_model_access_groups, only_model_access_groups). These
  expose router fallback chains and access-group naming. Other query
  params are preserved byte-for-byte; the request still returns 200.
• Strips every outbound x-litellm-* response header (model
  deployment IDs, cache-hit flags, response cost, ratelimit-remaining)
  before they leave the proxy.

When the header is absent, the middleware is a no-op — internal
services keep using override headers and full observability fields.

How it's installed

litellm_extras/entrypoint.py is a wrapper around the upstream
litellm CLI that imports litellm.proxy.proxy_server first
(constructing the FastAPI app), calls add_middleware, then
delegates to the existing Click run_server CLI. Python's module
cache means uvicorn's from .proxy_server import app returns the same
mutated instance.

e2e/_config/docker-compose.yml is switched to invoke
python -m litellm_extras.entrypoint in entrypoint:. Production
deploys should mirror the same swap.

Streaming safety

The middleware is pure ASGI, not BaseHTTPMiddleware. It does NOT
buffer response bodies — header changes happen only on the single
http.response.start message; http.response.body chunks pass
through verbatim. Streaming TTFT is unaffected, verified by both unit
test (4-chunk send order assertion) and e2e (streaming_phase > 200ms
on a real Anthropic streaming call).

Test plan

• Unit tests (tests/test_litellm/test_public_req_middleware.py) — 12 tests:

  • internal passthrough leaves scope/response untouched
  • public mode strips inbound x-litellm-* headers
  • public mode strips outbound x-litellm-* headers
  • streaming body chunks pass through one-by-one with correct more_body
  • /v1/models?include_metadata=...&team_id=keep → only team_id reaches inner app
  • /v1/models?include_metadata=true&only_model_access_groups=1 → empty query string
  • /v1/models?team_id=alpha&scope=expand → byte-identical passthrough
  • Marker value other than "1" treated as internal
  • Mixed-case X-Public-Req and X-Litellm-Tags both handled
  • Non-HTTP (websocket / lifespan) scope passes through

• E2E case 18 (e2e/cases/18_public_req_middleware.md, paid) — 8 assertions:

  • A1 SSE stream ≥ 3 chunks
  • A2 streaming_phase = wall - ttfb > 200ms (proves no buffering)
  • A3 public response: 0 x-litellm-* headers
  • A4 internal response: ≥ 1 x-litellm-* header (control)
  • A5 public /v1/models?include_metadata=true returns 200 with same body as bare
  • A6 internal /v1/models?include_metadata=true returns 200 with different body
  • A7 internal x-litellm-spend-logs-metadata reaches spend_logs (control)
  • A8 public x-litellm-spend-logs-metadata absent from spend_logs (DB-layer verification)

• Full e2e suite (e2e/tools/run-all-cases) — 18/18 PASS, no regressions

• uv run black . clean

• uv run ruff check clean