fix: stop chainlit's Socket.IO transport from tripping the rate limiter

docs/assets/env_example.env

@@ -75,6 +75,21 @@ AUTH_TOKEN=or-openrag-1234
 
 # # Rate limiting, activated by default (recommended).
 # RATE_LIMIT_ENABLED=false
+# # Path prefixes the limiter skips (CSV, matched by prefix — keep the trailing
+# # slash so siblings like /chainlithack stay limited). Unset uses the default shown
+# # here; set-but-empty disables the exemption entirely.
+# RATE_LIMIT_EXEMPT_PATHS=/chainlit/,/assets/
+
+# # Reverse-proxy trust. The client IP behind rate limits, and the scheme behind the
+# # OIDC cookie `Secure` flag, are only read from X-Forwarded-* headers when the peer
+# # is listed here. The bundled admin-ui proxy is NOT on loopback, so keeping the
+# # 127.0.0.1 default makes every chat user share a single rate-limit bucket keyed on
+# # the proxy's address. Prefer your compose/k8s proxy subnet (e.g. 172.16.0.0/12).
+# # Avoid `*` unless the API port is unreachable except through the proxy: `*` trusts
+# # X-Forwarded-For from ANY peer, so a caller reaching the API directly can forge the
+# # header and evade the per-IP brute-force limit on /auth/*. See env_vars.md
+# # (UVICORN_FORWARDED_ALLOW_IPS / Rate Limiting) for the full proxy-trust guidance.
+# UVICORN_FORWARDED_ALLOW_IPS=172.16.0.0/12
 
 
 # ──────────────── Chainlit: Chat interface ────────────────

docs/content/docs/documentation/env_vars.md

@@ -553,7 +553,7 @@ The following environment variables configure the FastAPI server and control acc
 | `DEFAULT_FILE_QUOTA` | `int` | `-1` | Default per-user file quota. `<0` disables quotas globally; `>=0` sets the default limit when a user has no explicit quota. |
 | `PREFERRED_URL_SCHEME` | `string` | `null` | URL scheme (`http` or `https`) used when generating URLs in API responses (e.g., `task_status_url`). When running behind a reverse proxy that terminates SSL, set this to `https` to ensure generated URLs use the correct scheme. If unset, the scheme from the incoming request is used. |
 | `CORS_EXTRA_ORIGINS` | `string` | _(unset)_ | Semicolon-separated list of additional origins allowed by CORS (e.g. `https://app.example.com;https://other.example.com`). Extends the default list without replacing it. |
-| `UVICORN_FORWARDED_ALLOW_IPS` | `string` | `127.0.0.1` | Comma-separated CIDRs/IPs (or `*`) whose `X-Forwarded-*` headers uvicorn trusts. **Required when OpenRAG runs behind a TLS-terminating reverse proxy that lives outside loopback** (typical docker-compose / k8s); otherwise `X-Forwarded-Proto` is dropped and OIDC cookies ship with `Secure=False` even over HTTPS. |
+| `UVICORN_FORWARDED_ALLOW_IPS` | `string` | `127.0.0.1` | Comma-separated CIDRs/IPs (or `*`) whose `X-Forwarded-*` headers uvicorn trusts. **Required when OpenRAG runs behind a reverse proxy that lives outside loopback** (typical docker-compose / k8s — including the bundled admin-ui proxy). Otherwise `X-Forwarded-Proto` is dropped and OIDC cookies ship with `Secure=False` even over HTTPS, and `X-Forwarded-For` is dropped so per-user rate limits collapse onto the proxy's single IP. **Set this to your proxy's subnet, not `*`** — see the proxy-trust caution under [Rate Limiting](#rate-limiting) for why `*` can be spoofed. |
 | `MAX_UPLOAD_SIZE_MB` | `int` | `1024` | Maximum accepted upload size, in MB. `0` or a negative value means unlimited. |
 | `MAX_PARTITIONS_PER_USER` | `int` | `100` | Maximum number of partitions a non-admin user may own. `-1` disables the cap (unlimited). Admin users always bypass it. |
 | `APP_UID` | `int` | `1000` | UID the API container drops to before running the app. Override when your host user is not UID 1000 and bind-mounted folders (`data/`, `logs/`) would otherwise not be writable by the container user. |
@@ -578,6 +578,25 @@ Limit values use the `<count>/<period>` format from the [`limits`](https://limit
 | `RATE_LIMIT_AUTH` | `str` | `60/minute` | Limit for `/auth/*` (login/callback/logout). Keyed on client IP because callers are unauthenticated there — keep it high enough that a shared corporate/NAT egress IP does not throttle a legitimate login rush. |
 | `RATE_LIMIT_CHAT` | `str` | `120/minute` | Limit for `/v1/*` (chat completions, tools). |
 | `RATE_LIMIT_AUTH_FAILURE` | `str` | `RATE_LIMIT_AUTH`, else `20/minute` | Separate, stricter budget for **failed** authentication attempts, keyed by client IP (brute-force protection). Falls back to `RATE_LIMIT_AUTH` when unset, then to `20/minute`. Disabled together with `RATE_LIMIT_ENABLED=false`. |
+| `RATE_LIMIT_EXEMPT_PATHS` | `str` | `/chainlit/,/assets/` | Comma-separated path prefixes the limiter skips, matched with `startswith`. These are auth-bypassed (Chainlit does its own header auth), so requests there carry no user and can only be keyed by IP. Chainlit's Socket.IO transport also issues one HTTP request per packet when it long-polls. Keep the trailing slash so a sibling like `/chainlithack` stays rate-limited rather than being swept into the `/chainlit` exemption. Set-but-empty (`RATE_LIMIT_EXEMPT_PATHS=`) removes all exemptions; `/auth/*` is never exempt. |
+
+:::caution[Behind a reverse proxy, trust the right client IP]
+The `/auth/*` limits and the failed-login budget (`RATE_LIMIT_AUTH_FAILURE`) are keyed **by client IP** — that is how brute-force protection tells attackers apart. Behind a proxy, uvicorn only sees the real client IP if it trusts the proxy's `X-Forwarded-For`, so `UVICORN_FORWARDED_ALLOW_IPS` must name the proxy. Get this wrong in either direction and the per-IP limits stop working:
+
+- **Too narrow** (default `127.0.0.1`, proxy outside loopback): every request keys on the proxy's single address, so one shared bucket throttles the whole deployment at once — one client's failed logins can lock out everyone.
+- **Too wide** (`*`): uvicorn trusts `X-Forwarded-For` from *any* peer. If the API port is reachable directly (the bundled compose publishes `APP_PORT` on `0.0.0.0`), an attacker sends a forged, rotating `X-Forwarded-For` and gets a fresh bucket per request — the brute-force limiter is fully bypassed.
+
+**Recommended:** set `UVICORN_FORWARDED_ALLOW_IPS` to the proxy's subnet (e.g. your compose/k8s network CIDR), **not** `*`. Reserve `*` for deployments where nothing untrusted can reach `APP_PORT` — keep the API on the internal network and expose only the admin-ui proxy (you don't need to publish `APP_PORT` at all).
+
+**Defense in depth (proxy config):** trusting `X-Forwarded-For` only helps if the header can't be forged *through* the proxy. The bundled nginx **appends** to the header (`proxy_add_x_forwarded_for`), and uvicorn reads the left-most value — which a client can inject. For a hard guarantee, have the edge proxy **overwrite** it with the real peer instead of appending:
+
+```nginx
+# openrag-admin.conf — edge proxy: replace any client-supplied X-Forwarded-For
+proxy_set_header X-Forwarded-For $remote_addr;
+```
+
+If nginx itself sits behind another load balancer, use the [real-ip module](https://nginx.org/en/docs/http/ngx_http_realip_module.html) (`set_real_ip_from <lb-subnet>; real_ip_header X-Forwarded-For;`) so the true client is resolved only from that trusted upstream.
+:::
 
 ### Admin UI
 

infra/compose/.env.example

@@ -75,6 +75,21 @@ AUTH_TOKEN=or-openrag-1234
 
 # # Rate limiting, activated by default (recommended).
 # RATE_LIMIT_ENABLED=false
+# # Path prefixes the limiter skips (CSV, matched by prefix — keep the trailing
+# # slash so siblings like /chainlithack stay limited). Unset uses the default shown
+# # here; set-but-empty disables the exemption entirely.
+# RATE_LIMIT_EXEMPT_PATHS=/chainlit/,/assets/
+
+# # Reverse-proxy trust. The client IP behind rate limits, and the scheme behind the
+# # OIDC cookie `Secure` flag, are only read from X-Forwarded-* headers when the peer
+# # is listed here. The bundled admin-ui proxy is NOT on loopback, so keeping the
+# # 127.0.0.1 default makes every chat user share a single rate-limit bucket keyed on
+# # the proxy's address. Prefer your compose/k8s proxy subnet (e.g. 172.16.0.0/12).
+# # Avoid `*` unless the API port is unreachable except through the proxy: `*` trusts
+# # X-Forwarded-For from ANY peer, so a caller reaching the API directly can forge the
+# # header and evade the per-IP brute-force limit on /auth/*. See env_vars.md
+# # (UVICORN_FORWARDED_ALLOW_IPS / Rate Limiting) for the full proxy-trust guidance.
+# UVICORN_FORWARDED_ALLOW_IPS=172.16.0.0/12
 
 
 # ──────────────── Chainlit: Chat interface ────────────────

infra/compose/nginx/openrag-admin.conf

@@ -14,7 +14,11 @@ map $http_x_forwarded_proto $forwarded_scheme {
     ''      $scheme;
 }
 
-# Preserve WebSocket upgrades for proxied apps such as Chainlit.
+# Echo the client's Upgrade intent through to the backend. Chainlit's Socket.IO
+# transport (/chainlit/ws/socket.io) opens on HTTP long-polling and then upgrades
+# to a WebSocket; without this the upgrade fails and it polls forever, issuing one
+# request per emitted packet. Must be a map, not a hardcoded `Connection: upgrade`
+# — that header on an ordinary request breaks the proxied connection.
 map $http_upgrade $connection_upgrade {
     default upgrade;
     ''      close;
@@ -70,6 +74,8 @@ server {
         proxy_set_header X-Real-IP         $remote_addr;
         proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
         proxy_set_header X-Forwarded-Proto $forwarded_scheme;
+        # WebSocket upgrades (chainlit's Socket.IO). nginx proxies over HTTP/1.0 by
+        # default, which cannot carry an upgrade, so 1.1 is required here.
         proxy_http_version 1.1;
         proxy_set_header Upgrade           $http_upgrade;
         proxy_set_header Connection        $connection_upgrade;

infra/scripts/entrypoint.sh

@@ -50,5 +50,13 @@ else
   if [[ "${UVICORN_RELOAD}" == "true" ]]; then
     RELOAD_ARGS+=("--reload")
   fi
-  uv run --no-dev "${ENV_ARGS[@]}" uvicorn api.main:app --host 0.0.0.0 --port "${APP_iPORT:-8080}" "${RELOAD_ARGS[@]}" --workers 1
+  # uvicorn only honours X-Forwarded-* from peers listed in --forwarded-allow-ips
+  # (default: 127.0.0.1), so a reverse proxy outside loopback — the admin-ui
+  # container, a k8s ingress — is ignored: request.client.host stays the proxy's
+  # address (collapsing every user into one rate-limit bucket) and
+  # X-Forwarded-Proto is dropped. Forward the same UVICORN_FORWARDED_ALLOW_IPS
+  # that api.main's __main__ block reads, so one documented variable covers both
+  # entrypoints — the bare `uvicorn` CLI otherwise only reads FORWARDED_ALLOW_IPS.
+  uv run --no-dev "${ENV_ARGS[@]}" uvicorn api.main:app --host 0.0.0.0 --port "${APP_iPORT:-8080}" "${RELOAD_ARGS[@]}" --workers 1 \
+    --proxy-headers --forwarded-allow-ips "${UVICORN_FORWARDED_ALLOW_IPS:-127.0.0.1}"
 fi

openrag/api/middleware/rate_limit.py

@@ -9,8 +9,19 @@
 ``api.dependencies.auth`` — trusted operators (admin UI polling, bulk scripts)
 should not be throttled.
 
+The Chainlit subtree is exempt: ``AuthMiddleware`` bypasses it (Chainlit runs
+its own header-auth callback), so no user ever lands on ``request.state`` and
+every request there keys on the client IP — which behind the admin-ui reverse
+proxy is a single container address shared by every chat user. A per-identity
+limit that cannot tell identities apart throttles the whole deployment at once,
+and Chainlit's Socket.IO transport at ``/chainlit/ws/socket.io`` issues one
+HTTP request per emitted packet whenever it falls back to long-polling.
+
 Env: RATE_LIMIT_ENABLED (true), RATE_LIMIT_DEFAULT (600/minute),
-RATE_LIMIT_AUTH (60/minute, /auth/*), RATE_LIMIT_CHAT (120/minute, /v1/*).
+RATE_LIMIT_AUTH (60/minute, /auth/*), RATE_LIMIT_CHAT (120/minute, /v1/*),
+RATE_LIMIT_EXEMPT_PATHS (/chainlit/,/assets/ — CSV of path prefixes). Any
+configured prefix that would cover /auth/ is dropped (with a warning) so the
+brute-force surface can never be exempted, even by operator misconfiguration.
 """
 
 import os
@@ -26,6 +37,22 @@
 
 logger = get_logger()
 
+# Path prefixes the limiter never touches, matched with ``str.startswith`` — so the
+# trailing slash is load-bearing. These are the auth-bypassed subtrees from
+# ``api.middleware.auth.is_bypass_path``: unauthenticated by design, so a request
+# there carries no user and can only ever be keyed by the proxy's IP. The prefixes
+# end in ``/`` so a sibling like ``/chainlithack`` is NOT swept into the ``/chainlit``
+# exemption — it stays rate-limited. Bare ``/chainlit`` (the 307 to ``/chainlit/``) is
+# therefore metered at the default tier, but that is one request per page load, so it
+# never bites. ``/auth/*`` is deliberately NOT here: it is the brute-force surface and
+# IP keying is the point.
+DEFAULT_EXEMPT_PREFIXES: tuple[str, ...] = ("/chainlit/", "/assets/")
+
+# The one prefix RATE_LIMIT_EXEMPT_PATHS is never allowed to cover, even via a
+# misconfigured operator override (e.g. "/auth/" or an overly broad "/"): it is
+# the brute-force surface and must always stay rate-limited.
+_PROTECTED_PREFIX = "/auth/"
+
 
 def _env_flag(name: str, default: bool) -> bool:
     val = os.environ.get(name)
@@ -34,13 +61,34 @@ def _env_flag(name: str, default: bool) -> bool:
     return val.strip().lower() in ("1", "true", "yes", "on")
 
 
+def _env_prefixes(name: str, default: tuple[str, ...]) -> tuple[str, ...]:
+    """CSV of path prefixes. Unset uses ``default``; set-but-empty disables exemptions."""
+    raw = os.environ.get(name)
+    if raw is None:
+        return default
+    return tuple(p.strip() for p in raw.split(",") if p.strip())
+
+
 class RateLimitMiddleware(BaseHTTPMiddleware):
     """Apply per-identity moving-window rate limits, tiered by path prefix."""
 
     def __init__(self, app):
         super().__init__(app)
         self.enabled = _env_flag("RATE_LIMIT_ENABLED", True)
         self._limiter = MovingWindowRateLimiter(MemoryStorage())
+        exempt = _env_prefixes("RATE_LIMIT_EXEMPT_PATHS", DEFAULT_EXEMPT_PREFIXES)
+        # Drop any prefix that overlaps "/auth/" in either direction: one that is
+        # itself a prefix of it (e.g. "/auth/", "/a", "/" — too broad, swallows the
+        # whole subtree) and one that starts with it (e.g. "/auth/login" — narrow,
+        # but still carves a brute-forceable route out of the auth tier). Either
+        # shape lets path.startswith() below exempt a request under /auth/.
+        unsafe = tuple(p for p in exempt if _PROTECTED_PREFIX.startswith(p) or p.startswith(_PROTECTED_PREFIX))
+        if unsafe:
+            logger.warning(
+                "Ignoring RATE_LIMIT_EXEMPT_PATHS entries that would exempt /auth/",
+                prefixes=",".join(unsafe),
+            )
+        self._exempt = tuple(p for p in exempt if p not in unsafe)
         # Only parse the limit configs when rate limiting is enabled: a malformed
         # RATE_LIMIT_* value must not crash boot when the feature is turned off
         # (``dispatch`` short-circuits before touching these when disabled).
@@ -54,6 +102,7 @@ def __init__(self, app):
                 default=str(self._default),
                 auth=str(self._auth),
                 chat=str(self._chat),
+                exempt=",".join(self._exempt) or "(none)",
             )
 
     def _limit_for(self, path: str):
@@ -83,13 +132,20 @@ async def dispatch(self, request: Request, call_next):
         if not self.enabled:
             return await call_next(request)
 
+        path = request.url.path
+
+        # Exempt prefixes are checked before the admin bypass on purpose: these
+        # paths never run through AuthMiddleware, so request.state.user is unset
+        # and _is_admin() is False even for an admin's browser session.
+        if self._exempt and path.startswith(self._exempt):
+            return await call_next(request)
+
         # Admins bypass rate limiting entirely, mirroring the file-quota bypass
         # in api.dependencies.auth. Unauthenticated paths (/auth/*) have no user
         # on request.state, so this only ever exempts an authenticated admin.
         if self._is_admin(request):
             return await call_next(request)
 
-        path = request.url.path
         limit, tier = self._limit_for(path)
         identity = self._identity(request)