BerriAI · neimaravila · May 8, 2026 · May 8, 2026 · Jun 22, 2026 · Jun 22, 2026
diff --git a/.github/workflows/guard-main-branch.yml b/.github/workflows/guard-main-branch.yml
@@ -31,12 +31,12 @@ jobs:
           echo "PR head repo: $HEAD_REPO"
           echo "PR head branch: $HEAD_REF"
           if [ "$HEAD_REPO" != "$BASE_REPO" ]; then
-            echo "::error::PRs to main must originate from the canonical repository ($BASE_REPO), not a fork ($HEAD_REPO). External contributors should open PRs against the 'litellm_oss_branch' branch instead."
+            echo "::error::PRs to main must originate from the canonical repository ($BASE_REPO), not a fork ($HEAD_REPO). External contributors should open PRs against the 'litellm_oss_staging' branch instead."
             exit 1
           fi
           if [ "$HEAD_REF" = "litellm_internal_staging" ] || [[ "$HEAD_REF" == litellm_hotfix_?* ]]; then
             echo "Allowed source branch."
             exit 0
           fi
-          echo "::error::PRs to main must originate from 'litellm_internal_staging' or a 'litellm_hotfix_*' branch. Got: '$HEAD_REF'. If this is a contribution, retarget the PR against 'litellm_oss_branch' instead."
+          echo "::error::PRs to main must originate from 'litellm_internal_staging' or a 'litellm_hotfix_*' branch. Got: '$HEAD_REF'. If this is a contribution, retarget the PR against 'litellm_oss_staging' instead."
           exit 1
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -29,7 +29,7 @@ If you ever make public-facing PR descriptions, comments, issues, commit message
 - don't use "—". Instead, reach for ";", ".", etc.
 - don't use the pattern "It's not X, it's Y", "You're not X, you're Y", etc.
 - don't use bulleted or numbered lists unless it would be nonsensical not to. Instead, prefer prose
-- don't add a trailing "." at the end of paragraphs (just like this file)
+- don't add a trailing "." at the end of paragraphs (just like this file). That means every paragraph, not just the last one (of the markdown file, PR description, GitHub comment, etc.). Rule of thumb: unless there's a sentence immediately after, don't add a "."
 - don't use →. Instead, prefer not to use arrows, and if need be, use -> instead
 
 Don't hesitate to use values in .env to get needed API keys and other secrets, as long as you never add them to conversation history, commit them, or include them in GitHub issues / PRs

diff --git a/Dockerfile b/Dockerfile
@@ -1,8 +1,8 @@
 # Base image for building
-ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:31da6565f35af6401031c1d7aa91dc84ac76c5c48edd17fb90f0ed9e3173c7a9
+ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:c61ac6919b811ea53c4782d69f1fe05218ba3c25d53f01b6ab7892e621bd4370
 
 # Runtime image
-ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:31da6565f35af6401031c1d7aa91dc84ac76c5c48edd17fb90f0ed9e3173c7a9
+ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:c61ac6919b811ea53c4782d69f1fe05218ba3c25d53f01b6ab7892e621bd4370
 ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.11.7@sha256:240fb85ab0f263ef12f492d8476aa3a2e4e1e333f7d67fbdd923d00a506a516a
 
 FROM $UV_IMAGE AS uvbin

diff --git a/backend/Dockerfile b/backend/Dockerfile
@@ -1,5 +1,5 @@
-ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:31da6565f35af6401031c1d7aa91dc84ac76c5c48edd17fb90f0ed9e3173c7a9
-ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:31da6565f35af6401031c1d7aa91dc84ac76c5c48edd17fb90f0ed9e3173c7a9
+ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:c61ac6919b811ea53c4782d69f1fe05218ba3c25d53f01b6ab7892e621bd4370
+ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:c61ac6919b811ea53c4782d69f1fe05218ba3c25d53f01b6ab7892e621bd4370
 ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.11.7@sha256:240fb85ab0f263ef12f492d8476aa3a2e4e1e333f7d67fbdd923d00a506a516a
 
 FROM $UV_IMAGE AS uvbin

diff --git a/docker/Dockerfile.database b/docker/Dockerfile.database
@@ -1,8 +1,8 @@
 # Base image for building
-ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:3258be472764337fd13095bcbb3182da170243b5819fd67ad4c0754590588b31
+ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:c61ac6919b811ea53c4782d69f1fe05218ba3c25d53f01b6ab7892e621bd4370
 
 # Runtime image
-ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:3258be472764337fd13095bcbb3182da170243b5819fd67ad4c0754590588b31
+ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:c61ac6919b811ea53c4782d69f1fe05218ba3c25d53f01b6ab7892e621bd4370
 ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.11.7@sha256:240fb85ab0f263ef12f492d8476aa3a2e4e1e333f7d67fbdd923d00a506a516a
 
 FROM $UV_IMAGE AS uvbin

diff --git a/docker/Dockerfile.non_root b/docker/Dockerfile.non_root
@@ -1,6 +1,6 @@
 # Base images
-ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:3258be472764337fd13095bcbb3182da170243b5819fd67ad4c0754590588b31
-ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:3258be472764337fd13095bcbb3182da170243b5819fd67ad4c0754590588b31
+ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:c61ac6919b811ea53c4782d69f1fe05218ba3c25d53f01b6ab7892e621bd4370
+ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:c61ac6919b811ea53c4782d69f1fe05218ba3c25d53f01b6ab7892e621bd4370
 ARG PROXY_EXTRAS_SOURCE=published
 ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.11.7@sha256:240fb85ab0f263ef12f492d8476aa3a2e4e1e333f7d67fbdd923d00a506a516a
 

diff --git a/examples/lar1_ollama_config.yaml b/examples/lar1_ollama_config.yaml
@@ -0,0 +1,45 @@
+model_list:
+  - model_name: agent-router
+    litellm_params:
+      model: ollama/qwen3.5:9b
+      api_base: http://127.0.0.1:11434
+    model_info:
+      id: cloud-smart
+      type: cloud-smart
+
+  - model_name: agent-router
+    litellm_params:
+      model: ollama/phi4-mini:latest
+      api_base: http://127.0.0.1:11434
+    model_info:
+      id: cloud-fast
+      type: cloud-fast
+
+  - model_name: agent-router
+    litellm_params:
+      model: ollama/llama3.2:3b
+      api_base: http://127.0.0.1:11434
+    model_info:
+      id: local
+      type: local
+
+  - model_name: agent-router
+    litellm_params:
+      model: ollama/lfm2.5-thinking:latest
+      api_base: http://127.0.0.1:11434
+    model_info:
+      id: deep
+      type: deep
+
+router_settings:
+  routing_strategy: lar1
+  routing_strategy_args:
+    confidence_threshold_low: 0.3
+    confidence_threshold_medium: 0.5
+    confidence_threshold_high: 0.7
+
+general_settings:
+  master_key: sk-lar1-demo
+
+litellm_settings:
+  set_verbose: true
diff --git a/gateway/Dockerfile b/gateway/Dockerfile
@@ -1,5 +1,5 @@
-ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:31da6565f35af6401031c1d7aa91dc84ac76c5c48edd17fb90f0ed9e3173c7a9
-ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:31da6565f35af6401031c1d7aa91dc84ac76c5c48edd17fb90f0ed9e3173c7a9
+ARG LITELLM_BUILD_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:c61ac6919b811ea53c4782d69f1fe05218ba3c25d53f01b6ab7892e621bd4370
+ARG LITELLM_RUNTIME_IMAGE=cgr.dev/chainguard/wolfi-base@sha256:c61ac6919b811ea53c4782d69f1fe05218ba3c25d53f01b6ab7892e621bd4370
 ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.11.7@sha256:240fb85ab0f263ef12f492d8476aa3a2e4e1e333f7d67fbdd923d00a506a516a
 
 FROM $UV_IMAGE AS uvbin

diff --git a/litellm-rust/Cargo.lock b/litellm-rust/Cargo.lock
diff --git a/litellm-rust/Cargo.toml b/litellm-rust/Cargo.toml
@@ -3,6 +3,7 @@ members = [
     "crates/core",
     "crates/providers",
     "crates/python-bridge",
+    "crates/ai-gateway",
 ]
 resolver = "2"
 
@@ -14,10 +15,13 @@ repository = "https://github.com/BerriAI/litellm"
 [workspace.dependencies]
 litellm-core = { path = "crates/core" }
 litellm-providers = { path = "crates/providers" }
+axum = "0.7"
 pyo3 = "0.23.5"
+rand = "0.8"
 reqwest = { version = "0.12", default-features = false, features = ["blocking", "json", "rustls-tls"] }
 serde = { version = "1.0", features = ["derive"] }
 serde_json = "1.0"
+subtle = "2"
 thiserror = "2.0"
 tokio = { version = "1", features = ["rt-multi-thread", "macros", "time"] }
 tokio-tungstenite = { version = "0.24", default-features = false, features = ["connect", "rustls-tls-native-roots"] }

diff --git a/litellm-rust/crates/ai-gateway/AGENTS.md b/litellm-rust/crates/ai-gateway/AGENTS.md
@@ -0,0 +1,50 @@
+# ai-gateway — folder architecture
+
+The Axum server that fronts the Rust gateway. It owns transport + config + auth
+only; deployment selection lives in `core::router`, transforms in `core`/`providers`.
+
+```
+src/
+  main.rs            # entrypoint: build AppState (router + master key), bind, serve
+  state.rs           # AppState — shared Arc<Router> + master_key
+  gil.rs             # GIL-activity tracker (records Python acquisitions)
+  auth/              # authentication as an axum extractor — added to handler args
+    mod.rs           #   RequireMasterKey: FromRequestParts, single master key (LITELLM_MASTER_KEY)
+  routes/            # one module per route, all matching the same template
+    AGENTS.md        #   ← the route template (read this before adding a route)
+    mod.rs           #   app(): merges every module's router()
+    health.rs        #   simple route (one file): router() + liveness/readiness
+    gil.rs           #   simple route (one file): router() + GET /health/gil
+    realtime/        #   route with logic → axum surface + a no-axum service:
+      mod.rs         #     router() + handler + WS<->events adapter (the axum surface)
+      service.rs     #     business logic (select deployment, call provider) — no axum, testable
+  python/            # Python interop (feature: python-config) — load-time only
+    mod.rs, config.rs, AGENTS.md
+```
+
+## Rules
+
+- **Routes follow one template.** Each route module exposes
+  `pub fn router() -> Router<AppState>`; `routes/mod.rs` only merges them. Simple
+  routes are one file; non-trivial routes are a folder (`handler`/`service`/
+  `transport`). See `routes/AGENTS.md`.
+- **Auth is an extractor.** Add `crate::auth::RequireMasterKey` to a handler's
+  args; it runs during extraction. Never re-implement the check per route.
+- **Handlers are thin.** A handler validates and delegates to its `service`. No
+  business logic, no provider calls, no transforms in handlers.
+- **State is shared and cheap to clone.** Long-lived handles live behind `Arc` in
+  `state.rs`; read env/config only in `main.rs` when building state.
+
+## Auth (interim)
+
+A single **master key** (`LITELLM_MASTER_KEY`), enforced by the
+`auth::RequireMasterKey` extractor: any caller presenting it as
+`Authorization: Bearer <key>` may invoke the gateway. Fails closed (500) when
+unset; constant-time compare. The server binds `127.0.0.1` by default (`HOST` to
+override). Full per-key auth + budgets/rate-limits are delegated to the Python
+proxy in a later phase. Health routes don't add the extractor (unauthenticated).
+
+## Python interop
+
+Anything that calls into Python lives in `python/` and is **load-time only** — see
+`python/AGENTS.md`. The realtime data path never takes the GIL.