Skip to content
Closed
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .circleci/scripts/unit_selection.sh
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ legacy_paths() {
llm-vertex-ai) echo tests/unit/llms/vertex_ai ;;
mcp-integration)
echo tests/unit/experimental_mcp_client
echo tests/unit/proxy/test_admin_mcp.py
echo tests/unit/proxy/_experimental/mcp_server
echo tests/unit/responses/mcp
echo tests/mcp_tests/test_proxy_mcp_e2e.py ;;
Expand Down
2 changes: 2 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ COPY litellm-proxy-extras/pyproject.toml litellm-proxy-extras/
RUN uv sync --frozen --no-install-project --no-install-workspace --no-default-groups --no-editable \
--extra proxy \
--extra proxy-runtime \
--group admin-mcp \
--extra extra_proxy \
--extra semantic-router \
--extra saml \
Expand All @@ -101,6 +102,7 @@ RUN sed -i 's/\r$//' docker/build_admin_ui.sh && chmod +x docker/build_admin_ui.
RUN uv sync --frozen --no-default-groups --no-editable \
--extra proxy \
--extra proxy-runtime \
--group admin-mcp \
--extra extra_proxy \
--extra semantic-router \
--extra saml \
Expand Down
3 changes: 3 additions & 0 deletions docker/Dockerfile.non_root
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,7 @@ RUN --mount=type=cache,target=/app/.cache/uv,id=litellm-uv-cache \
uv sync --frozen --no-install-project --no-install-workspace --no-default-groups --no-editable \
--extra proxy \
--extra proxy-runtime \
--group admin-mcp \
--extra extra_proxy \
--extra semantic-router \
--extra saml \
Expand Down Expand Up @@ -111,6 +112,7 @@ RUN --mount=type=cache,target=/app/.cache/uv,id=litellm-uv-cache \
uv sync --frozen --no-default-groups --no-editable \
--extra proxy \
--extra proxy-runtime \
--group admin-mcp \
--extra extra_proxy \
--extra semantic-router \
--extra saml \
Expand All @@ -121,6 +123,7 @@ RUN --mount=type=cache,target=/app/.cache/uv,id=litellm-uv-cache \
uv sync --frozen --no-default-groups --no-editable \
--extra proxy \
--extra proxy-runtime \
--group admin-mcp \
--extra extra_proxy \
--extra semantic-router \
--extra saml \
Expand Down
65 changes: 65 additions & 0 deletions docker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,71 @@ This guide provides instructions for building and running the LiteLLM applicatio
- Docker
- Docker Compose

## Enable the built-in admin MCP

The images built from `Dockerfile` and `docker/Dockerfile.non_root` include the
[LiteLLM Admin MCP](https://github.com/BerriAI/liteadmin-mcp). It is disabled by
default. Enable it on the existing LiteLLM service and restart the container:

```yaml
environment:
LITELLM_ENABLE_ADMIN_MCP: "true"
PROXY_BASE_URL: "https://litellm.example.com"
```

Keep your existing database, authentication, configuration mount and HTTPS reverse
proxy. The MCP endpoint is `https://litellm.example.com/admin/mcp`, on the same
port as LiteLLM. If the gateway is served under a URL prefix, include that prefix
in the endpoint, for example `https://litellm.example.com/gateway/admin/mcp`

Connect an MCP client that supports Streamable HTTP and bearer headers:

```json
{
"mcpServers": {
"litellm-admin": {
"url": "https://litellm.example.com/admin/mcp",
"headers": {
"Authorization": "Bearer <your-personal-proxy-admin-key>"
}
}
}
}
```

Each caller needs a current `proxy_admin` identity, including for read operations.
The connector calls this gateway's management API in process with the caller's
credential. It does not use `LITELLM_BASE_URL` or a shared `LITELLM_API_KEY`

The integration mounts the connector's own ASGI application and uses the standard
`httpx2.ASGITransport` for calls back into the gateway. LiteLLM's outbound HTTP
handler uses `httpx`, while the connector requires `httpx2`. A small ASGI adapter
preserves the original caller's address, scheme and policy headers, which a direct
transport mount would replace with loopback defaults. It retains the complete
gateway middleware and authentication stack; it does not implement HTTP requests
or MCP protocol handling. Nested calls share their parent request's admission
slot, and active requests drain before the connector is closed

`PROXY_BASE_URL` supplies the trusted public origin for Host and Origin validation.
Set `LITELLM_MCP_PUBLIC_URL` to an HTTPS origin if the admin MCP uses a different
public hostname. Without either setting, only the connector's loopback hosts are
accepted. Keep the existing `/mcp` endpoint for the MCP gateway; enabling admin MCP
reserves the `/admin` prefix, including any existing MCP server alias named `admin`

Set `LITELLM_ADMIN_READ_ONLY=true` to disable writes, or
`LITELLM_ADMIN_TOOLS=list_keys,list_teams` to restrict the available operations.
These restrictions apply in addition to the gateway's authorization checks

Embedded deployments return complete results by default, so requests can reach
different workers or replicas. `LITELLM_ADMIN_RESPONSE_VIEW=compact` enables the
connector's paged results. Compact results, including explicit per-call requests
for them, require subsequent `read_admin_result` calls to reach the same worker

The connector source is pinned to a commit and archive checksum in the
`admin-mcp` dependency group and `uv.lock`. Updates ship with the LiteLLM image;
starting the server does not download code. For source development with Python
3.12 or later, install it with `uv sync --extra proxy --group admin-mcp`

## Building and Running the Application

To build and run the application, you will use the `docker-compose.yml` file located in the root of the project. This file is configured to use the `Dockerfile.non_root` for a secure, non-root container environment.
Expand Down
113 changes: 113 additions & 0 deletions litellm/proxy/admin_mcp.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
import os
from collections.abc import AsyncGenerator
from contextlib import asynccontextmanager
from contextvars import ContextVar
from typing import Final
from urllib.parse import urlsplit

from fastapi import FastAPI
from starlette.datastructures import Headers
from starlette.requests import Request
from starlette.routing import Mount
from starlette.types import ASGIApp, Receive, Scope, Send

_REQUEST_HEADERS: Final = frozenset(
{
b"authorization",
b"cookie",
b"content-length",
b"content-type",
b"transfer-encoding",
b"connection",
b"accept",
b"accept-encoding",
b"mcp-protocol-version",
b"mcp-session-id",
}
)


class _CallerContext:
def __init__(self, app: ASGIApp, caller: ContextVar[Request]) -> None:
self.app = app
self.caller = caller

async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
if scope["type"] != "http":
await self.app(scope, receive, send)
return
token: Final = self.caller.set(Request(scope))
try:
await self.app(scope, receive, send)
finally:
self.caller.reset(token)


@asynccontextmanager
async def admin_mcp_lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
enabled: Final = os.environ.get("LITELLM_ENABLE_ADMIN_MCP", "false").strip().lower()
if enabled in ("false", "0", ""):
yield
return
if enabled not in ("true", "1"):
raise ValueError("LITELLM_ENABLE_ADMIN_MCP must be true or false")

try:
import httpx2
from litellm_admin_mcp.config import ( # pyright: ignore[reportMissingTypeStubs] # upstream has no py.typed marker
Config,
env_bool,
)
from litellm_admin_mcp.gateway import ( # pyright: ignore[reportMissingTypeStubs] # upstream has no py.typed marker
Gateway,
)
from litellm_admin_mcp.server import ( # pyright: ignore[reportMissingTypeStubs] # upstream has no py.typed marker
create_http_app,
)
except ImportError as exc:
raise RuntimeError(
"Admin MCP requires Python 3.12+ and the admin-mcp dependency group. "
"Use a LiteLLM image that bundles it, or run uv sync --extra proxy --group admin-mcp."
) from exc

from litellm.proxy.shutdown.graceful_shutdown_manager import GracefulShutdownManager

public_url: Final = urlsplit(os.environ.get("LITELLM_MCP_PUBLIC_URL") or os.environ.get("PROXY_BASE_URL", ""))
config: Final = Config(
base_url="http://localhost",
public_url=f"{public_url.scheme}://{public_url.netloc}" if public_url.netloc else "",
read_only=env_bool("LITELLM_ADMIN_READ_ONLY"),
allowed_tools=frozenset(
name.strip() for name in os.environ.get("LITELLM_ADMIN_TOOLS", "").split(",") if name.strip()
),
response_view=os.environ.get("LITELLM_ADMIN_RESPONSE_VIEW", "full").strip(),
schema_mode=os.environ.get("LITELLM_ADMIN_SCHEMA_MODE", "full").strip(),
)
caller: Final[ContextVar[Request]] = ContextVar("admin_mcp_caller")

async def management_api(scope: Scope, receive: Receive, send: Send) -> None:
request: Final = caller.get()
caller_headers: Final = tuple(pair for pair in request.headers.raw if pair[0] not in _REQUEST_HEADERS)
api_headers: Final = tuple(pair for pair in Headers(scope=scope).raw if pair[0] in _REQUEST_HEADERS)
headers: Final = list(caller_headers + api_headers) # mutable-ok: ASGI middleware modifies headers
gateway_scope: Final[Scope] = {
**scope,
"client": request.client,
"scheme": request.url.scheme,
"headers": headers,
}
await app(gateway_scope, receive, send)
Comment thread
greptile-apps[bot] marked this conversation as resolved.
Resolved
Comment thread
cursor[bot] marked this conversation as resolved.
Resolved

async with httpx2.AsyncClient(transport=httpx2.ASGITransport(app=management_api)) as client:
admin_app: Final = create_http_app(Gateway(config, client))
route: Final = Mount("/admin", app=_CallerContext(admin_app, caller), name="admin_mcp")
async with admin_app.router.lifespan_context(admin_app):
app.router.routes.insert(0, route)
try:
yield
finally:
GracefulShutdownManager.start_shutdown()
try:
await GracefulShutdownManager.wait_for_drain()
finally:
app.router.routes.remove(route)
16 changes: 16 additions & 0 deletions litellm/proxy/middleware/admission_control_middleware.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import asyncio
import os
from collections.abc import Callable, Mapping
from contextvars import ContextVar
from dataclasses import dataclass
from functools import lru_cache
from typing import Annotated, Final, Protocol, TypeAlias, runtime_checkable
Expand Down Expand Up @@ -39,6 +40,11 @@ class AdmissionControlStats:
rejected_total: int


class _AdmissionLease:
def __init__(self) -> None:
self.active: bool = True


@runtime_checkable
class _Gauge(Protocol):
def inc(self, amount: float = 1) -> None: ...
Expand Down Expand Up @@ -140,12 +146,18 @@ def __init__(
self.app = app
self.get_settings = get_settings
self.state = state
self._lease: ContextVar[_AdmissionLease | None] = ContextVar("admission_lease", default=None)

async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
if scope["type"] != "http":
await self.app(scope, receive, send)
return

inherited_lease: Final = self._lease.get()
if inherited_lease is not None and inherited_lease.active:
await self.app(scope, receive, send)
return

settings: Final = self.get_settings()
if settings is None or _get_route_path(scope) in _EXEMPT_PATHS:
await self.app(scope, receive, send)
Expand Down Expand Up @@ -178,9 +190,13 @@ async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
state.record_dequeue()
state.record_admission()

lease: Final = _AdmissionLease()
token: Final = self._lease.set(lease)
try:
await self.app(scope, receive, send)
finally:
lease.active = False
self._lease.reset(token)
semaphore.release()
state.record_release()

Expand Down
Loading
Loading