feat(integrations): Add TokenJam as named callback - #27869
Closed
anilmurty wants to merge 1060 commits into
Closed
Conversation
… codex/skills-containers-tenant-guard # Conflicts: # litellm/proxy/auth/auth_utils.py
…ice-account-isolation fix(proxy): isolate managed resources for service-account API keys
…enant-guard chore(proxy): tighten resource ownership checks
secret_fields (containing raw HTTP headers including Authorization Bearer tokens) was being included in proxy_server_request['body'] because the body snapshot was a copy.copy(data) of the full request dict. This body gets serialized and persisted in the LiteLLM_SpendLogs table, exposing user credentials in the database. Root cause: data['secret_fields'] was set before the body snapshot at data['proxy_server_request']['body'] = copy.copy(data), so the full raw headers (including auth tokens) ended up in the snapshot. Fix (defense in depth): 1. Exclude 'secret_fields' when creating the body snapshot in litellm_pre_call_utils.py (primary fix) 2. Strip 'secret_fields' in _sanitize_request_body_for_spend_logs_payload as a secondary safeguard secret_fields remains available on the live data dict for legitimate downstream consumers (MCP, Responses API). Co-authored-by: Krrish Dholakia <krrish-berri-2@users.noreply.github.com>
…l_key_deactivation fix(scim): revoke virtual keys when SCIM deprovisions a user
…s-in-spend-logs-a532 fix(security): prevent secret_fields from leaking into spend logs
The Python 3.13 CCI smoke matrix surfaces a partially-initialized-module
ImportError when loading the managed files hook chain:
litellm.proxy.hooks/__init__ (mid-import)
-> enterprise.enterprise_hooks
-> litellm_enterprise.proxy.hooks.managed_files
-> litellm.llms.base_llm.managed_resources.isolation
-> litellm.proxy.management_endpoints.common_utils
-> litellm.proxy.utils (re-enters litellm.proxy.hooks)
The except ImportError block in hooks/__init__.py silently swallowed the
failure, leaving managed_files unregistered and POST /files returning
500 "Managed files hook not found".
Two-layer fix:
- Inline the 3-line _user_has_admin_view check in isolation.py instead
of importing it from litellm.proxy.management_endpoints.common_utils.
litellm.llms.* should not depend on litellm.proxy.* — removing this
layering violation breaks the cycle at its root.
- Define PROXY_HOOKS and get_proxy_hook before the conditional
enterprise import in litellm/proxy/hooks/__init__.py, so any future
re-entry resolves the public names instead of hitting an
ImportError on a partially-initialized module.
Also fold in two unrelated CCI repairs surfaced in the same staging run:
- tests/otel_tests/test_key_logging_callbacks.py: per-key
gcs_bucket_name / gcs_path_service_account are now stripped by
initialize_dynamic_callback_params, so the GCS client falls through
to the env-only branch. Update the assertion to match the new
"GCS_BUCKET_NAME is not set" message.
- .circleci/config.yml: tests/pass_through_tests now resolves
google-auth-library@10.x via the @google-cloud/vertexai 1.12.0 bump,
which uses dynamic ESM imports Jest 29 cannot load without
--experimental-vm-modules. Pass that flag in the Vertex JS test step.
Adds tests/test_litellm/proxy/hooks/test_proxy_hooks_init.py as a
regression guard: managed_files / managed_vector_stores must register,
and isolation.py must not transitively import litellm.proxy.utils.
- Move _user_has_admin_view to litellm.proxy._types as
user_api_key_has_admin_view (single source of truth). common_utils.py
and isolation.py both import from there now, removing the duplicated
role-check that could silently diverge if new admin roles are added.
- Add pytest.importorskip("litellm_enterprise") to the two regression
tests that assert managed_files / managed_vector_stores are registered;
those keys come from ENTERPRISE_PROXY_HOOKS so the tests would fail
unconditionally in a checkout without the enterprise extra installed.
Ruff F401 flagged the aliased import as unused within common_utils.py because the name is consumed only by external modules (~15 callers across guardrails, spend tracking, MCP, agents, management endpoints). Add `# noqa: F401 re-exported` so the alias survives lint while keeping a single source of truth in litellm.proxy._types.
…ze hook - Add image_generation/http_utils.azure_deployment_image_generation_json_body; call from azure.py (keeps AzureChatCompletion focused on chat). - Rename finalize_image_edit_multipart_data to finalize_image_edit_request_data with docstring covering multipart and JSON POST payloads (review feedback). Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
…heck_otel_spans /otel-spans now requires proxy admin (returns 401 'Only proxy admin can be used to generate, delete, update info for new keys/users/teams. Route=/otel-spans' for non-admin callers). Switch the GET call to use the master key sk-1234 while keeping the generated key for the chat-completion request that produces the spans.
…_reasoning_effort feat(proxy): add health_check_reasoning_effort for model health checks
The /otel-spans endpoint returns process-wide spans and tags
most_recent_parent by max start_time. After tightening that route to
proxy_admin (sk-1234), the GET /otel-spans request itself emits auth
spans that beat the chat-completion spans on start_time, so
most_recent_parent now points at the request's own auth trace
(['postgres', 'postgres']) and the >=5-span assertion fails.
Pick the chat-completion trace by content: it is the only trace whose
span list is a superset of {postgres, redis, raw_gen_ai_request,
batch_write_to_db}. Verified locally end-to-end against
otel_test_config.yaml + OTEL_EXPORTER=in_memory: 3/3 runs green.
…t-image-body fix(azure): omit model from deployment image gen and image edit bodies
The Azure o-series tests were excluded from the conftest's VCR auto-marker because of a respx/vcrpy transport-patching conflict, but the only respx reference in the file was an unused `MockRouter` import. Drop the dead import and remove the file from the conflict set so cassettes record on first run and replay thereafter, eliminating the 60-95s live Azure latency that was crashing xdist workers under --timeout=120 thread-mode timeouts.
/metrics now requires auth by default; tests/otel_tests/test_prometheus.py makes 4+ unauthenticated GETs against http://0.0.0.0:4000/metrics, so every prometheus test in CI now fails the metric assertion. Set require_auth_for_metrics_endpoint: false in otel_test_config.yaml to opt out for this test job, which scrapes /metrics directly. Verified locally: 8/8 prometheus tests green (one flaky retry on test_proxy_success_metrics that pre-dates this PR). Also drop the -x stop-on-first-failure flag from the otel test command so all failures in the job surface in a single CI run rather than hiding behind whichever one trips first.
…erman-35cf02 [Fix] CI: Enable VCR replay for test_azure_o_series
…-6e46e7 [Fix] Proxy: Break managed-resources import cycle on Python 3.13
[Infra] Build UI
The cimg/python:3.12-browsers base image already ships every Chromium system dependency Playwright needs (libnss3, libatk-bridge2.0-0, libcups2, etc. — the install log shows them all as "already the newest version"). Passing --with-deps to `npx playwright install` therefore runs an apt-get update + install for nothing, but pays the full cost of hitting Ubuntu mirrors. On a recent run those mirrors stalled hard: apt-get update alone took 6m53s at 81.5 kB/s with several archives returning connection refused. Drop --with-deps and persist ~/.cache/ms-playwright alongside node_modules so the Chromium binary is also reused across runs. Bump the cache key to v2 so the existing v1 entry (which only contained node_modules) is not loaded and skipped over the new browser path.
…itellm_/vigorous-albattani-2b7480
…ilds PRISMA_CLI_BINARY_TARGETS="debian-openssl-3.0.x" was hardcoded in docker/Dockerfile.non_root by BerriAI#17695. On a buildx linux/arm64 leg this forces prisma to download the amd64 schema-engine into an arm64 image, so 'prisma migrate deploy' fails at startup with 'Could not find schema-engine binary'. Removing the env lets prisma auto-detect per build platform: amd64 builds still resolve to debian-openssl-3.0.x (Wolfi falls back to debian, same binary as before), and arm64 builds now correctly fetch linux-arm64-openssl-3.0.x. The offline-cache pre-warm goal of BerriAI#17695 is preserved — only which binaries fill the cache changes. Fixes BerriAI#19458
…tani-2b7480 [Perf] CI: Skip Redundant Playwright Apt Install in E2E UI Job
…intock-62a296 [Fix] Docker: Remove Hardcoded Prisma Binary Target For Multi-Arch Builds
- Narrow /root/.cache COPY in Dockerfile to /root/.cache/prisma{,-python}
only — drops ~660MB of uv build cache including a setuptools wheel
that surfaced as CVE-2024-6345 / CVE-2025-47273 even though it was
never on the runtime sys.path.
- DiskCache: switch to dc.JSONDisk to neutralize the pickle code path
(CVE-2025-69872, no upstream fix). Values must be JSON-serializable;
cleanup get_cache to skip the now-dead json.loads(dict) branch by
guarding on isinstance(str).
- pyproject.toml: drop diskcache pin from [caching] extra (no fixed
version exists). Stub kept so `pip install litellm[caching]` doesn't
warn; users who want disk caching install diskcache themselves.
- Bump black 24.10.0 → 26.3.1 (CVE-2026-32274) + apply 296-file mechanical
reformat. Black is dev-only (not in the runtime image), but bumping
clears the manifest-scan finding.
- Refresh ui/litellm-dashboard/package-lock.json to pick up next 16.2.4
(was 16.1.7, GHSA-q4gf-8mx6-v5v3), uuid 14.0.0, postcss 8.5.13.
- Refresh litellm-js/spend-logs/package-lock.json to pick up
hono 4.12.16 (GHSA-458j-xx4x-4375).
- uv lock: gitpython 3.1.46 → 3.1.49 (clears two High GHSAs),
langchain-text-splitters 1.1.1 → 1.1.2.
- Add tests/test_litellm/caching/test_disk_cache.py covering JSONDisk
enforcement, dict/string round-trip, TTL, increment, delete/flush.
Net delta on combined trivy + grype scans: 17 findings → 4 (all
remaining 4 are Wolfi system python-3.13 CVEs marked WONTFIX upstream
in CPython 3.14; CVE-2026-3298 is Windows-unreachable on Linux).
Existing on-disk caches written by the previous pickle-format Disk
will silently miss after upgrade — diskcache is intended to be
ephemeral so impact is recreate-on-next-write.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ut diskcache
- Add black to liccheck.ini Authorized Packages (MIT-licensed).
- pytest.importorskip("diskcache") at top of test_disk_cache.py so
the test skips cleanly when diskcache isn't installed (it's no longer
pulled in by the dev group after the CVE-2025-69872 mitigation).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Iterate user/key/team/team_member/org/end_user/tag spend dicts in sorted order inside each Prisma transaction so concurrent pods acquire row locks in the same order, avoiding PostgreSQL deadlocks under load.
…sts (BerriAI#27813) OpenAI returns 'The model dall-e-3 does not exist' for the test account, breaking test_openai_img_gen_health_check and test_image_generation. Switch to gpt-image-1, matching the existing TestOpenAIGPTImage1 pattern.
…Path [Fix] Lazy feature loading under SERVER_ROOT_PATH returns 404
…riAI#27775) * fix(gemini): normalize response_schema on native generateContent The /v1beta/models/{model}:generateContent passthrough forwarded generationConfig.response_schema verbatim, so schemas containing $defs, $ref, anyOf-with-null, default, or title were rejected by Gemini even though /chat/completions already handles them. GoogleGenAIConfig.transform_generate_content_request now calls a new _normalize_response_schema helper that mirrors the chat/completions path: Gemini 2.0+ models get the schema promoted to responseJsonSchema via _build_json_schema (preserving $defs/$ref natively), older models keep responseSchema but the schema is flattened with _build_vertex_schema. VertexAIGoogleGenAIConfig (which overrides the transform entirely) calls the same helper before building the request. * fix(gemini): preserve caller-supplied responseJsonSchema when responseSchema co-present Previously, when both responseJsonSchema and responseSchema were present on Gemini 2.0+, _normalize_response_schema processed responseJsonSchema first (no-op normalization) then unconditionally promoted responseSchema to responseJsonSchema, clobbering the caller-supplied value. Now skip the promotion (and drop the redundant responseSchema) when the caller already supplied responseJsonSchema. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * chore: strip restating comments from response-schema normalize Drop the docstring on _normalize_response_schema and the two inline comments that just restated what the surrounding code/asserts already say. Function name + variable names carry the intent; PR description covers the why-it-exists context. * perf(gemini): drop redundant deepcopy on responseJsonSchema normalize _build_json_schema is a no-op (returns its argument unchanged), so the deepcopy + round-trip on the responseJsonSchema branch allocated a full schema copy on every request with no observable effect. Forward the caller's value as-is, and just move the popped responseSchema value when promoting on Gemini 2.0+. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * style: remove unneeded comment * fix(gemini): drop unsupported responseJsonSchema for older models * test(gemini): add parity test between native and chat schema normalization Per @Sameerlite review: lock the two Gemini schema-normalization paths together. If either GoogleGenAIConfig._normalize_response_schema (native generateContent) or VertexGeminiConfig.apply_response_schema_transformation (/chat/completions) drifts, the parity test fails — forcing both to be updated together. * fix(google_genai): preserve key naming convention in _normalize_response_schema When the input schema key is snake_case (response_schema), the promoted JSON schema key should also be snake_case (response_json_schema) instead of mixing in camelCase (responseJsonSchema). This matters for the Vertex AI google_genai path which converts all keys to snake_case before calling _normalize_response_schema. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>
…th when flag set (BerriAI#27716) * feat(proxy): skip disable_background_health_check models on GET /health when flag set Co-authored-by: Cursor <cursoragent@cursor.com> * fix comment * fix greptile comments * Fix health check fallback kwargs * Format health endpoint * Harden direct health check kwargs compatibility for monkeypatched perform_health_check Replace substring-based TypeError detection with unexpected-keyword checks and a short retry chain (full kwargs, instrumentation only, filter only, minimal) so partial stubs work regardless of which optional kwarg fails first. Add proxy unit tests for legacy three-arg stubs and single-kwarg variants. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * fix black --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
…ocks (BerriAI#27850) * fix(bedrock-converse): drop blank-text fallback for empty thinking blocks Claude Code with extended thinking replays prior assistant turns that include an empty thinking block (`thinking=""`, `signature=""`) alongside tool_use blocks. The unsigned-reasoning fallback in `add_thinking_blocks_to_assistant_content` was emitting `BedrockContentBlock(text="")`, which Bedrock Converse rejects with: "The text field in the ContentBlock object at messages.X.content.0 is blank." Guard the fallback with a strip() check, matching the existing empty-text guards elsewhere in `_bedrock_converse_messages_pt`. * style: remove unneeded comments
…iAI#27847) * fix(mcp): surface upstream 401 for token-forwarding MCP servers For MCP servers configured with extra_headers: [Authorization], the gateway forwards the client token directly to the upstream. When that token is rejected (expired or invalid) the upstream returns 401, but the MCP SDK starts the SSE stream with 200 OK before calling handlers, so the 401 can't be returned mid-stream. Fix: add a pre-flight httpx probe in handle_streamable_http_mcp — before the SDK opens the session — so the gateway can still return HTTP 401 with WWW-Authenticate: Bearer authorization_uri=<gateway-discovery-url> when the upstream rejects the token. The probe fails-open (returns 200) on network errors so a transient hiccup does not block valid requests. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): parallelize pre-flight auth probes and use HEAD to avoid side effects - Extract forwarded_auth outside the pass-through server loop (was called N times for the same scope value) - Gather all upstream auth probes concurrently with asyncio.gather instead of sequentially; eliminates N×5 s worst-case latency - Switch probe from POST+initialize JSON-RPC body to HEAD request; HEAD carries the Authorization header so the upstream rejects invalid tokens with 401 but never allocates a session or writes an audit entry Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): use get_async_httpx_client in _probe_upstream_auth Replaces bare httpx.AsyncClient with the project-standard get_async_httpx_client(httpxSpecialProvider.MCP) to satisfy the ensure_async_clients_test code coverage check and avoid the +500 ms per-request overhead of creating a new client on every probe call. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(mcp): extract pre-flight probe into _check_passthrough_upstream_auth Moves the parallel upstream auth probe logic out of handle_streamable_http_mcp into a dedicated helper to satisfy Ruff PLR0915 (Too many statements > 50). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): gate pre-flight probes on authorized server set to prevent bypass _check_passthrough_upstream_auth was resolving user-supplied server names directly before authorization ran, letting any permitted LiteLLM key trigger an upstream HEAD probe to a server it was not allowed to use. Changes: - Call _get_allowed_mcp_servers inside the helper so only servers the caller's key is authorized for are probed. - Move the call site to after toolset scoping so the auth context is fully resolved before the probe list is built. - Thread user_api_key_auth into the helper signature (replaces the raw mcp_servers name list). Co-authored-by: Cursor <cursoragent@cursor.com> * Add async HTTP HEAD support Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(mcp): use Scope type annotation in _get_forwarded_auth_from_scope Co-authored-by: Cursor <cursoragent@cursor.com> * Fix MCP upstream auth probe method Co-authored-by: Yassin Kortam <yassin@berri.ai> * Remove unused AsyncHTTPHandler head method Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(mcp): exclude has_client_credentials servers from pre-flight auth probe _prepare_mcp_server_headers skips caller Authorization when the server uses OAuth client-credentials (M2M), but the pre-flight probe was still selecting those servers and forwarding the caller's raw token in the HEAD request. Exclude servers with has_client_credentials from the probe list to match the actual downstream header-preparation logic. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): propagate upstream 403 as 403, not 401 with WWW-Authenticate Per RFC 9110, 401 means "go get new credentials." Mapping an upstream 403 to a gateway 401 causes OAuth clients to restart the authorization flow, obtain a fresh token with identical scopes, hit 403 again, and loop indefinitely. 401 from upstream → gateway 401 + WWW-Authenticate (re-authorize) 403 from upstream → gateway 403 (no WWW-Authenticate hint) Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): skip auth probe when Authorization may be the LiteLLM proxy key The pre-flight upstream probe must not forward the caller's Authorization header when it could itself be the LiteLLM proxy API key. Restrict the probe to requests that supply x-litellm-api-key explicitly — only then is the Authorization header unambiguously the upstream OAuth token the caller wants forwarded. * Fix MCP ASGI HTTPException propagation Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(mcp): use public AsyncHTTPHandler.post() in auth probe Use AsyncHTTPHandler.post() and catch httpx.HTTPStatusError explicitly so the 401/403 we want to surface is not silently swallowed by the broad fail-open except Exception block. Avoids reaching into the handler's private client attribute, which would silently regress to fail-open if AsyncHTTPHandler is ever refactored. * Fix MCP auth probe tests Co-authored-by: Yassin Kortam <yassin@berri.ai> * test(mcp): add coverage for httpx.HTTPStatusError path in auth probe AsyncHTTPHandler.post() calls raise_for_status() internally, so a real upstream 401/403 lands as httpx.HTTPStatusError. Add a test that exercises that specific exception path so a regression that swallows the error in the broad fail-open except Exception would be caught. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: claude-bot <claude-bot@anthropic.com>
…timodal pricing (BerriAI#27848) * fix(cost): align vertex_ai/gemini-embedding-2-preview with Vertex multimodal pricing Co-authored-by: Cursor <cursoragent@cursor.com> * fix(cost): align vertex_ai/gemini-embedding-2 GA source URL with preview Per Greptile review on BerriAI#27848: GA entry referenced ai.google.dev while the preview entry was updated to the canonical Vertex AI pricing page. Both share identical pricing values; sync the source URL for consistency. https://claude.ai/code/session_01W8jRwstnmduadGw8Z8egxe --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude <noreply@anthropic.com>
…erriAI#27834) * feat(mcp): add delegate_auth_to_upstream flag for PKCE passthrough Adds an opt-in per-server flag that lets clients (e.g. VS Code) complete PKCE directly with an upstream OAuth2 MCP server, instead of LiteLLM double-gating with its own API-key/SSO check. Only honored when auth_type=oauth2 and the operator explicitly sets the flag; mixed-target or non-oauth2 requests fail closed. - Adds the field to Pydantic models, Prisma schema, and a migration - New MCPRequestHandler._target_servers_delegate_auth_to_upstream gate that runs only when no x-litellm-api-key is present, so authenticated users still get user_id resolution + stored-credential lookup - Anonymous callers now see delegate servers in get_allowed_mcp_servers (scoped to delegate servers only; the upstream still enforces auth) - mcp_management_endpoints: allow anonymous /authorize and /token for delegate servers so VS Code can complete PKCE without a LiteLLM session - UI toggle (shown only for oauth2) + payload/view wiring - Tests covering: oauth2 on/off, non-oauth2 with flag, mixed targets, no resolvable target, explicit key precedence, and 401 emission Co-authored-by: Cursor <cursoragent@cursor.com> * Enforce oauth2 for delegated MCP auth bypass Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(mcp): close secondary Authorization bypass for delegate servers The delegate-auth bypass gated only on the primary `x-litellm-api-key` header, so a LiteLLM key sent via `Authorization: Bearer sk-...` (the secondary header) was silently dropped — skipping spend tracking and rate limiting. Gate on the resolved litellm_api_key (which considers both headers) so the bypass fires only when neither is present. Also update the existing "Authorization header present" test to reflect that an upstream OAuth token now flows through the existing oauth2 fallback (LiteLLM auth attempt → fail → anonymous), not via the delegate branch. Co-authored-by: Cursor <cursoragent@cursor.com> * Avoid duplicate MCP OAuth credential lookup Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(mcp): block delegate bypass for M2M and internal-only servers Two security issues flagged in code review: 1. High – client_credentials (M2M) servers must not be delegatable: LiteLLM auto-fetches the upstream token using stored credentials, so allowing anonymous bypass would let any external caller invoke tools authenticated as LiteLLM's service account. Fix: check `server.has_client_credentials` in `_target_servers_delegate_auth_to_upstream`, the anonymous allow-list in `get_allowed_mcp_servers`, and `_mcp_oauth_user_api_key_auth`. 2. Medium – internal-only servers exposed to public internet: The anonymous delegate allow-list was not filtering by `available_on_public_internet`, so external callers with an upstream OAuth token could invoke tools on servers marked internal-only. Fix: add `available_on_public_internet` guard to the anonymous delegate server list in `get_allowed_mcp_servers`. Tests added for both cases. Co-authored-by: Cursor <cursoragent@cursor.com> * Require public MCP delegate auth servers Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(mcp): align delegate auth path parsing with downstream routing `_extract_target_server_names_from_path` used a naive segments-based split while `server.py::_get_mcp_servers_in_path` uses a regex that allows server names with one embedded slash and comma-separated lists. With the old parser, a request to `/mcp/<delegated>/<garbage>` was parsed as targeting `<delegated>` by the auth gate (bypassing LiteLLM auth) while the routing layer parsed it as `<delegated>/<garbage>` — when that name did not resolve, the request fell back to the anonymous allow-list, which can include `allow_all_keys` servers that normally require a LiteLLM key. Replace the parser with the same regex logic as `_get_mcp_servers_in_path` so auth gating sees the exact target name(s) downstream routing sees. Add regression tests covering parser parity and the specific extra-path-segment bypass attempt. https://claude.ai/code/session_01SjyPmwfmrq8fveFgw9iHW9 * fix(mcp): close header/path TOCTOU in MCP delegate auth gate `_target_servers_delegate_auth_to_upstream` and `_target_servers_use_oauth2` trusted the `x-mcp-servers` header when present, but `server.py::extract_mcp_auth_context` overrides that header with the path-derived list for `/mcp/...` routes. An attacker could set `x-mcp-servers: <delegated>` while pointing the URL path at a non-delegate server, flipping the auth gate without changing the target downstream routing actually uses. Extract a shared `_resolve_target_server_names` helper that mirrors the downstream override (path-derived names for `/mcp/...` routes, header value otherwise). Add regression tests covering the TOCTOU attempt and the helper's path-vs-header precedence. https://claude.ai/code/session_01SjyPmwfmrq8fveFgw9iHW9 * Fix delegated MCP OAuth test mock Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(mcp): drop unreachable /{server}/mcp branch in auth path parser `_extract_target_server_names_from_path` also matched the ``/{server_name}/mcp`` form, but the downstream parser ``_get_mcp_servers_in_path`` only handles ``/mcp/...`` — and ``dynamic_mcp_route`` in ``proxy_server`` rewrites ``/{name}/mcp`` to ``/mcp/{name}`` on the scope before the MCP handler runs. Parsing the un-rewritten form on the auth side was therefore unreachable in production, and contradicted the docstring's claim of mirroring the downstream parser — exactly the kind of mismatch that risks a future header/path TOCTOU if any new entry point skips the rewrite. Drop the branch; the canonical ``/mcp/...`` path matches both parsers. Update the regression test to assert the new behavior. https://claude.ai/code/session_01SjyPmwfmrq8fveFgw9iHW9 * Fix MCP path auth target resolution Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(mcp): require auth for refresh_token grants on delegate-auth servers `_mcp_oauth_user_api_key_auth` gates the unauthenticated PKCE flow for ``delegate_auth_to_upstream`` servers, but the bypass applied to BOTH ``/authorize`` and ``/token`` regardless of grant type. ``mcp_token`` accepts ``grant_type=refresh_token`` as well as ``authorization_code``, and ``exchange_token_with_server`` attaches the server's stored ``client_secret`` to whatever is forwarded upstream. An unauthenticated caller holding a refresh token issued to that OAuth client could mint fresh upstream access tokens through LiteLLM. Limit the anonymous bypass on ``/token`` to ``grant_type=authorization_code`` (the only grant PKCE actually protects via ``code_verifier``); fall through to normal LiteLLM auth for ``refresh_token`` and any other grant. ``/authorize`` continues to allow anonymous PKCE redirects. https://claude.ai/code/session_01SjyPmwfmrq8fveFgw9iHW9 * fix(ui): clear delegate_auth_to_upstream when switching off oauth2 The ``delegate_auth_to_upstream`` form field is rendered inside an ``isOAuth2 && (...)`` conditional, so the Form.Item unmounts when the user changes ``auth_type`` away from ``oauth2``. The follow-up ``form.setFieldValue("delegate_auth_to_upstream", false)`` runs after the field has already deregistered, so ``onFinish`` receives ``undefined`` and the fallback ``?? mcpServer.delegate_auth_to_upstream`` preserved the old ``true``. The flag then persisted in the database for a non-oauth2 server and silently re-activated if ``auth_type`` was later switched back to ``oauth2``. In the edit payload, force the flag to ``false`` whenever ``auth_type !== oauth2``; only trust the form value (and the existing DB fallback) when the server is actually oauth2. Backend defense-in-depth already ignores the flag for non-oauth2 servers, but the DB state should stay clean too. https://claude.ai/code/session_01SjyPmwfmrq8fveFgw9iHW9 * Fix MCP delegate auth reset on edit Co-authored-by: Yassin Kortam <yassin@berri.ai> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: Claude <claude@anthropic.com>
…etion transformation (BerriAI#27727) * fix(responses): preserve cache_control in Responses API -> Chat Completion transformation cache_control injected by AnthropicCacheControlHook was silently dropped when _transform_responses_api_content_to_chat_completion_content rebuilt content blocks with only {type, text}. Now copies cache_control through so Anthropic prompt caching works correctly when using client.responses.create with cache_control_injection_points. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(responses): preserve cache_control for input_image and input_file blocks Extends the cache_control fix to image and file content blocks, which were also silently dropping cache_control during the Responses API -> Chat Completion transformation. Adds tests for all three content block types. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Babysitter <claude@anthropic.com>
Author
Contributor
Greptile SummaryThis PR adds TokenJam as a new named callback in LiteLLM, enabling users to enable TokenJam observability with a single
Confidence Score: 5/5Safe to merge — the integration is isolated, additive, and follows established patterns in the codebase The change is purely additive: a new logger class, two small wiring additions in logging.py, and a type literal update. No existing behaviour is modified. The async hooks correctly offload blocking I/O to a thread pool. No files require special attention
|
| Filename | Overview |
|---|---|
| litellm/integrations/tokenjam/tokenjam.py | New TokenJamLogger implementation — correct pattern, async hooks use get_event_loop() (deprecated since 3.10) instead of get_running_loop() |
| litellm/init.py | Adds "tokenjam" to _custom_logger_compatible_callbacks_literal — minimal, correct change |
| litellm/litellm_core_utils/litellm_logging.py | Wires tokenjam into both _init_custom_logger_compatible_class and get_custom_logger_compatible_class following the same pattern as other loggers |
| tests/test_litellm/integrations/test_tokenjam.py | 7 fully-mocked unit tests covering env var injection, missing SDK graceful degradation, success/failure delegation, error swallowing, and async delegation — no real network calls |
| litellm/integrations/tokenjam/init.py | Empty package init — correct |
Reviews (3): Last reviewed commit: "fix(tokenjam): offload sync hooks to exe..." | Re-trigger Greptile
Contributor
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| or getattr(user_api_key_auth, "api_key", None) | ||
| ) | ||
| ) | ||
| if is_anonymous: |
| deployment = llm_router.get_deployment(model_id=model_id) | ||
| except Exception as e: | ||
| verbose_proxy_logger.error( | ||
| f"Error getting deployment for model_id {model_id}: {e}" |
| from typing import Any, Optional | ||
|
|
||
| from litellm._logging import verbose_logger | ||
| from litellm.integrations.custom_logger import CustomLogger |
| if isinstance(callback, SMTPEmailLogger): | ||
| return callback | ||
| elif logging_integration == "tokenjam": | ||
| from litellm.integrations.tokenjam.tokenjam import TokenJamLogger |
Comment on lines
+15
to
+18
| from litellm.llms.vertex_ai.common_utils import ( | ||
| _build_vertex_schema, | ||
| supports_response_json_schema, | ||
| ) |
Comment on lines
+317
to
+320
| from litellm.proxy.health_check import ( | ||
| health_check_filter_kwargs_from_general_settings, | ||
| perform_health_check, | ||
| ) |
Comment on lines
+45
to
+49
| from litellm.exceptions import ( | ||
| BadRequestError, | ||
| RateLimitError, | ||
| ServiceUnavailableError, | ||
| ) |
| ) | ||
| from litellm.integrations.custom_logger import CustomLogger, Span | ||
| from litellm.responses.utils import ResponsesAPIRequestUtils | ||
| from litellm.router_utils.cooldown_cache import CooldownCacheValue |
Contributor
Greptile SummaryThis PR adds TokenJam as a new named callback in LiteLLM, enabling users to enable observability logging with
Confidence Score: 3/5Safe to merge once the async hooks are fixed; the integration correctly silences errors and guards against a missing SDK, but the async methods currently block the event loop on every async LiteLLM call. The async hooks call the sync counterparts directly, so every async LiteLLM call that logs to TokenJam will block the asyncio event loop for the duration of the HTTP round-trip to the TokenJam server. In production async deployments this stalls all concurrent in-flight requests while the logging call completes. The rest of the wiring — import guard, env-var defaults, singleton caching, error swallowing — is implemented correctly and matches the established patterns in the codebase. litellm/integrations/tokenjam/tokenjam.py — the async logging methods need to offload the blocking emit call to a thread executor.
|
| Filename | Overview |
|---|---|
| litellm/integrations/tokenjam/tokenjam.py | New TokenJamLogger adapter; async hooks delegate to blocking sync I/O which will stall the asyncio event loop on every async LiteLLM call. |
| litellm/litellm_core_utils/litellm_logging.py | Wires "tokenjam" into both _init_custom_logger_compatible_class and get_custom_logger_compatible_class following established patterns; looks correct. |
| litellm/init.py | Adds "tokenjam" to the _custom_logger_compatible_callbacks_literal; minimal and correct. |
| tests/test_litellm/integrations/test_tokenjam.py | Seven mocked unit tests covering env var flow, import failure graceful handling, success/failure delegation, error swallowing, and async delegation; all mocked and no real network calls. |
| litellm/integrations/tokenjam/init.py | Empty package init; no issues. |
Reviews (2): Last reviewed commit: "feat(integrations): Add TokenJam as name..." | Re-trigger Greptile
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
| # no key supplied). | ||
| pass | ||
| else: | ||
| return UserAPIKeyAuth() |
Contributor
There was a problem hiding this comment.
Medium: Delegated OAuth token exchange does not enforce PKCE
This returns anonymous auth for any authorization_code token exchange, but exchange_token_with_server attaches the configured client secret and only sends code_verifier if the caller provided one. An attacker who obtains an authorization code for LiteLLM's OAuth client can redeem it through this endpoint without a LiteLLM session when the upstream issuer does not enforce PKCE; require a code_verifier here, and require a matching code_challenge on the anonymous /authorize path, before taking the bypass.
Contributor
TokenJam named callback addedThis PR registers a new TokenJam callback and adds a logger adapter that forwards success/failure callback payloads to the TokenJam SDK using endpoint and ingest-secret environment configuration. I reviewed the callback registration path and dynamic callback handling in the PR diff and did not identify a concrete attacker-controlled security issue introduced by these changes. Status: 1 open |
Author
anilmurty
changed the base branch from
litellm_internal_staging
to
litellm_oss_staging
May 13, 2026 21:39
Author
|
Companion docs PR (required for documentation_test_env_keys to pass): BerriAI/litellm-docs#121 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relevant issues
None — new integration.
Pre-Submission checklist
tests/test_litellm/directory — 7 mocked unit tests in
tests/test_litellm/integrations/test_tokenjam.pymake test-unit@greptileaiand received a Confidence Score of at least 4/5 beforerequesting a maintainer review
Type
🆕 New Feature
Changes
This PR adds TokenJam as a named callback in LiteLLM. After this PR, users can enable TokenJam
logging with one line:
What TokenJam is
TokenJam is an open-source, local-first, OTel-native observability and token-economics platform for autonomous AI agents and
coding agents. It captures token usage, cost, and call patterns for any LiteLLM call and exposes the data through a CLI, a
local web UI, and an MCP server.
Files changed
_init_custom_logger_compatible_class() and get_custom_logger_compatible_class()
Implementation notes
installed, a warning is logged via verbose_logger.warning() and events are silently dropped.
by Langfuse, AgentOps, MLflow, etc.
and sidebars.js is no longer present. Happy to add docs in whatever location/format the maintainers prefer.
Testing
tests/test_litellm/integrations/test_tokenjam.py (7 tests, all mocked):
Verified end-to-end against tokenjam==0.2.2 on PyPI.
Screenshots / Proof of Fix
$ python -m pytest tests/test_litellm/integrations/test_tokenjam.py -v
....... [100%]
7 passed in 0.12s