Repository navigation
feat(anthropic): workload identity federation and pluggable identity sources - #44448
Conversation
…sources Backend half of #38818 (internal copy of the fork PR #38013), rebuilt as one commit on top of litellm_internal_staging without the dashboard changes. Deployments on anthropic/ without a static api_key can exchange an OIDC workload assertion for a short-lived sk-ant-oat01 token through a shared RFC 7523 JWT-bearer engine. The assertion comes from a mounted token file, an env token, a LiteLLM-signed issuer, or Keycloak, chosen per deployment, per named credential, or through ANTHROPIC_IDENTITY_SOURCE. The federation fields are server-owned: refused inline in request bodies and on POST /model/new, proxy-admin only on credentials, and the token exchange is pinned to api.anthropic.com unless LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS adds a host. GET /credentials/{name}/jwks exports the public key set of a LiteLLM-signed credential for the Claude Console. The OpenAI federation trio from #39613 rides along on the backend side with the same server-owned handling. Fixes #28607 Resolves LIT-6107 Co-authored-by: derhornspieler <15236687+derhornspieler@users.noreply.github.com>
…s and accept host:port allowlist entries The files handler enabled workload identity on batch-result downloads but never received the deployment's litellm_params, so a deployment authenticating through a named credential could only mint from process-wide env vars. It now threads litellm_params through to the auth header the way the batch retrieve path already does. LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS entries written as host:port were read by urlsplit as a scheme, so the allowlist kept the raw entry while the exchange compared bare hostnames and refused the gateway. Entries are now parsed as network locations whether or not they carry a scheme.
…gs funnel imports without a cycle
…iod so the router suffix reads cleanly
…thropic_workspace_id keeps working The Bedrock Claude Platform route already reads anthropic_workspace_id from optional_params, so banning that spelling as a server-owned federation parameter broke a pre-existing client capability. The federation field is now anthropic_federation_workspace_id (env ANTHROPIC_FEDERATION_WORKSPACE_ID), which restores the base branch's behavior for Bedrock callers, drops the Bedrock-specific hint from the refusal message, and deletes the unconditional ban constant that no longer had a reader
…assertion Anthropic accepts each identity assertion exactly once, so two uvicorn workers reading the same token file both minting from it means the second exchange is denied with jti_reused. Minted tokens now land in a per-user 0700 cache directory guarded by a file lock, so workers on the same host reuse one exchange until the token expires or the assertion rotates. A 401 is only retried when the re-read assertion actually differs, and the denial hint explains jti_reused. LITELLM_TOKEN_EXCHANGE_CACHE_DIR moves the cache and an empty value disables it
…itellm_anthropic_wif_backend # Conflicts: # tests/test_litellm/test_lazy_imports.py
…itellm_anthropic_wif_backend # Conflicts: # tests/test_litellm/llms/anthropic/experimental_pass_through/adapters/test_anthropic_experimental_pass_through_adapters_transformation.py
An empty or whitespace-only ANTHROPIC_API_KEY counted as set, so a federated deployment sent an empty x-api-key on every call instead of minting a token. Blank values now read as unset, and a real static key on a federated deployment logs once that it outranks federation and nothing is being federated. The exchange-host allowlist matched hostnames only, so a second process on another port of an allowed host was trusted with the workload's identity token. An entry that names a port now trusts that port alone, while a bare host still trusts every port. The shared token store exists so the workers reading one projected token file do not each spend its single-use jti. A source that mints its own assertion per exchange shares nothing with another worker, so it no longer writes a live token to disk for a lookup that can never hit.
The 401 denial hint now also says federation ignores ANTHROPIC_WORKSPACE_ID, which the Bedrock Claude platform provider already reads.
A buffered write only reaches the disk when the handle closes, so a full disk surfaces at close and left the staging file behind holding a usable token.
The admin gate read the stored deployment, so a team admin lost edit, delete and Test Connection on any deployment carrying federation params. It now returns early unless the submitted fields touch the federation surface, and a Test Connection probe that points the deployment at its own api_base is still refused, with the 403 no longer wrapped into a 500 The rest of the same review pass: POST /model/new refuses only a blocking value of `blocked`, so a client that always sends `blocked: false` is not turned away; a request body can no longer pick which federated identity to mint as by naming a stored credential; an advisory refresh the executor refuses disarms the entry instead of wedging the identity until the follower timeout; the static-key shadow warning resolves its env fallback inside the cache instead of once per request; credential writes drop nulls before storing them; the token exchange validates the endpoint URL before reading an assertion and keeps refusing redirects across a client heal; /health hides every server-owned federation field from non-admins; and the async create_file and create_batch paths say which setting is missing when the provider resolves no URL
reject_federated_credential_reference runs from is_request_body_safe, which
pre_db_read_auth_checks calls on every route, so it also fired on POST
/model/new, /model/update, /model/{id}/update and /health/test_connection. A
proxy admin could no longer attach a federated credential to a deployment over
the API or the Admin UI, leaving a static config.yaml entry as the only way to
configure the feature the rejection told the caller to go configure, and
_reject_non_admin_wif_write never got to make the call it exists to make.
is_request_body_safe now takes the route and skips only the credential-reference
check on the routes that reach can_user_make_model_call. Federation fields typed
inline into a body stay refused everywhere, and a call naming a federated
credential still cannot pick the identity it mints as.
… sets The health check module hand-copied the five workload identity fields whose value is a credential, so a shared proxy surface named provider-specific parameters and a newly added secret-bearing field would have gone on being displayed until someone remembered both places WIF_SECRET_BEARING_KEYS now sits beside the key sets it splits out of, types/utils derives secret_bearing_wif_litellm_params from it, and the health layer splats that tuple the same way it already splats the admin-only one
completion() forwards the Anthropic and OpenAI federation kwargs again now that main inlines its forward tuple, and the forward tests drive completion() instead of checking the funnel's key set. The health tests follow main's display allowlist, the Bedrock workspace test opts into client-side credentials so the unconditional federation ban is what it exercises, stacked comprehensions and suppressions the newer checker reports as unused are cleared, and the child interpreter tests run isolated.
Resolve the test_anthropic_chat_handler.py conflict to main's thinking-block assertions and bring the branch back under the lint and type gates: split the stacked comprehensions LIT014 now flags, drop the suppression comments LIT013 reports as unused, run child interpreters through the isolated helper, and validate untyped litellm_params dicts with model_validate at the construction sites in the files this branch touches instead of unpacking them, since every new federation field on CredentialLiteLLMParams otherwise adds one unknown argument diagnostic per site
…gistry main's registry test (#43298) now fails the build for any credential-named deployment param without a classification. The five federation fields that carry a token, a token file path, or a signing or client secret reference are Unplanted, matching WIF_SECRET_BEARING_KEYS; the four remaining Keycloak settings name a URL, a client id, an auth method, or a scope and are NotSecret
…ves and chart their metrics Register the 18 Anthropic and 3 OpenAI federation params as frozen ConnectionSettings leaves so the owned-kwarg registry, the kwargs funnel and the request-body ban list read one declaration. Pass the deployment api_base through to the count-tokens handler instead of a pre-suffixed URL, which doubled the /count_tokens path on main's prompt-cache predictor. Add the five litellm_anthropic_wif_* families to the all-metrics Grafana dashboard.
…_backend # Conflicts: # litellm/llms/openai/workload_identity.py
|
|
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
The credential PATCH handler checked server-owned workload identity federation fields only on the values the caller sent, while a body that named a deployment through model_id had its credential values resolved after that check. A non-admin could therefore copy a federated deployment's WIF fields onto an ordinary credential. Resolve the incoming values first and run the non-admin gate on them, matching the POST path
GitHub Actions
Error
GitHub Actions
Error
|
bugbot run |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
…ared auth header Count-tokens walked its own credential ladder: a static key, else skip minting when ANTHROPIC_AUTH_TOKEN is set, else mint a federated token. With only the auth token set it forwarded nothing and the proxy silently fell back to its local tokenizer while chat on the same deployment authenticated with that token. The handler now takes the auth header that AnthropicModelInfo.aget_auth_header resolves, the same ladder chat, files, batches and skills use, and merges the oauth beta a minted or consumer token carries with the token-counting beta
| *AWS_CREDENTIAL_KWARGS_KEYS, | ||
| # A federated deployment holds no api_key, so without these the fetch that reads a | ||
| # finished batch's output has nothing to authenticate with and its cost is never billed. | ||
| *sorted(ANTHROPIC_WIF_KWARGS_KEYS), |
There was a problem hiding this comment.
llms/ and couples shared code to these providers’ authentication schemes. The repository requirement must be satisfied before merging.
Rule Used: What: Avoid writing provider-specific code outside of the llms/ directory. Why: This practice ensures better maintainability and reduces complexity over time. Good: ```python # Handle provider-specific logic within llms/vertex_ai/transformation.py ... (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
There was a problem hiding this comment.
This tuple already enumerates AWS credential keys, and litellm_params types already name every provider's fields. The Anthropic entries follow both precedents.
There was a problem hiding this comment.
That’s a fair distinction. This extractor is already an explicit credential-forwarding boundary and enumerates provider-specific authentication keys (including AWS_CREDENTIAL_KWARGS_KEYS); adding the centralized ANTHROPIC_WIF_KWARGS_KEYS set is consistent with that pattern. Since the federation keys are declared in the shared workload-identity/types layer rather than handwritten in this batch module, this does not introduce a separate Anthropic implementation here. I’m withdrawing the provider-logic finding for this change.
| **auth_header, | ||
| "anthropic-beta": merge_anthropic_beta_headers( | ||
| auth_header.get("anthropic-beta"), ANTHROPIC_TOKEN_COUNTING_BETA_VERSION | ||
| ), |
There was a problem hiding this comment.
Count-tokens requests using a consumer sk-ant-oat token previously sent anthropic-dangerous-direct-browser-access. The new shared auth-header path sends the bearer token and beta header but omits that header. This changes existing behavior without a user-controlled flag, contrary to the repository directive to avoid backwards-incompatible changes without one. The repository requirement must be satisfied before merging.
Rule Used: What: avoid backwards-incompatible changes without user-controlled flags Why: This breaks current behaviour for users using existing functionality Example of BAD: this PR (#22164) introduced run_post_custom... (source)
There was a problem hiding this comment.
That header only acknowledges browser CORS. The base branch's shared auth helper already sends sk-ant-oat bearers without it for files, batches and skills.
There was a problem hiding this comment.
You're right. I conflated the OAuth beta required for sk-ant-oat with anthropic-dangerous-direct-browser-access, which only opts a client into Anthropic's browser CORS behavior.
The count-tokens endpoint is server-to-server, and the shared auth helper intentionally uses the same bearer-token behavior already used by files, batches, and skills without that CORS header. Since the relevant behavior—the bearer authorization and OAuth beta—is preserved, I’m withdrawing this finding; no change is needed.
| # LiteLLM All Prometheus Metrics dashboard | ||
|
|
||
| Every `litellm_*` metric family the proxy can expose on `/metrics` (136 families across 97 panels), grouped into rows: proxy traffic, latency, spend and tokens, cache, LLM API deployments, key and team rate limits, budgets, guardrails, MCP, managed files and batches, users and teams, the Redis circuit breaker, the spend log cleanup job, and the `prometheus_system` service callback metrics (per-service latency, request and failure rates, spend update queue sizes). Panel titles are the metric names so you can grep the JSON for the metric you care about | ||
| Every `litellm_*` metric family the proxy can expose on `/metrics` (141 families across 97 panels), grouped into rows: proxy traffic, latency, spend and tokens, cache, LLM API deployments, key and team rate limits, budgets, guardrails, MCP, managed files and batches, users and teams, the Redis circuit breaker, the spend log cleanup job, and the `prometheus_system` service callback metrics (per-service latency, request and failure rates, spend update queue sizes). Panel titles are the metric names so you can grep the JSON for the metric you care about |
There was a problem hiding this comment.
The changed README documents LiteLLM’s dashboard and exposed metric families in this repository. The repository directive requires product documentation to live in litellm-docs, and this cookbook README has no applicable exception. That requirement must be satisfied before merging.
Rule Used: Prevent documentation about the LiteLLM product from being added - needs to be in litellm-docs repo instead. This does NOT include any documentation about how to contribute to the LiteLLM repo, README.md, or code comments (source)
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
There was a problem hiding this comment.
This README predates the PR and catalogs the cookbook dashboard's own panels. The diff only bumps its metric family count from 136 to 141.
There was a problem hiding this comment.
You're right — this README is an existing cookbook artifact documenting the dashboard JSON, and this change only updates its metric-family count to match the newly added metrics. It does not add new LiteLLM product documentation or bypass the litellm-docs repository. I’m withdrawing this finding.
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit fce9d8f. Configure here.
…models Main added the same discovery hook as BaseLLMModelInfo.discover_models (#44448), so the PR's get_models_for_deployment hook and the HTTPHandler header widening are dropped for main's versions and the Bedrock override now reads the deployment's region and credentials from the mapping the proxy passes. The inference profile pager is a bounded for loop (20 pages of 1000) instead of recursion, which the recursion detector flagged.
This branch was successfully deployed
3 participants
Backend half of #38818 (itself the internal copy of the fork PR #38013 by @derhornspieler), targeting
main. Everything underlitellm/,tests/andterraform/from that PR is here; the dashboard (ui/), the credential form variants and thePOST /provider/models/discoverendpoint are left out and follow in a separate PR, so this PR ships the config.yaml, env var andPOST /credentialssetup paths onlyTLDR
Supersedes #39935 (identical commits, same tip fce9d8f)
Problem this solves:
How it solves it:
GET /credentials/{name}/jwksexports the public key set of a LiteLLM-signed credential for the Console/healthnever displays the federation token, the secret references, or the token file pathslitellm_anthropic_wif_*andlitellm_anthropic_wif_cache_*Prometheus families and charted in the all-metrics Grafana dashboardPATCH /credentials/{name}runs the same proxy-admin gate over federation fields it copies from a deployment named throughmodel_id, matchingPOST /credentialsUser Flow
Before: a platform team with a no-static-secrets policy cannot point LiteLLM at an Anthropic federation rule
model: anthropic/claude-haiku-4-5in config.yaml with noapi_key, and set the federation ids on the proxy's environment{"model": "claude-haiku-4-5-wif", "messages": [...]}Missing Anthropic API Key - A call is being made to anthropic but no key is set either in the environment variables or via paramsAfter: the same team configures the provider once in config.yaml and every deployment behind it authenticates without a static key
fdrl_..., an org id, and ansvac_...credential_listentry withcustom_llm_provider: anthropicand the identity source's fields, secrets asos.environ/...references, plus the three Console ids, and point deployments at it throughlitellm_credential_name(or setANTHROPIC_IDENTITY_SOURCEand theANTHROPIC_*ids in the environment with no credential at all)ANTHROPIC_API_KEYorANTHROPIC_AUTH_TOKEN, the same POST answers 401 naminganthropic_organization_idandanthropic_federation_rule_idinstead of silently trying a static key; with either of those set the static key answers instead, since it outranks federationRelevant issues
Fixes #28607
Docs: BerriAI/litellm-docs#1000 documents the config.yaml and environment flow this PR ships
Linear ticket
Resolves LIT-6107
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Config-only rig, no dashboard. Two proxy pairs, every proxy in its own worktree and venv with 2 uvicorn workers,
STORE_MODEL_IN_DB=Trueagainst a Postgres of its own (soPOST /model/newpersists),LITELLM_MASTER_KEY=sk-1234underLITELLM_DANGEROUSLY_PERMIT_WEAK_OR_UNSET_MASTER_KEY=true, and aLITELLM_LICENSEso no premium gate gets in the way. The first pair boots fromwif_config.yaml(litellm/proxy/dev_config.yamlplus the block below): Before at b024950 (the merge base) on port 60131, After at fce9d8f on port 60132.anthropic-wif-issuernames a real federation issuer, federation rule and service account in a Claude Console organization: the issuer was registered with the JWKS the first case exports, the rule is enabled for all workspaces, so the credential also names the workspace to mint for, and the only workspace the service account belongs to is the organization's default workspace, whichGET /v1/organizations/workspacesdoes not list (its id comes fromGET /v1/organizations/service_accounts/{id}/workspaces). A direct exchange with the same assertion got Anthropic's opaque 401 withoutworkspace_idand 200 with it.anthropic-wif-noidscarries no Console ids. The third deployment points at a local nginx on127.0.0.1:60135that forwards toapi.anthropic.com, standing in for a customer's egress gateway, and the exchange host is allowlisted ashost:port. NoANTHROPIC_API_KEYorANTHROPIC_AUTH_TOKENin either environment and no.envin either worktree, sinceproxy_cli.pyloads one when present and a static key would silently win;ISSUER_SIGNING_KEY_PEMholds the P-256 private key behind that JWKS. The After side's chat, messages, responses and gateway calls below are real federated calls: the proxy signs its own assertion, exchanges it for ansk-ant-oat01token and the model answers. The count-tokens cases go throughPOST /v1/messages/count_tokens, the route that reaches the provider's counter (POST /utils/token_counteralways answers the local estimate), with a body carrying a system prompt and one tool: the local estimate is 82 and a direct call to Anthropic's own count-tokens endpoint for that body returns 590The four cases from
/healththrough the credential PATCH need the database-backed team flow, so they run on the second pair booted the same way (Before on port 60133, After on port 60134) fromrig_config.yaml, which adds thewif-inline-token-cfgdeployment below and setslitellm_key_header_name: X-Team-Key. On each side the team admin key comes fromPOST /team/new {"team_alias":"rig-team"}, thenPOST /team/member_addwith{"team_id":"<team>","member":{"user_id":"rig-team-admin","role":"admin"}}, thenPOST /key/generate {"user_id":"rig-team-admin","team_id":"<team>","key_alias":"rig-team-admin"};$TEAM_ADMIN_KEYbelow is that key. Theblockedcase repeats those three calls for a second team so the aliases stay unique, and$TEAM2_ADMIN_KEYis that team's admin key. The credential PATCH case creates an internal user withPOST /user/new {"user_id":"gate-internal-user","user_role":"internal_user"}andPOST /key/generate {"user_id":"gate-internal-user","key_alias":"gate-internal-user","allowed_routes":["/credentials*"]};$USER_KEYbelow is that key, and$MIDis the id the first step of that case reads for thewif-inline-token-cfgdeploymentThe last two cases run on a third pair booted from
wif_config_at.yamlbelow, a singleclaude-haiku-4-5-auth-tokendeployment ofanthropic/claude-haiku-4-5with noapi_keyand no credential: Before on port 60151, After on port 60152, same flags and a Postgres per side. That pair's environment holdsANTHROPIC_AUTH_TOKENset to an Anthropic API key and noANTHROPIC_API_KEY, the shape a customer gets from a token-based login or a gateway that hands out bearer tokensBefore (b024950)
JWKS export for a LiteLLM-signed credential
Run
Observed
/v1/chat/completions on a federated deployment
Run
Observed
/v1/messages/count_tokens on a federated deployment
Run (the federated deployment counts a body with a system prompt and a tool)
Observed (no credential reaches Anthropic, so the proxy answers its local estimate: 82 tokens for a body Anthropic itself counts at 590)
/v1/messages on a federated deployment
Run
Observed
/v1/responses on a federated deployment
Run
Observed
/v1/chat/completions through an egress gateway allowlisted as host:port
Run
Observed
Federated credential missing the Console ids
Run
Observed
Federation field typed into a chat body
Run
Observed
Federation fields typed into POST /model/new, next to the sanctioned credential
Run (a federation field straight in the deployment's
litellm_params)Observed
Run (the same proxy admin attaches the stored federated credential instead)
Observed
Run (which of the two the proxy now serves)
Observed
GET /health with a team admin key on a deployment carrying an inline identity token
Run
Observed (the raw identity token and the Console ids come back to a team admin; the
errorfield's stack trace and the raw request dict are trimmed)Team admin edits on a federated team deployment
Run (the proxy admin creates the team-scoped federated deployment)
Observed
Run (the team admin sets rpm on it)
Observed (the edit lands, it sets no federation field)
Run (the same team admin points it at a static key instead)
Observed (the credential swap lands)
Run (cleanup, the team admin deletes it)
Observed
Setting
blockedon a new team deploymentRun (the team admin creates a deployment with
blocked: true)Observed (the flag is dropped and the deployment lands unblocked)
Run (the proxy admin sends the same flag)
Observed (the flag is dropped for the proxy admin too)
PATCH /credentials/{name} copying a federated deployment's fields through model_id
Run (the proxy admin reads the id of the config deployment that carries federation fields inline)
Observed
Run (the proxy admin creates a plain credential holding only an
api_key)Observed
Run (an internal user whose key allows only
/credentials*PATCHes that credential naming the federated deployment throughmodel_id)Observed (the PATCH lands, the merge base has no federation fields to copy)
Run (which fields the credential holds now)
Observed (the credential still holds only
api_key)Run (the proxy admin sends the same PATCH)
Observed (lands the same way)
api_key)/v1/messages/count_tokens with only
ANTHROPIC_AUTH_TOKENsetRun (a keyless deployment on a proxy whose environment holds only
ANTHROPIC_AUTH_TOKEN, counting a body with a system prompt and a tool)Observed (the auth token never reaches the count-tokens call and the proxy answers the same local estimate of 82)
Run (chat on the same deployment)
Observed (chat authenticates with the same auth token on both sides)
After (fce9d8f)
JWKS export for a LiteLLM-signed credential
Run
Observed
/v1/chat/completions on a federated deployment
Run
Observed
/v1/messages/count_tokens on a federated deployment
Run (the federated deployment counts a body with a system prompt and a tool)
Observed (Anthropic's own count, 590, the number a direct call to Anthropic's count-tokens endpoint returns for this body, authenticated with the minted federated token)
/v1/messages on a federated deployment
Run
Observed
/v1/responses on a federated deployment
Run
Observed
/v1/chat/completions through an egress gateway allowlisted as host:port
Run
Observed
Federated credential missing the Console ids
Run
Observed
Federation field typed into a chat body
Run
Observed
Federation fields typed into POST /model/new, next to the sanctioned credential
Run (a federation field straight in the deployment's
litellm_params)Observed
Run (the same proxy admin attaches the stored federated credential instead)
Observed
Run (which of the two the proxy now serves)
Observed
GET /health with a team admin key on a deployment carrying an inline identity token
Run
Observed (the token and the Console ids are gone and the error says raw tokens must be
oidc/references; theerrorfield's stack trace and the raw request dict are trimmed)Team admin edits on a federated team deployment
Run (the proxy admin creates the team-scoped federated deployment)
Observed
Run (the team admin sets rpm on it)
Observed (the edit lands, it sets no federation field)
Run (the same team admin points it at a static key instead)
Observed (the credential swap is refused)
Run (cleanup, the team admin deletes it)
Observed
Setting
blockedon a new team deploymentRun (the team admin creates a deployment with
blocked: true)Observed (the team admin is refused)
Run (the proxy admin sends the same flag)
Observed (the proxy admin's flag sticks)
PATCH /credentials/{name} copying a federated deployment's fields through model_id
Run (the proxy admin reads the id of the config deployment that carries federation fields inline)
Observed
Run (the proxy admin creates a plain credential holding only an
api_key)Observed
Run (an internal user whose key allows only
/credentials*PATCHes that credential naming the federated deployment throughmodel_id)Observed (the internal user is refused, naming the first federation field the copy would have landed)
Run (which fields the credential holds now)
Observed (the credential is untouched)
Run (the proxy admin sends the same PATCH)
Observed (the proxy admin's copy lands)
api_key)/v1/messages/count_tokens with only
ANTHROPIC_AUTH_TOKENsetRun (a keyless deployment on a proxy whose environment holds only
ANTHROPIC_AUTH_TOKEN, counting a body with a system prompt and a tool)Observed (the same 590 from Anthropic, authenticated with the auth token as a Bearer)
Run (chat on the same deployment)
Observed (chat authenticates with the same auth token on both sides)
Type
New Feature
Caveats (if any)
Severe
ANTHROPIC_API_KEY, thenANTHROPIC_AUTH_TOKEN, then federation, and it holds for chat,/v1/messages,/v1/responses, files, batches and model discovery alike. It is the order the Anthropic SDK itself uses, and inverting it would break every deployment that sets both, so it staysproxy_cli.pycallsload_dotenv(), so a.envin the proxy's working directory puts the key back even after the shell unset it, and a fleet that wants no static secrets has to clean that file toojti, so dropping it would break the multi-worker case this PR exists for. Entries are 0600 in a 0700 uid-owned directory, a staging file a failed write leaves behind is now unlinked so no live token survives it, andLITELLM_TOKEN_EXCHANGE_CACHE_DIRset to an empty string turns the store offMedium
POST /provider/models/discoverfrom feat(anthropic): workload identity federation, pluggable identity sources, and provider-level setup (internal copy of #38013) #38818 come back in a follow-up PR; until then an admin sets federation up through config.yaml, env vars orPOST /credentials, and a credential row created through the dashboard's generic form has no federation fields to fillMissing Anthropic API Key401 for up to one credential refresh interval on a multi-worker proxy without Redislitellm_credential_namedeployment (LIT-6901); config.yaml credentials are loaded by every worker at boot and do not hit it,POST /credentialsones dooidc/env/value or a token file that rotates slower than the rule's token lifetime (60 s to 24 h, wizard default 10 min) getsjti_reusedon the re-mint until it rotates, and the 401 hint says so; the LiteLLM-signed internal issuer and Keycloak mint a fresh assertion every timeeu.api.openai.comandus.api.openai.comis exercised only up to the upstream geography refusal, no geography-restricted project was available to see a 200 on a regional hostLow
ANTHROPIC_FEDERATION_WORKSPACE_ID, not the shorterANTHROPIC_WORKSPACE_IDthe Console reference uses, because the Bedrock Claude platform provider already reads that shorter name on a released path; the 401 hint says so when the workspace is ambiguousLITELLM_OIDC_ALLOWED_CREDENTIAL_DIRS, and the token exchange only talks to api.anthropic.com unless the operator adds hosts toLITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS, so a gateway or test double needs that set. Entries are a host or ahost:port: naming a port trusts that port alone, leaving it off trusts every port on the host. The list is server-owned, read from the environment only, and cannot be set through the model or credential APIsLITELLM_TOKEN_EXCHANGE_CACHE_DIR, default$TMPDIR/litellm-token-exchange-<uid>), so workers on one host share one exchange and each host mints its ownPOST /model/newfor the proxy admin too, not only for team admins: the sanctioned paths are a named credential (POST /credentialsthenlitellm_credential_name) or config.yaml. Changing the credentials of a federated deployment, meaning its federation fields, its api_base or the credential attached to it, stays proxy-admin only; the team admin who owns it can still edit everything else on it, delete it, and press Test Connection/v1/chat/completionsand every other LLM route, so a caller cannot pick which federated identity the proxy mints as/anthropic/*passthrough no longer forwards the caller's own credential headers upstream once the deployment has a server-side Anthropic credential:authorization,x-api-keyand the rest of the headers the proxy accepts a LiteLLM key in are stripped, plusgeneral_settings.litellm_key_header_namewhen one is set. Base forwarded them all, so a client that relied on its own header reaching Anthropic through the relay has to stop sending the server credential instead. With no server credential nothing is stripped and BYOK is unchangedCredentialItem-InputandCredentialItem-Outputinstead of oneCredentialItem, becausecredential_values_to_deleteis excluded from serialization. The wire shape is unchanged; a generated client pinned to the old schema name regenerates.ui/litellm-dashboard/src/lib/http/schema.d.tsis regenerated for the same reason and is the only file underui/this PR touchesapi_basegets a different synthesized per-request model id than before, since the federation disable flag now rides those params. Cosmetic, the id is not persistedblockedto true onPOST /model/newfor a team deployment gets 403 where base answered 200 and silently dropped the flag;blocked: falsestill answers 200, since that is the state every create lands in anyway, and a proxy admin still gets 200 with the flag honored/health/test_connectionanswers the model auth gate's own 403 or 400 where base wrapped the same refusal into a 500Failed to test connection; for a team admin that includes a probe whose body redirects a federated deployment'sapi_baseor swaps its credential, while an unmodified probe of that deployment still runscredential_valuesonPOST /credentialsandPATCH /credentials/{name}is dropped rather than stored, where base stored it; removing a value goes throughcredential_values_to_delete. Storing the null is net-negative: the federation resolver refuses a foreign variant's field by key, so one storedanthropic_issuer_url: nullwedged every deployment naming that credentialanthropic_identity_token_file: ""and the other variant fields) count as unset, the same blank-as-unset rule the static key follows, so a config template that leaves a field empty falls through to the next identity source instead of failing on an empty path or an unexpected variantANTHROPIC_FEDERATION_RULE_IDexported into a running proxy's environment after a deployment's first static-key call is not noticed until restart. Re-reading it per request is net-negative: it would be a secret-manager read on every static-key Anthropic call for a variable almost nobody setstags, so it cannot change a federation fieldapi_basenow count tokens throughANTHROPIC_API_BASE/ANTHROPIC_BASE_URLthe way chat does; a gateway there that serves/v1/messagesbut not/v1/messages/count_tokensgets the logged local-tokenizer fallback where the base counted through Anthropic's host directlysk-ant-oattoken on count-tokens no longer getsanthropic-dangerous-direct-browser-access, which the base branch's count-tokens header ladder added for those tokens: count-tokens now builds its headers through the shared auth helper, the same one files, batches, skills and federated chat use, and none of those send that headerconfig.yamlanswers 404 and leaves it serving, staging's fix(proxy): 404 a credential delete that matched nothing, and raise instead of return #36260 rule carried through the federation admin gatemonkeypatch.setattr68 times rather than injecting a dependency, which the repo prefers. Fixing it is net-negative: most of those calls patch module-level singletons (litellm.api_key,litellm.credential_list,litellm.module_level_client) that have no constructor to inject through, so removing them means reshaping settled provider modules well outside this PR. The token exchange engine, the one piece here that does have a seam, is tested through itosv-scanwas red on the pre-existing gitpython advisory inuv.lockon feat(anthropic): workload identity federation, pluggable identity sources, and provider-level setup (internal copy of #38013) #38818; this PR does not touch dependenciesPATCH /credentials/{name}with amodel_idnaming a deployment that does not exist now answers 404Model not foundbefore looking the credential up, where base answered 404Credential not found in DB.when the credential was missing too: same status, different message, because the deployment's values are resolved first so the gate can inspect themlitellm_credential_nameexposes no federation fields throughmodel_id: named credentials are resolved per request and never merged into the router's copy of the deployment, so a PATCH copying from it carries only what the deployment holds inline, which config.yaml deployments do and API-created ones do notFinal Attestation
Note
High Risk
Changes authentication across Anthropic and shared token-exchange infrastructure, including credential precedence, host allowlists, and on-disk minted-token caching—mistakes could break auth or leak org-scoped tokens.
Overview
Adds Anthropic workload identity federation: a new
litellm/llms/anthropic/wif.pypath and sharedlitellm/llms/base_llm/auth/RFC 7523 JWT-bearer engine (token file/env, internal ES256 issuer, Keycloak client-credentials, on-disk multi-worker cache) mint short-livedsk-ant-oat01tokens when static keys are absent.Auth plumbing now threads
litellm_paramsand WIF kwargs throughget_litellm_params, batch credential extraction (so federated batch output fetch can bill), and Anthropic chat, messages passthrough, files, batches, skills, and count-tokens—using sharedget_auth_header/aget_auth_header, stripping caller credential headers when the server mints, and asyncvalidate_environment/avalidate_anthropic_messages_environmentso exchanges do not block the event loop.Operational touches: Grafana all-metrics dashboard panels for
litellm_anthropic_wif_*and cache families;HTTPHandleroptionalfollow_redirects=Falsefor Keycloak; async file/batch create paths await async credential validation.Related fixes in the same diff: count-tokens and model discovery honor deployment
api_baseandANTHROPIC_AUTH_TOKEN; batches useGenericLiteLLMParams.model_validate.Reviewed by Cursor Bugbot for commit fce9d8f. Bugbot is set up for automated code reviews on this repo. Configure here.