feat(anthropic): workload identity federation, pluggable identity sources, and provider-level setup (internal copy of #38013) - #38818
Open
mateo-berri wants to merge 89 commits into
Conversation
…oken exchange engine Adds a provider-agnostic JWT-bearer token exchange engine (litellm/llms/base_llm/auth/) with two-tier refresh, single-flight minting, negative caching, response caps, and RFC 6749 error redaction, plus the Anthropic WIF adapter and wiring: env and litellm_params config, sync and async facades, and beta-header merging fixes on the skills, files, batches, messages, and passthrough surfaces WIF is the lowest credential tier, so a deployment that does not configure it behaves exactly as before. Verified end to end against the live token endpoint from a container: both /chat/completions and /v1/messages complete over a minted sk-ant-oat01 token with no api_key configured anywhere Two protocol details were confirmed against the live endpoint rather than the docs. The exchange needs no anthropic-beta header, so none is sent. service_account_id mints fine when omitted for a single-service-account rule, so it stays optional Resolves #28607
…nthropic providers The federation exchange derives its token endpoint from the deployment's api_base and returns an Anthropic-org credential, so it must only ever run for Anthropic itself. It did not. VertexAIAnthropicConfig inherits AnthropicModelInfo.validate_environment, and the MiniMax and Tencent /v1/messages configs inherit the Anthropic validate step, so a proxy configured with the ANTHROPIC_* federation variables POSTed the workload's OIDC assertion to those providers' own hosts, confirmed against a real deployment base Eligibility is now declared per class and read from that class's own __dict__, so a subclass written for another provider inherits nothing, and the get_auth_header facades refuse to mint unless a caller states that it authenticates against Anthropic's own API. The six federation fields join the banned request-body parameters: they select which server-side secret is read and, with api_base, where it is sent, so a caller-supplied value was an exfiltration primitive for any environment variable or mounted token file. The exchange client no longer follows redirects, since only the initial token URL is validated, and a base that already ends in /v1 no longer yields a doubled /v1/v1/oauth/token
…olled request paths An independent review of the previous commit found the narrowing incomplete. The federation fields select which server-side secret is read, and together with api_base decide where it is sent, so they are deployment decisions on every surface: - They are rejected from any request body unconditionally, ahead of the general banned-parameter check, because both client-side credential opt-ins would otherwise re-enable them. The inert workspace id keeps its existing behaviour, since Bedrock Claude Platform already accepts that spelling in a request body. - /health/test_connection takes a litellm_params object that never reached the request-body check, and its existing guard only covers os.environ references, not oidc ones. It rejects them now. - A client-redirected api_base clears the federation fields and marks the deployment, so a token is not minted for a caller-chosen host. The mark is what stops the environment-configured path, which cannot be cleared out of a dictionary Batch retrieval now threads litellm_params, so it authenticates the same ways every other Anthropic surface does rather than failing ahead of the transformation that resolves them The exchange engine always publishes a result for its single-flight entry, so an unexpected failure can no longer leave every later caller waiting on a leader that never finishes. Error bodies are rendered from structured fields only, and a body that echoes the submitted assertion is dropped rather than logged and returned. Endpoint normalization now works on the URL path, so a single-label host is left alone and a pathological base cannot exhaust the stack
…eration The workload assertion could only come from a mounted file or an environment variable, which assumes a platform that already projects one. Two more sources sit behind an explicit anthropic_identity_source discriminator, and its absence keeps today's resolver exactly as it was: - internal_issuer signs a short-lived ES256 assertion with an operator-supplied key, resolved through the usual secret reference so it can live in a secret manager. Its JWKS is exported for the operator to register with Anthropic, without which the source cannot be used at all. - keycloak fetches the assertion from a client_credentials grant, with the client secret likewise held behind a reference rather than in the configuration The engine gains an optional assertion source on the spec, so a source that needs more than a string can supply one without the reference ever carrying a secret: it stays a hash of the non-secret fields and remains what the cache keys on and what errors name. Failures carry a redacted detail, so a Keycloak hop is diagnosable rather than collapsing into one opaque message The server-owned field list is now derived from one definition, so every field added here is rejected from request bodies and cleared on a client base override without a second edit. Credential params carry the federation fields too, which is what the files, batches, and passthrough surfaces read Verified against the live token endpoint: an ES256 assertion from internal_issuer, with its exported JWKS registered on the federation issuer, mints a token. Note that a freshly registered inline JWKS takes up to about a minute to become usable
…eration Federation was configurable but only per model, and only by hand. This adds the provider-level flow the feature was missing, reusing what already exists rather than introducing parallel concepts Provider metadata gains optional credential variants: a selector plus per-variant field lists, so a provider whose credential shape branches describes that in the same metadata every provider already publishes, and the existing generic renderer drives it with no provider-specific branch. Anthropic publishes five, one per way of authenticating. Fields holding a secret reference render as text, not as password inputs that would invite pasting the secret itself Model discovery could not see a credential configured through litellm_params, only through the environment, so a federated provider had nothing to discover. Providers now expose discovery that receives the deployment params, and an admin-only endpoint discovers a provider's models from a stored credential by name. The credential fields are never accepted in that request, since choosing which server-side secret is read is not a caller's decision Enable and disable reuse the existing blocked flag rather than a second notion of the same thing, which meant letting model creation persist it so a model can be created already disabled instead of being created live and then paused. Alternate names reuse the existing public name and group alias. One credential now feeds many models, so switching a credential's auth variant has to be able to remove the previous variant's fields, which credential updates could not express before The Admin UI gains an Add Provider flow over all of it: pick the provider and auth method, fill that method's fields, export the JWKS when the identity is self-signed, discover the provider's models, then enable, rename, and alias them before creating. Two pre-existing UI defects surfaced while building it: a variant-capable provider could render through the stale flat field list before its metadata loaded and leave a validation rule behind that blocked submission, and the credential editor left the previous variant's fields in place when the variant changed
…ject /model/block and /model/unblock declare the proxy model table as their response model, and FastAPI validates the returned row against it with from_attributes. The before-validator that parses the JSON string columns assumed a mapping, so it called .get on the row object and both endpoints answered 500 even though the block itself had already been applied The validator now passes an object through untouched and keeps parsing the string columns it exists for, which is the case the database actually produces. Found while exercising the provider setup's enable and disable toggle against a live proxy
…types The discovery and JWKS routes were missing from the generated types. Regenerating on a developer machine can drop the enterprise routes when that package is not importable, which turns the sync check into a large deletion; this spec was dumped from the built container image, so the diff is only the two added endpoints
…-encode Keycloak credentials The retry path re-read the assertion from its source to decide whether an error body reflected it back. A rotating source (a token file rewritten between attempts, a Keycloak fetch that mints a fresh token) hands back a different value, so the assertion the failing request carried stayed in the error text. _Unauthorized now carries the assertion the attempt used and the redaction runs against that, which also drops one extra read of the secret per failure Separately, RFC 6749 2.3.1 wants the client id and secret form-urlencoded before the base64, so a reserved character in either one no longer corrupts what the far side decodes back out of Basic auth
…g the event loop The async file and batch handlers called the provider's sync validate_environment directly. For the workload identity tier that hook performs a blocking token exchange, so a mint stalled the whole loop. They now go through one facade that awaits the provider's async hook when it has one and offloads the sync hook to a worker thread otherwise, so every other surface keeps its existing behaviour Batch retrieval also copies the litellm_params it hands down rather than sharing the caller's dict, and get_anthropic_headers moves its credential selection into a helper so the four auth tiers read as four branches instead of one nested chain
… fields These fields pick which server-side secret is read and where the resulting assertion is sent, so a team admin who can otherwise manage a team-scoped deployment must not be able to set them. The create, update and patch model paths and both credential write paths now reject them for anyone below proxy admin, mirroring the existing blocked-flag gate. The field list is derived from anthropic_wif_litellm_params rather than copied, so a new federation field is covered the day it is added Also fixes two things CI caught: credential_values_to_delete is a PATCH instruction rather than part of the credential, so it stays out of dumps that feed config loading and the Prisma write, and the provider discovery route is registered in the backend allowlist
…es for failed rows The tab list put add-provider ahead of auto-routers, which reordered the tabs an existing test pins. It now sits after auto-routers, where the admin-only tabs start The wizard also wrote a model_group_alias entry for every discovered row it tried, including rows whose model creation had just failed, leaving an alias pointing at a deployment that does not exist. Alias additions are now taken from the rows that actually got created
Both posters built their HTTPHandler inline, so nothing exercised the guarantees that matter there: redirects stay disabled, an HTTPStatusError comes back as its response rather than escaping, and a None response raises instead of being dereferenced. Each now takes a handler factory that defaults to the real one, which is the same injection seam the rest of this package already uses for its poster and secret reader Coverage for litellm/llms/base_llm/auth goes from 90% to 97%: client_credentials from 79% to 98%, with only its assert_never arm left, and token_exchange from 90% to 96%
…l schema Excluding credential_values_to_delete from dumps makes pydantic emit separate input and output schemas for CredentialItem, since the field is accepted on a write and absent from every read. The generated types now say so: the credential read routes return the output shape, which no longer carries the PATCH-only field, and only the write route accepts it
…fetime The advisory and mandatory refresh windows were flat 120s and 30s, while the fallback lifetime for a token minted without expires_in is 60s. Such a token was therefore born inside its own advisory window, so every request armed another background exchange against the provider's token endpoint and the cache never settled. Any real expires_in of 120s or less did the same Each window is now the smaller of its flat value and a fraction of the observed lifetime (half for advisory, an eighth for mandatory), so a 60s token is served until roughly its half life and then refreshed once. Tokens of 240s and above hit the flat values and keep exactly the previous behaviour. The tradeoff is deliberate: a 60s token may now be served with as little as 7.5s left, where before the mandatory wall was 30s, which for a 60s token meant refreshing at half its life on every path
…d identity surface The admin gate only inspected the incoming credential_values, so it was fail-open three ways. A non-admin could PATCH with an empty value map and name a federation field in credential_values_to_delete, dropping it off the stored credential. A PATCH carrying no federation field could still mutate a credential whose stored values held them, because the stored state was never consulted. DELETE had no gate at all The gate now takes the effective surface of the operation: the fields the payload sets, the keys it names for deletion, and the fields already on the credential being changed. Presence is keyed on the field name rather than a non-null value, matching how the resolver and get_litellm_params decide a field is set, so clearing a field to null no longer walks past the gate and wedges every deployment referencing that credential Create, delete and a rename that targets another credential resolve the existing record through hydrate_named_credential, which reads the in-memory config list before the database. A credential declared in config.yaml has no row, so a DB-only lookup let a non-admin shadow or drop one. POST semantics for credentials carrying no federation field are unchanged
…ed passthrough token The Anthropic passthrough route forwards the caller's headers, so once the server owned the credential the caller's own key travelled to Anthropic alongside the minted bearer. Two credentials arrived in one request and the caller's could win, which defeats the point of server-owned federation The set of headers the server owns is now derived from SpecialHeaders, the same enumeration the proxy accepts a LiteLLM key in, plus the configured litellm_key_header_name, rather than the two names it listed before. That closes the same leak for x-litellm-api-key, api-key, x-goog-api-key and Ocp-Apim-Subscription-Key. The non-passthrough path already stripped these, so this brings the two into line Bring-your-own-key is untouched: with no server credential the caller's key is the only one there is, so it still forwards
check_workflow_startup_safety rejects a job whose deadline can preempt pytest: setup can take 35 minutes plus 5 of runner overhead, so a 20 minute pytest budget needs the job capped at 60, not 55. Three entries in the unit matrix still sat at 55 and failed the guard on every pull request, including on litellm_internal_staging itself Raising those three to 60 matches the ten entries already at that value and turns the code-quality job green. This is unrelated to the rest of the branch, kept as its own commit so it can be dropped or cherry-picked independently
…ervice A deployment that mints short-lived tokens on a two tier refresh was unobservable: the exchange emitted one warning and no metrics, so the first sign of a failing token endpoint was user visible 401s. Registering the exchange as a service type puts it on the same path redis and postgres already use, so success and failure counters and a latency histogram reach /metrics and OTel spans from one emit, with no per backend wiring Each attempt reports its call type, separating a cold mint from a mandatory refresh and from a background advisory refresh, and tokens served from cache are counted on their own service so a cache hit never fakes an exchange latency. Failures carry the ExchangeError variant as a low cardinality error class The sink is a constructor parameter like the poster and the clock, so tests inject a fake rather than patching. It hands events to a single worker thread, which keeps the sync path free of the async hooks and of any need for a running loop, and every emission is wrapped so a broken metrics backend can never fail or delay a mint. Labels carry only the variant name and the redacted summary the error path already produces, never an assertion, a token or a secret
…cannot exhaust memory Every exchange and cache hit queued an event on the metrics executor with no ceiling, so a telemetry backend that stalls turned request volume into unbounded queued work inside the proxy. Reported by a review bot and independently by the adversarial verifiers on the change that introduced it The sink now tracks its own backlog and drops events once it reaches a thousand, on the principle that losing a metric sample is cheaper than losing the process. A rejected submit releases its slot too, so a failing executor cannot leak the count and wedge the sink shut. Metrics stay best-effort throughout: nothing here can fail or delay a mint
…id not need The repo's recursion guard rejects unignored recursive functions, and this branch had added three. Two were plain iteration written as recursion: the model list walked its pages recursively, and the base URL stripped chat suffixes by calling itself. The third mattered more: a token exchange follower whose leader published nothing called get_token again, so a contended entry grew the stack one frame per failed leader All three are loops now. The suffix strip re-strips trailing slashes on each pass, which the recursion got for free by re-entering a function that begins by stripping them, and which a doubled suffix needs to keep matching The passthrough header tests also pin SERVER_ROOT_PATH. They drive the real relay, and a sibling test leaving that variable set re-prefixes the route, so the request 404s before any header is built and the assertion never runs
…l write The gate resolved the existing credential through hydrate_named_credential, which answers from the in-memory list and only reads the database when memory has no entry. That order is right when serving a request and wrong when deciding authorization: a pod whose copy predates an admin adding federation fields sees none of them and lets a non-admin through. In a multi-pod deployment the window is a config reload, and with store_model_in_db off it never closes Authorization now goes through its own helper that reads the in-memory entry and the row and takes the union, so the gate refuses whenever either side says the credential is server-owned. Resolution keeps memory-first, which is still correct for serving
…stinct identities Eviction skipped every in-flight entry and then removed exactly one, so a burst of distinct federation identities pushed the map past max_entries and left it there: each later insert evicted one and added one, holding the high-water mark for the life of the process. The cap the parameter advertises was effectively advisory It now evicts as many as the overshoot needs, soonest-to-expire first, and treats an entry with no token as evictable rather than skipping it. An entry a leader owns or a follower waits on is still never a candidate, since dropping one would break single flight, so a moment when every entry is in flight can still over-insert; that residue is bounded by the concurrent mints themselves and clears as they finish
The recursion-to-loop rewrite carried its Final annotations onto locals that are now assigned once per iteration, which basedpyright reports as an error: a Final cannot be assigned within a loop. Each is a genuine per-iteration rebind, so they carry rebind-ok with the reason instead
The exchange derived its token URL from the deployment's api_base and sent the signed assertion wherever that pointed. Any write path that could set api_base on a WIF deployment could therefore redirect a valid assertion to a host of its choosing, and gating each of those paths individually has no termination condition as new ones are added. Enforce it where the exchange is actually built instead: the host must be api.anthropic.com or an entry in LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS, matched on the parsed hostname so a lookalike like api.anthropic.com.evil.test cannot pass by containing the real one. Both the sync and async call sites check before any assertion leaves the process.
…d exists The Add Provider wizard required anthropic_federation_rule_id up front, but with LiteLLM as the issuer that id does not exist yet: the operator gets it from Anthropic after registering the JWKS URL the credential itself publishes. So the step that needed a rule id could not be reached without one. Mark the field optional for wif_internal_issuer and let the JWKS step PATCH it in once it is known. The Keycloak flow is unchanged, where the rule id is known in advance.
A council reviewed the four findings the review bots raised on the wizard and ruled these three worth fixing before human review, since all of them are in code this PR introduces. Creation no longer treats a failed deployment lookup as an empty list. That lookup is what makes a retry skip rows already created, so swallowing its error turned a partial-failure retry into duplicate deployments. It now stops and says why, with the rows still on screen. Whether the wizard creates or updates a credential is derived from the name it actually saved rather than a flag that was set once and never cleared. Renaming after a save used to take the update path against a name the server had never seen, which dead-ended the flow with no way forward but a reload. Create is disabled while any model name is blank, instead of submitting an empty model_name and producing a deployment that cannot be addressed. Each fix carries a regression test that fails when the fix is reverted.
effectiveSessionRole deliberately reports a proxy_admin_viewer session as "Admin", so gating the tab on all_admin_roles showed a read-only admin a wizard whose every step is a write the proxy then refuses. The first step solicits a provider API key, so a viewer types a real secret into a form that 403s on submit. Only the raw-role isViewOnly separates the two, which is the same mechanism the Playground already uses for this carve-out. Both conjuncts of the gate are covered: dropping either one fails a test. The five older admin panels on this page have the same gap, but they predate this PR and fixing them is a page-wide change, so they are left for a follow-up.
…, and normalize the discovery base Two defects the review bots found in this PR's own code, both confirmed against the implementation before fixing. The files surface built its headers by merging the resolved credential over the caller's, which leaves a caller-supplied x-api-key in place next to a minted federation Bearer. The chat surface already stripped those, so this was an inconsistency rather than a new rule. That stripping is now one helper both surfaces call, and it applies whatever the credential turned out to be, since a header that authenticates the caller to LiteLLM should never reach Anthropic. Batches is unaffected: it passes no caller headers through. Model discovery appended /v1/models to the configured base without normalizing it, so a deployment whose api_base already ended in /v1, or in the /v1/messages URL an operator copied out of the docs, asked for /v1/v1/models and failed. It now reuses the same suffix stripping the token exchange derives its URL from, so both agree on what the deployment's base is.
…e surface gaps Three defects the review bot found, all confirmed against the code before fixing. anthropic_workspace_id was carved out of the request-body ban as inert. It is not inert: it is the scope the federation token is minted for, and router.py merges request kwargs over deployment params, so a caller who set it chose the scope instead of the administrator. Proven against Anthropic before fixing, whose token endpoint answered "workspace_id is not a well-formed wrkspc_ tagged ID" on a value that came from the request body. The carve-out is gone, so it is banned like every other minting parameter. That does not cost the Bedrock Claude Platform route anything. It reads a workspace from workspace_id or aws_workspace_id as well, neither of which is a federation parameter or in all_litellm_params, so both still reach it from a request body, and an administrator-configured anthropic_workspace_id still resolves from the deployment. The refusal names those two spellings so that caller is not left guessing. Batches had the same caller-credential merge that files did, on its create path. The retrieve path in the handler passes no caller headers, which is why the earlier pass over this missed it. merge_anthropic_beta_headers now accepts a list as well as a comma-separated string. The Skills surface handled a list-valued anthropic-beta before it shared this helper, and calling .split() on one raises.
…/litellm into litellm_internal_copy_38013
Contributor
Author
Contributor
Author
|
bugbot run |
Contributor
|
Too many files changed for review (129 files, 100 file limit). Bypass the limit by tagging |
mateo-berri
enabled auto-merge
September 5, 2026 03:11
auto-merge was automatically disabled
September 5, 2026 03:21
Pull request was closed
mateo-berri
enabled auto-merge
September 5, 2026 03:22
…/litellm into litellm_internal_copy_38013
Contributor
Author
|
bugbot run |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Empty identity fields treated as set
- Filtered out empty strings alongside None in the identity-source params comprehension and in the internal-issuer _claims comprehension, so blank optional fields and leftover other-variant keys are treated as unset.
Or push these changes by commenting:
@cursor push 2cd0c2ce9f
Preview (2cd0c2ce9f)
diff --git a/litellm/llms/anthropic/wif.py b/litellm/llms/anthropic/wif.py
--- a/litellm/llms/anthropic/wif.py
+++ b/litellm/llms/anthropic/wif.py
@@ -160,7 +160,7 @@
legacy_ref: Final = _resolve_assertion_ref(litellm_params)
return (legacy_ref, None) if legacy_ref is not None else None
params: Final[Mapping[str, object]] = MappingProxyType(
- {key: value for key, value in (litellm_params or _EMPTY_PARAMS).items() if value is not None}
+ {key: value for key, value in (litellm_params or _EMPTY_PARAMS).items() if value not in (None, "")}
)
match source_kind:
case AnthropicIdentitySourceKind.internal_issuer.value:
diff --git a/litellm/llms/base_llm/auth/internal_issuer.py b/litellm/llms/base_llm/auth/internal_issuer.py
--- a/litellm/llms/base_llm/auth/internal_issuer.py
+++ b/litellm/llms/base_llm/auth/internal_issuer.py
@@ -38,7 +38,7 @@
("exp", issued_at + config.ttl_seconds),
("jti", str(uuid.uuid4())),
)
- if value is not None
+ if value not in (None, "")
}
)You can send follow-ups to the cloud agent here.
Contributor
Author
|
bugbot run |
Contributor
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 82afbbd. Configure here.
This was referenced Sep 5, 2026
Contributor
Author
|
Backend half of this PR, config-only and without the dashboard, is now #39935 on litellm_internal_staging. The dashboard returns in a follow-up |
mateo-berri
disabled auto-merge
September 6, 2026 01:02
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Internal copy of #38013 (fork branch
derhornspieler:litellm_anthropic_wif_oidc_28607, superseded at 57b9e94), opened on alitellm_head so CircleCI runs the legacy suites before merge. Commits and authorship are carried over unchangedTLDR
Problem this solves:
How it solves it:
User Flow
Before: a platform team with a no-static-secrets policy cannot point LiteLLM at an Anthropic federation rule
model: anthropic/claude-haiku-4-5{"model": "claude-haiku-4-5", "messages": [...]}Missing Anthropic API Key - A call is being made to anthropic but no key is set either in the environment variables or via paramsAfter: the same team configures the provider once and every discovered model authenticates without a static key
fdrl_..., an org id, and ansvac_...os.environ/KC_CLIENT_SECRET, never a pasted secret. Test Connection stays disabled with a hint to add the credential first, since federation values can only be tested once saved; only an API key and API base can be tested before savinggpt-5.6through it on the models page with the key field no longer required, and POST http://localhost:4000/v1/chat/completions with{"model": "gpt-wif", ...}answers 200 with the reply and real token usage. Before, the Add Model form refused to submit without a key and a keyless deployment answered 500The api_key client option must be setRelevant issues
Fixes #28607
Docs: BerriAI/litellm-docs#1000
Linear ticket
Resolves LIT-6107
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Federated end to end (contributor's run at 197fd1e, before at fde3075)
Four models on
anthropic/, noapi_keyanywhere, federation ids from the environment, and the identity token projected at a path inside the OIDC file allowlist:curl -s localhost:4100/v1/chat/completions -d '{"model":"claude-haiku-4-5-wif","messages":[{"role":"user","content":"Reply with exactly: WIF OK"}],"max_tokens":16}'{"error":{"message":"litellm.AuthenticationError: Missing Anthropic API Key - A call is being made to anthropic but no key is set either in the environment variables or via params. ..."}}. After:content: WIF OK | model: claude-haiku-4-5-wif | tokens: 20/v1/messageswithclaude-sonnet-5-wif. Before:{"type":"error","error":{"type":"authentication_error","message":"x-api-key header is required"}}. After:content: WIF OK | usage: 19 in / 7 outcurl -s localhost:4100/v1/messages/count_tokens -d '{"model":"claude-haiku-4-5-wif","messages":[{"role":"user","content":"count these tokens please"}]}'. Before: no credential, the count never reaches Anthropic. After:{"input_tokens":11}, Anthropic's own countcurl -s localhost:4100/v1/chat/completions -d '{"model":"claude-haiku-4-5-wif", ..., "anthropic_identity_token":"attacker-supplied"}'. Before: the field is neither refused nor honoured and the call fails for want of a key. After:{"error":{"message":"Authentication Error, Rejected Request: anthropic_identity_token is a server-owned workload identity federation parameter and cannot be set in a request body; configure it on the deployment instead.","type":"auth_error","code":"401"}}curl -s localhost:4100/public/providers/fields. Before:credential_variants: None. After:credential_variants: ['api_key', 'wif_token', 'wif_token_file', 'wif_internal_issuer', 'wif_keycloak']The federated 200 legs were not re-run at the tip: the federation rule, service account, and identity token live only in a Claude Console organization the reviewer cannot provision, so the review runs below cover every leg that does not need a live federation rule. The tip changes nothing on the path those legs exercise beyond the three fixes proven below
Review run at f2240df vs base e585aab, carried to ff59884 since the later commits only touch the dashboard wizard, the lazy import of two proxy-extra packages, and the
ANTHROPIC_IDENTITY_SOURCEenv precedence, which the run never set, and to 8402ae9 which mergeslitellm_internal_staginginto the branch (one import-order conflict inmodel_management_endpoints.py) plus the LIT002 budget accommodations that merge forced: two immutable rewrites incommon_utils.pyand eight suppressions on the legacy files/batches call sites inllm_http_handler.py, and to c4d2f03 whose only delta is an unrelated one-line CI fix (LIT-6519): the savings baseline test now pins the local cost map via thelocal_model_cost_mapfixture, since main's registry update gave grok-4 a cache-read rate and broke the test's precondition on every branch reading the remote map, and to 068c1d1 which mergeslitellm_internal_stagingagain after #38863 landed the canonical fix for that test (a map-derived baseline pick), the one conflict resolved by taking staging's version oftest_savings.pywholesale so the interim fixture pin is superseded (two proxies,--num_workers 2each, real Anthropic key on the control models, real spend), and to c6f585e which fixes defect 8 below in the token counter, touching onlytoken_counter.pyand its regression tests; its count-tokens legs were re-driven live in the defect 8 A/B, and to c813cc7 which mergeslitellm_internal_staginga third time (seven conflicts, all resolved by keeping both sides: the WIF-aware_list_modelsincommon_utils.py, thecheck_potential_json_strguard inmodel.py, the credential-hydration and encrypt-decrypt imports plus the blocked-flag and attach-credential checks inmodel_management_endpoints.py, the server-owned-header import next to staging's httpx client import inllm_passthrough_endpoints.py, staging's# test-quality-okpatch forms in its test, the WIF secret-pointer test alongside staging's tier-fallback class intest_router.py, and the models-and-endpoints tab tests rewritten on staging'sVIEW_ONLY_ADMINfixture with the Add Provider tab assertions folded in) plus the two accommodations that merge forced: ruff's blank-line trim inmodel_management_endpoints.py, and the Add Provider integration test's row lookup moving from.closest("tr")to arowrole query since staging ratcheted the dashboardtesting-library/no-node-accessbudget to 716 on 2026-09-01 and the old lookup was the 717th hit, with the merge tip re-driven in full on the rig rather than carried (46 S legs against the base, three of them new: S15k for a team admin registering a model through a stored credential, and S24a to S24c forfallbacksgiven as a config entry or as request-body dicts with and without a federation field inside; D01 to D16 with the proxy booted through a dotenv-patching wrapper so the worktree.envcannot re-supplyANTHROPIC_API_KEY; and the same 46 S legs on a worktree with staging a677242 merged in, identical to the tip apart from noise; the rig'sLITELLM_LICENSEno longer verifies, so the team-admin legs ran on the non-premium path), and to 412890f, which fixes defect 9 below in the count-tokens URL builder, its handler call and their regression tests and touches nothing else, re-driven in full again: 54 S legs against base 2ffe6a1 (the eight new S25 legs drive count tokens and chat on deployments whoseapi_baseis a bare host or ends in/v1/messages), D01 to D16 under the wrapper, and a merge of staging 987ab76 into the tip that applies clean, with no commit on staging touchingcount_tokens/since the base, and to 272ccfd, which fixes defect 10 below by giving the token exchange and the count-tokens URL one shared base resolver and touches nothing else, re-driven in full: 59 S legs against base 2ffe6a1 (the five new S25 and S26 legs drive count tokens and chat on a deployment withapi_base: ""and chat on federated deployments with an empty or bogus base), D01 to D16 plus D04b for a federated deployment withapi_base: ""under the wrapper, the same S25 and D legs at 412890f for the before side, and six E legs on both sides withANTHROPIC_API_BASE=https://gateway.invalidexported, and to c0739de, the fourthlitellm_internal_stagingmerge, made from GitHub's Update branch, which merged staging's rename of the SSO helper intest_auth_utils.pyas one call to the old name (thelintandproxy-authchecks caught it), to 55e303e, which repairs that one test line and nothing else, and to 4a785c8, the fifth staging merge (one conflict, intest_proxy_server.py, where both sides appended tests at the end of the file and both were kept), and to 0a6f606, the sixth staging merge (two conflicts: the import block oftypes/proxy/public_endpoints/public_endpoints.py, resolved as the union of both sides, andtest_public_endpoints.py, where both sides appended tests at the end of the file and both were kept, and where taking staging's tightenedreportUnnecessaryComparisonceiling in place of the branch's stale one surfaced four dead comparisons this PR had added: the two token-posterresponse is Noneguards now run through onerequire_posted_responsehelper whose parameter is typed optional, so the guard and the tests that cover it stay, and the twocomplete_api_base is Noneguards the PR duplicated into its new async file and batch wrappers are gone, since both URL builders are declared to returnstr), with the tip re-driven at 55e303e, at the fifth merge and at the sixth: the same 59 S legs against the base, identical to 272ccfd apart from noise, and D01 to D16 plus D04b with D12 and D13 repeated three times each (at the sixth merge the federation error text names the keycloak host asREDACTED, staging's #39380 at work, a body diff only), and to cb03462, the seventh staging merge, which brings in one Azure storage commit with no file in common with this PR and applies clean, and to 3794ba9, which resolves the count-tokens static key throughAnthropicModelInfo.get_api_keyso a secret-manager-only key is honored, touching onlytoken_counter.pyand its regression tests; the tip was re-driven in full at both, the same 59 S legs and D01 to D16 plus D04b, with no status difference between the two tips and only the documented noise in the bodies, and to 9036b90, the eighth staging merge, made after #36260 landed on staging and turned the branch CONFLICTING (four conflicts, all resolved by keeping both sides:DELETE /credentials/{name}incredential_endpoints/endpoints.pynow runs the federation admin gate first and then staging'sCredentialsRepository.delete_by_name, so a never-stored or config-only credential answers staging's 404 with the in-memory entry kept; thetest_endpoints.pyfixture rewritten around a prisma mock whosefind_uniquereturns None with the repository patched at the endpoints import site only; staging's single-line# test-quality-okpatch forms kept intest_model_management_endpoints.py; and both the federation secret-pointer test and staging's reasoning-effort class kept intest_router.py) plus the accommodation the TQ008 patch budget forced (four displaced suppressions restored, one fixture patch swapped for the prisma mock, and apatch.dictintest_llm_pass_through_endpoints.pyswapped formonkeypatch.setitem). The merge tip was driven live on a DB-backed proxy with two uvicorn workers and real Anthropic and OpenAI spend: 25 targeted legs on the conflicted endpoint (admin create and delete of a federation credential 200, a never-stored name 404, a config-only credential 404 whileGET /credentialsstill lists it and chat through it still answers 200, an internal user refused 401 at the route on both deletes, a federation field inline in a chat body 401,/public/providers/fieldslisting the five Anthropic auth variants; a GET on the other worker within milliseconds of the delete still listed the row until staging's config sync tick cleared it, pre-existing and not a regression), then the same 59 S legs and D01 to D16 plus D04b in full, with no status difference from 3794ba9 apart from latency; the rig's license still does not verify, so the team-admin legs again ran on the non-premium pathSame config, same 30 requests in the same order on both sides, S01 to S23 in the run log. Identical on both sides: liveliness and readiness,
/v1/chat/completions,/v1/messages,/v1/responses,/v1/messages/count_tokensand/anthropic/v1/messagespassthrough on a static-key deployment and on an env-key deployment (all 200,pong), key generate, chat, key info spend and delete, the spend log row,/v1/model/info,/v1/models,/ui/,/health. The differences are the feature:POST /v1/chat/completionswith"anthropic_federation_rule_id":"fdrl_x"in the body (S10). Base:400 AnthropicException - invalid_request_errorafter the field reached Anthropic as an unknown parameter. Tip:401 Authentication Error, Rejected Request: anthropic_federation_rule_id is a server-owned workload identity federation parameter and cannot be set in a request body; configure it on the deployment instead.POST /provider/models/discoverwith{"custom_llm_provider":"anthropic"}(S13). Base:404. Tip:200with the live model list; the same call carryinganthropic_federation_rule_id(S13b) is refused with the same401POST /model/newcarrying federation fields inline, as a team admin (S15e) and as the proxy admin (S15h): both401naming the field. A team admin onPOST /credentials(S15g):401 Only proxy admin can be used to generate, delete, update info for new keys/users/teams. Route=/credentials. A team admin creating a plain deployment (S15f) still gets200GET /public/providers/fields(S12). Base: nocredential_variantskey. Tip:credential_variants: [api_key, wif_token, wif_token_file, wif_internal_issuer, wif_keycloak]on the anthropic entry, every other provider unchangedANTHROPIC_API_KEYis set in the proxy environment (S11) answers200 pongon both sides: the static key wins over federation, see CaveatsDefects found in review, before at a7ed600 and after at f2240df, the dashboard defect after at 57b9e94, the base install defect after at f8b3184, the precedence and error-detail defects after at ff59884, the count-tokens
api_basedefect after at 412890f, and the count-tokens env-base defect after at 272ccfd (proxy booted withoutANTHROPIC_API_KEY,--num_workers 2,router_settings: disable_cooldowns: true)Config:
claude-wif-keycloak(keycloak,anthropic_keycloak_client_secret_ref: os.environ/KC_SECRET, token url pointing at an unreachablehttps://127.0.0.1:9/...),claude-wif-tokenfile(token file insideLITELLM_OIDC_ALLOWED_CREDENTIAL_DIRS),claude-wif-issuer(internal_issuer inline,anthropic_issuer_signing_key_ref: os.environ/ISSUER_KEYholding a P-256 PEM),claude-wif-issuer-cred(litellm_credential_namepointing at acredential_listentry mirroring the docs example), federation ids asos.environ/references, every one a placeholder (fdrl_rigbogus...) since no live rule is availablePOST /v1/messagesonclaude-wif-keycloak(D03). Before:401 anthropic_identity_source is 'keycloak', but anthropic_issuer_audience, anthropic_issuer_signing_key_ref, anthropic_issuer_subject, anthropic_issuer_ttl_seconds, anthropic_issuer_url belongs to a different identity source, so the endpoint could never federate:/v1/messageshands the resolver every deployment field with the unset ones asNone, and the resolver rejected them as foreign. After: the request reaches the identity source,401 Could not obtain the OIDC identity token (unreadable) from oidc/keycloak/...: could not reach the keycloak token endpoint https://127.0.0.1:9/realms/rig/protocol/openid-connect/token: ConnectError. The same flip onclaude-wif-issuer-cred(D09) and on a credential-backed model (D13)POST /v1/chat/completionsonclaude-wif-keycloak(D02) and onclaude-wif-issuer(D07) andclaude-wif-issuer-cred(D08). Before:401 ... keycloak client secret <withheld: not a secret reference> could not be readand401 ... internal_issuer signing key <withheld: not a secret reference> could not be read: config.yaml loading expandedos.environ/KC_SECRETinto the raw secret, and the identity source, which dereferences that pointer itself at use time, then saw a value that is not a reference. After: D02 reaches Keycloak (ConnectErrorfrom the unreachable rig endpoint), D07 and D08 mint and send the assertion and get Anthropic's own answer,401 The token endpoint returned HTTP 400: error: invalid_request_error - federation_rule_id is not a well-formed fdrl_ tagged ID, which is exactly the placeholder rule id. The token-file model (D04, D05) returns that same Anthropic answer on both sides, and a credential created throughPOST /credentials(D10 to D13) was never affected because the credentials API never expanded the referenceAsync
create_fileandcreate_batchvalidated the provider credential on the event loop before dispatching, so a federated token exchange during an async file upload or batch create blocked every other request on that worker. No curl shows this deterministically; the regression tests intests/test_litellm/llms/custom_httpx/test_llm_http_handler.pyassert the async hook is awaited (and a sync hook runs off the loop thread) and fail on the previous handlerThe Add Provider wizard on the models page kept the previous provider's form values after switching providers, so an Anthropic key typed on step two travelled into the OpenAI credential when the operator went back and picked OpenAI. The wizard was fixed to reset its form on a real provider change, and at 3a46fa2 the wizard is removed altogether; the LLM Credentials modal, now the only place a credential is created, does the same reset and its unit tests cover it
A base
pip install litellm(no extras) could no longerimport litellm: the Anthropic provider now importslitellm/llms/base_llm/auth/jwt_signing.py, which imported PyJWT and cryptography at module level, and both ship only in theproxyextra (CircleCIbase_sdk_installcaught it on the internal copy). The module now imports them inside the two functions that sign or load keys and raises anImportErrornaminglitellm[proxy]when they are missing, so the internal issuer identity source fails at use time instead of every base SDK user failing at import;tests/base_sdk_tests/check_base_sdk_install.pypasses 7/7 on a wheel installed into a fresh no-extras venv at f8b3184, and the subprocess test intest_jwt_signing.pyblocks both packages and fails on the previous moduleSetting
ANTHROPIC_IDENTITY_SOURCEin the proxy environment forced every deployment without a param-level kind onto that identity source, including deployments whose params configure a legacyanthropic_identity_token_fileoranthropic_identity_tokenref: those failed closed on the missing issuer or keycloak fields, with no kind value to opt back out (found by bugbot). Kind resolution now lets deployment params win: a param-levelanthropic_identity_sourcefirst, a param-level token or token-file ref keeps that deployment on legacy resolution, and the env kind governs only deployments that set no identity params of their own; the regression tests intest_anthropic_wif.pypin a param token ref beating the env kind and fail on the previous resolver. A live A/B on the rig (env kind exported, no ambient key, D14) shows it end to end: the before proxy answers401 Invalid anthropic_identity_source configurationnaming the three missingInternalIssuerSourcefields without ever reading the token file, and the after proxy reads the param token file, reaches Anthropic's token endpoint, and returns the placeholder-rule401 federation_rule_id is not a well-formed fdrl_ tagged IDThe
ImportErrorfrom defect 5's lazy import surfaced as a bareunreadablewith no detail, because the exchange engine captured a message only forValueError(found by bugbot)._read_assertionnow catchesImportErrorthe same way, so the 401 names thelitellm[proxy]install hint instead of an unexplained unreadable ref; the regression test intest_token_exchange.pyasserts the detail carrieslitellm[proxy]and fails on the previous handlerPOST /v1/messages/count_tokenson a federated deployment (found by bugbot). The token counter treated only a deploymentapi_keyorANTHROPIC_API_KEYas static credentials, so a proxy relying onANTHROPIC_AUTH_TOKENstill minted a federation token for counting, and the mint sat outside the counter's error handling, so a failed mint raised past the endpoint's local-tokenizer fallback:/v1/messages/count_tokensand/utils/token_counter?call_endpoint=trueboth answered500 Internal server error: litellm.AuthenticationError .... The counter now skips minting whenever a static key orANTHROPIC_AUTH_TOKENis present, mirroring chat'svalidate_environment, and a failed mint degrades like any other Anthropic counting error (error=True, status 401), so the endpoint falls back to the local tokenizer. Live A/B at 068c1d1 vs c6f585e on the bogus-federation deployment: before, both routes return 500, and they still mint withANTHROPIC_AUTH_TOKENexported; after, both return 200 with the local count (tokenizer_type: huggingface_tokenizer) and the auth-token boot never mints; the static-key control counts 11 tokens on both sides. The regression tests intest_count_tokens_oauth.pyfail on the previous counterPOST /v1/messages/count_tokensandPOST /utils/token_counter?call_endpoint=trueon a deployment carrying anapi_base(found by bugbot at c813cc7). Defect 8's fix started forwarding the deploymentapi_baseto the count-tokens handler, but the handler took that value as the complete count-tokens URL, so a bare-host base posted to the host root and got Anthropic's 404, and a base written as.../v1/messagesposted the count body to the messages endpoint and got a 400; either way the counter quietly fell back to the local tokenizer. The handler now derives<base>/v1/messages/count_tokensthrough the same suffix-stripping helper the token exchange and model discovery already use, so a base with or without/v1or/v1/messageslands on the count endpoint. On the rig (claude-haiku-basehostwithapi_base: https://api.anthropic.com,claude-haiku-basemsgswithapi_base: https://api.anthropic.com/v1/messages,--num_workers 2): base 2ffe6a1 and 412890f both answertokenizer_type: anthropic_apion/utils/token_counter?call_endpoint=trueand{"input_tokens":12}on/v1/messages/count_tokensfor both deployments; c813cc7 answerstokenizer_type: huggingface_tokenizerfor both while its log carriesAnthropic CountTokens error: status=404andstatus=400; chat on both deployments is200 pongon every side. The regression tests intest_anthropic_count_tokens_transformation.pydrive the handler through respx againsthttps://gateway.exampleand fail on the previous handlerPOST /utils/token_counter?call_endpoint=trueandPOST /v1/messages/count_tokenson a deployment without anapi_base(found by bugbot at 412890f). Defect 9's fix built the count-tokens URL from the deploymentapi_basealone, so a deployment with none skipped theANTHROPIC_API_BASE/ANTHROPIC_BASE_URLthat chat and the token exchange both follow, and anapi_base: ""posted the count body to the bare relative path. The token exchange URL, its cache key and the count-tokens URL now come from one resolver: the deploymentapi_base, else the env base, else Anthropic's host, with a trailing slash or a chat-appended/v1/messagesstripped. On the rig (--num_workers 2):claude-haiku-baseempty(api_base: "") answerstokenizer_type: huggingface_tokenizeron/utils/token_counter?call_endpoint=trueat 412890f withAnthropic CountTokens error: status=500, message=CountTokens processing error: /v1/messages/count_tokensin the log, andanthropic_apiat 272ccfd like base 2ffe6a1;/v1/messages/count_tokensanswers{"input_tokens":12}on every side because that endpoint falls back to the local tokenizer silently. WithANTHROPIC_API_BASE=https://gateway.invalidexported, chat onclaude-haikufails withCannot connect to host gateway.invalid:443on both sides, and at 272ccfd the count fails the same way and falls back (huggingface_tokenizer), where base 2ffe6a1 still counted through Anthropic's host (anthropic_api); the explicitclaude-haiku-basehostdeployment counts throughanthropic_apiand chats200 pongon every side. Chat on a federated deployment withapi_base: ""(D04b) reaches the token endpoint on both sides, since chat resolves the base before minting; only the count path handed the mint the raw value. The regression tests intest_anthropic_count_tokens_transformation.pyandtest_anthropic_wif.pyfail on the previous sourceObserved alongside, none caused by this PR:
/v1/messages/count_tokensmaps provider auth failures to 500 for non-federated deployments too, pre-existing; the federated leg is fixed as defect 8 and the general mapping is being fixed in fix(proxy): return provider auth errors from /v1/messages/count_tokens instead of masking or 500 #38902/model/newmodel can miss its/credentialscredential on the other worker, pre-existing (on the D rig, chat and messages on a federated model registered through a keycloak credential answer the federation error orMissing Anthropic API Keyacross six repeats of the same request, whichever worker picks it up)ANTHROPIC_API_KEYsilently wins over deployment federation, by designblockedflag on keys, pre-existingNinth and tenth staging merges, with #39613 folded in
Staging merged in twice more after 9036b90. The ninth merge resolved two conflicts (staging's heuristic_v2 auto-router limit from #39468 and the
write_row/ team-model test replacement), then the stacked #39613 (OpenAI federation from the Add Model and LLM Credentials forms, LIT-6869, approved on its own PR at fe6a535) was merged into this branch at 10d8449, and the tenth merge resolved one conflict intests/test_litellm/llms/custom_httpx/test_llm_http_handler.pyby keeping both sides' tests.make checkpasses at the merged tree, and the two pytest runs over every test directory the PR touches come back with only environment-bound reds (A, 6461 unit tests: 6458 passed, 13 skipped, 1 xfailed, and 2 that fail only because litellm's import-timeload_dotenv()reads the worktree's gitignored.env(test_aretrieve_batch_missing_api_key_raisessees a realANTHROPIC_API_KEY,test_openai_env_base[OPENAI_API_BASE]seesOPENAI_BASE_URLwin), both green with the file moved aside; B:test_dual_cache14 passed against a local Redis, the fourTestMigrateDeployAttemptAccountingtests are red on staging's own test-unit proxy-extras shard at a5b3bc8 too (LIT-6918), and the Responses e2e suite targets CI's booted proxy, so the CircleCI run at the tip covers it)Same rigs as the earlier review runs, at the merged tree (
--num_workers 2, Postgres, random ports, real spend): the S run's 59 legs and the D run's 23 legs answer the same statuses as at 9036b90, and the Q run's 25 legs differ only on Q11, aGET /credentialsracing a/model/newon the other worker, the pre-existing cross-worker credential window (LIT-6901). A D run booted with--detailed_debugproves staging's verbose-line redaction (#39526, #39538) on the federation fields: the kwargs line printsanthropic_keycloak_client_secret_ref='REDACTED',anthropic_issuer_signing_key_ref='REDACTED',anthropic_identity_token_file='REDACTED'andanthropic_keycloak_token_url='REDACTED', and the whole log has zero hits for the keycloak secret, either PEM header,client_secret=orsigning_key=, while the ids (organization, federation rule, keycloak client id, issuer url, subject and audience) print in the clearOpenAI federation legs at the merged tree (#39613's A to I, re-run)
The proxy for these legs strips
OPENAI_API_KEY,OPENAI_BASE_URL,OPENAI_IDENTITY_PROVIDER_ID,OPENAI_SERVICE_ACCOUNT_IDandOPENAI_IDENTITY_TOKEN_FILEafterload_dotenv()(the worktree's.envotherwise puts the static key back inside the proxy, see Caveats), and a control deploymentopenai/gpt-5.6with no key proves the env is clean: four chats in a row answerAuthenticationError: OpenAIException - The api_key client option must be set. With the federation credential (identity provider id, service account id, token file) stored throughPOST /credentials: Test Connect200 successwith OpenAI's rate-limit headers,POST /model/newthrough the credential200andGET /model/infoshowing onlylitellm_credential_name, chat200 ok, streaming200,POST /v1/responses200, embeddings200(spend rows onhttps://api.openai.com/v1, real cost), Discover Models200, the trio typed inline on/model/new401naming the field, a non-admin key on/credentials401, and the chat, streaming, Responses and embeddings legs again 45s later all200. The EU credential (api_base: https://eu.api.openai.com/v1): the chat sent 2s after/credentialson the two-worker proxy answers 500The api_key client option must be setwhen the worker that has not refreshed its credential list picks it up (LIT-6901's window), and on a fresh boot every chat reacheshttps://eu.api.openai.com/v1with the federation bearer and gets OpenAI'sThis endpoint is only accessible by projects with geography restrictions enabled(twice as upstream 401, twice as 403, proxied as such), the same result as #39613's leg I, whilegpt-wifon the default host answers 200 alongside. The dashboard walkthrough for the OpenAI forms is in #39613's description and stands: no dashboard file changed after itEleventh staging merge (37f2de1)
Staging moved to a53c550 (#39661) and the PR went CONFLICTING again on
litellm/llms/openai/workload_identity.pyand its test: #39652 moved the OpenAI host check intois_openai_backed_api_baseinlitellm/llms/openai/common_utils.py(it acceptsapi.openai.comand any*.api.openai.com, PrivateLink and regional hosts included), a superset of this branch's_OPENAI_REGIONAL_HOST_SUFFIXhandling, so the merge keeps staging's helper and drops the branch-local constants;_targets_openai_apinow returnsparsed.scheme == "https" and is_openai_backed_api_base(api_base). The resolved test module keeps staging'stest_openai_backed_api_base_allowsandtest_lookalike_or_plaintext_api_base_disablesin place of the branch's regional and lookalike cases.make checkpassed on the staged merge. The PR-touched unit-test groups ran locally in CI's shard shape (-n 4 --reruns 1):test_anthropic_skills_transformation.pyplustest_router.pygreen;batches integrations litellm_core_utils router_utils types models7763 passed and 1 failed (test_websearch_streaming_conversion);llms/anthropic llms/base_llm/auth llms/custom_httpx llms/openai3353 passed and 1 failed (test_aretrieve_batch_missing_api_key_raises);proxy21651 passed and 52 failed. Re-running those 54 reds serially with the local.envmoved aside leaves 23: 21test_semantic_tool_filter.pycases (No module named 'semantic_router', an optional dependency not installed locally) and 2test_guardrail_coverage.pysecret-detection cases (No module named 'detect_secrets'), and all of them fail identically on a pure staging tree at a53c550 through the same venv, so none is this branch'sQA at the eleventh-merge tree (37f2de1)
Same rigs as the tenth-merge run, on the merged tree with
--num_workers 2, Postgres and random ports. The OpenAI federation legs (port 29518) answer the same statuses as at 055eaee modulo ids and rate-limit counters: the keyless control answers 500 four times, B to F 200, G and H 401, the E rerun 200, and leg I (the EU credential) answers 200 on its first two calls and then OpenAI's 403This endpoint is only accessible by projects with geography restrictions enabledon chat, the same answer the EU re-run at 10d8449 got, so the request reacheshttps://eu.api.openai.com/v1with the federation bearer and OpenAI refuses the project, not the proxy. Every keyless spend row issuccesswith a realapi_baseand non-zero cost. The Anthropic S run (59 legs) and D run (23 legs) against a fresh Postgres 16 container answer the same leg counts and statuses as at 055eaee exceptS14b-credential-get, 404 at the tenth merge and 200 here:GET /credentials/by_namereads the worker-locallitellm.credential_list, so on two workers the GET answers 200 only when it lands on the worker that served thePOST /credentials, the pre-existing cross-worker credential window already listed under Caveats (the base answered 404 on the same leg, the original head 200, and nothing underlitellm/proxy/credential_endpoints/changed between the two merge tips). The D12 and D13 legs flip betweenMissing Anthropic API Keyand the keycloakConnectErrorfederation failure for the same reason, both 401, as in every earlier runCircleCI at 37f2de1
The legacy suites ran at the eleventh-merge tip after a
run-cilabel cycle, 40 of 42 jobs green,llm_translation_testing,local_testing_part1,local_testing_part2,pass_through_unit_testing,base_sdk_install,build_docker_database_imageandauth_ui_unit_testsamong them (upload-coveragedoes not run behind a red job). The two reds are staging's:litellm_router_unit_testingontest_no_linear_scans_in_router, red onlitellm_internal_stagingitself since #39468 and tracked as LIT-6911, ande2e_ui_testingontests/users/searchUsers.spec.ts(the search placeholder #39604 renamed, LIT-6919), neither touching a file this PR changes. At the tenth-merge tip 055eaee the same two jobs were the only reds oncepass_through_unit_testing, which flaked on a single run, passed on rerun. On the GitHub side all 33 required checks are green at both tips; the non-requiredosv-scanstays red on the pre-existing gitpython advisory inuv.lock, the same result on the base branchTwelfth staging merge, discovery api_base fix and empty-credential-value fix (7a9ee45, 56cf0cd, 1caa2fa)
7a9ee45 answers a Bugbot thread: an empty
api_baseon an OpenAI deployment is treated as unset when discovering models instead of being parsed as a host. 56cf0cd merges staging at c4e9076 (#39037's Anthropic error envelope, the SCIM default-team fixes, the Vertex passthrough default location, the router pre-call context-window count and the daily spend summary grouping) and resolveslitellm/router.py,llm_passthrough_endpoints.pyand their two test modules by hand, keeping both sides' tests; this PR's ownrouter.pydelta stays the five-line secret-pointer guard. 1caa2fa keeps an empty credential value empty after decryption:decrypt_value_helper(...) or valuehanded the stored ciphertext back for a value that decrypts to the empty string, so a field cleared through the wizard came back as its encrypted form on the next read, anddecrypted_or_storednow falls back to the stored value only when decryption answersNone(a config.yaml value that was never encrypted). 5e5c436 is the wizard commit described below. At the tip the five pytest modules these commits touch pass locally (384 tests acrosstest_anthropic_wif.py,test_public_endpoints.py,test_credential_hydration.py,test_proxy_config.pyandtest_openai_workload_identity.py), the 22 vitest cases inanthropicFederation.test.tsandAddProviderPanel.integration.test.tsxpass, and every required GitHub check, the lint and code-quality jobs included, is greenThirteenth staging merge (669f028)
Staging moved to 2e73400 (#39399's Python version matrix, the per-worker admission control 503s from #39352, the SSE keepalives from #39273 and the pattern-router eviction from #39664) and the PR went CONFLICTING on
tests/test_litellm/llms/anthropic/experimental_pass_through/messages/test_mcp_handler.py: #39399's cross-version fix rewrote the two parenthesizedwith (...)blocks this branch had reformatted, and the merge keeps staging's form of both. The five code files both sides touch (_health_endpoints.py,llm_passthrough_endpoints.py,proxy_server.py,router.py,utils.py) change in different functions on each side, so no line this PR adds moved.make checkpasses on the merge and the seven pytest modules that hold the federation code pass at the merged tree (594 passed). The fail-closed D rig from the 5e5c436 run was driven again at 669f028 (--num_workers 2, Postgres 16, port 41873): the same 22 legs answer the same statuses, every chat and/v1/messagesthrough a LiteLLM-signed credential with no ids answers 401 naminganthropic_federation_rule_id and anthropic_organization_id, the rule-id-only legs name the singular field, the token-endpoint legs carry the denial hints, and the two calls that land in the cross-worker credential window (LIT-6901) and answer the genericMissing Anthropic API Key401 are the first two token-endpoint legs here where they were two of the fail-closed legs at 5e5c436. 3a46fa2, the credential-modal commit described below, follows this mergeFourteenth staging merge (4592e71)
Staging moved to d23bec8 (122 commits, among them #39801's health probe by named credential, the chronic-test repairs in dbf8fe0 and a5a7867, and 7d3b03d's redis event-loop stall test) and 3a46fa2 merged clean with no conflicts. On the nine files both sides touch (
_health_endpoints.py,proxy_server.py,router.py,types/utils.py,coverage_allowlist.txt,test_streaming_iterator.py,test_public_endpoints.py,networking.tsx,schema.d.ts) this PR's added and removed lines are the same before and after the merge, so the merge changed nothing this PR does.make checkpasses at the tip, the mapped suites pass there (62 pytest across the credential endpoints, credential hydration,TestDiscoverProviderModelsand the redis stall test, 43 vitest acrossCredentialModal,credential_form_helpersand the models page), and the fail-closed legs re-driven on a fresh two-worker rig at port 40886 (22 legs) answer 401 on every chat and/v1/messagescall that is meant to fail closed and 200 on every credential, model and read stepFifteenth staging merge (216da96)
Staging moved to 7573632 (142 commits, among them #39461's
Anyreduction, #39780's proxy-infra Python 3.10 fix, #39848's guardrail cost field for the Datadog integration, #39842's spend-log test for LIT-6949, #35853's retry policy for 503s and #36575's stateless MCP follow-up) and 4592e71 went CONFLICTING on one import line oflitellm/llms/anthropic/files/transformation.py: #39461 droppedAnyfrom thetypingimport and typedparamsasFinal[dict[str, str]], while this branch addsfrom collections.abc import Mappingon the neighbouring line for_finalize_headers, so the merge keeps staging's narrowed import and the branch'sMappingimport together. On the fourteen files both sides touch (batch_utils.py, the Anthropic messages and files transformations,model_management_endpoints.py,proxy_server.py,fallback_event_handlers.py,types/router.py,types/utils.py,utils.py, three test modules,networking.tsxandschema.d.ts) this PR's added and removed lines are the same before and after the merge, so the merge changed nothing this PR does.make checkpasses at the tip and the mapped suites pass there (97 pytest across the credential endpoints, credential hydration,TestDiscoverProviderModels, the Anthropic files transformation and the redis stall test, 43 vitest acrossCredentialModal,credential_form_helpersand the models page). Staging has since moved to aea5358 (#39843, #39844), whose one shared filelitellm/llms/openai/responses/transformation.pymerges clean with this tip (git merge-tree), so the PR reads MERGEABLE there without another merge.Bugbot finding at 216da96, fixed at 82afbbd
Bugbot's pass on the fifteenth merge found one issue:
_resolve_identity_sourceinlitellm/llms/anthropic/wif.pyfiltered onlyNoneout of a deployment's federation params, so a blank string left behind by an API client's PATCH counted as set, a blankanthropic_keycloak_client_idon a LiteLLM-signed deployment tripped the foreign-field check, and a blankanthropic_issuer_ttl_secondsreached pydantic's integer field and failed validation. The fix treats""as unset in both the resolver and_build_variant, which the JWKS export route calls too, so a blank optional field on a saved credential no longer 400s the public key block. Two regression tests cover it,test_blank_optional_and_foreign_fields_count_as_unseton the resolver andtest_jwks_export_treats_blank_optional_fields_as_unseton the export route, both red on 216da96 (the first with the foreign-field error, the second with the 400) and green at the fix; the non-admin key-presence gate onPOST /credentialsis an authorization rule and stays as it was.make checkpasses at the tip and the two mapped test files pass there (141 tests). The merge base is unchanged at 7573632 and the tip still merges clean with stagingRegister issuer step, before at 1caa2fa and after at 5e5c436
Superseded: the wizard these legs drove is removed at 3a46fa2, see the next section. The backend legs (fail-closed discovery, the refused workspace id, the variant metadata) still hold and are re-run there
The proxy runs from the PR worktree with
ISSUER_SIGNING_KEY_PEMholding a P-256 PEM,ANTHROPIC_API_KEYunset, master keysk-1234, and the dashboard dev server on port 3000. No federation rule exists for this rig, so the ids are placeholders and every leg that reaches Anthropic gets its own400for the malformedfdrl_. The before side swaps only the two backend files this commit changes into the running proxy; the dashboard before side is the code as it stoodBefore (1caa2fa)
Discover before the Console has generated the ids
curl -s localhost:4000/credentials -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{"credential_name":"wif-qa","credential_info":{"custom_llm_provider":"anthropic"},"credential_values":{"anthropic_identity_source":"internal_issuer","anthropic_issuer_url":"https://litellm.local/qa","anthropic_issuer_subject":"litellm-wif-demo","anthropic_issuer_audience":"https://api.anthropic.com","anthropic_issuer_signing_key_ref":"os.environ/ISSUER_SIGNING_KEY_PEM"}}', observed{"success":true,"message":"Credential created successfully"}curl -s localhost:4000/credentials/wif-qa/jwks -H "Authorization: Bearer sk-1234" | jq -c '.keys[0] | {kty, crv, alg, use}', observed{"kty":"EC","crv":"P-256","alg":"ES256","use":"sig"}curl -s -w '\nHTTP %{http_code}' localhost:4000/provider/models/discover -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{"custom_llm_provider":"anthropic","litellm_credential_name":"wif-qa"}', observed{"detail":{"error":"Model discovery failed: ANTHROPIC_API_BASE/ANTHROPIC_BASE_URL or ANTHROPIC_API_KEY/ANTHROPIC_AUTH_TOKEN (or workload identity federation via ANTHROPIC_FEDERATION_RULE_ID/ANTHROPIC_ORGANIZATION_ID/ANTHROPIC_IDENTITY_TOKEN_FILE) is not set. Please set the environment variable, to query Anthropic's/modelsendpoint."}}andHTTP 502: the credential's identity source is skipped and the message points at environment variables the admin never meant to useDiscover once the ids are saved
curl -s -X PATCH localhost:4000/credentials/wif-qa -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{"credential_name":"wif-qa","credential_info":{"custom_llm_provider":"anthropic"},"credential_values":{"anthropic_organization_id":"00000000-0000-4000-8000-000000000000","anthropic_federation_rule_id":"fdrl_qa_fake","anthropic_service_account_id":"svac_qa_fake"}}', observed{"success":true,"message":"Credential updated successfully"}curl -s -w '\nHTTP %{http_code}' localhost:4000/provider/models/discover -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{"custom_llm_provider":"anthropic","litellm_credential_name":"wif-qa"}', observed{"detail":{"error":"Model discovery failed: litellm.AuthenticationError: Anthropic workload identity federation failed. The token endpoint returned HTTP 400: error: invalid_request_error - federation_rule_id is not a well-formed fdrl_ tagged ID"}}andHTTP 502: the placeholder rule id reached Anthropic's token endpointWorkspace id typed into the discover body
curl -s -w '\nHTTP %{http_code}' localhost:4000/provider/models/discover -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{"custom_llm_provider":"anthropic","litellm_credential_name":"wif-qa","anthropic_workspace_id":"default"}', observed{"error":{"message":"Authentication Error, Rejected Request: anthropic_workspace_id is a server-owned workload identity federation parameter and cannot be set in a request body; configure it on the deployment instead. On the Bedrock Claude Platform route, pass workspace_id or aws_workspace_id instead.","type":"auth_error","param":"None","code":"401"}}andHTTP 401Credential form metadata for the LiteLLM-signed variant
curl -s localhost:4000/public/providers/fields | jq -c '.[] | select(.provider=="Anthropic") | .credential_variants.variants[] | select(.id=="wif_internal_issuer") | {optional_field_keys, last_four: (.field_keys[-4:])}', observed{"optional_field_keys":["anthropic_federation_rule_id"],"last_four":["anthropic_issuer_subject","anthropic_issuer_audience","anthropic_issuer_ttl_seconds","anthropic_issuer_signing_key_ref"]}: the Authentication step demands Organization ID before the Console can produce it, and the variant has no Service Account ID or Workspace ID field at allDashboard wizard
401(workspace_id_requiredin the Console's authentication history), and adding the Workspace ID means Back twice, a re-save, and retyping the rule idAfter (5e5c436)
Discover before the Console has generated the ids
curl -s localhost:4000/credentials -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{"credential_name":"wif-qa","credential_info":{"custom_llm_provider":"anthropic"},"credential_values":{"anthropic_identity_source":"internal_issuer","anthropic_issuer_url":"https://litellm.local/qa","anthropic_issuer_subject":"litellm-wif-demo","anthropic_issuer_audience":"https://api.anthropic.com","anthropic_issuer_signing_key_ref":"os.environ/ISSUER_SIGNING_KEY_PEM"}}', observed{"success":true,"message":"Credential created successfully"}curl -s localhost:4000/credentials/wif-qa/jwks -H "Authorization: Bearer sk-1234" | jq -c '.keys[0] | {kty, crv, alg, use}', observed{"kty":"EC","crv":"P-256","alg":"ES256","use":"sig"}curl -s -w '\nHTTP %{http_code}' localhost:4000/provider/models/discover -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{"custom_llm_provider":"anthropic","litellm_credential_name":"wif-qa"}', observed{"detail":{"error":"Model discovery failed: litellm.AuthenticationError: anthropic_identity_source is 'internal_issuer', but anthropic_federation_rule_id and anthropic_organization_id are not set. Copy them from the federation rule's detail page under Settings > Workload identity in the Claude Console, or set ANTHROPIC_FEDERATION_RULE_ID and ANTHROPIC_ORGANIZATION_ID."}}andHTTP 502Discover once the ids are saved
curl -s -X PATCH localhost:4000/credentials/wif-qa -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{"credential_name":"wif-qa","credential_info":{"custom_llm_provider":"anthropic"},"credential_values":{"anthropic_organization_id":"00000000-0000-4000-8000-000000000000","anthropic_federation_rule_id":"fdrl_qa_fake","anthropic_service_account_id":"svac_qa_fake"}}', observed{"success":true,"message":"Credential updated successfully"}curl -s -w '\nHTTP %{http_code}' localhost:4000/provider/models/discover -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{"custom_llm_provider":"anthropic","litellm_credential_name":"wif-qa"}', observed{"detail":{"error":"Model discovery failed: litellm.AuthenticationError: Anthropic workload identity federation failed. The token endpoint returned HTTP 400: error: invalid_request_error - federation_rule_id is not a well-formed fdrl_ tagged ID"}}andHTTP 502: the placeholder rule id reached Anthropic's token endpointWorkspace id typed into the discover body
curl -s -w '\nHTTP %{http_code}' localhost:4000/provider/models/discover -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{"custom_llm_provider":"anthropic","litellm_credential_name":"wif-qa","anthropic_workspace_id":"default"}', observed{"error":{"message":"Authentication Error, Rejected Request: anthropic_workspace_id is a server-owned workload identity federation parameter and cannot be set in a request body; configure it on the deployment instead. On the Bedrock Claude Platform route, pass workspace_id or aws_workspace_id instead.","type":"auth_error","param":"None","code":"401"}}andHTTP 401Credential form metadata for the LiteLLM-signed variant
curl -s localhost:4000/public/providers/fields | jq -c '.[] | select(.provider=="Anthropic") | .credential_variants.variants[] | select(.id=="wif_internal_issuer") | {optional_field_keys, last_four: (.field_keys[-4:])}', observed{"optional_field_keys":["anthropic_organization_id","anthropic_federation_rule_id"],"last_four":["anthropic_organization_id","anthropic_federation_rule_id","anthropic_service_account_id","anthropic_workspace_id"]}: the Authentication step no longer needs the ids Anthropic generates later, and the variant carries all four in the order the Register issuer step shows themDashboard wizard
Driven in headless Chromium against the dev server, then repeated by hand for the screenshots
sk-1234, open http://localhost:3000/models-and-endpoints/, Add Provider, Anthropic, namewif-qa-ui, NextWorkload Identity Federation (LiteLLM-signed), Issuer URLhttps://litellm.local/qa, Subjectlitellm-wif-demo, Audiencehttps://api.anthropic.com, Signing Key Referenceos.environ/ISSUER_SIGNING_KEY_PEM, every id left blank, Save credential: the toast says the credential is saved"kty": "EC") with Copy JWKS, then Organization ID, Federation Rule ID, Service Account ID and Workspace ID, each with a hint on where it lives in the Console and when it is needed, the lineStill needed before discovery: Organization ID, Federation Rule ID.and Next disabled00000000-0000-4000-8000-000000000000, Federation Rule IDfdrl_qa_fakeand Service Account IDsvac_qa_fake: the line disappears and Next enablesDiscovery failedwithModel discovery failed: litellm.AuthenticationError: Anthropic workload identity federation failed. The token endpoint returned HTTP 400: error: invalid_request_error - federation_rule_id is not a well-formed fdrl_ tagged ID, andGET http://localhost:4000/credentials/by_name/wif-qa-uialready carries the three idsdefaultthere, Back again: the Authentication step shows all four values, Save changes, and the Register issuer step still showsdefault400, and the credential now carries all four ids. Delete the credential from LLM Credentials afterwardsOne home for provider credentials, before at 5e5c436 and after at 3a46fa2
The proxy runs from the PR worktree with
ISSUER_SIGNING_KEY_PEMholding a P-256 PEM,ANTHROPIC_API_KEYunset, master keysk-1234, and the dashboard dev server on port 3000. No federation rule exists for this rig, so the ids are placeholders and every leg that reaches Anthropic gets its own400for the malformedfdrl_. The credential in the curl legs is created the way the modal creates it, withcustom_llm_providerin the dashboard's display casingBefore (5e5c436)
Models page
sk-1234, open http://localhost:3000/models-and-endpoints/: the tab strip is All Models, Add Model, Auto-Routers, Add Provider, LLM Credentials, Pass-Through Endpoints, Health Status, Model Retry Settings, Model Group Alias, Model Access Group Budgets, Price Data Reload. Add Provider and LLM Credentials both create an Anthropic federation credential; the JWKS and model discovery live only in the wizardJWKS and discovery for a credential the modal created
curl -s localhost:4000/credentials -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{"credential_name":"wif-qa-modal","credential_info":{"custom_llm_provider":"Anthropic"},"credential_values":{"anthropic_identity_source":"internal_issuer","anthropic_issuer_url":"https://litellm.local/qa","anthropic_issuer_subject":"litellm-wif-demo","anthropic_issuer_audience":"https://api.anthropic.com","anthropic_issuer_signing_key_ref":"os.environ/ISSUER_SIGNING_KEY_PEM"}}', observed{"success":true,"message":"Credential created successfully"}curl -s -w '\nHTTP %{http_code}' localhost:4000/credentials/wif-qa-modal/jwks -H "Authorization: Bearer sk-1234", observed{"detail":{"error":"No anthropic credential named 'wif-qa-modal'."}}andHTTP 404: the route compared the storedAnthropicagainstanthropicand missedcurl -s -w '\nHTTP %{http_code}' localhost:4000/provider/models/discover -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{"custom_llm_provider":"anthropic","litellm_credential_name":"wif-qa-modal"}', observed{"detail":{"error":"Credential 'wif-qa-modal' is configured for provider 'Anthropic', not 'anthropic'."}}andHTTP 400: same comparison, so only curl-created credentials could be discovered by namecurl -s -X DELETE localhost:4000/credentials/wif-qa-modal -H "Authorization: Bearer sk-1234", observed{"success":true,"message":"Credential deleted successfully"}After (3a46fa2)
Models page
Driven in headless Chromium against the dev server. The steps below are the ones to repeat by hand for the screenshots
sk-1234, open http://localhost:3000/models-and-endpoints/: the tab strip is All Models, Add Model, Auto-Routers, LLM Credentials, Pass-Through Endpoints, Health Status, Model Retry Settings, Model Group Alias, Model Access Group Budgets, Price Data Reload. No Add ProviderFill in the credential values first.wif-qa-cred, Provider Anthropic, Authentication methodWorkload Identity Federation (LiteLLM-signed), Issuer URLhttps://litellm.local/qa, Subjectlitellm-wif-demo, Audiencehttps://api.anthropic.com, Signing Key Referenceos.environ/ISSUER_SIGNING_KEY_PEM, ids blank: Test Connection stays disabled with the hintAdd the credential first, then test it from Edit. Only an API key and API base can be tested before saving., Add Credential: the toast saysCredential added successfullyandGET http://localhost:4000/credentials/by_name/wif-qa-credcarries the issuer fields withanthropic_identity_source: internal_issuerPublic JWKSwith the Console instructions, the key set ("kty": "EC") and a Copy JWKS button; Test Connection is enabledModel discovery failed: litellm.AuthenticationError: anthropic_identity_source is 'internal_issuer', but anthropic_federation_rule_id and anthropic_organization_id are not set. Copy them from the federation rule's detail page under Settings > Workload identity in the Claude Console, or set ANTHROPIC_FEDERATION_RULE_ID and ANTHROPIC_ORGANIZATION_ID., so the request reached the identity source rather than a static key00000000-0000-4000-8000-000000000000, Federation Rule IDfdrl_qa_fake, Service Account IDsvac_qa_fake: Test Connection disables with the hintUpdate the credential first. Test Connection checks the saved values., Update Credential: the toast saysCredential updated successfullyand the credential carries the three idsModel discovery failed: litellm.AuthenticationError: Anthropic workload identity federation failed. The token endpoint returned HTTP 400: error: invalid_request_error - federation_rule_id is not a well-formed fdrl_ tagged ID, the placeholder rule id reached Anthropic's token endpoint. Delete the credential from the row menu afterwardsJWKS and discovery for a credential the modal created
curl -s localhost:4000/credentials -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{"credential_name":"wif-qa-modal","credential_info":{"custom_llm_provider":"Anthropic"},"credential_values":{"anthropic_identity_source":"internal_issuer","anthropic_issuer_url":"https://litellm.local/qa","anthropic_issuer_subject":"litellm-wif-demo","anthropic_issuer_audience":"https://api.anthropic.com","anthropic_issuer_signing_key_ref":"os.environ/ISSUER_SIGNING_KEY_PEM"}}', observed{"success":true,"message":"Credential created successfully"}curl -s -w '\nHTTP %{http_code}' localhost:4000/credentials/wif-qa-modal/jwks -H "Authorization: Bearer sk-1234", observed{"keys":[{"crv":"P-256","kty":"EC","x":"...","y":"...","use":"sig","alg":"ES256","kid":"..."}]}andHTTP 200curl -s -w '\nHTTP %{http_code}' localhost:4000/provider/models/discover -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{"custom_llm_provider":"anthropic","litellm_credential_name":"wif-qa-modal"}', observed{"detail":{"error":"Model discovery failed: litellm.AuthenticationError: anthropic_identity_source is 'internal_issuer', but anthropic_federation_rule_id and anthropic_organization_id are not set. Copy them from the federation rule's detail page under Settings > Workload identity in the Claude Console, or set ANTHROPIC_FEDERATION_RULE_ID and ANTHROPIC_ORGANIZATION_ID."}}andHTTP 502: the request reached the identity sourcecurl -s -X DELETE localhost:4000/credentials/wif-qa-modal -H "Authorization: Bearer sk-1234", observed{"success":true,"message":"Credential deleted successfully"}Test Connection's inline path, what the add-mode button sends for an API key
curl -s -w '\nHTTP %{http_code}' localhost:4000/provider/models/discover -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{"custom_llm_provider":"openai","api_key":"sk-not-a-real-key"}', observed{"detail":{"error":"Model discovery failed: Failed to get models: { \"error\": { \"message\": \"Incorrect API key provided: sk-not-a*****-key. ...\", \"type\": \"invalid_request_error\", \"code\": \"invalid_api_key\" } }"}}andHTTP 502, the text the modal shows in its red alertcurl -s localhost:4000/provider/models/discover -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{"custom_llm_provider":"openai","api_key":"'"$OPENAI_API_KEY"'"}' | jq -c '{models: (.models | length), first3: .models[:3]}', observed{"models":132,"first3":["text-embedding-ada-002","whisper-1","gpt-3.5-turbo"]}, the list the modal summarizes in its green alertFederation field typed into the discover body
curl -s -w '\nHTTP %{http_code}' localhost:4000/provider/models/discover -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{"custom_llm_provider":"anthropic","litellm_credential_name":"wif-qa-modal","anthropic_workspace_id":"default"}', observed{"error":{"message":"Authentication Error, Rejected Request: anthropic_workspace_id is a server-owned workload identity federation parameter and cannot be set in a request body; configure it on the deployment instead. On the Bedrock Claude Platform route, pass workspace_id or aws_workspace_id instead.","type":"auth_error","param":"None","code":"401"}}andHTTP 401: this is why add mode refuses to test a federation credential inline and points at EditQA at 5e5c436
Same rigs as the eleventh-merge run, on the tip with
--num_workers 2, Postgres 16 in Docker and random ports. The Anthropic S run (59 legs) and D run (23 legs) answer the same statuses as at 37f2de1, the differences being reply text, generated ids, and S12's provider-fields output, where the four issuer tooltips changed andoptional_field_keysfor the LiteLLM-signed variant now lists the organization and rule ids. The OpenAI federation legs (port 58122) answer the same statuses as before modulo the two-worker window: the keyless control 500 four times, B to D 200, E's first pass chat 500 and Responses 401 on the worker that had not refreshed its credential list while embeddings answered 200, the E rerun 45s later 200 three times, F 200, G and H 401, and leg I (the EU credential) 200, 200 and OpenAI's 403 geography refusal on chatThe fail-closed change was driven before and after on the D rig with a LiteLLM-signed credential that has no organization or rule id (
POST /credentialswithanthropic_identity_source: internal_issuer, a/model/newthrough it, then four chats and two/v1/messages). At 1caa2fa all six calls answer the generic 401Missing Anthropic API Key. At 5e5c436 four of the six answer 401anthropic_identity_source is 'internal_issuer', but anthropic_federation_rule_id and anthropic_organization_id are not set. Copy them from the federation rule's detail page under Settings > Workload identity in the Claude Console, or set ANTHROPIC_FEDERATION_RULE_ID and ANTHROPIC_ORGANIZATION_ID.and the other two, picked up by the worker that had not refreshed its credential list yet, still answer the generic message (Caveats). A PATCH adding both ids the way the wizard's Next does turns every chat into the token endpoint's 401, at the tip with the new hints (Anthropic answers every denied exchange with the same 401; the reason (for example workspace_id_required or jti_reused) is only shown in the Claude Console under Settings > Workload identity, in the rule's authentication history.plus the workspace and service account hints, four of four; at 1caa2fa the same 401 carries no hint). A PATCH withcredential_values_to_delete: ["anthropic_federation_rule_id"]then turns the chats into the singularbut anthropic_federation_rule_id is not setfour times out of four at the tip (the generic message four times out of four at 1caa2fa), andGET /credentials/by_nameshows the organization id kept and the rule id gone on both sidesCircleCI at 5e5c436
The legacy suites ran at the tip after a
run-cilabel cycle (workflow 3781f6f6): 40 of 43 jobs green,llm_translation_testing,local_testing_part1,local_testing_part2,pass_through_unit_testing,proxy_e2e_anthropic_messages_tests,auth_ui_unit_tests,e2e_ui_testing_server_root_pathandusing_litellm_on_windowsamong them, withupload-coveragenot run behind the reds. The two reds are the same staging reds as at 37f2de1:litellm_router_unit_testingontest_no_linear_scans_in_router, which namesconfig_deployments()andheuristic_v2_router_limit_violation(), the twomodel_listscans staging's #39468 added and which fail the same way onlitellm_internal_staging(LIT-6911), ande2e_ui_testingontests/users/searchUsers.spec.ts(narrows the table to the matching email, the placeholder #39604 renamed, LIT-6919), a file this PR does not touch. On the GitHub side all 33 required checks are green; the documentation and code-quality jobs went red once on the fleet-wide docs rename (LIT-6929) and are green on rerun, andosv-scanstays red on the pre-existing gitpython advisory, the same result on the base branchQA at 4592e71
Same rigs, head at the tip and base at the new merge base d23bec8, each with
--num_workers 2on random ports (head S 34974 and D 44914, base S 41256 and D 43632, OpenAI federation 33026, dashboard proxy 47657 with the Next dev server on 37887). The shared local Postgres on 5432 ran out of connections part way through the first head S run and took that proxy down, so every rig moved to a per-run Postgres 16 container on 27801 and the run started over. The Anthropic S run (59 legs) and D run (23 legs) answer the same statuses on head and base as at 5e5c436, the differences being reply text and generated ids, and the health-check legs differ between head and base only where the head adds a route or closes a hole: the fail-closed 401 on H06, the JWKS route answering 200 (H13) and a 404 with a reason (H07) where the base answers 405, discovery answering 200 with 11 models (H08) and, for placeholder federation ids, the 502 that relays Anthropic's 400 (H14) where the base 404s, plus a display-casedcustom_llm_providertyped straight into a discover body (something the dashboard never sends, it always posts the canonical id) getting a 400 where the base 404s. The OpenAI federation legs ran with a re-minted 24 hour subject token after the previous one had expired (an expired token reads like a regression and is not one): the 15 legs answer the statuses recorded at 5e5c436 except two that improved, E's first pass answering 200 on Responses where it answered 401 and D's/model/infolisting both keyless deployments where it listed none, both inside the LIT-6901 two-worker window. Dashboard leg at the head: the Next dev server on port 37887 pointed at the head proxy on 47657 loaded /models-and-endpoints/, the LLM Credentials tab rendered its Add Credential button, and the Add New Credential modal opened with its provider select, with zero console errors and zero 4xx/5xx requests (?tab=llm-credentialsis not a deep link on this page, the tab is clicked)CircleCI at 4592e71
The legacy suites ran at the tip (workflow c93d2804): 43 of 44 jobs green,
litellm_router_unit_testing,e2e_ui_testing,e2e_ui_testing_server_root_path,llm_translation_testing,local_testing_part1,local_testing_part2andusing_litellm_on_windowsamong them, so the two staging reds recorded at 5e5c436 are green here. The one red isproxy_store_model_in_db_testsontest_missing_model_parameter_curl[chat], red onlitellm_internal_stagingitself since its 03:09 UTC pipeline on 2026-09-04; LIT-6949 tracks it. On the GitHub side every required check is green (33 of 33).proxy-infra / Run tests (Python 3.10)was red onTestNumericFormFields::test_qualifiers_and_optionality_are_unwrapped, a non-required job of #39399's Python matrix that was red on staging's own runs at d23bec8 and after (LIT-6947, fixed by #39780); a close/reopen on 2026-09-05 refreshed the merge ref to a110b1c (this tip into df3b8a6) and the job is green thereQA at 216da96
Same rigs at the fifteenth merge, head at the tip and base at the new merge base 7573632, two workers each on random ports with the per-run Postgres 16 container on 27801 (head S 52568 and D 55848, base S 31493 and D 46115, OpenAI federation 40326, fail-closed 54615, dashboard proxy 52328 with the Next dev server on 38042). The 59 S legs and 23 D legs answer the same statuses on head and base as at 4592e71: the S differences are the nine legs where the head adds a route or closes a hole (S10 401 for a server-owned field in a body where the base 400s, S11 and S26a and S26b 200 on federated deployments with a bogus or empty base where the base 400s, S13 200 and S13b 401 on discovery where the base 404s, S15e and S15h 401 for a team admin or the proxy admin posting federation fields inline where the base 200s, and S23's cleanup 400 on the deployment the head refused to create) plus S14b, where the head's
GET /credentials/by_nameanswered 404 on the worker that had not synced the credential yet (the LIT-6901 window, the same leg the 4592e71 run explains). The 16 health-check S legs and 11 D legs differ from the base exactly as before (H06 401, H07 404, H08 200 twice, H08b 400, H13 200 twice, H14 502 twice against the base's 200, 405, 404, 404, 405, 404) and the 22 fail-closed legs answer as at 4592e71: 401 naming the missing ids on every chat and messages call, 200 on the credential edits around them. The OpenAI federation run: 16 keyless calls 200 (store credential, Test Connect, Add Model, chat, streaming, Responses and embeddings, discovery through the credential with and without the empty api_base), the two inline-trio and non-admin legs 401, the control keyless static deployment 500 four times as it must, and one chat in the first pass of E plus the EU leg's chat 500 inside the two-worker window. The EU leg was driven twice more at the tip: the immediate variant split 2 to 3 between OpenAI's geography refusal and the missing-key 500 across five calls, and with the credential stored 20s before the model all four calls reachedeu.api.openai.comand got OpenAI's geography refusal, so the window is LIT-6901's (the router resolveslitellm_credential_nameper request from the worker-local list, upstream code this PR does not touch) and the repro is noted on that ticket. Dashboard leg: the dev server on 38042 pointed at the head proxy on 52328 loaded /models-and-endpoints/, the LLM Credentials tab rendered Add Credential, the modal opened with its provider select, zero console errors and zero 4xx/5xx requestsQA at 82afbbd
Same rigs at the Bugbot fix, head at the tip and base at the unchanged merge base 7573632, two workers each on random ports with the per-run Postgres 16 container on 27801 (head S 49107 and D 50944, base S 28968 and D 38770, OpenAI federation 27307, fail-closed 34735, dashboard proxy 31256 with the Next dev server on 48870). Every leg answers the status it answered at 216da96: the 59 S and 23 D legs on the head are status-identical to the previous tip's run, the head-vs-base differences are the same nine legs, and the base run moved only on S14b, its
GET /credentials/by_nameanswering 404 in the same LIT-6901 window the head hit at the previous tip (the window is upstream's, so it shows on either side). The 27 health-check legs differ from the base as before, with the twoby_namereads right afterPOST /credentials(H03b, H12b) landing in that window on this run, 404 on the worker that had not synced yet where the previous run's reads hit the worker that had. The 22 fail-closed legs are status-identical, and the OpenAI federation run is the same 16 keyless 200s, the same two 401s and the control's four 500s, with E's first pass clean this time and the EU leg's chat still inside the window. Dashboard leg: the first attempt's Next dev server answered 404 on every route,/login/included, serving the not-found page out of the persistent Turbopack cache under.next/devthat the previous leg's teardown had left half-written (nothing in this PR, the dev server is the same tree at both tips); with that directory cleared the dev server on 48870 pointed at the head proxy on 31256 loaded /models-and-endpoints/, the LLM Credentials tab rendered Add Credential, the modal opened with its provider select, zero console errors and zero 4xx/5xx requestsCircleCI at 82afbbd
The legacy suites ran at the tip (workflow 56bc1c76, 05:54Z to 06:07Z): all 43 jobs green,
proxy_store_model_in_db_testsamong them, since #39842's fix for LIT-6949 sits in the merge base now, andlitellm_router_unit_testing,e2e_ui_testing,e2e_ui_testing_server_root_path,llm_translation_testing,local_testing_part1,local_testing_part2andusing_litellm_on_windowswith it. On the GitHub side every check is green, the 33 required ones included, withproxy-infra / Run tests (Python 3.10)green on the tip directly (the tip runs against staging at or after #39780, so no close/reopen was needed). Bugbot reviewed the tip at 05:56Z and found no new issues; Greptile's only status is its 03:08Z "Too many files" skipType
New Feature
Caveats (if any)
Medium
OPENAI_API_KEYin the proxy env orapi_keyon the deployment disables it silently, the same precedence the Anthropic side keeps as a Low below; the token-file field's tooltip says so, and a mixed fleet needs the env var goneproxy_cli.pycallsload_dotenv(), so a.envin the proxy's working directory carryingOPENAI_API_KEY(orANTHROPIC_API_KEY) puts the static key back even when the shell unset it; a no-static-secrets proxy needs that file clean too, and the docs PR gets the notelitellm_credential_namedeployment (LIT-6901), but the UI only offers OpenAI federation through a credential, so every federated deployment sits in that window; seen on the EU leg above, 2s after/credentials, and again at 216da96, where the same second split between the missing-key 500 and OpenAI's geography refusal and every call reached OpenAI once both workers held the credential (repro noted on LIT-6901)Missing Anthropic API Key401 instead of the fail-closed message naming the missing ids: 2 of the 6 calls sent right afterPOST /credentialson the two-worker rig; once both workers hold the credential every call names the idseu.api.openai.comandus.api.openai.comis exercised only up to the upstream geography refusalLow
TestServiceAccountIdIsOptional), and the workspace id is required only when the rule is enabled in more than one workspace, so both stay optional with that guidance in their tooltipsfdrl_,svac_andwrkspc_ids is not driven live at 3a46fa2, the rig has no federation rule; the placeholder ids reach Anthropic's token endpoint and get its400, and theworkspace_id_requiredrecovery (fill Workspace ID, Update Credential, Test Connection) is covered by the unit tests and the headless run, not by a Console-backed exchange401above), so the button stays disabled with a hint to add the credential and test it from Editapi_keyor an ambientANTHROPIC_API_KEY/ANTHROPIC_AUTH_TOKENtakes precedence over the deployment's federation fields, silently (S11): the same precedence every provider uses, and the docs PR gets a note; unset the env var on a no-static-secrets proxyLITELLM_OIDC_ALLOWED_CREDENTIAL_DIRS, and the token exchange only talks to api.anthropic.com unless the operator adds hosts toLITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS, so a gateway or test double needs that setPOST /model/newfor the proxy admin too, not only for team admins: the sanctioned paths are a named credential (POST /credentialsthenlitellm_credential_name) or config.yaml. Editing a federated deployment stays proxy-admin only, including changing its api_base or attaching a credential that carries federation valuesCredentialItem-InputandCredentialItem-Outputinstead of oneCredentialItem, becausecredential_values_to_deleteis excluded from serialization. The wire shape is unchanged; a generated client pinned to the old schema name regenerates. Keeping one name would mean splitting the request and response models, more churn than the rename costsapi_basegets a different synthesized per-request model id than before, since the federation disable flag now rides those params. Cosmetic, the id is not persistedanthropic_workspace_idis refused in a request body even on a Bedrock deployment, where it is harmless; it stays refused for uniformitytags, so it cannot change a federation field/public/agents/fieldsdoes not list the federation variants, so the agents page offers API key only; the models page doesapi_basenow count tokens throughANTHROPIC_API_BASE/ANTHROPIC_BASE_URLthe way chat does; a gateway there that serves/v1/messagesbut not/v1/messages/count_tokensgets the logged local-tokenizer fallback where the base counted through Anthropic's host directly. Special-casing the count would put it on a host the operator pointed the proxy away from, so it staysANTHROPIC_AUTH_TOKEN: with only that set it logs the local-tokenizer fallback, exactly as the base branch did, since the base readapi_key/ANTHROPIC_API_KEYand nothing else. The guard that skips federation in that case keeps the count's credential tier matching chat's_static_auth_header, which also prefers the auth token over federation, so the two surfaces never authenticate as different principals for one request. Sending that token as a Bearer on the count is a separate gap this PR neither introduces nor widensconfig.yamlanswers 404 and leaves it serving, staging's fix(proxy): 404 a credential delete that matched nothing, and raise instead of return #36260 rule carried through the federation admin gate; the dashboard's delete button on such a row shows the 404 rather than silently succeeding as it did before that staging changeGET /credentialsracing the delete on another worker can still list it; pre-existing publish-on-write behavior on staging, observed once on the rig and cleared on the next readosv-scanis red on the pre-existing gitpython 3.1.58 advisory inuv.lock, the same result on the base branch; this PR does not touch dependenciesci/circleci: proxy_store_model_in_db_testsontest_missing_model_parameter_curl[chat], red onlitellm_internal_stagingitself since 2026-09-04 (LIT-6949, fixed by test(store_model_in_db): assert the 400 contract in the unknown-model spend log test #39842, which the fifteenth merge brought in), andproxy-infra / Run tests (Python 3.10)ontest_qualifiers_and_optionality_are_unwrapped(LIT-6947, fixed by fix(proxy): strip every TypedDict qualifier before numeric form-field detection #39780, which the same merge brought in). Thelitellm_router_unit_testingred recorded at 5e5c436 stays greenFinal Attestation
The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR
068c1d1 passes /live-pr-risk
c6f585e passes /live-pr-risk
412890f passes /live-pr-risk
272ccfd passes /live-pr-risk
55e303e passes /live-pr-risk
4a785c8 passes /live-pr-risk
cb03462 passes /live-pr-risk
3794ba9 passes /live-pr-risk
9036b90 passes /live-pr-risk
055eaee passes /live-pr-risk
37f2de1 passes /live-pr-risk
5e5c436 passes /live-pr-risk
4592e71 passes /live-pr-risk
216da96 passes /live-pr-risk
82afbbd passes /live-pr-risk
Note
High Risk
Changes authentication across Anthropic and OpenAI paths (token minting, header handling, batch billing) and introduces new federation configuration with host allowlisting—security-critical infrastructure.
Overview
Adds Anthropic workload identity federation (WIF): deployments without a static
api_keycan mint short-livedsk-ant-oat01tokens via a shared RFC 7523 JWT-bearer exchange engine, with identity from mounted OIDC files, inline refs, a LiteLLM-signed internal issuer, or Keycloak client_credentials.Auth plumbing threads new
anthropic_*/openai_*WIF keys throughlitellm_params(forwarded, optional, request-banned). Anthropic surfaces (chat,/v1/messages, files, batches, skills, count-tokens) resolve credentials throughget_auth_header/validate_environment, strip caller-supplied auth headers when federation mints, and run blocking token exchange off the async event loop. Batch retrieve and batch output billing now passlitellm_paramsso federated deployments authenticate when fetching results.Operator flows:
discover_models()on providers enables live/v1/modelsdiscovery with deployment credentials; proxy exposesPOST /provider/models/discover(allowlisted). OpenAI gains parallel WIF config for client creation and model discovery. Sharedresolve_anthropic_basealigns token exchange, discovery, and count-tokens URLs acrossapi_baseshapes.Security: exchange hosts are allowlisted (
LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS); federation fields are server-owned (not client request bodies).Reviewed by Cursor Bugbot for commit 82afbbd. Bugbot is set up for automated code reviews on this repo. Configure here.