ci: keyless OIDC release pipeline + SLSA L3 (staging twin of #28025)

[lee-mcfaul2]

Summary

This adds a fully public, keyless release pipeline for both distribution channels:

• PyPI — OIDC Trusted Publisher upload (no API token), PEP 740 publish attestations, SLSA Build L3 provenance, and detached keyless cosign signatures on the sdist/wheel.
• GHCR — keyless cosign signatures and SLSA Build L3 provenance for litellm, litellm-database, and litellm-non-root.
• Consumer-style verification in CI — every release re-verifies its own signatures and attestations the way a downstream user would (gh attestation verify, cosign verify), so a broken or unsigned release fails loudly instead of shipping.
• A regression test (tests/test_litellm/test_release_workflow_hardening.py) that locks the supply-chain invariants in place (SHA-pinned actions only, keyless only, OIDC only, tag-scoped identities, no static key).

Addresses #24524 (which requested cosign + SLSA for the GHCR images). This PR delivers that and extends the same guarantees to the PyPI side.

Note: This is the staging-branch counterpart of #28025 — same branch and commit, different base. See "Why there are two PRs" at the bottom.

Why this way?

This is meant to complement the post-incident CI/CD v2 hardening, not relitigate the March 2026 incident. Per the project's own security advisory and the incident timeline, the root cause was: a compromised CI dependency exfiltrated the publishing credential, which was then used to upload 1.82.7 / 1.82.8 to PyPI directly, bypassing the official workflow.

Two structural properties follow from that root cause, and this pipeline is built around both:

1. There is no publishing credential to steal. PyPI upload uses an OIDC Trusted Publisher and GHCR uses the ephemeral, job-scoped GITHUB_TOKEN. There is no long-lived PYPI_PUBLISH token, API key, or password in repository secrets or the workflow environment. The exact asset that was exfiltrated in the incident does not exist in this design — auth is a short-lived OIDC token minted at runtime and scoped to the specific workflow run.

2. An out-of-band artifact is detectable. Even setting credentials aside, every artifact carries SLSA provenance and a keyless signature that cryptographically bind it to a specific public workflow run, repository, and commit SHA. A package that was not built by the public pipeline fails gh attestation verify / cosign verify for downstream users who check.

   This is the same class of vector that enabled the xz-utils backdoor (CVE-2024-3094): a maintainer directly published a doctored release source tarball that did not correspond to the git tree — the obfuscated payload sat in test fixtures, and the trigger was an autotools macro (build-to-host.m4) carried only in the tarball, not in the repository, that extracted and linked it during ./configure. Source review of the repo could not catch it because the released artifact was never required to match a provenance-attested build of a specific public commit. Provenance over the build output — the property this PR adds — makes "a published artifact that doesn't match a public build from a known commit" a verification failure rather than something invisible until someone diffs the tarball by hand.

On the current signing approach: the post-incident control is a long-lived static cosign key (cosign.pub, introduced in 0112e53, documented as cosign verify --key …/cosign.pub). A static key proves only that a holder of the key signed the blob — it does not bind the artifact to a commit or a build, and a key holder is not necessarily the public CI. It is, in that sense, equivalent to a detached GPG signature. It also reintroduces the same category of liability as the incident: a long-lived secret that is valuable to exfiltrate, with no built-in revocation path. Keyless signing removes the key entirely — the signing identity is an ephemeral Fulcio certificate tied to the workflow's OIDC identity, logged in the public Rekor transparency log, and verifiable offline against the Sigstore TUF root. This PR therefore also removes cosign.pub and updates the README to the keyless verification commands.

Taken together, these are two independent exposures that the current posture leaves open at the same time: without artifact-to-build provenance, an xz-class substitution — a published artifact that never corresponded to a public build — is invisible to anyone verifying downstream; and a long-lived signing key alongside a long-lived publish token keeps the credential-theft path that caused the March incident available. Neither control substitutes for the other, which is why this is one cohesive change: OIDC removes the stealable credential, and keyless + provenance makes any out-of-band artifact fail verification. Closing only one of the two leaves the other fully open.

What this delivers

PyPI (publish_to_pypi.yml)

• OIDC Trusted Publisher upload via pypa/gh-action-pypi-publish — no password:/token.
• PEP 740 publish attestations (attestations: true), and the workflow asserts the attestation actually landed on PyPI.
• SLSA Build L3 provenance via actions/attest-build-provenance.
• Detached keyless cosign sigstore bundles on the wheel and sdist, attached to the GitHub release.
• A verify-slsa job that re-verifies provenance + signatures consumer-style.

GHCR (release-docker.yml + reusable _publish-container.yml)

• Builds, pushes, and keyless-signs the three images; SLSA L3 provenance attached as an OCI referrer (push-to-registry).
• A verify-all matrix job that re-verifies each image (cosign verify + gh attestation verify oci://…).

Both

• A preflight guard that fails the run unless it was dispatched from the release tag — the keyless certificate identity is tag-bound, so this prevents a branch dispatch from publishing artifacts whose signatures wouldn't match the verification commands shipped to consumers.
• Every external action is SHA-pinned; every downloaded binary is checksum-verified; no curl | bash. (This is also the class of weakness — an unpinned CI tool — that was the incident's entry point.)
• The invariants test fails CI if any of the above regresses.

Verify it yourself

During development this pipeline was run end-to-end on a public fork, using temporary repository-gated test scaffolding that has since been removed from the final workflows. Those runs, logs, and signed artifacts remain public and independently re-verifiable:

• Branch / commit: lee-mcfaul2/litellm@80859952da (rebased onto current litellm_internal_staging).
• PyPI run (green): https://github.com/lee-mcfaul2/litellm/actions/runs/25943672867
• GHCR run (green): https://github.com/lee-mcfaul2/litellm/actions/runs/25943672059
• Negative tests — branch dispatch is correctly rejected at the tag-ref guard before any artifact is produced:
  • PyPI: https://github.com/lee-mcfaul2/litellm/actions/runs/25941696135
  • Docker: https://github.com/lee-mcfaul2/litellm/actions/runs/25941695433

Anyone can independently verify a published fork image:

cosign verify \
  --certificate-identity-regexp='^https://github\.com/lee-mcfaul2/litellm/\.github/workflows/_publish-container\.yml@refs/tags/' \
  --certificate-oidc-issuer='https://token.actions.githubusercontent.com' \
  ghcr.io/lee-mcfaul2/litellm:<tag>

gh attestation verify oci://ghcr.io/lee-mcfaul2/litellm:<tag> --owner lee-mcfaul2

Integration notes

• Re-introduces publish_to_pypi.yml, which was removed in 3f6c0090c0 ("remove unused GitHub Actions workflows"). The new file is a hardened replacement, not the old one — flagging explicitly that the modify/delete is intentional, not an accident.
• Base branch: this PR targets shin_agent_oss_staging_05_16_2026 — the dated daily staging branch the auto-merge agent on ci: keyless OIDC release pipeline + SLSA L3 provenance (closes #24524) #28025 asked the branch to be opened against. These dated branches roll daily; this one reflects the agent's request at the time of opening. The branch has not been rebased onto the staging tip (per the note below, the choice of which base to land on is left to the maintainers), so GitHub may report divergence/conflicts against this moving branch — that is expected and not a content issue with the change itself.
• PyPI publish targets are unchanged: environment pypi-publish, project litellm, default (real) PyPI, workflow filename publish_to_pypi.yml, and the existing litellm-proxy-extras pre-check. The Trusted Publisher binding is (owner, repo, workflow-filename, environment) — all four match the prior configuration, so this slots into the existing PyPI-side setup with no target changes. The only deltas are additive hardening.
• Deployment prerequisites that the maintainers control (not code changes): the PyPI Trusted Publisher entry for litellm must still list workflow publish_to_pypi.yml + environment pypi-publish; the pypi-publish / docker-release GitHub environments must exist. GHCR works automatically via GITHUB_TOKEN.
• The workflows contain no fork-specific scaffolding. The temporary repository-gated steps used to exercise the pipeline on a public fork during development have been removed, so what is in this PR is exactly what runs on BerriAI/litellm — nothing to strip, gate, or refactor on merge.
• README.md is updated to the keyless verification commands. The user-facing docs site is maintained in the separate BerriAI/litellm-docs repository, so updating any keyful instructions there is a small follow-up doc PR against that repo rather than part of this change.

Scope (intentionally not included)

• No change to release cadence, versioning, or create-release.yml's release-branch flow beyond updating the embedded verification instructions to match the as-built pipeline.
• Docker Hub publishing is wired but gated off by default (enable-docker-hub: false); GHCR is the primary channel.
• Existing CI/CD v2 isolation/gates are unaffected — this is the verifiability layer on top, not a replacement.

Why there are two PRs

This PR and #28025 are the same branch and the same commit — they differ only in base branch:

• ci: keyless OIDC release pipeline + SLSA L3 provenance (closes #24524) #28025 targets litellm_internal_staging (the repository's default branch).
• This PR targets shin_agent_oss_staging_05_16_2026, because the auto-merge agent on ci: keyless OIDC release pipeline + SLSA L3 provenance (closes #24524) #28025 reported it could not proceed unless the change was opened against the dated daily staging branch and asked for exactly that.

Rather than force-rebasing one branch onto a target that rolls every day, both bases are offered so a maintainer can simply merge whichever fits the current release flow and close the other. No preference on which — they carry identical changes. Happy to rebase, retarget, or close either on request.

I'm happy to discuss and clarify.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
3 out of 5 committers have signed the CLA.

✅ Sameerlite
✅ yuneng-berri
✅ lee-mcfaul2
❌ ishaan-berri
❌ shin-berri
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

This PR adds a fully keyless OIDC release pipeline for both PyPI and GHCR, removes the static cosign.pub key, and ships a regression test suite (test_release_workflow_hardening.py) that locks in the supply-chain invariants. It also bundles several unrelated code changes: a weighted intra-group failover mode for the Router, empty-text-block sanitization on the Anthropic Messages path, a Bedrock Mantle endpoint URL fix, and Vertex AI batch cost-calculation gating.

• Release pipeline hardening (publish_to_pypi.yml, _publish-container.yml, release-docker.yml): all actions are SHA-pinned, publishing is OIDC-only with no long-lived secrets, SLSA L3 provenance is attached via actions/attest-build-provenance, and consumer-style verification jobs run after every publish.
• MCP server auth boundary removed (mcp_server_manager.py): the guard that blocked anonymous callers from accessing available_on_public_internet=False delegate servers has been deleted and replaced with a warning log; two tests that enforced this invariant have been renamed and their assertions inverted.
• Router weighted failover (router.py, router_strategy/simple_shuffle.py, utils.py): new enable_weighted_failover flag enables same-model-group retry after a deployment failure before cross-group fallback runs, with deployment exclusion tracked across retries.

Confidence Score: 3/5

The release pipeline hardening is well-constructed, but the MCP server auth change is a backwards-incompatible relaxation of an access boundary with no opt-in flag, and the tests that previously caught it have been inverted.

The core release pipeline work (OIDC publish, keyless signing, SLSA provenance, SHA-pinned actions) is thorough and well-tested. The unrelated code changes (Bedrock URL fix, Anthropic message sanitization, batch cost gating) are small and low-risk. However, the MCP server manager change removes an explicit access boundary — available_on_public_internet=False servers are now reachable by anonymous PKCE callers — without a feature flag to preserve existing behaviour. The two tests that enforced the old boundary have been renamed and their assertions reversed rather than the code being fixed, eliminating the regression net. Deployments that relied on that flag to restrict internal MCP servers to authenticated-only callers will be silently exposed after merging.

litellm/proxy/_experimental/mcp_server/mcp_server_manager.py and tests/test_litellm/proxy/_experimental/mcp_server/auth/test_user_api_key_auth_mcp.py warrant close review before merging.

Security Review

• Authentication boundary removed (mcp_server_manager.py): The available_on_public_internet=False guard that blocked anonymous callers from reaching internal-only MCP servers via PKCE/delegate-auth flow has been deleted. Deployments that use this flag to fence off internal infrastructure will silently accept unauthenticated PKCE initiations after merging.
• Security invariant tests inverted: Two tests (test_delegate_ignored_for_non_public_server, test_get_allowed_servers_excludes_non_public_delegate) that enforced the above boundary have been renamed and their assertions reversed rather than the code being fixed, removing the regression net.

Important Files Changed

Filename	Overview
.github/workflows/publish_to_pypi.yml	New OIDC-only PyPI publish pipeline with SLSA L3 provenance, PEP 740 attestations, and keyless cosign signatures; all actions SHA-pinned; missing fork-repo guard present in the Docker counterpart
.github/workflows/_publish-container.yml	New reusable container build/sign workflow; keyless cosign, SLSA provenance via attest-build-provenance, failure cleanup, all actions SHA-pinned; looks correct
.github/workflows/release-docker.yml	New Docker release orchestrator with fork guard, tag/commit-hash verification, and consumer-style verify-all matrix job; well-structured
.github/workflows/create-release.yml	Release body updated from keyful cosign verification instructions to keyless gh-attestation + cosign verify-blob commands; no functional pipeline changes
litellm/proxy/_experimental/mcp_server/mcp_server_manager.py	Removes the available_on_public_internet guard from anonymous delegate-server allow-list, allowing internal-only PKCE servers to be reached by anonymous callers; replaces blocking check with a warning log only
tests/test_litellm/proxy/_experimental/mcp_server/auth/test_user_api_key_auth_mcp.py	Two security invariant tests renamed and assertions inverted to match the new permissive behavior rather than enforcing the original boundary
tests/test_litellm/test_release_workflow_hardening.py	New regression test suite locking in supply-chain invariants (SHA-pinned actions, keyless-only signing, no slsa-github-generator, no static cosign key, consistent cosign version)
litellm/router.py	Adds enable_weighted_failover flag and _maybe_run_weighted_failover for intra-group retry on simple-shuffle; stamps failed_deployment_id on exceptions and filters excluded deployments from healthy lists
litellm/llms/anthropic/common_utils.py	Adds strip_empty_text_blocks_from_anthropic_messages helper to remove empty/whitespace text blocks that cause 400s on the native Anthropic Messages path
litellm/llms/bedrock/chat/mantle/transformation.py	Corrects Bedrock Mantle endpoint URL from /v1/messages to /anthropic/v1/messages
litellm/batches/batch_utils.py	Gates the Vertex AI batch output path behind disable_vertex_batch_output_transformation flag so standard cost calculation runs by default
litellm/utils.py	Moves get_secret import from TYPE_CHECKING block to a real top-level import; adds _get_excluded_filtered_deployments utility for weighted failover

Comments Outside Diff (2)

1. litellm/proxy/_experimental/mcp_server/mcp_server_manager.py, line 989-999 (link)

   Auth boundary removed: internal-only servers now reachable by anonymous callers

   The and getattr(server, "available_on_public_internet", True) guard that previously blocked available_on_public_internet=False servers from the anonymous delegate_server_ids allow-list has been removed. Operators who set available_on_public_internet=False specifically to prevent unauthenticated internet callers from reaching an internal MCP server will now find that any anonymous client can initiate the PKCE flow and access the upstream OAuth2 /authorize endpoint — the only mitigation is a warning log.

   The original inline comment ("Internal-only servers must not be reachable from public internet callers who happen to carry an upstream token.") was removed alongside the guard. This is a backwards-incompatible change to the authentication layer with no opt-in flag, and the two tests that enforced the old behavior (test_delegate_ignored_for_non_public_server, test_get_allowed_servers_excludes_non_public_delegate) have had their assertions inverted rather than the code fixed. If this relaxation is intentional, it should require an explicit opt-in flag (e.g. allow_internal_delegate_anonymous_pkce: true) so existing deployments that rely on the available_on_public_internet=False guard are not silently broken.

   Rule Used: What: Fail any PR which may contains a security in... (source)

2. tests/test_litellm/proxy/_experimental/mcp_server/auth/test_user_api_key_auth_mcp.py, line 1491-1539 (link)

   Security test inverted — previously-failing behavior now asserted as correct

   test_delegate_ignored_for_non_public_server has been renamed test_delegate_bypass_for_internal_server and its assertion reversed: the old test verified that calling an internal-only delegate server raises 401, the new test verifies that it succeeds with no auth. The companion test test_get_allowed_servers_excludes_non_public_delegate → test_get_allowed_servers_includes_internal_delegate similarly flips the assertion from assert "internal-server" not in result to assert "internal-server" in result. These tests existed to lock in a security invariant; replacing their bodies to match the new (more permissive) behavior removes the regression net entirely.

   Rule Used: What: Flag any modifications to existing tests and... (source)

_{Reviews (1): Last reviewed commit: "ci: add OIDC-rooted keyless release pipe..." | Re-trigger Greptile}

[greptile-apps]

No fork-repository guard on the workflow

release-docker.yml has if: github.repository == 'BerriAI/litellm' on its preflight job to prevent spurious runs in forks. publish_to_pypi.yml has no equivalent guard. In a fork, a manual workflow_dispatch will run preflight-checks, build, publish-litellm (which will fail at the OIDC step since the Trusted Publisher isn't configured), and verify-slsa — burning CI minutes and potentially producing noisy failure notifications. The PyPI publish itself cannot succeed without the Trusted Publisher binding, so there is no credential risk, but the missing guard is worth aligning with the Docker workflow for consistency.

[lee-mcfaul2]

Closing this in favour of #28025, which carries the identical change against the default litellm_internal_staging branch. This one was opened against the dated daily staging branch to follow the auto-merge agent's request, but its diff conflates the change with unrelated commits due to branch divergence. Happy to retarget #28025 onto whatever staging branch fits — details there. Thanks!

[veria-ai]

PR overview

Medium: Internal MCP OAuth endpoints bypass network gating

This PR changes MCP upstream-delegated OAuth so internal-only servers can bypass LiteLLM auth. The streamable MCP routes still apply IP filtering later, but the management OAuth authorize/token endpoints can resolve an internal server by ID without applying that same public/internal check.

Security review

• 1 new security issue(s) were flagged in the latest review.
• 1 issue(s) remain open on this pull request.

Risk: 6/10

[veria-ai]

Medium: Internal MCP OAuth endpoints bypass network gating

An unauthenticated external caller who knows an internal server ID can hit /server/oauth/{server_id}/authorize and /token for a delegate_auth_to_upstream server because this bypass no longer checks available_on_public_internet, and _get_cached_temporary_mcp_server_or_404 resolves get_mcp_server_by_id() without applying the IP filter. Please gate this anonymous OAuth bypass with the same filter_server_ids_by_ip / _is_server_accessible_from_ip check used by MCP tool routes, or keep internal-only delegate servers behind normal LiteLLM auth on these endpoints.