ci: keyless OIDC release pipeline + SLSA L3 provenance (closes #24524)

[lee-mcfaul2]

Summary

This adds a fully public, keyless release pipeline for both distribution channels:

• PyPI — OIDC Trusted Publisher upload (no API token), PEP 740 publish attestations, SLSA Build L3 provenance, and detached keyless cosign signatures on the sdist/wheel.
• GHCR — keyless cosign signatures and SLSA Build L3 provenance for litellm, litellm-database, and litellm-non-root.
• Consumer-style verification in CI — every release re-verifies its own signatures and attestations the way a downstream user would (gh attestation verify, cosign verify), so a broken or unsigned release fails loudly instead of shipping.
• A regression test (tests/test_litellm/test_release_workflow_hardening.py) that locks the supply-chain invariants in place (SHA-pinned actions only, keyless only, OIDC only, tag-scoped identities, no static key).

Closes #24524 (which requested cosign + SLSA for the GHCR images). This PR delivers that and extends the same guarantees to the PyPI side.

Why this way?

This is meant to complement the post-incident CI/CD v2 hardening, not relitigate the March 2026 incident. Per the project's own security advisory and the incident timeline, the root cause was: a compromised CI dependency exfiltrated the publishing credential, which was then used to upload 1.82.7 / 1.82.8 to PyPI directly, bypassing the official workflow.

Two structural properties follow from that root cause, and this pipeline is built around both:

1. There is no publishing credential to steal. PyPI upload uses an OIDC Trusted Publisher and GHCR uses the ephemeral, job-scoped GITHUB_TOKEN. There is no long-lived PYPI_PUBLISH token, API key, or password in repository secrets or the workflow environment. The exact asset that was exfiltrated in the incident does not exist in this design — auth is a short-lived OIDC token minted at runtime and scoped to the specific workflow run.

2. An out-of-band artifact is detectable. Even setting credentials aside, every artifact carries SLSA provenance and a keyless signature that cryptographically bind it to a specific public workflow run, repository, and commit SHA. A package that was not built by the public pipeline fails gh attestation verify / cosign verify for downstream users who check.

   This is the same class of vector that enabled the xz-utils backdoor (CVE-2024-3094): a maintainer directly published a doctored release source tarball that did not correspond to the git tree — the obfuscated payload sat in test fixtures, and the trigger was an autotools macro (build-to-host.m4) carried only in the tarball, not in the repository, that extracted and linked it during ./configure. Source review of the repo could not catch it because the released artifact was never required to match a provenance-attested build of a specific public commit. Provenance over the build output — the property this PR adds — makes "a published artifact that doesn't match a public build from a known commit" a verification failure rather than something invisible until someone diffs the tarball by hand.

On the current signing approach: the post-incident control is a long-lived static cosign key (cosign.pub, introduced in 0112e53, documented as cosign verify --key …/cosign.pub). A static key proves only that a holder of the key signed the blob — it does not bind the artifact to a commit or a build, and a key holder is not necessarily the public CI. It is, in that sense, equivalent to a detached GPG signature. It also reintroduces the same category of liability as the incident: a long-lived secret that is valuable to exfiltrate, with no built-in revocation path. Keyless signing removes the key entirely — the signing identity is an ephemeral Fulcio certificate tied to the workflow's OIDC identity, logged in the public Rekor transparency log, and verifiable offline against the Sigstore TUF root. This PR therefore also removes cosign.pub and updates the README to the keyless verification commands.

Taken together, these are two independent exposures that the current posture leaves open at the same time: without artifact-to-build provenance, an xz-class substitution — a published artifact that never corresponded to a public build — is invisible to anyone verifying downstream; and a long-lived signing key alongside a long-lived publish token keeps the credential-theft path that caused the March incident available. Neither control substitutes for the other, which is why this is one cohesive change: OIDC removes the stealable credential, and keyless + provenance makes any out-of-band artifact fail verification. Closing only one of the two leaves the other fully open.

What this delivers

PyPI (publish_to_pypi.yml)

• OIDC Trusted Publisher upload via pypa/gh-action-pypi-publish — no password:/token.
• PEP 740 publish attestations (attestations: true), and the workflow asserts the attestation actually landed on PyPI.
• SLSA Build L3 provenance via actions/attest-build-provenance.
• Detached keyless cosign sigstore bundles on the wheel and sdist, attached to the GitHub release.
• A verify-slsa job that re-verifies provenance + signatures consumer-style.

GHCR (release-docker.yml + reusable _publish-container.yml)

• Builds, pushes, and keyless-signs the three images; SLSA L3 provenance attached as an OCI referrer (push-to-registry).
• A verify-all matrix job that re-verifies each image (cosign verify + gh attestation verify oci://…).

Both

• A preflight guard that fails the run unless it was dispatched from the release tag — the keyless certificate identity is tag-bound, so this prevents a branch dispatch from publishing artifacts whose signatures wouldn't match the verification commands shipped to consumers.
• Every external action is SHA-pinned; every downloaded binary is checksum-verified; no curl | bash. (This is also the class of weakness — an unpinned CI tool — that was the incident's entry point.)
• The invariants test fails CI if any of the above regresses.

Verify it yourself

During development this pipeline was run end-to-end on a public fork, using temporary repository-gated test scaffolding that has since been removed from the final workflows. Those runs, logs, and signed artifacts remain public and independently re-verifiable:

• Branch / commit: lee-mcfaul2/litellm@80859952da (rebased onto current litellm_internal_staging).
• PyPI run (green): https://github.com/lee-mcfaul2/litellm/actions/runs/25943672867
• GHCR run (green): https://github.com/lee-mcfaul2/litellm/actions/runs/25943672059
• Negative tests — branch dispatch is correctly rejected at the tag-ref guard before any artifact is produced:
  • PyPI: https://github.com/lee-mcfaul2/litellm/actions/runs/25941696135
  • Docker: https://github.com/lee-mcfaul2/litellm/actions/runs/25941695433

Anyone can independently verify a published fork image:

cosign verify \
  --certificate-identity-regexp='^https://github\.com/lee-mcfaul2/litellm/\.github/workflows/_publish-container\.yml@refs/tags/' \
  --certificate-oidc-issuer='https://token.actions.githubusercontent.com' \
  ghcr.io/lee-mcfaul2/litellm:<tag>

gh attestation verify oci://ghcr.io/lee-mcfaul2/litellm:<tag> --owner lee-mcfaul2

Integration notes

• Re-introduces publish_to_pypi.yml, which was removed in 3f6c0090c0 ("remove unused GitHub Actions workflows"). The new file is a hardened replacement, not the old one — flagging explicitly that the modify/delete is intentional, not an accident.
• Rebased onto current litellm_internal_staging; opens with zero merge conflicts.
• Base branch: this targets litellm_internal_staging (the repository default). The dated litellm_oss_staging_<MM_DD_YYYY> branch that create_daily_staging_branch.yml is meant to produce twice daily is not currently available — the most recent in that series is litellm_oss_staging_04_27_2026, and the scheduled workflow has been failing on every run over at least 2026-05-13 → 2026-05-15. Happy to retarget to whatever branch the current release flow expects; flagging it here since the documented daily target isn't being generated right now.
• PyPI publish targets are unchanged: environment pypi-publish, project litellm, default (real) PyPI, workflow filename publish_to_pypi.yml, and the existing litellm-proxy-extras pre-check. The Trusted Publisher binding is (owner, repo, workflow-filename, environment) — all four match the prior configuration, so this slots into the existing PyPI-side setup with no target changes. The only deltas are additive hardening.
• Deployment prerequisites that the maintainers control (not code changes): the PyPI Trusted Publisher entry for litellm must still list workflow publish_to_pypi.yml + environment pypi-publish; the pypi-publish / docker-release GitHub environments must exist. GHCR works automatically via GITHUB_TOKEN.
• The workflows contain no fork-specific scaffolding. The temporary repository-gated steps used to exercise the pipeline on a public fork during development have been removed, so what is in this PR is exactly what runs on BerriAI/litellm — nothing to strip, gate, or refactor on merge.
• README.md is updated to the keyless verification commands. The user-facing docs site is maintained in the separate BerriAI/litellm-docs repository, so updating any keyful instructions there is a small follow-up doc PR against that repo rather than part of this change.

Scope (intentionally not included)

• No change to release cadence, versioning, or create-release.yml's release-branch flow beyond updating the embedded verification instructions to match the as-built pipeline.
• Docker Hub publishing is wired but gated off by default (enable-docker-hub: false); GHCR is the primary channel.
• Existing CI/CD v2 isolation/gates are unaffected — this is the verifiability layer on top, not a replacement.

I'm happy to discuss and clarify.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR replaces the existing static-key cosign signing setup with a fully keyless, OIDC-based supply-chain hardening layer for both PyPI and GHCR distribution channels. cosign.pub is removed, publish_to_pypi.yml and release-docker.yml are introduced with OIDC Trusted Publisher upload, PEP 740 publish attestations, SLSA Build L3 provenance, and detached keyless cosign bundles backed by a regression test suite.

• PyPI pipeline: build job is deliberately isolated from id-token permission; a separate publish job holds the OIDC token and wraps upload, SLSA provenance, cosign signing, and a consumer-style verify-slsa job.
• GHCR pipeline: reusable _publish-container.yml builds, signs, and attests three images; release-docker.yml orchestrates them behind a canonical-repo guard and a tag\u2192commit binding check.
• Test harness: test_release_workflow_hardening.py asserts SHA-pinned actions, keyless-only cosign, no static passwords, and consistent cosign version \u2014 all via local file reads only.

Confidence Score: 5/5

Safe to merge — the core publishing and signing logic is sound and no unauthorized PyPI or GHCR publish is possible.

All three distribution pipelines correctly isolate build from publish permissions, use ephemeral OIDC tokens rather than long-lived secrets, and include consumer-style post-publish verification. The two observations are minor: fork-specific routing code that is inert on the canonical repo, and an incomplete regex-escape for tag characters that only matters for non-standard tag formats litellm does not currently use.

publish_to_pypi.yml retains fork-specific lee-mcfaul2/litellm routing that is dead code on the canonical repo and was already removed from release-docker.yml.

Important Files Changed

Filename	Overview
.github/workflows/_publish-container.yml	New reusable workflow: builds multi-arch images, keyless-signs with cosign, attaches SLSA L3 provenance via actions/attest-build-provenance, and self-verifies. SHA-pinned throughout; provenance: false intentional and documented inline.
.github/workflows/release-docker.yml	New orchestrator workflow: dispatches to _publish-container.yml for three images, verifies tag→commit binding, runs consumer-style verify-all matrix job. Canonical-repo guard correctly applied at preflight.
.github/workflows/publish_to_pypi.yml	New PyPI publishing workflow with OIDC Trusted Publisher, PEP 740 attestations, SLSA provenance, and keyless cosign signing. Fork-specific lee-mcfaul2/litellm references remain in seven places unlike release-docker.yml which had its fork allowlist removed.
.github/workflows/create-release.yml	Release body updated from keyful cosign instructions to keyless verify commands. No structural workflow changes.
README.md	Verification instructions updated from keyful to keyless. Change is minimal and accurate.
cosign.pub	Static ECDSA public key deleted — replaced by keyless Sigstore signing. Correct and intentional.
tests/test_litellm/test_release_workflow_hardening.py	New regression test suite for supply-chain invariants. All assertions read local YAML files only — no network calls.

_{Reviews (2): Last reviewed commit: "ci: add OIDC-rooted keyless release pipe..." | Re-trigger Greptile}

[veria-ai]

PR overview

Keyless release signing pipeline added

This PR adds OIDC-based PyPI and container release workflows with SHA-pinned actions, tag-bound cosign verification, SLSA attestations, and scoped job permissions. I checked the workflow dispatch inputs, tag/commit validation, privileged job boundaries, artifact handoff, registry publishing, and generated verification instructions and did not find a concrete new security issue.

Security review

• No new security issues were flagged in the latest review.
• No review issues remain open on this pull request.

Risk: 2/10

[oss-pr-review-agent-shin]

🤖 litellm-agent: This PR is currently BLOCKED from merge.

Score: 2/5 ❌

Why blocked:

• 1 PR-related CI failure (Greptile gate: score 3/5 below required 4/5 — request a Greptile review (@greptileai) and resolve its comments before maintainer review.) (pr_related_failures, -2 pts)
• Greptile 3/5 (greptile_low, -1 pts)

Details: Score docked for: 1 PR-related CI failure (Greptile gate: score 3/5 below required 4/5 — request a Greptile review (@greptileai) and resolve its comments before maintainer review.); Greptile 3/5.

Fix the issues above and push an update — the bot will re-review automatically.

Note: This bot is still in beta and might not always work as expected. Please share any feedback via Slack.

[lee-mcfaul2]

@greptileai retrigger request

[oss-pr-review-agent-shin]

🤖 litellm-agent: Auto-merge skipped — the staging branch shin_agent_oss_staging_05_15_2026 has 3 commit(s) not in your branch. Merging as-is would produce a confusing diff on the staging PR.

Please rebase your branch onto shin_agent_oss_staging_05_15_2026 and push; the agent will re-review automatically.

[lee-mcfaul2]

Thanks for taking the time to look at this, and apologies for the noise on what's a busy project — a short status to make a maintainer's review as easy as possible.

Where this stands. This is a small, self-contained change: 4 CI workflows, the README, removal of the static cosign.pub, and one regression test (7 files). Greptile re-reviewed it at 5/5, the Veria review passed, and 29 of 30 checks are green.

The one failing check looks unrelated to this change. misc / Run tests fails on tests/test_litellm/interactions/test_openapi_compliance.py::TestResponseCompliance::test_interaction_response_fields. That test fetches a live third-party spec over the network at runtime (httpx.get("https://ai.google.dev/static/api/interactions.openapi.json", timeout=5.0)) and asserts a field (steps) that the currently-served spec doesn't contain. The behaviour is deterministic rather than flaky: with no outbound network the test pytest.skips (and shows green); when CI can reach the endpoint the assertion fails. It reproduces on an unmodified checkout of the base branch and appears to trace to commit 3d67f00ed (2026-05-08, "align openapi compliance with upstream rename outputs->steps"). This PR changes no source, response models, or OpenAPI specs, so it shouldn't be able to affect that test — flagging it in case it's useful for your own CI, not as anything blocking here.

About the second PR (#28038). The auto-merge agent on this PR asked for the change to be opened against the dated daily staging branch, so I opened #28038 against shin_agent_oss_staging_05_16_2026 to follow that. Because that branch has diverged from this PR's base, its diff ends up conflating this change with ~37 unrelated commits, which isn't helpful to review. To keep things tidy I've closed #28038 — this PR (#28025), against the default litellm_internal_staging, is the single canonical change. It's happy to live on whatever base suits your release flow; just say the word and I'll rebase or retarget.

Next steps. Since the staging branches roll daily, I'll hold off on further pushes or rebases until someone has a chance to weigh in — purely to avoid adding churn against a moving target on your side. The change is complete and verified (including independent reproduction on a public fork, linked above), thanks again for the project and for taking a look.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.