feat: add github cosign for releases - #24683
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| cosign verify \ | ||
| --certificate-identity-regexp "https://github.com/BerriAI/litellm/" \ | ||
| --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \ | ||
| "ghcr.io/berriai/${{ matrix.image }}:${{ github.event.inputs.tag }}" |
Check failure
Code scanning / zizmor
code injection via template expansion Error
| --certificate-identity-regexp "https://github.com/BerriAI/litellm/" \ | ||
| --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \ | ||
| --type spdxjson \ | ||
| "ghcr.io/berriai/${{ matrix.image }}:${{ github.event.inputs.tag }}" |
Check failure
Code scanning / zizmor
code injection via template expansion Error
| cache-to: type=gha,mode=max | ||
|
|
||
| - name: Install cosign | ||
| uses: sigstore/cosign-installer@3454372f43399081ed03b604cb2d021dabca52bb # v4.1.1 |
Check warning
Code scanning / zizmor
detects commit SHAs that don't match their version comment tags Warning
|
|
||
| steps: | ||
| - name: Install cosign | ||
| uses: sigstore/cosign-installer@3454372f43399081ed03b604cb2d021dabca52bb # v4.1.1 |
Check warning
Code scanning / zizmor
detects commit SHAs that don't match their version comment tags Warning
| - name: Download published package from PyPI | ||
| run: | | ||
| pip download --no-deps --no-binary :all: \ | ||
| litellm==${{ needs.preflight-checks.outputs.version }} \ |
Check notice
Code scanning / zizmor
code injection via template expansion Note
| litellm==${{ needs.preflight-checks.outputs.version }} \ | ||
| -d verify/ | ||
| pip download --no-deps --only-binary :all: \ | ||
| litellm==${{ needs.preflight-checks.outputs.version }} \ |
Check notice
Code scanning / zizmor
code injection via template expansion Note
|
|
Greptile SummaryThis PR adds GitHub Cosign signing for LiteLLM releases. It introduces a new Key issue found:
Confidence Score: 4/5Safe to merge with one blocking issue: the embedded cosign.pub key-based verification in release notes will be broken. The core infrastructure (workflow structure, permissions, input validation) is solid, but the release notes template embeds a verification command that will fail for every release — it references a non-existent cosign.pub file and uses a different signing paradigm than the documentation. This is a P1 correctness defect on the primary feature being shipped. .github/workflows/create-release.yml lines 53-57 — the cosign verification snippet references a missing public key file and uses a different signing approach than the docs
|
| Filename | Overview |
|---|---|
| .github/workflows/create-release.yml | New release workflow with solid structure (draft-then-publish, input validation, pinned action SHA), but embeds a key-based cosign verification command referencing a non-existent cosign.pub file that conflicts with the keyless signing described in documentation |
| docs/my-website/docs/proxy/docker_quick_start.md | Adds keyless OIDC-based cosign verification docs for Docker images and PyPI packages; overly broad regexp issue already flagged in prior review threads |
| docs/my-website/docs/proxy/release_cycle.md | Minor update adding a reference to Sigstore signing documentation — no issues |
| scripts/install.sh | New one-line install script with proper POSIX compatibility, Python version checking, and interactive setup wizard prompt — no issues |
Sequence Diagram
sequenceDiagram
participant Maintainer
participant GH as GitHub Actions
participant DockerBuild as Docker Build Workflow
participant GHCR as ghcr.io
participant Sigstore as Sigstore/Fulcio
participant User
Maintainer->>GH: workflow_dispatch(tag, commit_hash)
GH->>GH: Validate tag format + SHA
GH->>GH: createRelease(draft=true)
GH->>GH: Prepend cosign section to release body
GH->>GH: updateRelease(draft=false)
Note over DockerBuild,Sigstore: Separate Docker signing flow (keyless)
DockerBuild->>GHCR: docker push image
DockerBuild->>Sigstore: cosign sign (OIDC keyless)
Sigstore-->>GHCR: Store signature
User->>GHCR: cosign verify --key cosign.pub (from release notes)
GHCR-->>User: ❌ Fails — no key-based signature exists
User->>GHCR: cosign verify --certificate-identity-regexp (from docs)
GHCR-->>User: ✅ Succeeds — keyless signature found
Reviews (2): Last reviewed commit: "chore: fixes" | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| cosign verify \ | ||
| --certificate-identity-regexp "https://github.com/BerriAI/litellm/" \ | ||
| --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \ | ||
| "ghcr.io/berriai/${{ matrix.image }}:${{ github.event.inputs.tag }}" |
There was a problem hiding this comment.
The regexp "https://github.com/BerriAI/litellm/" matches the identity of any workflow in the repository — not just the release workflow. An image signed by a compromised or unrelated workflow would pass this check, defeating the core supply-chain assurance sigstore is meant to provide.
The PyPI workflow correctly uses an exact --cert-identity flag pinned to the specific workflow file and branch. The Docker verification should follow the same pattern — use --certificate-identity (not the regexp variant) and pin to release_docker.yml@refs/heads/main.
The same issue applies to the SBOM attestation verification on lines 145–149.
| - name: Verify SBOM attestation | ||
| run: | | ||
| cosign verify-attestation \ | ||
| --certificate-identity-regexp "https://github.com/BerriAI/litellm/" \ | ||
| --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \ | ||
| --type spdxjson \ | ||
| "ghcr.io/berriai/${{ matrix.image }}:${{ github.event.inputs.tag }}" |
There was a problem hiding this comment.
Same issue as the image signature verification above: --certificate-identity-regexp "https://github.com/BerriAI/litellm/" accepts a signature from any workflow in the repository. Use --certificate-identity (exact match) pinned to the specific workflow file path and branch, consistent with how the PyPI verification is implemented.
| cosign verify \ | ||
| --certificate-identity-regexp "https://github.com/BerriAI/litellm/" \ | ||
| --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \ | ||
| ghcr.io/berriai/litellm:main-latest | ||
| ``` |
There was a problem hiding this comment.
The user-facing verification command uses --certificate-identity-regexp "https://github.com/BerriAI/litellm/", which accepts a signature from any workflow in the repository. The documentation tells users this proves the artifact "was built by the official BerriAI/litellm CI", but the regexp is much weaker than that claim implies.
Compare with the PyPI section (line 914), which correctly pins --cert-identity to the exact workflow file. The Docker verification commands should do the same — swap the regexp flag for an exact --certificate-identity pinned to release_docker.yml@refs/heads/main.
The same change is needed for the SBOM attestation command on lines 899–903.
| tags: | | ||
| type=raw,value=${{ github.event.inputs.tag }} | ||
| type=raw,value=main-latest |
There was a problem hiding this comment.
main-latest tag is unconditionally overwritten for every dispatch
type=raw,value=main-latest is always included in the pushed tags, meaning any workflow_dispatch invocation — including nightly, RC, or hotfix releases — will silently overwrite main-latest. Users relying on that tag for production deployments could inadvertently pick up a pre-release image.
Consider making this conditional, for example by only adding main-latest when the tag doesn't contain -nightly or similar pre-release markers.
| - name: Verify image signature | ||
| run: | | ||
| cosign verify \ | ||
| --certificate-identity-regexp "https://github.com/BerriAI/litellm/" \ | ||
| --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \ | ||
| "ghcr.io/berriai/${{ matrix.image }}:${{ github.event.inputs.tag }}" |
There was a problem hiding this comment.
The verify-signatures job verifies by the mutable tag value (${{ github.event.inputs.tag }}). Because tags can be overwritten, this verification does not strictly confirm the just-signed digest is still what that tag points to.
The build-and-push job captures the exact digest in steps.push.outputs.digest. Consider passing it as a job output and verifying by digest instead of tag for stronger guarantees.
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. |
4 participants
Relevant issues
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
tests/test_litellm/directory, Adding at least 1 test is a hard requirement - see detailsmake test-unit@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewDelays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
CI (LiteLLM team)
Branch creation CI run
Link:
CI run for the last commit
Link:
Merge / cherry-pick CI run
Links:
Type
🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test
Changes