feat(fips): build grpcio from source against system OpenSSL and drop its scanner waiver by devin-ai-integration[bot] · Pull Request #42912 · BerriAI/litellm

devin-ai-integration · 2026-09-24T08:15:23Z

TLDR

Problem this solves:

The FIPS image ships grpcio's PyPI wheel, which statically bundles BoringSSL
The bundled-crypto gate waived that file, so the build could not catch it
Vertex AI over grpc stays in the customer's worst case, so grpcio cannot simply be dropped

How it solves it:

Dockerfile.fips builds grpcio from source with GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1, so cygrpc links the system libcrypto.so.3
The compile lives in its own stage keyed on the version read from uv.lock, so it is cached until grpcio moves
The build asserts the result links libcrypto.so.3 and carries no BoringSSL string, then strips it (546 MB to 14 MB)
scan_bundled_crypto.sh no longer waives cygrpc*.so, so a BoringSSL grpcio fails the gate again
Two integration tests in the CircleCI management shard pin the gate behavior and prove a two worker proxy serves all three LLM endpoints, and keeps serving after one worker is killed, with grpcio, polars and google.cloud absent
docker/Dockerfile.fips is allowlisted in .github/ci-coverage-allowlist.yml because no pull request job holds the Chainguard apk token the image needs

User Flow

Before: a FIPS operator builds the image and gets a green crypto gate while grpcio still carries its own BoringSSL

They run docker build -f docker/Dockerfile.fips . and the fips-check stage prints PASS: all N ELF binaries accounted for
objdump -p /app/.venv/lib/python3.13/site-packages/grpc/_cython/cygrpc*.so shows no libcrypto.so.3, grep -a BoringSSL matches
Any Vertex AI grpc traffic from the proxy negotiates TLS through BoringSSL, outside the validated module

After: the same build compiles grpcio against the FIPS OpenSSL and the gate would fail if it ever regressed

They run the same docker build -f docker/Dockerfile.fips .; the grpcio stage compiles the wheel once (about 6 minutes on 8 cores) and is cached afterwards
fips-check imports grpc._cython.cygrpc, objdump -p lists NEEDED libcrypto.so.3, and the gate prints PASS
Rebuilding with the PyPI wheel instead now ends the build with VIOLATION .../cygrpc.cpython-313-ARCH-linux-gnu.so [BoringSSL]

Linear ticket

Resolves LIT-8435

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Setup: the Chainguard FIPS base needs a registry token this run does not have, so the grpcio build stage was replayed byte for byte on python:3.13-slim (build-essential libssl-dev zlib1g-dev binutils, uv 0.11.7), and the scanner was run on the result. Integration rig: real proxy with two workers, Postgres, Redis and the scripted upstream from tests/integration/_support/upstream.py, no real provider

Before (merge-base `09ebb28` with the pre-fix scanner from 41802 tip `e43e984`)

Scanner gate on the PyPI grpcio wheel

uv run --no-sync python -m pytest tests/integration/configuration/test_fips_bundled_crypto_scan.py tests/integration/configuration/test_proxy_without_grpc_stack.py -p no:pytest-retry -p no:rerunfailures --hypothesis-seed=8435 --integration-order-seed=8435, run id 2026-09-24T09:13:56Z
test_bundled_boringssl_grpcio_fails_the_fips_gate FAILED: WAIVED .../grpc/_cython/cygrpc.cpython-313-x86_64-linux-gnu.so [BoringSSL], PASS: all 1 ELF binaries accounted for, exit 0 instead of 1; the other 7 nodes passed

grpcio build against system OpenSSL

objdump -p .venv/lib/python3.12/site-packages/grpc/_cython/cygrpc*.so | grep NEEDED on the PyPI wheel: no libssl.so.3 or libcrypto.so.3
grep -ac BoringSSL on that file: 1

After (`fa694bd`)

Audit matrix, two runs with identical seeds, identical order, 0 skipped, 0 retries

Run ids 2026-09-24T09:23:31Z (8 passed, 62.71s) and 2026-09-24T09:24:35Z (8 passed, 62.92s). Nodes under tests/integration/configuration/:

Cell	Node	Base	Head
bundled BoringSSL grpcio at inventory path fails	`test_fips_bundled_crypto_scan.py::test_bundled_boringssl_grpcio_fails_the_fips_gate`	FAIL (waived)	PASS
libcrypto.so.3 linked ELF at inventory path is clean	`::test_system_openssl_linked_grpcio_at_its_inventory_path_is_clean`	PASS	PASS
unreviewed ELF outside inventory fails	`::test_unreviewed_elf_outside_the_inventory_fails_the_gate`	PASS	PASS
empty tree passes	`::test_empty_tree_passes`	PASS	PASS
POST /v1/chat/completions, upstream body equality	`test_proxy_without_grpc_stack.py::test_chat_completions_serves_without_the_grpc_stack`	PASS	PASS
POST /v1/chat/completions stream, last chunk id equals upstream id	`::test_streaming_chat_completion_serves_without_the_grpc_stack`	PASS	PASS
POST /v1/messages and POST /v1/responses	`::test_messages_and_responses_serve_without_the_grpc_stack`	PASS	PASS
chaos: SIGKILL one of two workers, survivor creates a model and serves 5 requests	`::test_surviving_worker_keeps_serving_without_the_grpc_stack`	PASS	PASS

Scanner gate on the PyPI grpcio wheel

The scanner now prints VIOLATION .../cygrpc.cpython-313-x86_64-linux-gnu.so [BoringSSL] and exits 1 for the PyPI wheel, CLEAN for a libcrypto.so.3 linked ELF at the inventory path, VIOLATION [unreviewed] outside it

grpcio build against system OpenSSL

docker build of the replayed grpcio stage: GRPCIO_VERSION 1.78.0 read from uv.lock, wheel compiled in 6m07s wall on 8 cores, pip wheel chain exit 0
objdump -p on the installed, stripped cygrpc: NEEDED libssl.so.3, NEEDED libcrypto.so.3, NEEDED libz.so.1, libstdc++.so.6, libm.so.6, libc.so.6; grep -a BoringSSL matches nothing; size 14,422,672 bytes
python -c "import grpc._cython.cygrpc; import grpc; grpc.secure_channel('127.0.0.1:1', grpc.ssl_channel_credentials()).close(); print(grpc.__version__)" prints 1.78.0
docker/scan_bundled_crypto.sh on that tree with the repo inventory: CLEAN .../cygrpc.cpython-313-x86_64-linux-gnu.so, PASS: all 1 ELF binaries accounted for

Proxy without grpcio, polars, google.cloud

Proxy booted with two workers under a sitecustomize that makes grpc, polars, google.cloud and google.api_core unimportable (python -c "import grpc" fails under it)
POST /v1/chat/completions non streaming: 200, chatcmpl- id, upstream received exactly one request with the sent body; streaming: last SSE chunk id equals the upstream's
POST /v1/responses: 200 with resp_ id and the upstream text; POST /v1/messages: 200 with one text block carrying the upstream text
Proxy log contains no ModuleNotFoundError for grpc or polars

Live A/B of the two wheels against real Google endpoints

The diff moves no Python under litellm/, so the live leg compares the PyPI grpcio 1.78.0 (bundled BoringSSL) against the source-built grpcio 1.78.0 (NEEDED libcrypto.so.3) in the same process shape

grpc.secure_channel to aiplatform.googleapis.com:443 and generativelanguage.googleapis.com:443 with grpc.ssl_channel_credentials(), channel_ready_future(...).result(timeout=15): READY on both wheels
litellm.completion(model="vertex_ai/gemini-2.5-flash", ...) with a real service account: 200 on both wheels, ids O--0au-wJfe00ekP4b7xwAM (PyPI) and V--0au2... (source built)
google.cloud.aiplatform ModelServiceClient.list_models over ModelServiceGrpcTransport: RPC succeeded on both wheels, so cygrpc TLS was on the wire through the system OpenSSL

Mutation check on the tests: re-adding the waiver, downgrading the VIOLATION line to WAIVED, deleting the cygrpc inventory line, adding import polars or import grpc at the top of proxy_server.py, and killing the master instead of a worker each turned the matching tests red. Waiving every path did not kill the unreviewed ELF test because that scanner branch never consults is_waived, so the kill rate is 6 of 7, not above 90 percent

Type

Infrastructure

Caveats (if any)

Medium

Stacked on feat(docker): add FIPS 140-3 image with build-time crypto verification #41802: the docker/ FIPS files come from litellm_fips_base_image, so this diff shows them until that PR lands
The real Chainguard build was not run here (no cgr.dev token in the Shared vault); the stage was replayed on python:3.13-slim with the identical commands
First FIPS build gets about 6 minutes slower on 8 cores; later builds reuse the cached layer until uv.lock moves grpcio

Low

CircleCI at fa694bd (pipeline 90280): integration-management (job 2211080, the shard that runs the two new files) and the database, replica, sdk, providers, cost and browser shards are green. Three shards are red with failures that are red on main at the same time: integration-mcp 2211079 (test_malformed_bodies_missing_users_and_foreign_servers_are_rejected, main 2210684), integration-extensions 2211088 (test_redis_outage_keeps_serving_in_memory_hits and test_passthrough_worker_sigkill_leaves_sibling_serving_and_logging, main 2210686 and 2210285), integration-accounting 2211081 and one rerun 2211256 (two different tests, each also failing on main in 2210281, 2210055, 2209497). local_testing_part1 and using_litellm_on_windows fail on main the same way (2210327, 2210297). This diff moves no Python under litellm/
uv venv --seed pulls pip for the sdist build; uv itself is pinned, pip is not
polars, ddtrace and Prisma stay waived; they belong to the A and D tickets. The proxy is proven to boot without polars and grpcio, so --no-install-package polars is safe when A decides to use it

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/2a0cd462da5846baa5db8b7c178f99d6
Open in Devin Desktop: https://app.devin.ai/desktop/session/2a0cd462da5846baa5db8b7c178f99d6?variant=devin
Requested by: @yucheng-berri

Dockerfile.fips builds on chainguard-base-fips with FIPS-provider-linked python/node/openssl from the org apk repo (CMVP #5132). A fips-check stage runs openssl-fips-test and an ELF allowlist gate: every native binary must be in the reviewed fips_elf_inventory.txt or carry an explicit bundled-crypto waiver, so new native deps fail the build until reviewed. Known offenders are waived pending the dependency-fix workstream and the report ships at /fips-scan-report.txt

…tellm_fips_c4_grpcio_system_openssl

…its scanner waiver Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration · 2026-09-24T08:15:26Z

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

Disable automatic comment, CI, and merge conflict monitoring

mateo-berri · 2026-09-24T08:15:57Z

bugbot run

… job holds) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

mateo-berri · 2026-09-24T08:17:05Z

bugbot run

greptile-apps · 2026-09-24T08:20:04Z

Confidence Score: 5/5

The PR appears safe to merge; no outstanding review finding remains.

Summary

This PR adds a FIPS image build that compiles grpcio against system OpenSSL and removes the scanner waiver for its bundled crypto.

Adds build-time linkage and bundled-crypto checks, plus an ELF inventory.
Adds scanner and proxy integration tests and records why the Dockerfile is exempt from PR build coverage.

_{Reviews (4) · Last reviewed commit: "test(fips): isolate the worker kill cell..."}

codecov · 2026-09-24T08:20:43Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

cursor

Stale Bugbot comment from a previous run.

cursor · 2026-09-24T08:22:41Z

+    */hf_xet/hf_xet.abi3.so) ;;
+    */_awscrt.abi3.so) ;;
+    */pyroscope/_native__lib*.so) ;;
+    */psycopg_binary.libs/libcrypto-*) ;;


psycopg libcrypto waiver never matches

High Severity

is_waived matches psycopg_binary.libs/libcrypto-* (hyphen), but the inventoried files are libcrypto.so.3 and libcrypto.so.1.1.1k. Those copies carry OpenSSL version strings and do not NEEDED libcrypto.so.3, so the gate reports VIOLATION and fails fips-check.

Additional Locations (1)

docker/fips_elf_inventory.txt#L81-L83

^{Reviewed by Cursor Bugbot for commit e707821. Configure here.}

The scanned path keeps the auditwheel hash (libcrypto-2a5143d4.so.3), so the hyphen pattern matches. Only the inventory line is normalized to libcrypto.so.3

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

mateo-berri · 2026-09-24T08:23:31Z

bugbot run

cursor

Stale Bugbot comment from a previous run.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

…tack Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

mateo-berri · 2026-09-24T08:51:53Z

bugbot run

mateo-berri · 2026-09-24T08:52:00Z

@greptileai

cursor

Stale Bugbot comment from a previous run.

cursor · 2026-09-24T08:56:44Z

+        bodies: Final = tuple(JSON_OBJECT.validate_json(response.content) for response in responses)
+        for response, body in zip(responses, bodies):
+            assert response.status_code == 200, response.text
+            assert isinstance(body["id"], str) and body["id"].startswith("chatcmpl-"), response.text


No retry after worker SIGKILL

Medium Severity

After SIGKILL, the test immediately calls scenario.model() and five chat completions on the reused owned.gateway client. Gateway.request does not catch transport errors, and Gateway.post asserts 200 on the first try. A reset keep-alive to the victim or a briefly unready respawn fails the test instead of waiting for the survivor to accept again.

^{Reviewed by Cursor Bugbot for commit c7cf432. Configure here.}

Fixed in fa694bd: a fresh httpx client probes the survivor inside eventually before the burst

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

mateo-berri · 2026-09-24T09:13:49Z

bugbot run

mateo-berri · 2026-09-24T09:13:50Z

@greptileai

cursor

✅ Bugbot reviewed your changes and found no new issues!

1 issue from previous review remains unresolved.

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit fa694bd. Configure here.}

mateo-berri · 2026-09-24T09:38:23Z

@greptileai

ojensen-berri and others added 3 commits September 18, 2026 12:29

Merge remote-tracking branch 'origin/litellm_fips_base_image' into li…

53cb157

…tellm_fips_c4_grpcio_system_openssl

feat(fips): build grpcio from source against system OpenSSL and drop …

401c320

…its scanner waiver Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration Bot requested a review from a team September 24, 2026 08:15

devin-ai-integration Bot assigned yucheng-berri Sep 24, 2026

ci: allowlist docker/Dockerfile.fips (requires Chainguard token no PR…

e707821

… job holds) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

greptile-apps Bot reviewed Sep 24, 2026

View reviewed changes

Comment thread docker/scan_bundled_crypto.sh

Comment thread docker/scan_bundled_crypto.sh

cursor Bot reviewed Sep 24, 2026

View reviewed changes

test(fips): spawn the no-grpc probe interpreter with -P (TQ009)

355986a

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

cursor Bot reviewed Sep 24, 2026

View reviewed changes

yucheng-berri and others added 2 commits September 24, 2026 08:49

fix(fips): run the grpcio version stage as root

d610c52

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

test(fips): prove a surviving worker keeps serving without the grpc s…

c7cf432

…tack Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

greptile-apps Bot reviewed Sep 24, 2026

View reviewed changes

Comment thread .github/ci-coverage-allowlist.yml

Comment thread tests/integration/configuration/test_proxy_without_grpc_stack.py Outdated

cursor Bot reviewed Sep 24, 2026

View reviewed changes

test(fips): isolate the worker kill cell on its own proxy

fa694bd

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

greptile-apps Bot reviewed Sep 24, 2026

View reviewed changes

Comment thread tests/integration/configuration/test_proxy_without_grpc_stack.py

cursor Bot reviewed Sep 24, 2026

View reviewed changes

devin-ai-integration Bot mentioned this pull request Sep 24, 2026

chore(techdebt): clear fresh tech debt from the last 24 hours (rolling, 2026-09-06 to 2026-09-24) #42710

Merged

5 of 7 tasks

Conversation

devin-ai-integration Bot commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Linear ticket

Pre-Submission checklist

Screenshots / Proof of Fix

Before (merge-base 09ebb28 with the pre-fix scanner from 41802 tip e43e984)

Scanner gate on the PyPI grpcio wheel

grpcio build against system OpenSSL

After (fa694bd)

Audit matrix, two runs with identical seeds, identical order, 0 skipped, 0 retries

Scanner gate on the PyPI grpcio wheel

grpcio build against system OpenSSL

Proxy without grpcio, polars, google.cloud

Live A/B of the two wheels against real Google endpoints

Type

Caveats (if any)

Medium

Low

Final Attestation

Uh oh!

devin-ai-integration Bot commented Sep 24, 2026

Uh oh!

mateo-berri commented Sep 24, 2026

Uh oh!

mateo-berri commented Sep 24, 2026

Uh oh!

greptile-apps Bot commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Sep 24, 2026

Codecov Report

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor Bot Sep 24, 2026

Choose a reason for hiding this comment

psycopg libcrypto waiver never matches

Uh oh!

devin-ai-integration Bot Sep 24, 2026

Choose a reason for hiding this comment

Uh oh!

mateo-berri commented Sep 24, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mateo-berri commented Sep 24, 2026

Uh oh!

mateo-berri commented Sep 24, 2026

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cursor Bot Sep 24, 2026

Choose a reason for hiding this comment

No retry after worker SIGKILL

Uh oh!

devin-ai-integration Bot Sep 24, 2026

Choose a reason for hiding this comment

Uh oh!

mateo-berri commented Sep 24, 2026

Uh oh!

mateo-berri commented Sep 24, 2026

Uh oh!

Uh oh!

devin-ai-integration Bot commented Sep 24, 2026 •

edited

Loading

Before (merge-base `09ebb28` with the pre-fix scanner from 41802 tip `e43e984`)

After (`fa694bd`)

greptile-apps Bot commented Sep 24, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading