fix(componentized): honor USE_DDTRACE in the gateway and backend deployments

[yassin-berriai]

TLDR

Problem this solves:

• Componentized deployments never run ddtrace-run
• Datadog APM gets no HTTP request spans from them
• Terraform overrides bypass the image entrypoint too

How it solves it:

• Shared entrypoint wraps uvicorn when USE_DDTRACE=true
• Terraform inlines the same check, so old images trace too
• Exports DD_TRACE_OPENAI_ENABLED=False like the monolith

Relevant issues

Fixes #34251

Linear ticket

Resolves LIT-4727

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

The end-user surface here is the container, so the proof runs the real images against a trace agent. The agent is a small HTTP server standing in for the Datadog agent; it msgpack-decodes the v0.5 payloads and prints the span names in each one, which is what makes the difference legible. No mocks inside the image; the gateway boots its real lifespan and serves real traffic

The headline is the span names. Before, the agent receives orphaned fastapi.serialize_response children and a fastapi.request count of 0. After, that count is 6, one per served request. That is what a Datadog user actually sees: today the split deployment produces traces with no root span, so APM has no route, method, status or end-to-end latency to show

Before is commit 0a42f288 and after is commit 203844df

# confirm the "before" tree really lacks the wiring
$ git grep -c ddtrace 0a42f288 -- gateway/Dockerfile backend/Dockerfile
(no output; zero matches)

$ docker build -f gateway/Dockerfile -t lit4727-ddt-gateway:base .    # at 0a42f288
$ python fake_dd_agent.py 18126 > agent.log &
$ docker run -d --name lit4727-ddt-gw-base -p 14727:4000 \
    --add-host host.docker.internal:host-gateway \
    -e USE_DDTRACE=true -e DD_TRACE_AGENT_URL=http://host.docker.internal:18126 \
    -e NUM_WORKERS=2 -e CONFIG_FILE_PATH=/app/config.yaml -e LITELLM_MASTER_KEY=sk-... \
    -v $PWD/config.yaml:/app/config.yaml:ro lit4727-ddt-gateway:base
$ for i in 1 2 3 4 5; do curl -s -o /dev/null -w "%{http_code} " http://127.0.0.1:14727/health/liveliness; done
200 200 200 200 200

$ grep /traces agent.log
POST /v0.5/traces bytes=751 spans=['fastapi.serialize_response', 'fastapi.serialize_response', 'fastapi.serialize_response', 'fastapi.serialize_response', 'fastapi.serialize_response']
$ grep -c fastapi.request agent.log
0

Child spans arrive, no root request span ever does. A trace with no root span carries no route, method, status or end-to-end latency, so the split deployment shows nothing usable in APM

Same commands against the fixed image:

$ docker build -f gateway/Dockerfile -t lit4727-ddt-gateway:fixed .   # at 203844df
$ docker run -d --name lit4727-ddt-gw-fixed -p 14727:4000 ...same flags... lit4727-ddt-gateway:fixed
$ for i in 1 2 3 4 5; do curl -s -o /dev/null -w "%{http_code} " http://127.0.0.1:14727/health/liveliness; done
200 200 200 200 200

$ grep /traces agent.log
POST /v0.5/traces bytes=688  spans=['http.request']
POST /v0.5/traces bytes=688  spans=['http.request']
POST /v0.5/traces bytes=1973 spans=['fastapi.request', 'fastapi.serialize_response', 'fastapi.request', 'fastapi.serialize_response', 'fastapi.request', 'fastapi.serialize_response', 'fastapi.request', 'fastapi.serialize_response', 'fastapi.request', 'fastapi.serialize_response', 'fastapi.request', 'fastapi.serialize_response']
$ grep -o fastapi.request agent.log | wc -l
6

Six root spans for six served requests, counting the readiness probe

ddtrace-run reaches every uvicorn worker, read straight out of the running container rather than inferred from the payload batching:

$ docker exec lit4727-ddt-gw-fixed sh -c 'for p in /proc/[0-9]*; do ... done'
pid=1  uvicorn gateway.main:app --workers 2 --host 0.0.0.0 --port 4000
    PYTHONPATH=/app/.venv/lib/python3.13/site-packages/ddtrace/bootstrap:/app
    DD_TRACE_OPENAI_ENABLED=False
pid=17 python -B -c from multiprocessing.spawn import spawn_main; ... --multiprocessing-fork
    PYTHONPATH=/app/.venv/lib/python3.13/site-packages/ddtrace/bootstrap:/app
    DD_TRACE_OPENAI_ENABLED=False
pid=18 python -B -c from multiprocessing.spawn import spawn_main; ... --multiprocessing-fork
    PYTHONPATH=/app/.venv/lib/python3.13/site-packages/ddtrace/bootstrap:/app
    DD_TRACE_OPENAI_ENABLED=False

The bootstrap directory is prepended to the image's own PYTHONPATH=/app rather than replacing it, and it survives uvicorn's spawn into both workers. This is the same arrangement the monolith already ships, where ddtrace-run litellm fronts a multi-worker uvicorn

The backend image, which has no worker knob:

$ for i in 1 2 3 4 5; do curl -s -o /dev/null -w "%{http_code} " http://127.0.0.1:14729/health/liveliness; done
200 200 200 200 200
$ grep /traces agent.log
POST /v0.5/traces bytes=675  spans=['http.request']
POST /v0.5/traces bytes=1960 spans=['fastapi.request', 'fastapi.serialize_response', ...x6]

With USE_DDTRACE unset the fixed image is unchanged from today; the wrapper execs uvicorn with the same argv and touches no environment:

$ docker run -d --name lit4727-ddt-gw-noddt -p 14728:4000 ...no USE_DDTRACE... lit4727-ddt-gateway:fixed
$ for i in 1 2 3 4 5; do curl -s -o /dev/null -w "%{http_code} " http://127.0.0.1:14728/health/liveliness; done
200 200 200 200 200
$ docker exec lit4727-ddt-gw-noddt sh -c '...'
cmd=/app/.venv/bin/python /app/.venv/bin/uvicorn gateway.main:app --workers 2 --host 0.0.0.0 --port 4000
PYTHONPATH=/app
$ grep -c /traces agent.log
0

The Terraform change was checked against real images rather than only in the unit tests, using the command string resolved straight out of cloudrun.tf. The image under test is ghcr.io/berriai/litellm-gateway:v1.86.0-dev, which is what both modules default to and which predates this PR, so it does not contain docker/component_entrypoint.sh at all:

$ CMD='if [ "$USE_DDTRACE" = "true" ]; then export DD_TRACE_OPENAI_ENABLED="False"; exec ddtrace-run uvicorn gateway.main:app --host 0.0.0.0 --port 4000 --workers 2; else exec uvicorn gateway.main:app --host 0.0.0.0 --port 4000 --workers 2; fi'

$ docker run --rm --entrypoint sh ghcr.io/berriai/litellm-gateway:v1.86.0-dev -c 'ls /app/docker/component_entrypoint.sh; command -v ddtrace-run'
ls: /app/docker/component_entrypoint.sh: No such file or directory
/app/.venv/bin/ddtrace-run

$ docker run -d --name pub -p 14734:4000 -e USE_DDTRACE=true ... \
    --entrypoint sh ghcr.io/berriai/litellm-gateway:v1.86.0-dev -c "$CMD"
$ for i in 1 2 3 4 5; do curl -s -o /dev/null -w "%{http_code} " http://127.0.0.1:14734/health/liveliness; done
200 200 200 200 200

$ docker exec pub sh -c 'tr "\0" "\n" < /proc/1/environ | grep -E "^(PYTHONPATH|DD_TRACE_OPENAI_ENABLED)="'
DD_TRACE_OPENAI_ENABLED=False
PYTHONPATH=/app/.venv/lib/python3.13/site-packages/ddtrace/bootstrap:/app

$ grep -o fastapi.request agent.log | wc -l
6

That is the point of inlining. An image published long before this PR, with no entrypoint script anywhere in it, starts emitting root request spans the moment the module is applied. Had Terraform called the script instead, this same deployment would either have failed to start or, with an existence check in front of it, come up silently untraced

Type

🐛 Bug Fix

Changes

USE_DDTRACE was not being wholly ignored in the componentized images, which is worth stating precisely because it explains why the symptom looks partial rather than absent. gateway/main.py and backend/main.py wrap proxy_server's lifespan, so ProxyStartupEvent._init_dd_tracer still runs ddtrace.patch_all(logging=True, openai=False), and litellm/litellm_core_utils/dd_tracing.py still binds the real tracer at import time, so litellm's own manual spans emit normally

What never gets installed is ddtrace's ASGI TraceMiddleware. The fastapi integration works by wrapping FastAPI.build_middleware_stack, and starlette builds that stack lazily on the first __call__, which is the lifespan scope. By the time patch_all runs inside the lifespan body the stack already exists, and adding middleware after startup raises. The monolith does not hit this because ddtrace-run installs its sitecustomize bootstrap at interpreter start, before import fastapi. The same timing argument covers the httpx, redis and aiohttp references litellm binds at module import

The componentized images bypassed docker/prod_entrypoint.sh entirely and exec'd uvicorn directly, so nothing ever wrapped the interpreter. Both now route through a new docker/component_entrypoint.sh that prefixes the command with ddtrace-run when the flag is on and execs it untouched otherwise. It takes the whole command rather than an app target on purpose: the gateway honors NUM_WORKERS and the backend deliberately does not, so leaving argv construction in each Dockerfile keeps that asymmetry intact instead of quietly granting the backend workers it never had

The DD_TRACE_OPENAI_ENABLED=False export carries over from docker/prod_entrypoint.sh, and it matters more under ddtrace-run than it looks. The bootstrap patches the openai integration before any litellm code runs, so the in-process patch_all(..., openai=False) cannot suppress it afterwards. Without the export every LLM call gets instrumented twice, once by ddtrace and once by litellm's own Datadog and LLM Observability loggers, and ddtrace additionally captures prompt and completion payloads that litellm's redaction settings are supposed to govern

The wrapping stays behind the USE_DDTRACE branch rather than becoming unconditional, since ddtrace-run always starts the tracer and a writer thread that tries to reach an agent whether or not one exists

Fixing the images alone would have left this half done, because this repo's own Terraform modules override the image entrypoint and would have kept exec'ing uvicorn directly. Those overrides are already shell strings that spell out the uvicorn invocation, so they now spell out the tracing decision too:

if [ "$USE_DDTRACE" = "true" ]; then export DD_TRACE_OPENAI_ENABLED="False"; exec ddtrace-run uvicorn gateway.main:app <args>; else exec uvicorn gateway.main:app <args>; fi

Inlining rather than calling the script is deliberate. Both modules take a caller-supplied image tag and both default to v1.86.0-dev, published long before this PR creates docker/component_entrypoint.sh. Terraform therefore cannot reference that file: exec'ing a path the image does not contain fails at container start, and testing for it first would only downgrade those deployments to silently untraced, which is the exact failure this change exists to remove. ddtrace-run on the other hand is already present in every componentized image ever published, because ddtrace ships in the proxy-runtime extra that both Dockerfiles have always installed. Writing the two-line decision inline means old images start tracing as soon as the module is applied, new images do the same, and nothing has to be released in a particular order

On ECS the gateway and backend prepend an S3 config fetch to that command when proxy_config is set; the gateway's other branch previously overrode entryPoint as an exec-form array, which cannot express a conditional, so it now uses the same sh -c shape as its sibling with the uvicorn args folded into the string. On Cloud Run both services join the command onto their existing startup fragments. The ECS backend has one branch that overrides nothing and inherits the image ENTRYPOINT, so it needed no change and got none. One comment went away with the branch it described, since it explained appending --workers through command, which that branch no longer does

The decision now lives in two places, docker/component_entrypoint.sh for image-default startup and the Terraform strings for override startup, and that duplication is the price of not depending on a file the caller's image may lack. The tests pin the two to the same contract rather than trusting them to stay in step

Tests live in tests/test_litellm/test_component_entrypoint.py and run in the misc unit shard. They drive the script with stub ddtrace-run and uvicorn executables on PATH and assert which one got exec'd, that the openai integration was disabled on that branch only, and that PYTHONPATH passes through untouched so ddtrace-run still has something to prepend to. A parametrized case pins the componentized gating to docker/prod_entrypoint.sh across unset, empty, false, True, TRUE, 1 and yes, so the two entrypoints cannot drift apart. The rest assert the wiring itself: both images invoke uvicorn through the entrypoint, both make it executable, the gateway keeps NUM_WORKERS and the backend stays single-process

The Terraform side gets the same treatment, and it is the only guard those files have, since no CI job lints or validates terraform/litellm/. Rather than matching strings, the tests resolve each launch command out of the .tf file, expand its interpolations, and run it against the same stub binaries under USE_DDTRACE unset, true, false and True, asserting it reaches the same verdict as docker/component_entrypoint.sh run under that same value: the same binary exec'd, and the openai integration disabled on the traced branch only. That is what keeps the two copies of the decision honest. Alongside it, every line naming a component ASGI target must honor the knob, the launch-site count is pinned so a deleted site cannot quietly shrink the scan, and the modules are asserted to contain no reference to the entrypoint script path at all

Reverting the Dockerfile wiring while leaving the script in place, which is exactly the shipped bug, fails four of these tests; removing only the DD_TRACE_OPENAI_ENABLED export fails one; overwriting PYTHONPATH in the wrapper fails one. Collapsing either module's conditional to a bare uvicorn exec fails three tests in that module, dropping only its DD_TRACE_OPENAI_ENABLED export fails two, and bypassing any one of the five individual launch sites fails that module's wiring assertion

The PYTHONPATH check is worth a note, because its first version was worthless and mutation testing is what exposed that. It originally used /app as the fixture value, the same thing the images themselves set. Mutating the wrapper to export PYTHONPATH="/app", which is precisely the bug the assertion exists to catch, left all tests green: the assertion was satisfied by the defect. The sentinel is now a value the wrapper could not plausibly invent, and the identical mutation fails

One thing this PR does not touch: docker/build_from_pip/Dockerfile.build_from_pip pins ddtrace yet has a bare ENTRYPOINT ["litellm"], so it carries the same defect. Keeping the scope to the two componentized images

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

This PR enables Datadog bootstrap tracing for componentized gateway and backend deployments while retaining compatibility with images that predate the new entrypoint script.

• Adds a shared tracing-aware component entrypoint and wires both component Dockerfiles through it.
• Replicates the tracing decision inline in ECS and Cloud Run launch overrides so overridden entrypoints still honor USE_DDTRACE.
• Adds tests for flag semantics, command forwarding, worker behavior, executable permissions, and Terraform launch-site coverage.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the deployment overrides now honor tracing without requiring older or custom images to contain the newly added entrypoint script.

Important Files Changed

Filename	Overview
docker/component_entrypoint.sh	Adds a transparent command wrapper that invokes ddtrace-run only when USE_DDTRACE=true and otherwise preserves the original command.
gateway/Dockerfile	Routes gateway startup through the shared wrapper while preserving worker expansion and CMD forwarding.
backend/Dockerfile	Routes backend startup through the shared wrapper without changing its single-process invocation.
terraform/litellm/aws/ecs.tf	Inlines tracing-aware launch commands in ECS overrides, avoiding a dependency on the new script in older or custom images.
terraform/litellm/gcp/cloudrun.tf	Adds the tracing-aware launch conditional as the final stage of the existing Cloud Run initialization chain.
tests/test_litellm/test_component_entrypoint.py	Exercises wrapper behavior and verifies that Docker and Terraform launch paths remain aligned.

_{Reviews (4): Last reviewed commit: "fix(docker): honor USE_DDTRACE in the co..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[yassin-berriai]

@greptileai please re-review at c00462c

Verified the entrypoint-override finding and fixed it. The Terraform modules did override the image ENTRYPOINT, so USE_DDTRACE=true stayed inert on those paths even with the wrapper shipped. Five launch sites now route through /app/docker/component_entrypoint.sh: on ECS the gateway and backend sh -c branches plus the gateway exec-form entryPoint, and on Cloud Run the gateway and backend arg strings. The path lives in a component_entrypoint local per module so the stacks cannot drift from the images.

The ECS backend_proxy_overrides else branch is {} and inherits the image ENTRYPOINT, so it was already covered and is deliberately unchanged.

Tests extended in tests/test_litellm/test_component_entrypoint.py: every line naming a component ASGI target must reference the entrypoint local, the launch-site count is pinned so a deleted site cannot silently shrink the scan, and the local is checked against the path the images ship. Each of the five sites was reverted individually to confirm the matching assertion fails. terraform fmt -check is clean on both modules.

[yassin-berriai]

@greptileai please re-review at d27dc73

Verified this and fixed it. Checked the modules own defaults, aws/variables.tf pins litellm-gateway:v1.86.0-dev and gcp/variables.tf defaults image_tag to the same, so the break was not limited to custom images; the stock configuration would have failed at container start on a missing file.

Both modules now build their command from a <component>_launch_cmd local shaped if [ -x <script> ]; then exec <script> uvicorn <app> <args>; else exec uvicorn <app> <args>; fi. Images without the script start exactly as before, images that ship it pick up tracing. The ECS non-proxy_config branch moved from an exec-form entryPoint array to sh -c, since an array cannot express the conditional.

Confirmed against real images with the command resolved out of cloudrun.tf: a pre-change image serves 200 with no missing-file error, and an image from this revision with USE_DDTRACE=true shows the ddtrace bootstrap prepended to PYTHONPATH and 6 fastapi.request spans for 6 requests. The same command without the guard fails with stat /app/docker/component_entrypoint.sh: no such file or directory.

Tests now resolve each launch command from the .tf and execute it against stub binaries twice, script present and absent, asserting ddtrace-run then bare uvicorn. Collapsing either guard back to an unconditional exec fails that module two execution tests. terraform fmt -check clean on both modules.

[yassin-berriai]

@greptileai please re-review at 203844d

Verified the compatibility concern and removed the file dependency entirely rather than guarding it. Checked the modules own defaults: aws/variables.tf pins litellm-gateway:v1.86.0-dev and gcp/variables.tf defaults image_tag to the same, so the stock configuration was affected, not only custom images.

Terraform no longer references /app/docker/component_entrypoint.sh in any form. The override strings already spell out the uvicorn invocation, so they now spell out the tracing decision inline: if [ "$USE_DDTRACE" = "true" ]; then export DD_TRACE_OPENAI_ENABLED="False"; exec ddtrace-run uvicorn <app> <args>; else exec uvicorn <app> <args>; fi. ddtrace-run is present in every componentized image ever published, since ddtrace ships in the proxy-runtime extra both Dockerfiles have always installed, so this needs no particular image version.

Confirmed against the published default tag itself. That image contains no entrypoint script (ls returns no such file) yet serves 200s under the new command and emits 6 fastapi.request spans for 6 requests, with the ddtrace bootstrap prepended to PYTHONPATH. An existence check would have left that same deployment starting but silently untraced, which is the failure this PR exists to remove.

The decision now lives in two places, the script for image-default startup and the Terraform strings for override startup. Tests pin them to the same contract: each Terraform command is resolved from the .tf, executed against stub binaries under USE_DDTRACE unset/true/false/True, and asserted to reach the same verdict as the script under the same value. Collapsing either conditional fails three tests, dropping either openai export fails two, bypassing any one of the five launch sites fails that module wiring assertion. terraform fmt -check clean.