fix(helm): give gateway and backend probes an explicit timeoutSeconds

[yassin-berriai]

TLDR

Problem this solves:

• Gateway and backend probes had no timeoutSeconds, so kubelet used 1s
• A busy single-worker pod can't answer either probe in 1s
• Stage sees both probes time out on pods that are serving fine
• That risks dropping healthy pods from the LB and restarting busy ones

How it solves it:

• Sets timeoutSeconds: 10 on both probes, gateway and backend
• Sets liveness failureThreshold: 6 so a busy pod is not restarted
• Adds a mutation-checked helm-unittest suite pinning those probe blocks

Relevant issues

Linear ticket

Resolves LIT-5054

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Field evidence (observed on the stage EKS cluster by a maintainer, before this change)

The deployed stage gateway spec carries the implicit default: timeoutSeconds: 1, periodSeconds: 10, failureThreshold: 3. Under closed-loop load Kubernetes logged probe timeouts repeatedly and reproducibly, across several pods and several runs, for both endpoints:

Warning  Unhealthy  pod/litellm-gateway-...  Readiness probe failed:
  Get "http://.../health/readiness": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

and the same for /health/liveliness. Per-request latency on that replica measured ~57-62ms serial and ~6s at 100 concurrent users, which is what pushes both probes past a 1s budget

The consequence is stated here as risk, not as an observed outage. A 100-user, 180s run against a settled single replica came back inside the 1% failure ceiling even while readiness probes were still timing out during it, so no specific request-error storm is attributed to the probe timeout. What stands on its own is that probes time out against a pod that is serving traffic correctly, and what that exposes the deployment to: readiness failures pull a healthy pod out of its load balancer during a burst, and liveness failures restart a merely busy pod, each removing capacity at the moment it is needed

Rendered probes, before (base commit de43328f63)

$ helm template rel ./helm/litellm -f helm/litellm/tests/values/required.yaml

          livenessProbe:
            httpGet:
              path: /health/liveliness
              port: http
            initialDelaySeconds: 10
            periodSeconds: 15
          readinessProbe:
            httpGet:
              path: /health/readiness
              port: http
            initialDelaySeconds: 5
            periodSeconds: 10

Rendered probes, after (commit bca4bd4b36)

$ helm template rel ./helm/litellm -f helm/litellm/tests/values/required.yaml

          livenessProbe:
            failureThreshold: 6
            httpGet:
              path: /health/liveliness
              port: http
            initialDelaySeconds: 10
            periodSeconds: 15
            timeoutSeconds: 10
          readinessProbe:
            httpGet:
              path: /health/readiness
              port: http
            initialDelaySeconds: 5
            periodSeconds: 10
            timeoutSeconds: 10

The full rendered diff is exactly two hunks, one for the gateway Deployment and one for the backend Deployment, both of the shape above. The ui Deployment renders byte-identically to before

helm-unittest (commit bca4bd4b36)

$ helm unittest -f 'tests/*.yaml' helm/litellm
### Chart [ litellm ] helm/litellm

 PASS  test billingMetrics wiring on gateway and backend	helm/litellm/tests/billing_metrics_tests.yaml
 PASS  test deployment volumes and volumeMounts	helm/litellm/tests/deployment_volumes_tests.yaml
 PASS  test pod disruption budgets and topology spread constraints	helm/litellm/tests/pdb_topology_spread_tests.yaml
 PASS  test liveness and readiness probe timeouts	helm/litellm/tests/probe_tests.yaml
 PASS  test redis coordination env vars	helm/litellm/tests/redis_env_tests.yaml

Charts:      1 passed, 1 total
Test Suites: 5 passed, 5 total
Tests:       44 passed, 44 total

$ helm unittest -f 'tests/*.yaml' helm/litellm-helm
Charts:      1 passed, 1 total
Test Suites: 11 passed, 11 total
Tests:       90 passed, 90 total

Mutation check, the other direction. Reverting helm/litellm/values.yaml to its pre-change state (restored afterwards from a copy taken before any edit, never with git checkout) and re-running the new suite fails 4 of its 5 tests and every assertion that is about the defaults:

$ helm unittest -f 'tests/probe_tests.yaml' helm/litellm    # values.yaml reverted
 FAIL  test liveness and readiness probe timeouts	helm/litellm/tests/probe_tests.yaml
		- asserts[0] `equal` fail  Path: ...containers[0].livenessProbe    (gateway)
		- asserts[1] `equal` fail  Path: ...containers[0].readinessProbe   (gateway)
		- asserts[0] `equal` fail  Path: ...containers[0].livenessProbe    (backend)
		- asserts[1] `equal` fail  Path: ...containers[0].readinessProbe   (backend)
		- asserts[0] `isNotNullOrEmpty` fail                               (liveness timeout guard)
		- asserts[1] `isNotNullOrEmpty` fail                               (readiness timeout guard)
		- asserts[2] `equal` fail                                          (liveness timeout == 10)
		- asserts[3] `equal` fail                                          (readiness timeout == 10)
		- asserts[0] `equal` fail                                          (liveness failureThreshold == 6)
Tests:       4 failed, 1 passed, 5 total

The fifth test sets its own probe overrides through set:, so it is deliberately insensitive to the defaults and passes in both directions; it pins that operators can still override the values rather than pinning the values themselves

helm lint (commit bca4bd4b36)

$ helm lint ./helm/litellm
==> Linting ./helm/litellm
[INFO] Chart.yaml: icon is recommended

1 chart(s) linted, 0 chart(s) failed

$ helm lint ./helm/litellm-helm
==> Linting ./helm/litellm-helm
[INFO] Chart.yaml: icon is recommended

1 chart(s) linted, 0 chart(s) failed

Type

🐛 Bug Fix

Changes

helm/litellm/values.yaml left timeoutSeconds unset on the gateway and backend probes, which means Kubernetes applies its default of 1 second. Both containers run a single uvicorn worker, so a pod is a single asyncio event loop and its per-request latency under closed-loop saturation is dominated by queueing rather than by work. Once queueing exceeds a second, both probes time out even though the pod is serving traffic correctly, which is a bug in the probe configuration regardless of what the request path is doing at the time: the probe is answering the question "is this pod broken" with a budget that a working pod cannot meet. With failureThreshold: 3 and periodSeconds: 10 a readiness probe in that state takes the pod out of its Service after roughly 30 seconds, and the liveness probe restarts it, so the exposure is losing healthy capacity during exactly the bursts that need it

Both endpoints were checked before picking numbers. /health/liveliness returns a constant after reading an in-process shutdown flag, so it does no I/O at all; the only way it can be slow is event-loop starvation. /health/readiness also calls _db_health_readiness_check, which does a real Postgres round trip (cached for 15s on success), so it is strictly the more expensive of the two and additionally exposed to database latency

Readiness gets timeoutSeconds: 10, and failureThreshold stays at the default 3. The measured saturated latency was ~6s, so a budget at or below 5s still trips during exactly the burst we are trying to tolerate; the probe's own latency distribution was never measured separately, only the fact that it exceeded 1s, so picking a number under the one measurement we do have would be precision we have not earned

10s equals periodSeconds, which is the ceiling, and that boundary is what keeps the eviction arithmetic intact. kubelet's prober worker drives doProbe from a time.Ticker of periodSeconds (pkg/kubelet/prober/worker.go) rather than sleeping between attempts, and a Go ticker coalesces ticks that arrive while the receiver is busy, so the interval between probe starts is max(periodSeconds, probeDuration) and not their sum. With the timeout at the period, probes run effectively back to back and three consecutive failures still take about 30 seconds. Raising the timeout above the period is what would break this: at timeoutSeconds: 15 with periodSeconds: 10 the probe duration becomes the cadence and eviction stretches to ~45s, which is the real reason for the timeoutSeconds <= periodSeconds rule

A pod that is genuinely down refuses the connection immediately rather than consuming the timeout, so the longer budget does not slow real failure detection either

Liveness gets the same timeoutSeconds: 10 plus failureThreshold: 6. Restarting a merely busy pod is strictly worse than leaving it busy: it throws away in-flight requests, pays a cold start, and hands the load to the remaining replicas. Because the liveness endpoint does no I/O, ten seconds without an answer already means the loop is wedged, and requiring six consecutive such failures (period 15s) means roughly 90 seconds of sustained unresponsiveness before a restart. Readiness therefore reacts in ~30s and liveness in ~90s, which is the ordering we want

The backend gets the same treatment on its own evidence rather than for symmetry with the gateway. backend/Dockerfile ends in ENTRYPOINT ["uvicorn", "backend.main:app"] with no --workers flag, so uvicorn's default of one worker applies and the backend is unconditionally single-loop; unlike the gateway it does not even expose a numWorkers knob to raise. It serves the same /health/readiness with the same Postgres round trip, so the identical omission has the identical consequence

The ui component is deliberately left on the Kubernetes default and is not touched by this PR. ui/Dockerfile builds a Next.js static export and serves it from nginx:1.27-alpine, so / is a file read off disk with no application runtime that could queue behind saturated work. The single-event-loop reasoning above does not transfer to it, there is no measurement suggesting it needs more than a second to serve a static file, and widening its budget tenfold without a reason of its own would be scope creep

helm/litellm-helm/values.yaml, the older chart, was checked and deliberately left alone: it already sets timeoutSeconds explicitly on its liveness, readiness, and startup probes, so the omission this PR fixes does not exist there. Its value is 5s, which is below the latency measured here, but retuning a published chart's defaults is a separate decision with a much wider blast radius than fixing a missing key, and it is not needed to resolve this ticket

Tests live in helm/litellm/tests/probe_tests.yaml and pin the full rendered probe block for the gateway and the backend, assert that neither single-event-loop component is left without an explicit timeout, assert the liveness/readiness threshold asymmetry, and assert that operators can still override any of it per component

A companion e2e PR, #35494, makes the throughput load test's failure messages surface this class of failure by name instead of reporting an undifferentiated throughput miss. That test only goes fully green in stage once this chart change ships and stage redeploys

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[yassin-berriai]

@greptileai please review; this adds an explicit probe timeoutSeconds to the chart plus a mutation-checked helm-unittest suite

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Greptile Summary

This PR gives gateway and backend health probes explicit ten-second timeouts and increases the liveness failure threshold to tolerate temporary event-loop saturation.

• Adds the new timeout and liveness-threshold defaults for both components.
• Adds Helm unit tests covering rendered probe blocks, gateway/backend parity, readiness/liveness asymmetry, and operator overrides.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
helm/litellm/values.yaml	Adds the intended gateway and backend probe defaults without changing UI probes or preventing component-specific overrides.
helm/litellm/tests/probe_tests.yaml	Adds focused regression tests that verify the rendered defaults and preserve operator override behavior.

_{Reviews (4): Last reviewed commit: "fix(helm): give gateway and backend prob..." | Re-trigger Greptile}

[yassin-berriai]

@greptileai re-review please; the body was corrected to drop an unproven causal claim and the commit was amended (now 7f15804), the diff itself is unchanged

[yassin-berriai]

@greptileai please re-review at 5fafea9. The ui component was dropped from the diff (it is nginx serving a static export, so the single-event-loop reasoning does not apply to it); gateway and backend are unchanged from the previous review.

[yassin-berriai]

@greptileai please re-review at bca4bd4. Only the commit message and PR body changed since the last review; the diff is identical to 5fafea9.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!