fix(proxy): honour allow_requests_on_db_unavailable in readiness probe

[Zerohertz]

TLDR

Problem this solves:

• allow_requests_on_db_unavailable had no effect on readiness
• A DB outage marked every replica NotReady in ~30s
• Kubernetes emptied the Service, so the fail-open never ran
• The probe could also blow its timeout on a hung DB

How it solves it:

• Readiness keeps its 200 when the flag is set
• Body still reports the real db status
• The DB lookup is bounded when serving without a DB
• Behaviour with the flag unset is unchanged

Relevant issues

Fixes #34934

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Before is captured at daf22ec8712e406f502da4268efcb1743d5564ff (the branch point), after at 771c09f6c04b48f0168abf17d9899e70eed974a0. Every call below hits a real OpenAI model and costs real money; nothing is mocked

1. Start a Postgres the proxy can lose on demand:

docker run -d --name litellm-db -e POSTGRES_PASSWORD=pw -p 5432:5432 postgres:16

2. Write proof_config.yaml:

model_list:
  - model_name: gpt-5.6
    litellm_params:
      model: openai/gpt-5.6
      api_key: os.environ/OPENAI_API_KEY

general_settings:
  master_key: sk-1234
  allow_requests_on_db_unavailable: true

3. Check out the branch point and start the proxy:

git checkout daf22ec8712e406f502da4268efcb1743d5564ff
DATABASE_URL=postgresql://postgres:pw@localhost:5432/postgres \
  python litellm/proxy/proxy_cli.py --config proof_config.yaml --detailed_debug \
  --use_v2_migration_resolver 2>&1 | tee before.log

4. Confirm the baseline in another shell:

curl -s -w '\nhttp=%{http_code} time=%{time_total}\n' http://localhost:4000/health/readiness

5. Take the database down:

docker stop litellm-db

6. Call readiness again. Before the fix this answers 503 even though the flag is on:

curl -s -w '\nhttp=%{http_code} time=%{time_total}\n' http://localhost:4000/health/readiness

7. Show the request layer is still perfectly willing to serve, which is what makes the 503 a self-inflicted outage:

curl -s -w '\nhttp=%{http_code}\n' -X POST http://localhost:4000/v1/chat/completions \
  -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' \
  -d '{"model":"gpt-5.6","messages":[{"role":"user","content":"reply with exactly: proof of fix"}]}'

8. Simulate the failover case where connections hang instead of being refused, then call readiness once more. Before the fix this outlasts a timeoutSeconds: 5 probe:

docker start litellm-db && sleep 5 && docker pause litellm-db
curl -s -w '\nhttp=%{http_code} time=%{time_total}\n' http://localhost:4000/health/readiness

9. Stop the proxy, move to this branch, and start it again with the database still paused:

docker unpause litellm-db && docker stop litellm-db
git checkout litellm_readiness_honor_db_unavailable_flag
DATABASE_URL=postgresql://postgres:pw@localhost:5432/postgres \
  python litellm/proxy/proxy_cli.py --config proof_config.yaml --detailed_debug \
  --use_v2_migration_resolver 2>&1 | tee after.log

10. Readiness now answers 200 with the real db status in the body, so the pod stays in the Service:

curl -s -w '\nhttp=%{http_code} time=%{time_total}\n' http://localhost:4000/health/readiness

11. Repeat the hung-connection case and confirm the probe returns inside the bound rather than waiting on the DB:

docker start litellm-db && sleep 5 && docker pause litellm-db
curl -s -w '\nhttp=%{http_code} time=%{time_total}\n' http://localhost:4000/health/readiness

12. Bring the database back and confirm readiness reports connected again:

docker unpause litellm-db
sleep 20 && curl -s -w '\nhttp=%{http_code}\n' http://localhost:4000/health/readiness

13. Confirm the default is untouched. Set allow_requests_on_db_unavailable: false in proof_config.yaml, restart the proxy, stop the database, and check that readiness goes back to 503:

docker stop litellm-db
curl -s -w '\nhttp=%{http_code}\n' http://localhost:4000/health/readiness

Type

🐛 Bug Fix

Changes

/health/readiness returned 503 whenever a configured Prisma DB was unreachable and never looked at general_settings.allow_requests_on_db_unavailable. Under the probe defaults the Helm chart ships (readinessProbe.path: /health/readiness, periodSeconds: 10, failureThreshold: 3) a database outage marks every replica NotReady inside about thirty seconds, Kubernetes drops all of them from the Service endpoints, and the request-layer fallback that the flag exists to provide never gets a request to run on. A recoverable blip becomes a total outage

The flag used to gate this. _db_health_readiness_check called PrismaDBExceptionHandler.handle_db_exception, which re-raised only when the flag was off, and health_readiness turned that into an HTTPException(503). #26134 dropped the call to fix a 503 loop, then #27003 re-added an unconditional 503, which left the flag with no say over readiness

_apply_db_readiness_status now makes that call in one place and both readiness paths share it, so the low-detail probe and the detailed payload reachable through allow_public_health_readiness_details cannot disagree. With the flag set an unreachable DB keeps the 200; the db field still carries the truth either way

Status code alone is not enough. PrismaClient.health_check retries under backoff over a query_raw that has no timeout of its own, so an unreachable DB can outlast the kubelet's timeoutSeconds and fail the probe on time whatever we would have returned. _readiness_db_status therefore bounds the lookup with asyncio.wait_for when the flag is set. Giving up early costs no recovery, since _db_health_watchdog_loop already owns reconnection and runs on its own schedule. The bound deliberately does not apply when the flag is unset, so a slow but reachable DB on the default path behaves exactly as it does today

The budget for that bound is the watchdog's own PRISMA_HEALTH_WATCHDOG_PROBE_TIMEOUT_SECONDS, surfaced as a PrismaClient.db_health_probe_timeout_seconds property rather than a new setting. Both are answering the same question, how long to wait on a DB liveness check, and an operator who tightens one wants the other tightened too. Note that the 5 second default is longer than some probe configurations allow, so a deployment running readinessProbe.timeoutSeconds below that should lower this knob

Four tests cover this. Three fail if the source change is reverted, and the fourth fails if the bound is applied unconditionally, so the "unchanged by default" half of the contract is pinned too. They assert on the returned status and body rather than on elapsed time, so there is nothing timing-sensitive to flake in CI

One thing is deliberately left out of scope. #26237 asks that a worker which never successfully loaded its router fail readiness, and it should keep failing even with this flag set, since "router state was never populated" is a different condition from "the DB is momentarily unreachable". That needs first-successful-load tracking in the startup path, so it belongs in its own PR

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[greptile-apps]

Greptile Summary

This PR makes database readiness behavior follow the existing fail-open setting

• Keeps readiness at 200 during database outages when fail-open is enabled
• Bounds database checks on the fail-open path using a configurable timeout
• Shares database status handling between public and detailed readiness responses
• Adds regression coverage for both flag states and timeout behavior

Confidence Score: 4/5

The timeout configuration needs validation before merging because malformed or infinite values can recreate the readiness outage this change is intended to prevent

The fail-open behavior and shared status handling are covered, but the newly exposed timeout setting can produce an HTTP 500 or disable the database bound on the affected readiness path

Files Needing Attention: litellm/proxy/health_endpoints/_health_endpoints.py

Important Files Changed

Filename	Overview
litellm/proxy/health_endpoints/_health_endpoints.py	Adds flag-aware, timeout-bounded database readiness handling, but invalid or non-finite timeout configuration can break the probe
tests/test_litellm/proxy/health_endpoints/test_health_endpoints.py	Adds focused regression tests for fail-open status handling, detailed responses, and conditional timeout behavior

_{Reviews (1): Last reviewed commit: "fix(proxy): honour allow_requests_on_db_..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 96.29630% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/utils.py	66.66%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[Solaris-star]

Reviewed this in depth — it's the more complete version of the readiness/HA-flag fix (I had attempted a narrower variant that only consulted the flag, so this supersedes that work). A few observations:

The timeout bound is the part that actually matters, and it's correct
Consulting allow_requests_on_db_unavailable alone isn't sufficient: PrismaClient.health_check retries under backoff over a query_raw with no timeout of its own, so an unreachable DB can outlast the kubelet's timeoutSeconds and fail the probe on time regardless of the status code we'd return. Bounding the lookup with asyncio.wait_for and reporting disconnected on timeout is what keeps the pod genuinely in rotation. Good catch.

The timeout asymmetry is intentional and right
The bound is only applied when the fallback flag is set; when it's off, _readiness_db_status awaits the check unbounded. That's correct — when fallback is disabled we want 503, and a probe that hangs until kubelet times out still yields NotReady, which is the desired outcome. Only the "must return 200 fast" path needs the budget.

Reusing the watchdog's probe budget is a nice touch
Exposing db_health_probe_timeout_seconds and sharing it between the watchdog and the readiness probe means an operator tightening one tightens both — they answer the same question (how long to wait on a DB liveness check). The docstring spells out exactly this rationale.

Tests
The coverage is solid: stays-ready-when-unreachable, the parametrized detailed-payload path (200/503), and the slow-client/probe-timeout test that proves the bound actually fires. The _slow_prisma_client(delay=0.3, timeout=0.05) case is a clean way to exercise the timeout without real network flakiness.

One minor, non-blocking thought: on the timeout path asyncio.wait_for cancels the in-flight health_check mid-backoff. In the test that's a mock so it's moot; in production asyncpg/prisma generally handle cancellation cleanly, but it's the one spot I'd keep an eye on under real DB-outage conditions.

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing Zerohertz:litellm_readiness_honor_db_unavailable_flag (771c09f) with litellm_internal_staging (daf22ec)}