fix(proxy): stop leaking master_key and database_url in startup DEBUG logs

[yucheng-berri]

Relevant issues

Linear ticket

Resolves LIT-4152

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review)

Screenshots / Proof of Fix

Steps run against a live proxy backed by real OpenAI (gpt-4o-mini), with the last-line-of-defense regex scrubber disabled (LITELLM_DISABLE_REDACT_SECRETS=true) so anything a log call site still writes into a record shows up raw. Distinctive LEAKMARKER strings stand in for the secrets so the grep is unambiguous; master_key and alert_to_webhook_url live in general_settings, and both a secret-named key and a plain one live in litellm_settings

Repro config /tmp/lit4152_live.yaml (no database, so startup does not need Postgres):

model_list:
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY

general_settings:
  master_key: sk-lit4152-LIVE-masterkey-LEAKMARKER0001
  alerting: ["slack"]
  alert_to_webhook_url:
    budget_alerts: https://hooks.slack.com/services/LEAKMARKER-webhook0002

litellm_settings:
  langfuse_secret_key: sk-lit4152-LIVE-langfusesecret-LEAKMARKER0003
  num_retries: 7

$ export LITELLM_DISABLE_REDACT_SECRETS=true
$ python litellm/proxy/proxy_cli.py --config /tmp/lit4152_live.yaml --port 4152 --detailed_debug > litellm.log 2>&1 &

$ curl -sS -o /dev/null -w "%{http_code}\n" http://localhost:4152/health/liveliness
200

$ curl -sS http://localhost:4152/v1/chat/completions \
    -H "Authorization: Bearer sk-lit4152-LIVE-masterkey-LEAKMARKER0001" \
    -H "Content-Type: application/json" \
    -d '{"model":"gpt-3.5-turbo","messages":[{"role":"user","content":"say ok"}]}' \
    | jq -r '.choices[0].message.content'
Ok!

$ for m in LEAKMARKER0001 LEAKMARKER-webhook0002 LEAKMARKER0003; do echo "$m: $(grep -c $m litellm.log)"; done
LEAKMARKER0001: 0
LEAKMARKER-webhook0002: 0
LEAKMARKER0003: 0

The debug lines still carry the operational signal, now with the credentials removed:

proxy_server.py - _alerting_callbacks: ['slack']
proxy_server.py - setting litellm.langfuse_secret_key=REDACTED
proxy_server.py - setting litellm.num_retries=7

The real request returns Ok!, no marker lands in the log stream even with the module regex scrubber off, and a plain operational setting such as num_retries keeps its real value so the debug line stays useful

The store_model_in_db config path needs a Postgres instance, so those rows (environment_variables in particular) are covered by regression tests that drive _update_config_from_db directly with the redaction filter disabled rather than by the live run above

Test results

$ pytest tests/test_litellm/proxy/proxy_server/test_lifecycle.py \
         tests/test_litellm/proxy/proxy_server/test_proxy_config.py \
         tests/test_litellm/proxy/common_utils/test_encrypt_decrypt_utils.py \
         tests/test_litellm/test_secret_redaction.py \
         tests/test_litellm/proxy/test_proxy_server.py -q
371 passed

Every new redaction regression runs with the module regex scrubber (_ENABLE_SECRET_REDACTION) disabled, so it proves the fix at the log call site rather than the last-line filter, and each one fails when its fix is reverted. test_secret_redaction.py (the regex net itself) and the broad test_proxy_server.py suite stay green, so the change adds redaction without regressing the existing defense-in-depth layer or the wider proxy surface

Type

🐛 Bug Fix

Changes

Several startup and config-load DEBUG log statements in litellm/proxy/proxy_server.py dumped secret-bearing values whenever the module-level SecretRedactionFilter was bypassed. Fixes to log leaks in this codebase have all been reactive per-endpoint (/get/config/callbacks, /model/info, cooldown cache, etc.), so the same bug keeps recurring on a new surface; this change fixes it at the call site

proxy_startup_event logged the raw WORKER_CONFIG blob, which docker/K8s deployments hand the proxy as a JSON string containing master_key, database_url, and provider API keys. It now routes through _redact_worker_config_for_logging, which delegates to the existing recursive _redact_secret_values_in_obj. Reusing that helper rather than a hand-rolled top-level pass means a credential nested under general_settings or another config object is redacted at any depth, and depth overrun fails closed

ProxyConfig._load_alerting_settings logged the whole general_settings dict under a label that only referred to the alerting callbacks. Copy-paste bug that happened to leak master_key, database_url, and everything else in general_settings. Now logs only the alerting callback list

ProxyConfig.load_config logged the resolved DB URL after secret-manager resolution. The stated purpose was to confirm the retrieval ran, which does not need the value. Now logs a value-less breadcrumb

ProxyConfig._update_config_from_db logged each DB param_value verbatim on the store_model_in_db=True path. For general_settings, router_settings, and litellm_settings the value now routes through the recursive redactor. The environment_variables row is different: its keys are operator-chosen env var names, so a connection string commonly sits under DATABASE_URL or REDIS_URL which the key-name matcher cannot recognize (url is not a sensitive segment, and the database_url allow-entry is lowercase while the real env var is uppercase). Every value in that row is therefore blanked while the variable names stay visible for signal

decrypt_value_helper logged Unable to decrypt value={value} at DEBUG, leaking the raw secret on the same environment_variables path whenever decryption failed (for example after a salt or master key change). It now drops the value; the key already identifies the failing pair

The litellm_settings apply loop logged setting litellm.<key>=<value> verbatim, leaking secret-named settings such as api_key or langfuse_secret_key. It now routes the value through _redact_general_setting_value, which masks a value only when the key name is secret-bearing and recurses into dict/list, so a plain setting like num_retries still logs its real value

The _EXTRA_SECRET_GENERAL_SETTINGS_FIELDS constant sits alongside SENSITIVE_DATA_MASKER so the logging and config-update paths share the same list

Regression tests in test_lifecycle.py and test_proxy_config.py disable _ENABLE_SECRET_REDACTION before asserting, so they exercise the source-level fix rather than the last-line-of-defense regex. Coverage includes dict-form and JSON-string-form worker_config, None and non-JSON passthrough, secret fields nested under a parent key, the _update_config_from_db general_settings and environment_variables paths (asserting a DATABASE_URL connection string never reaches a log record while the variable names stay visible), the decrypt_value_helper breadcrumb on a value that fails to decrypt, and the litellm_settings loop asserting both that a secret key is redacted and that a plain num_retries stays visible

---

Note

Medium Risk
Touches many DEBUG logging paths on config load and DB overlay; behavior change is redaction-only at log sites, but mistakes could hide useful debug signal or miss a leak path.

Overview
Stops proxy startup/config DEBUG logs from writing credentials when the log redaction filter is off (LIT-4152).

Adds _redact_worker_config_for_logging so WORKER_CONFIG (dict or JSON string) is masked before proxy_startup_event logs it, using the existing recursive _redact_secret_values_in_obj (including nested general_settings and extra secret field names like database_url).

ProxyConfig paths now redact at the call site: _load_alerting_settings logs only the alerting list (not all of general_settings); database_url resolution logs breadcrumbs without the URL; load_config applies _redact_general_setting_value for litellm_settings debug lines (secrets REDACTED, plain settings like num_retries unchanged); _update_config_from_db logs param_value via _redact_config_param_value_for_logging (full value redaction for environment_variables, recursive redaction elsewhere). _EXTRA_SECRET_GENERAL_SETTINGS_FIELDS is hoisted next to module init for shared use.

decrypt_value_helper no longer includes the raw ciphertext/value in the “unable to decrypt” DEBUG message (key name only).

Regression tests assert no leaks with _ENABLE_SECRET_REDACTION disabled.

^{Reviewed by Cursor Bugbot for commit bd6ae9e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[yucheng-berri]

@greptileai

[CLAassistant]

All committers have signed the CLA.

[greptile-apps]

Greptile Summary

This PR fixes a recurring class of startup/config-load DEBUG log statements in litellm/proxy/proxy_server.py that leaked raw secrets (master_key, database_url, provider API keys, webhook URLs) whenever the module-level SecretRedactionFilter was bypassed.

• Six specific call sites are hardened: the raw WORKER_CONFIG blob, the _load_alerting_settings copy-paste that logged all of general_settings, the resolved database_url breadcrumb, the _update_config_from_db per-param log, the decrypt_value_helper failure message, and the litellm_settings apply loop.
• Redaction is done at the call site by routing values through the existing recursive _redact_secret_values_in_obj helper (for dicts/lists) or by logging only the value-less breadcrumb (for the DB URL), so the last-line SecretRedactionFilter is no longer the single safety net.
• Regression tests disable _ENABLE_SECRET_REDACTION before asserting, proving the source-level fix rather than the downstream filter.

Confidence Score: 5/5

Safe to merge — changes are confined to log call sites and introduce no behavioral differences for the proxy's request or config-apply paths.

All six hardened log sites redact correctly via the existing recursive helper or by dropping the value entirely. The new helper functions have no side effects on the actual config values — only the strings passed to verbose_proxy_logger.debug change. Regression tests disable the module-level filter before asserting, so they prove the fix at the source rather than relying on the downstream safety net. Non-secret settings such as num_retries are verified to remain visible, confirming surgical redaction. No functional, API, or schema changes are made.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/proxy/proxy_server.py	Six log call sites hardened to redact secrets before they reach a log record; _EXTRA_SECRET_GENERAL_SETTINGS_FIELDS moved earlier so the new top-of-file helpers can reference it; two new functions (_redact_worker_config_for_logging, _redact_config_param_value_for_logging) added with correct forward references.
litellm/proxy/common_utils/encrypt_decrypt_utils.py	Single-line fix: removes the raw value from the decrypt-failure DEBUG message, keeping only the key name for debugging signal.
tests/test_litellm/proxy/proxy_server/test_lifecycle.py	Adds five focused regression tests for _redact_worker_config_for_logging covering dict, JSON-string, None, nested, and non-string-value shapes; all disable the module regex scrubber to prove source-level fixes.
tests/test_litellm/proxy/proxy_server/test_proxy_config.py	Adds three regression tests: _load_alerting_settings no longer logs raw general_settings, _update_config_from_db redacts param values at DEBUG, and the litellm_settings apply loop masks secret keys while keeping plain values visible.
tests/test_litellm/proxy/common_utils/test_encrypt_decrypt_utils.py	Adds a decrypt-failure log regression test; cosmetic reformatting of three existing assertions does not weaken any coverage.

_{Reviews (4): Last reviewed commit: "fix(proxy): stop the decrypt-failure deb..." | Re-trigger Greptile}

[greptile-apps]

Greptile Summary

This PR fixes three DEBUG log statements in proxy_server.py that leaked master_key, database_url, and other secret-bearing values whenever the module-level SecretRedactionFilter was bypassed. It introduces two helper functions (_redact_config_dict_for_logging, _redact_worker_config_for_logging) and adds focused regression tests that run with the last-line-of-defense scrubber disabled.

• proxy_startup_event: raw WORKER_CONFIG blob is now routed through _redact_worker_config_for_logging, which combines the segment-based SensitiveDataMasker with an explicit whole-value replacement for fields the segment masker misses (database_url, alert_to_webhook_url, etc.).
• ProxyConfig._load_alerting_settings: the mislabelled log line that dumped the entire general_settings dict now logs only the alerting callback list.
• ProxyConfig.load_config: the resolved DB URL is no longer emitted; a value-less breadcrumb replaces it.

Confidence Score: 5/5

Safe to merge. The change is narrowly scoped to log statements; no request-path logic, no schema changes, and no behavioral changes for callers.

Three leaky DEBUG log statements are fixed at the source, each with a dedicated regression test that exercises the fix with the module-level filter disabled. The helper functions handle all realistic input shapes and fall back safely for non-dict inputs. The constant hoisting is a pure refactor with no behavioral change. No existing tests are weakened.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/proxy/proxy_server.py	Adds _redact_config_dict_for_logging and _redact_worker_config_for_logging helpers; fixes three leaky DEBUG log statements; hoists _EXTRA_SECRET_GENERAL_SETTINGS_FIELDS to module top so both helpers and _is_secret_general_setting_field share the same definition. Logic is correct and handles all expected input shapes (None, file-path string, JSON string, dict).
tests/test_litellm/proxy/proxy_server/test_lifecycle.py	Adds four regression tests for _redact_worker_config_for_logging and _redact_config_dict_for_logging covering dict input, JSON-string input, None/non-JSON passthrough, and non-string webhook/URL values. All tests disable _ENABLE_SECRET_REDACTION to verify the fix at the source rather than relying on the filter.
tests/test_litellm/proxy/proxy_server/test_proxy_config.py	Adds a regression test for _load_alerting_settings that captures raw log records with SecretRedactionFilter disabled and asserts neither master_key nor database_url appears while confirming the alerting list is still present.

_{Reviews (1): Last reviewed commit: "fix(proxy): stop leaking master_key and ..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 76.92308% with 6 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/proxy_server.py	76.00%	6 Missing ⚠️

📢 Thoughts on this report? Let us know!

[yucheng-berri]

CI status summary

Greptile is at 5/5 on both passes. Failures on the checks tab break down as:

ci/circleci: image_gen_testing and ci/circleci: local_testing_part1 are integration jobs that hit real LLM provider APIs. Neither exercises anything this PR touches; my change is scoped to three DEBUG log statements in litellm/proxy/proxy_server.py's startup path plus two module-level helpers, and the same two jobs pass on other recent open PRs. Nothing under tests/local_testing/ or tests/image_gen_tests/ imports _load_alerting_settings, _redact_worker_config_for_logging, _redact_config_dict_for_logging, or _EXTRA_SECRET_GENERAL_SETTINGS_FIELDS. Requesting a retry on both

codecov/patch is at 89.47% (2 lines uncovered per the latest codecov comment). The two lines are the verbose_proxy_logger.debug(...) call at proxy_server.py:896 inside proxy_startup_event and one adjacent line; that handler is an async FastAPI lifecycle hook that unit tests cannot enter without launching the full app. The five new regression tests all pass locally and cover the two helpers plus the _load_alerting_settings fix; codecov's initial reading of 26% was before three of the coverage-upload shards finished, and it now sits at 89.47% as reported in the codecov PR comment

[yucheng-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit e2ede3e. Configure here.}

[yucheng-berri]

@greptileai

[yucheng-berri]

bugbot run

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: DB env vars still log secrets
  • DB config debug logging now redacts every value for environment_variables rows and the regression test covers DATABASE_URL with source redaction disabled.

_{You can send follow-ups to the cloud agent here.}

[yucheng-berri]

@greptileai

[yucheng-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 44318b0. Configure here.}

[yucheng-berri]

Live proxy verification — no regressions, no backward-incompatible changes

Exercised every changed call site on a live proxy (real OpenAI gpt-4o-mini) with the last-line scrubber disabled (LITELLM_DISABLE_REDACT_SECRETS=true and _ENABLE_SECRET_REDACTION=False) so any residual leak at a call site shows raw. Distinctive LEAKMARKER strings + the real Postgres password stand in for secrets.

Live results

Path	How	Result
Startup worker_config (CLI --config)	live proxy + real request	request OK; master_key auth enforced (wrong key and literal REDACTED → rejected); 0 leaks
Docker entrypoint (uvicorn app + WORKER_CONFIG JSON string)	live proxy + real request	request OK; string-branch redaction confirmed; 0 leaks
_load_alerting_settings	live	logs _alerting_callbacks: ['slack'], not the full general_settings dict
litellm_settings loop	live	langfuse_secret_key=REDACTED, num_retries=7 still visible
db_url breadcrumb	live w/ Postgres (real DB password as marker)	resolved password count 0 in logs; proxy still serves
_update_config_from_db	scrubber-off capture, adversarial rows	environment_variables → all values REDACTED, keys visible; general/router/litellm_settings recursively redacted; scalars pass through; 0 leaks
decrypt_value_helper	real call, forced decrypt failure	logs Unable to decrypt value for key: <k> — raw value absent from every line incl. exception stack

Backward-compat / regression probes (all pass)

• Log-only, no mutation — _redact_secret_values_in_obj rebuilds dicts/lists via comprehensions; the live worker_config/param objects are unmutated. Confirmed behaviorally: master_key auth still works with its real value, so the applied config is untouched.
• Union annotation safe — requires-python >=3.10,<3.14; str | dict[...] imports cleanly (runtime-validated on 3.11 and 3.13).
• Admin bypass unchanged — is_full_admin=True still returns raw (the /config/field/info non-log path).
• Edge cases — None / empty / non-JSON string / JSON-list / scalar pass through; depth-overrun fails closed; non-dict environment_variables handled defensively.

Test suites

• PR's own regressions: 157 passed
• Broad proxy suite (test_proxy_server.py + proxy_server/ + common_utils/): 1021 passed, 0 failed (14 deselected: one DB-E2E file that needs a live Prisma query engine, unrelated — the PR touches no key-rotation/prisma/db files).

Minor, non-blocking observation

The load_config db_url breadcrumb this PR made value-less is often unreachable in practice: general_settings resolves os.environ/ values earlier, so database_url no longer starts with os.environ/ by the time that branch runs. The hardening is still correct defense-in-depth; the DB-password redaction is what actually protects that path, and it's confirmed above.

[yucheng-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit bd6ae9e. Configure here.}