fix(proxy): make the per-reload credential sync linear in the number of credentials

[timrsmith]

TLDR

Problem this solves:

• Every config reload re-syncs the credentials table with two nested scans, quadratic in credential count
• With 20k credentials each reload blocks the worker's event loop for 16 to 34s on main
• Requests queue, /health/liveliness times out and Kubernetes kills the pod mid-request

How it solves it:

• delete_credentials keeps names from one frozenset instead of rebuilding a name list per credential
• upsert_credentials indexes the current list once, first occurrence per name, instead of rescanning per credential
• Same result and order as before; the slowest health probe drops from 33.8s to 69ms at 20k credentials

Replaces #43923, which closed when its head branch was renamed

User Flow

Before: a developer calling a proxy whose database holds thousands of credentials sees stalls of tens of seconds, and the pod restarts under them

1. They send POST https://litellm-domain/v1/chat/completions with a 30-token prompt
2. Most calls return in about 1s, but about once every 30s a call hangs for tens of seconds before the first byte while x-litellm-overhead-duration-ms says 20
3. Meanwhile GET https://litellm-domain/health/liveliness also hangs, the liveness probe times out and the pod is restarted, so in-flight calls fail

After: the same proxy answers every call promptly and health probes never miss

1. They send the same POST https://litellm-domain/v1/chat/completions
2. Every call returns in about 1s, and GET https://litellm-domain/health/liveliness sampled once a second for 60s never takes over 1s
3. No liveness probe failures and no pod restarts

Relevant issues

Fixes #43922

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/litellm_core_utils/test_credential_accessor.py tests/unit/proxy/proxy_server/test_proxy_config.py -k credential -v
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

The two scale tests go red on the previous code: with the fix reverted, test_upsert_credentials_scales_linearly_with_credential_count fails with "upsert of 20000 credentials took 24.2s" and the delete test fails the same way. The three behavioural tests pass on both versions, which is the point of a performance-only rewrite

Screenshots / Proof of Fix

Local litellm-database proxy on localhost:4000 against Postgres with store_model_in_db: true. Credentials were created with a curl loop, then /health/liveliness was curled once a second for 60 seconds. Nothing here calls an LLM provider: the bug is in the reload job, and the health endpoint is what the kubelet sees

for i in $(seq 1 20000); do
  curl -s -o /dev/null -X POST http://localhost:4000/credentials -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' \
    -d "{\"credential_name\":\"e2e-run-$i\",\"credential_values\":{\"aws_access_key_id\":\"AKIA$i\",\"aws_secret_access_key\":\"secret-$i\",\"aws_region_name\":\"us-east-1\"},\"credential_info\":{\"custom_llm_provider\":\"bedrock\"}}"
done
for i in $(seq 1 60); do echo "$(date +%H:%M:%S) $(curl -s -o /dev/null -w '%{time_starttransfer}' http://localhost:4000/health/liveliness)"; sleep 1; done

Before (657bb77)

20,000 credentials, 60 liveliness probes at 1/s

credentials in DB: 19997
15:41:48 15.9s
15:42:10 24.0s
15:42:40 23.1s
15:43:11 22.8s
15:43:41 22.4s
15:44:11 23.4s
15:44:40 23.3s
15:45:11 23.6s
15:45:41 33.8s
samples=60 stalls_over_1s=9 max=33.8s

Only probes that waited over 1s are listed. One reload every 30s, and every one of them froze the worker for 16 to 34s, so the 60 probes took almost five minutes to complete

After (13f25d1, identical diff measured at d1bdd4d before rebasing onto main)

20,000 credentials, 60 liveliness probes at 1/s

credentials in DB: 19997
samples=60 stalls_over_1s=0 max=0.069s

Same database, same 20,000 credentials, same probe loop across two reloads. The slowest probe took 69ms

Caveats

Six checks fail on this PR and the same six fail on main's own head b41715b, so they are not caused by this change. code-quality and documentation report undocumented CLICKHOUSE_* variables and the schema.d.ts check reports a new trace_ref query parameter, all from the tracing feature in #43915. proxy-endpoints fails two batches tests on input_file_id, proxy-infra fails the component allowlist test on three new routes, and misc fails four interactions OpenAPI compliance tests. None of those paths read or write credentials

Codecov's patch status briefly reported the two delete_credentials body lines as missed while uploads were still landing; the proxy-server shard's coverage.xml artifact had them at 1 hit each, and codecov now reports all modified lines covered. The earlier schema-migration failure was test_db_push_timeout_hint_names_the_per_command_budget, which failed on main itself and on a dozen branches after #44034 until main fixed it; this head is rebased past the fix

Still on the serving loop after this fix, as follow-ups: get_credentials re-reads the config file just to learn the config credential names and decrypts every credential value even when no row changed, about 0.25s at 6k rows. Request-path lookups in CredentialAccessor.find_credential and the Rust preflight scan stay linear per request; a name-keyed index there needs a single writer for litellm.credential_list, which several call sites reassign directly today

Type

🐛 Bug Fix

[greptile-apps]

Confidence Score: 5/5

[Medium risk] Optimizes credential sync performance in the proxy.

The PR appears safe to merge based on the reviewed changes.

Summary

This PR replaces repeated credential-list scans during reload with a name index for upserts and a name set for deletion, and adds behavioral and scale tests.

_{Reviews (1) · Last reviewed commit: "refactor(proxy): build the credential na..."}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing timrsmith:litellm_credential_sync_linear (13f25d1) with main (d7e6154)}

[chrikrah]

@timrsmith I would merge this at 6aa2db3. Your tests bound the sync end to end; in process, upsert_credentials stays linear up to 40,000 credentials, at 13 ms for 20,000 and 28 ms for 40,000. Your revert result reproduces here too.

Five of the six red checks also fail on main at 632b69b. The sixth, the schema.d.ts check, compares the OpenAPI spec, and this diff adds no route or model.

$ python probe_upsert.py      # upsert_credentials at 6aa2db3, n refreshed in place
5000 0.0028
10000 0.0056
20000 0.0132
40000 0.0279

$ pytest tests/unit/litellm_core_utils/test_credential_accessor.py \
    tests/test_litellm/proxy/proxy_server/test_proxy_config.py -k credential -q
10 passed, 276 deselected in 9.42s

# credential_accessor.py and proxy_server.py reverted to b41715b, tests kept
AssertionError: delete_credentials over 20000 credentials took 10.2s
AssertionError: upsert of 20000 credentials took 15.6s
2 failed, 8 passed, 276 deselected in 33.11s
# Python 3.13; not run: the rest of tests/test_litellm/proxy

@yassin-berriai, you merged #39876, the last rework of this credential sync. Could this go into the next release, given the liveness restarts in #43922?

[timrsmith]

Codecov's two missed lines are hit in the proxy-server shard's coverage.xml artifact; details in the Caveats section.