fix(proxy): unregister logging callbacks removed from the stored config

[yuneng-berri]

TLDR

Problem this solves:

• Deleting a logging callback reports success but it keeps exporting
• The deleted callback stays listed as read-only in the Admin UI

How it solves it:

• The proxy remembers which callbacks it registered from the stored config
• Each config sync unregisters those the stored config no longer lists
• Callbacks from the YAML file or code are never touched
• An entry the sync replaced (a code-registered string) is put back on delete
• A failed config load skips the sync instead of acting on an empty config

User Flow

Before: an admin deletes a logging callback, gets a success message, and the proxy keeps exporting to it

1. They add langfuse_otel with POST http://localhost:4000/config/update and traffic starts reaching Langfuse
2. They delete it with POST http://localhost:4000/config/callback/delete and get Successfully deleted callback: langfuse_otel
3. GET http://localhost:4000/get/config/callbacks still lists langfuse_otel, now marked read only, on every worker
4. New chat completions keep exporting to Langfuse

After: the delete sticks on every worker

1. They add langfuse_otel with POST http://localhost:4000/config/update and traffic starts reaching Langfuse
2. They delete it with POST http://localhost:4000/config/callback/delete and get Successfully deleted callback: langfuse_otel
3. GET http://localhost:4000/get/config/callbacks returns an empty list on every worker within 30 seconds
4. New chat completions export nothing, and adding the callback back makes exports resume

Affected release

The read-only listing appears since v1.102.0-rc.1 (#38974). The leftover exports are older and also happen on v1.101.0

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

The resync that runs after a delete, and every 30 seconds on each worker, could only add callbacks. So the callback object stayed in litellm's in-memory callback lists after its config entry was gone, and /get/config/callbacks shows runtime-only callbacks as read only. The bug itself is older than v1.102.0: v1.101.0 also keeps exporting after the delete, but its callback list hid the leftover

Greptile on the rc backport found that registering langfuse_otel from the DB swaps a code-registered string entry for a logger instance, so the delete dropped the code-registered callback too. The second fix commit records replaced entries and restores them, covered by test_db_config_sync_restores_a_code_callback_it_replaced. A restart between the add and the delete was also checked on this branch: 10 chats after the delete produced 0 exports and every worker listed nothing. The new tests fail on main and pass here. Four of five planned source mutations are caught. The survivor removes and re-adds every still-configured callback on each sync, which ends in the same state

Screenshots / Proof of Fix

Proxy started with --num_workers 2 (the After run waits 30 seconds after the first worker is live so both workers finish startup), a local Postgres, STORE_MODEL_IN_DB=True, a mock_response model named mock-gpt, and LANGFUSE_OTEL_HOST pointing at a local OTLP capture server that counts export POSTs. No real provider calls are needed since the bug is in callback registration, not the LLM call. "OTLP exports" below counts export batches the capture server received

Before (8e6d99d)

1. curl -s localhost:4000/config/update -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d '{"litellm_settings":{"success_callback":["langfuse_otel"]}}' returns {"message":"Config updated successfully"}, and 6 chats produce 1 export batch
2. curl -s localhost:4000/config/callback/delete -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d '{"callback_name":"langfuse_otel"}' returns {"message":"Successfully deleted callback: langfuse_otel","removed_callback":"langfuse_otel","remaining_callbacks":[],...}
3. curl -s localhost:4000/get/config/callbacks -H "Authorization: Bearer $LITELLM_MASTER_KEY" still returns [{"name": "langfuse_otel", ..., "type": "success", "read_only": true}]
4. After waiting 35 seconds for every worker's periodic sync, 10 chats produce 3 export batches, and 4 more /get/config/callbacks calls across workers all return ["langfuse_otel"]

After (7026833)

1. Same /config/update call returns {"message":"Config updated successfully"}, and 6 chats produce 2 export batches
2. Same /config/callback/delete call returns the same success message
3. /get/config/callbacks returns []
4. After waiting 35 seconds, 10 chats produce 0 export batches, and 4 more /get/config/callbacks calls across workers all return []
5. Re-adding with the same /config/update call, then waiting 35 seconds, 10 chats produce 2 export batches

Type

🐛 Bug Fix

Caveats (if any)

Medium

• A callback listed in both the YAML file and the DB survives a DB delete
  • The YAML registered it first, so the sync never owned it
• Other workers drop a deleted callback on their next sync, up to 30 seconds later
• A DB-added callback is now unregistered when the YAML file starts setting the same key
  • YAML wins that key, so a restart already dropped it before this PR

Low

• Non-string entries in the stored callback lists are now ignored instead of registered

[greptile-apps]

Confidence Score: 5/5

[High risk] Adds callback registration tracking and unregistration logic to proxy config.

The PR appears safe to merge based on this review

Summary

The PR reconciles runtime logging callbacks with stored configuration, removes DB-owned callbacks after deletion, restores code entries displaced by DB registration, and skips reconciliation when config loading fails

• Adds regression tests for deletion, source preservation, restoration, and failed config reads

_{Reviews (4) · Last reviewed commit: "fix(proxy): restore code-registered call..."}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[yuneng-berri]

@greptileai review

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_callback_delete_sticks (7026833) with main (8e6d99d)}

[devin-ai-integration]

QA: before (8e6d99d) fails, after (b8eed45) passes

Live e2e run, no mocks. Each leg ran two separate proxy processes with --num_workers 2 sharing one Postgres database (STORE_MODEL_IN_DB=True), every request alternating between the two processes. The model is openai/gpt-5.4-nano on a real OpenAI key. The export sink is a real self-hosted Langfuse v3 (official docker compose) and exports are counted through Langfuse's own read API, GET /api/public/traces?tags=<tag>, where the tag comes from the chat request's metadata.tags. The callback was only ever added through /config/update, never in the YAML

Chat used in both legs:

curl -s http://localhost:$PORT/v1/chat/completions -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
  -d '{"model":"gpt-mini","messages":[{"role":"user","content":"say hi"}],"max_tokens":10,"metadata":{"tags":["<leg>-<phase>"]}}'

Before (merge base 8e6d99d, ports 4101/4102)

$ curl -s -X POST localhost:4101/config/update -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d '{"litellm_settings":{"success_callback":["langfuse_otel"]}}'
{"message":"Config updated successfully"}
(35s later) GET /get/config/callbacks on 4101 and 4102 both list {"name":"langfuse_otel", ..., "type":"success"}
6 chats tagged before-added alternating 4101/4102
$ curl -s -u $LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY "localhost:3000/api/public/traces?tags=before-added"
{"data":[...],"meta":{"page":1,"limit":50,"totalItems":6,"totalPages":1}}

$ curl -s -X POST localhost:4101/config/callback/delete -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" -d '{"callback_name":"langfuse_otel"}'
{"message":"Successfully deleted callback: langfuse_otel","removed_callback":"langfuse_otel","remaining_callbacks":[],"deleted_at":"2026-09-27T07:47:12.456805"}

(35s later, 4 reads alternating 4101/4102, all identical)
$ curl -s localhost:4101/get/config/callbacks -H "Authorization: Bearer $LITELLM_MASTER_KEY"
{"status":"success","callbacks":[{"name":"langfuse_otel","variables":{...},"type":"success","read_only":true}], ...}

10 chats tagged before-deleted alternating 4101/4102
$ curl -s -u $LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY "localhost:3000/api/public/traces?tags=before-deleted"
{"data":[...],"meta":{"page":1,"limit":50,"totalItems":10,"totalPages":1}}

The deleted callback stays listed as read only on both processes and all 10 chats after the delete still land in Langfuse

After (PR tip b8eed45, ports 4103/4104)

$ curl -s -X POST localhost:4103/config/update ... -d '{"litellm_settings":{"success_callback":["langfuse_otel"]}}'
{"message":"Config updated successfully"}
(35s later) GET /get/config/callbacks on 4103 and 4104 both list langfuse_otel
6 chats tagged after-added alternating 4103/4104
$ curl -s -u $LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY "localhost:3000/api/public/traces?tags=after-added"
{"data":[...],"meta":{"page":1,"limit":50,"totalItems":6,"totalPages":1}}

$ curl -s -X POST localhost:4103/config/callback/delete ... -d '{"callback_name":"langfuse_otel"}'
{"message":"Successfully deleted callback: langfuse_otel","removed_callback":"langfuse_otel","remaining_callbacks":[],"deleted_at":"2026-09-27T07:47:20.591361"}

(35s later, 4 reads alternating 4103/4104, all identical)
$ curl -s localhost:4103/get/config/callbacks -H "Authorization: Bearer $LITELLM_MASTER_KEY"
{"status":"success","callbacks":[], ...}

10 chats tagged after-deleted alternating 4103/4104
$ curl -s -u $LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY "localhost:3000/api/public/traces?tags=after-deleted"
{"data":[],"meta":{"page":1,"limit":50,"totalItems":0,"totalPages":0}}

$ curl -s -X POST localhost:4103/config/update ... -d '{"litellm_settings":{"success_callback":["langfuse_otel"]}}'
{"message":"Config updated successfully"}
(35s later) both processes list langfuse_otel again, 6 chats tagged after-readded
$ curl -s -u $LANGFUSE_PUBLIC_KEY:$LANGFUSE_SECRET_KEY "localhost:3000/api/public/traces?tags=after-readded"
{"data":[...],"meta":{"page":1,"limit":50,"totalItems":6,"totalPages":1}}

Observations

• Right after the delete, both legs still list the callback (read_only: true); the after leg clears it on the next sync tick, within 30s
• Delete response says success while exports continue for up to one sync interval; PR leaves this alone
• Langfuse v4 compose disables GET /api/public/traces; v3 was used for the read API

[yuneng-berri]

@greptileai review

[yucheng-berri]

@greptileai review latest head

[yucheng-berri]

bugbot run

[cursor]

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.