fix(proxy): resolve team callbacks when key logging is empty and report GCS flush failures

[devin-ai-integration]

TLDR

Problem this solves:

• A key saved with an empty logging list silences its team's and the config's callbacks
• Failed GCS uploads were dropped silently, nothing retried them
• /key/health only looked at key metadata, so it said healthy while nothing reached GCS

How it solves it:

• An empty key logging list now means unset: the team's callbacks apply, then the config's
• A team's empty logging list still means the team turned logging off
• Callback entries a request would ignore now make /key/health say unhealthy
• /key/health tests the callbacks a real request would use
• For gcs_bucket it flushes the queue now and reports whether its own test event reached the bucket. Another key's queued failure does not make this key unhealthy, the flush drains everything queued rather than one batch, and it waits for any periodic flush already in flight so a failure that flush hits is not missed
• Failed GCS batches stay queued and go out on the next flush

Behavior change, no flag:

• A key stored with "metadata": {"logging": []} used to log nowhere. After this PR it logs through its team's callbacks, then the config's. The Admin UI writes that empty list on every key created without a logging integration, so it never expressed intent to disable, and litellm_disabled_callbacks remains the way to turn callbacks off on a key. A team's logging: [] (what /team/{id}/disable_logging writes) still disables

Review notes:

• Two Greptile threads were withdrawn by Greptile after discussion: cross-team attribution of GCS failures (process-wide queue predates this PR) and reporting healthy after a periodic flush failed but the health flush's retry delivered the event

User Flow

Before: a team admin sets GCS logging for the team in the proxy config, then a member creates a key from the Admin UI and none of that key's requests ever land in the bucket

1. The admin puts default_team_settings with success_callback: ["gcs_bucket"] for team-gcs in the config and restarts the proxy
2. A member creates a key for team-gcs at https://litellm-domain/ui/?page=api-keys without picking a logging integration, so the key is stored with "metadata": {"logging": []}
3. They send POST https://litellm-domain/v1/chat/completions with that key and get a 200 with a normal chatcmpl-... response
4. The bucket never gets an object for that request
5. They send POST https://litellm-domain/key/health with that key and get {"key":"healthy","logging_callbacks":{"callbacks":[],"status":"healthy",...}}, which says nothing is wrong
6. A key with no logging metadata at all does log to GCS, but when the bucket rejects one upload those events are gone: the next flush is silent and /key/health still says healthy

After: the same key logs to the team's bucket, and /key/health tells the truth about GCS

1. The admin keeps the same config and the member creates the same key with "metadata": {"logging": []}
2. They send POST https://litellm-domain/v1/chat/completions with that key and get the same 200
3. Within one flush interval the bucket has a YYYY-MM-DD/batch-....ndjson object containing that chatcmpl-... id
4. They send POST https://litellm-domain/key/health with that key and get {"key":"healthy","logging_callbacks":{"callbacks":["gcs_bucket"],"status":"healthy",...}}
5. If GCS is unreachable, the same /key/health returns {"key":"unhealthy",...,"details":"... GCS upload failed for the /key/health event and 0 other event(s), 0 uploaded ..."}
6. Once GCS is reachable again the next flush uploads the events that failed earlier, so the bucket still gets that request

Relevant issues

Linear ticket

Resolves LIT-7072

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.). All 89 checks pass at the current tip acba499, including CodeQL, see Review and CI status below
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

This is the live audit run against a real Google Cloud Storage bucket, replacing the earlier fake-gcs-server run. Every request below is a real provider call. Two proxies ran side by side from the same config and the same key set: the Before leg from the merge base with main on :20173, the After leg from this PR's tip on :20172, each with its own Postgres database, 2 workers, LITELLM_DISABLE_NO_REDIS_WARNING=true, --use_v2_migration_resolver, no --detailed_debug, GCS_FLUSH_INTERVAL=5. The destination is a real GCS bucket (name redacted as <BUCKET>) written with a real service account through the proxy's own GCS logger; objects land under litaudit-7072gcs/<leg>/. Traffic to storage.googleapis.com:443 goes through a local TCP forwarder (hosts override) so the chaos cases can cut or slow the destination; the forwarder only copies bytes and TLS still terminates at Google. No mocks anywhere in the product path: the only mock_response is the one POST /key/health itself sends by design

Config:

model_list:
  - model_name: gpt-5.6
    litellm_params:
      model: openai/gpt-5.6
      api_key: os.environ/OPENAI_API_KEY
  - model_name: claude-sonnet-5
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: broken-model
    litellm_params:
      model: openai/gpt-5.6
      api_key: sk-invalid-litaudit
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
litellm_settings:
  default_team_settings:
    - team_id: team-gcs
      success_callback: ["gcs_bucket"]
      failure_callback: ["gcs_bucket"]

Keys, created the way the Admin UI does it ("metadata":{"logging":[]}) unless stated otherwise. $K_YAML_EMPTY is on team-gcs (callbacks from the YAML above); $K_YAML_NULL is the same team with "logging": null; $K_YAML_MISSING is the same team with no logging key; $K_TEAM_EMPTY is on team-meta, whose metadata.logging is [{"callback_name":"gcs_bucket","callback_type":"success_and_failure","callback_vars":{}}]; $K_TEAM_SUCCESS and $K_TEAM_FAILURE are on teams whose callback_type is success and failure; $K_KEY_ONLY carries its own success_and_failure entry on a team with no settings; $K_KEY_OVER_TEAM carries a success entry on team-meta; $K_DEPRECATED is on a team using the deprecated metadata.callback_settings shape; $K_TEAM_DISABLED is on a team whose metadata.logging is []; $K_NO_TEAM has no team. Every "landed" claim was checked by listing the real bucket through the storage JSON API and matching the response id inside the .ndjson payloads (streaming /v1/responses payloads carry an internal id, so those cells match on a unique marker embedded in the prompt instead; the base leg has the same id shape). Team and key metadata sit in a 60 s per-worker in-memory cache (no Redis), so settings changes are asserted after a 65 s wait. The full matrix (76 executed rows, exact commands, ids, object names) is in the audit report shared with the requester

curl -s -X POST localhost:20172/key/generate -H "Authorization: Bearer $MASTER" \
  -d '{"team_id":"team-gcs","key_alias":"K_YAML_EMPTY","metadata":{"logging":[]}}'
# -> {"key_alias":"K_YAML_EMPTY", ..., "team_id":"team-gcs", ..., "metadata":{"logging":[]}, ...}

Before (d4a72e7)

Empty key logging on a team with YAML callbacks

1. curl -s -X POST localhost:20173/v1/chat/completions -H "Authorization: Bearer $K_YAML_EMPTY" -H 'Content-Type: application/json' -d '{"model":"gpt-5.6","messages":[{"role":"user","content":"Reply with the single word ok. marker litaudit-base-B1-1789419396"}]}' -> 200, "id":"chatcmpl-EO8161wxcOHYOJDFA5VloXOXrWuf7"
2. Bucket listing under litaudit-7072gcs/base/ after the flush interval: no object contains that id

Empty key logging on a team with metadata callbacks

1. Same request on :20173 with $K_TEAM_EMPTY -> 200, "id":"chatcmpl-EO8175NyqLOmgmxxoGGU87bGoKXHf"
2. No object contains that id

Control key with no logging entry

1. Same request on :20173 with $K_YAML_MISSING -> 200, "id":"chatcmpl-EO819rFy5osHtEViX1LrcDm9uOVLD"
2. 2026-09-14/batch-1789419402442-66e6cf9f.ndjson contains that id, so the team's callbacks work when the key says nothing at all

/key/health for empty and null key logging

1. curl -s -X POST localhost:20173/key/health -H "Authorization: Bearer $K_YAML_EMPTY" -> 200 {"key":"healthy","logging_callbacks":{"callbacks":[],"status":"healthy","details":"No logger exceptions triggered, system is healthy. Manually check if logs were sent to [] "}}
2. curl -s -X POST localhost:20173/key/health -H "Authorization: Bearer $K_YAML_NULL" -> 500 {"error":{"message":"Key health check failed: 'NoneType' object is not iterable","type":"internal_server_error","param":"None","code":"500"}}

/key/health while the destination is unreachable

1. Forwarder set to down (every connection to storage.googleapis.com:443 refused), 6 s wait
2. curl -s -X POST localhost:20173/key/health -H "Authorization: Bearer $K_YAML_MISSING" -> 200 {"key":"healthy","logging_callbacks":null}, nothing names the failed upload
3. Forwarder back to ok

Endpoints, clients, modes, sad paths, edge cases, chaos and Admin UI

1. Not run on the Before leg. The five cases above are the rows whose outcome this PR changes; the After leg below runs the full matrix at the tip

After (cd322fa)

Empty key logging on a team with YAML callbacks

1. curl -s -X POST localhost:20172/v1/chat/completions -H "Authorization: Bearer $K_YAML_EMPTY" -H 'Content-Type: application/json' -d '{"model":"gpt-5.6","messages":[{"role":"user","content":"Reply with the single word ok. marker litaudit-head-H1-1789419421"}]}' -> 200, "id":"chatcmpl-EO81WhenjXChQ56fdwxefqN8REmfi"
2. litaudit-7072gcs/head/2026-09-14/batch-1789419426166-7aac9fe9.ndjson contains that id, status: success, exactly one object
3. Same with "stream": true (-N, stream read to the end) -> chatcmpl-EO81WASVgvHw3tDxugwRqIMeohvld, one object

Empty key logging on a team with metadata callbacks

1. openai.OpenAI(base_url="http://localhost:20172", api_key=$K_TEAM_EMPTY).chat.completions.create(model="gpt-5.6", ...) -> chatcmpl-EO81Y2urEkytVdqS1mpcs6zI8QHG1, one object (batch-1789419426166-7aac9fe9.ndjson)
2. Same call with stream=True -> chatcmpl-EO81ZXLe4fP9JyBqO15Ia9f7qXvmg, one object (batch-1789419427053-65c762c6.ndjson)

Control key with no logging entry

1. Same curl on :20172 with $K_YAML_MISSING -> 200, chatcmpl-EO81t2D2flmQ8dCPEbsfI4HkVTUNM, one object (batch-1789419446817-4227c0f8.ndjson), unchanged from Before

/key/health for empty and null key logging

1. curl -s -X POST localhost:20172/key/health -H "Authorization: Bearer $K_YAML_EMPTY" -> 200 {"key":"healthy","logging_callbacks":{"callbacks":["gcs_bucket"],"status":"healthy","details":"No logger exceptions triggered, system is healthy. Manually check if logs were sent to gcs_bucket"}}
2. Same for $K_YAML_NULL -> 200, callbacks: ["gcs_bucket"], healthy (was a 500 Before)
3. Same for $K_YAML_MISSING, $K_TEAM_EMPTY, $K_KEY_ONLY and $K_DEPRECATED -> 200, callbacks: ["gcs_bucket"], healthy
4. $K_TEAM_DISABLED, $K_NO_TEAM and a key on a team with no settings -> 200 {"key":"healthy","logging_callbacks":null}, so an explicit team disable stays disabled

/key/health while the destination is unreachable

1. Forwarder set to down, then a 36 request burst (below) and curl -s -X POST localhost:20172/key/health -H "Authorization: Bearer $K_YAML_EMPTY" -> 200 {"key":"unhealthy","logging_callbacks":{"callbacks":["gcs_bucket"],"status":"unhealthy","details":"Logger exceptions triggered, system is unhealthy: GCS upload failed for the /key/health event and 18 other event(s), 0 uploaded GCS Bucket error logging batch payload to GCS bucket: Cannot connect to host storage.googleapis.com:443 ..."}}
2. Forwarder back to ok, one flush interval later the same call -> 200 healthy, and the health event itself is in the bucket
3. Real 404: a third head instance on :20174 with GCS_BUCKET_NAME set to a bucket that does not exist. A chat completion with $K_YAML_EMPTY -> 200 chatcmpl-EO8EIeMEhyrsNQ0mj3Z2InaBbyuYH (caller unaffected), then /key/health -> 200 {"key":"unhealthy",...,"details":"... GCS upload failed for the /key/health event and 1 other event(s), 0 uploaded GCS Bucket error logging batch payload to GCS bucket: Client error '404 Not Found' for url 'https://storage.googleapis.com/upload/storage/v1/b/<missing bucket>/o...'"}
4. Real 403: a fourth head instance on :20175 pointed at a public bucket this service account cannot write. Chat completion -> 200 chatcmpl-EO8F2um3q2B86S8zS7HRVZZmM7c62, /key/health -> 200 unhealthy with the 403 Forbidden text from GCS in details

Endpoints, clients, modes, sad paths, edge cases, chaos and Admin UI

1. Every endpoint and client with a key whose logging the fix has to resolve, one request per cell, each id found in exactly one object in the real bucket:

endpoint	stream	client	key	response id or marker
/v1/chat/completions	no	curl	$K_YAML_EMPTY	chatcmpl-EO81WhenjXChQ56fdwxefqN8REmfi
/v1/chat/completions	yes	curl	$K_YAML_EMPTY	chatcmpl-EO81WASVgvHw3tDxugwRqIMeohvld
/v1/chat/completions	no	openai sync	$K_TEAM_EMPTY	chatcmpl-EO81Y2urEkytVdqS1mpcs6zI8QHG1
/v1/chat/completions	yes	openai sync	$K_TEAM_EMPTY	chatcmpl-EO81ZXLe4fP9JyBqO15Ia9f7qXvmg
/v1/chat/completions	no	openai async	$K_KEY_ONLY	chatcmpl-EO81as7hNsb9kP6UBhnkMLFj9qY3f
/v1/chat/completions	yes	openai async	$K_KEY_ONLY	chatcmpl-EO81boiPbtxkJs9b75v0G45mX1nsP
/v1/messages	no	curl	$K_YAML_EMPTY	msg_011Cf3zzh8rXJp7CddWd72WA
/v1/messages	yes	curl	$K_TEAM_EMPTY	msg_011Cf3zznFRf2yQQtNMseR4q
/v1/messages	no	anthropic sync	$K_YAML_NULL	msg_011Cf3zzr19uYKER6YH6dKRk
/v1/messages	yes	anthropic sync	$K_TEAM_MISSING	msg_011Cf3zzvaV5MKssWS8yS2nr
/v1/messages	no	anthropic async	$K_KEY_ONLY	msg_011Cf4111b6khdsMCFRP2WQz
/v1/messages	yes	anthropic async	$K_YAML_MISSING	msg_011Cf4115z26ssZ17BuuSs3n
/v1/responses	no	curl	$K_YAML_EMPTY	resp_2cLWn-ROqOw6wY7wG4lIvDGaC0W30yco4bI6fY3AKfKiiX8Y...
/v1/responses	yes	curl	$K_TEAM_EMPTY	marker litaudit-head-H14-...
/v1/responses	no	openai sync	$K_TEAM_SUCCESS	resp_DfM298wOeouAIw4XsLeVqfIZ3LStiibxI3IuA_ZhwVzJfPY8...
/v1/responses	yes	openai sync	$K_KEY_OVER_TEAM	marker litaudit-head-H16-...
/v1/responses	no	openai async	$K_DEPRECATED	resp_KvL10PY6AKUI5gBBj3n6djUdZ8zXw9tCSLZCxca0MLHYbSO_...
/v1/responses	yes	openai async	$K_YAML_NULL	marker litaudit-head-H18-...
/v1/chat/completions	no	curl	$K_DEPRECATED	chatcmpl-EO81s151pPyHvk7Bgw02nofEhMCPp

2. Callback modes, invalid provider key via broken-model: success team uploads the 200 (chatcmpl-EO81uHQceKdrj1OzTnveBZOxqp20z) and skips the 401; failure team skips the 200 (chatcmpl-EO81yCLAVjcF9ESo5gVXe4wj5rZcV) and uploads the error (one failure object, batch-1789419452009-eab89d1b.ndjson); success_and_failure via YAML uploads both; $K_KEY_OVER_TEAM (key success on a success_and_failure team) uploads no failure object, so a key that says something still wins. After the first 401 the router cools the deployment down, so later cells saw its 429 No deployments available instead of the 401; both reached the caller in the response body
3. Explicit disable: $K_TEAM_DISABLED chat (chatcmpl-EO81yqnoJ7kYn0ex8hvseWwapPaQ4) and /v1/messages (msg_011Cf412RChmTXBxEp3Hf1a5) -> 200, zero objects. $K_NO_TEAM and a key on a team with no settings -> 200, zero objects
4. Sad paths, all with the proxy staying up and an unrelated key still returning 200 afterwards: unknown model on all three endpoints -> 400 Invalid model name passed in model=no-such-model-litaudit, no failure object (the router rejects it before any provider call, same as Before); max_tokens: 99999999 -> 400 max_tokens is too large: 99999999. This model supports at most 128000, one failure object with the marker; a key logging entry without callback_vars -> request 200 and /key/health 200 {"key":"unhealthy",...,"details":"Invalid callback metadata, requests ignore these entries: not_a_callback: callback_vars Field required"}; a well formed entry naming a callback no logger implements -> request 200, zero objects, /key/health says healthy for ["not_a_callback"], unchanged from Before and tracked as LIT-7693; a key callback_vars.gcs_bucket_name pointing at a missing bucket -> the upload still goes to the process bucket because that var is request-blocked, unchanged from Before and tracked as LIT-7692; an entry with no callback_name -> request 200, /key/health 500 Key health check failed: callback_name is required in key_logging, same handler as Before (every health exception becomes a 500), listed under caveats
5. Edge cases: five identical requests with $K_YAML_EMPTY -> five distinct ids, one object each. POST /team/team-plain/callback adding gcs_bucket while a request fires every 8 s: every request from 0.9 s after the update landed once. POST /team/team-plain/disable_logging while traffic flows: one request 18.6 s after the disable still landed (the other worker's 60 s team cache, same window as Before), nothing after the TTL. POST /key/update switching $K_TEAM_EMPTY's own logging to failure and back to [] with a 65 s wait after each update: success objects 1 / 0 / 1
6. Chaos, forwarder cut to the real bucket at the start of a 36 request burst across all three endpoints, mixed streaming, five keys, both workers: 36/36 callers got 200, /key/health during the cut reported unhealthy (step 1 of the previous case), after ok every one of the 36 ids landed exactly once, zero duplicates, zero missing. Slow destination (3 s added to every chunk sent to GCS): 18/18 landed exactly once and three back to back /key/health calls racing the periodic flush returned healthy in 10.1 s, 5.9 s and 5.0 s, no deadlock. SIGKILL of one of the two head workers mid burst: the other worker kept serving, the supervisor spawned a replacement, 16/24 callers got 200, 15 of those landed once, the one event still sitting in the killed worker's in-memory queue was lost. Proxy restart mid burst: 24/24 callers got 200 before the stop, the 24 events still queued in memory did not survive the process, no duplicates, proxy back in 24.4 s and serving 200
7. Admin UI at http://localhost:20172/ui (admin plus master key). ?page=teams, open team-meta, Logging Settings tab: gcs_bucket, Success & Failure. ?page=api-keys, open K_YAML_EMPTY, Settings, Logging Settings: no integrations (the dashboard renders the stored "logging": [] as an empty state; the API transcript above shows the literal metadata). ?page=logs, click the row for chatcmpl-EO81WhenjXChQ56fdwxefqN8REmfi: Success, gpt-5.6, cost and token counts, team team-gcs. Playground, virtual key K_YAML_EMPTY, /v1/chat/completions, gpt-5.6, prompt Reply with exactly: confirmed litaudit-ui-20260914-225430 -> reply confirmed litaudit-ui-20260914-225430, response id chatcmpl-EO8lVw2TZYJjLcVKHsXtDMCxUgj1a, and that marker is in exactly one object in the real bucket (litaudit-7072gcs/head/2026-09-14/batch-1789422277664-a1c91631.ndjson, 82 objects listed, 1 match). The same viewer for the $K_TEAM_DISABLED request shows 82 objects and 0 matches. Google Cloud console needs an interactive Google login this session does not have, so the bucket pages are a local read-only page that lists the real bucket through the storage JSON API with the same service account. The dashboard has no button that calls POST /key/health, so that endpoint is proven with the curl steps above. Screenshots and the recording are in the PR comment linked below, kept out of the description on purpose

Not run: the OTel v2 destination fallback (no OTel collector in this rig, unit tested only)

Type

🐛 Bug Fix

Caveats (if any)

Medium

• /key/health now runs the 2 second test log for keys whose callbacks come from the team or config, not only from key metadata. It used to return instantly for those keys
• A batch that keeps failing (bad credentials for one team's bucket) is retried every flush until the queue fills, then the oldest event is dropped with an error log. Before, it was dropped on the first failure

Low

• The unhealthy details string still carries the captured logger traceback, same as before. The GCS verdict for the health check's own event is prepended to it
• GCSFlushResult now carries sent_ids / failed_ids; sent and failed counts remain as properties, and flush_queue() still returns None
• Both GCS flush paths now hold the logger's flush_lock, so /key/health waits for a periodic flush that is mid-upload instead of racing it. flush_queue_and_report() takes every event off the queue before uploading, sends them in batch_size groups, and leaves any failed group on the queue for the next flush instead of retrying it in the same call. The periodic flush still sends one batch per interval
• Disabling callbacks on a key stays litellm_disabled_callbacks; an empty key logging list no longer does that. The OTel v2 destination resolver follows the same rule
• After /team/{id}/disable_logging, a proxy worker keeps serving the team's old callbacks until its 60 second team cache expires, same as before this PR (seen once at 18.6 s in the audit)
• GCS events wait in process memory until the next flush, so a worker kill or a proxy restart loses whatever is still queued, same as before this PR (1 of 15 and 24 of 24 events in the chaos cases)
• A key logging entry with no callback_name makes /key/health return 500 with a clear message instead of 400, same handler as before this PR
• A well formed key logging entry naming a callback no logger implements still reports healthy (LIT-7693), and a per key callback_vars.gcs_bucket_name still cannot redirect the upload (LIT-7692). Both unchanged from the base and out of scope here
• tests/test_litellm/integrations/gcs_bucket/test_gcs_bucket.py is new because gcs_bucket.py had no mapped test file (test_gcs_bucket_base.py covers the base class)

Review and CI status (at cd322fa)

• Base and branch: this PR replaces fix(proxy): resolve team callbacks when key logging is empty and report GCS flush failures #40922, same commits, same tip cd322fa. fix(proxy): resolve team callbacks when key logging is empty and report GCS flush failures #40922 was opened against litellm_internal_staging, then moved to main on request, where the required Guard main branch check rejects any head that is not litellm_internal_staging or litellm_hotfix_*, so the branch was pushed again as litellm_hotfix_7072_team_callbacks. Review threads and the Greptile and Bugbot history also live on fix(proxy): resolve team callbacks when key logging is empty and report GCS flush failures #40922; the visual evidence for the real GCS run is a comment on this PR
• Merging into main: main later grew a conflicting import line at the top of test_key_management_endpoints.py, so 8bc772d merges main into the branch and keeps both imports (asyncio from this PR, SimpleNamespace from main). No product code changed in that merge; the audit and review evidence below are for cd322fa, the last commit that touched product code. CI at 8bc772d: 89 checks pass, 1 skipped (stage-mirror e2e), 0 fail; CodeQL passes there. main then added a CustomKeyPolicyRequest import and new bulk-update tests at the end of the same test file, so acba499 merges main again and keeps both sides (the import next to this PR's ProxyConfig import, main's tests after this PR's health tests). Again no product code changed; the focused key-management tests pass at that tip (49 passed) and CI at acba499 is 89 pass, 1 skipped, 0 fail. The schema.d.ts sync check that failed against staging (two stale soft_budget doc lines that main had already regenerated, chore(ui): regenerate schema.d.ts after soft_budget left the user endpoint docstrings #40987, now closed) does not apply to main
• CI at cd322fa against main (this PR): 88 checks pass, 1 skipped (stage-mirror e2e, no e2e files changed), 1 fails (CodeQL, below, not in main's required checks). Every required check passes, including the schema.d.ts sync check. Against staging (on fix(proxy): resolve team callbacks when key logging is empty and report GCS flush failures #40922) the same tip passed 83 checks and failed only that schema check
• Greptile (this PR): 4/5 at cd322fa with two threads. One asks that the empty key logging behavior change be flagged, now stated in the TLDR above. The other says concurrent health checks share the root logger; the base branch attaches the same handler to the root logger for every health check, so this PR does not change that. Both stood down in-thread, Greptile withdrew both on re-trigger, and its second summary at cd322fa is 5/5 with no outstanding findings. On fix(proxy): resolve team callbacks when key logging is empty and report GCS flush failures #40922 the same tip also scored 5/5 with no open threads
• Cursor Bugbot: "found no new issues" at cd322fa on this PR (run triggered 2026-09-14 19:48 UTC, review submitted 19:53 UTC). The same sha also passed Bugbot on fix(proxy): resolve team callbacks when key logging is empty and report GCS flush failures #40922
• CodeQL: passes at 8bc772d. At cd322fa the job failed with 12 alerts, all from the py/cyclic-import and py/unsafe-cyclic-import queries, and all in key_management_endpoints.py or at the other end of a cycle it already sits in (proxy_server.py:526-531, internal_user_endpoints.py:54-56). None is a runtime problem. The CustomLogger alert (line 165) points at an import under TYPE_CHECKING, which never executes. The encrypt_callback_vars alert (line 74) is the same module-level import main has at the same line, minus names this PR stopped using. The four "begins an import cycle" notes are function-level imports of proxy_server, litellm_pre_call_utils, litellm_logging and gcs_bucket; the file already imports proxy_server at 19 function-level sites on main for the same reason (proxy_server imports this module), and import litellm loads litellm_logging and gcs_bucket before any proxy module. The three internal_user_endpoints and three proxy_server alerts describe that pre-existing cycle. CodeQL also fails on another open main PR at the time of writing (feat(router): cool deployment down until quota-reset time stated in error body #41090)
• Veria: no Veria review, comment, or check-run on fix(proxy): resolve team callbacks when key logging is empty and report GCS flush failures #40922. Of the 21 most recent internal-branch PRs into litellm_internal_staging, Veria reviewed one (fix(mcp): require admission for delegated OAuth #40923, an MCP auth change), so a missing Veria verdict on this PR is treated as not applicable, not as pending
• Devin Review (this PR, at cd322fa): two threads, both stood down in-thread and resolved. One says a non-200 JSON response from GCS is counted as sent because _log_json_data_on_gcs returns the body without raising; not reachable, AsyncHTTPHandler.post calls raise_for_status() first, and a MockTransport returning 403 makes _send_grouped_batch return (0, 1) and _send_single_log_item return False. The other repeats the fix(proxy): resolve team callbacks when key logging is empty and report GCS flush failures #40922 thread that a failure-only callback goes untested because the health check's mock completion always succeeds. That is how the base branch behaves too, this PR only changes which callbacks the health check selects and how GCS results are read back, and treating an absent event id as a failure would misreport events a periodic flush already uploaded
• CodSpeed: no performance change. Codecov patch: pass

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

ran /live-pr-risk and found no regressions/backward incompatible risks

• d45f42f passes /live-pr-risk; 62b9130 differs from it only in where one import lives, and ba9e47e plus cd322fa only add annotations to test helpers and swap where one isinstance target is imported from

Link to Devin session: https://app.devin.ai/sessions/42a8e9b831c141108fd010826df31fe7
Open in Devin Desktop: https://app.devin.ai/desktop/session/42a8e9b831c141108fd010826df31fe7?variant=devin
Requested by: @yucheng-berri

---

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[devin-ai-integration]

Real GCS audit at cd322fa: Admin UI Playground request, its object in the real bucket, and the disabled team's zero matches

Playground request through the empty logging key

Real bucket, one object matches the Playground marker

Real bucket, the curl request id chatcmpl-EO81WhenjXChQ56fdwxefqN8REmfi

Real bucket, disabled team request, zero matches

Team logging settings, key logging settings, Logs detail

Recording of the walk-through

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_hotfix_7072_team_callbacks (acba499) with main (c8114ba)}

[greptile-apps]

Greptile Summary

This PR corrects callback inheritance for keys with empty logging metadata and makes GCS delivery failures visible and retryable.

• Treats an empty key-level logging list as unset while preserving an empty team-level list as an explicit disable state.
• Makes /key/health resolve the same effective callbacks as normal requests and report invalid callback metadata.
• Adds serialized GCS flushing, per-event results, and requeueing of failed uploads.
• Adds coverage for callback precedence, GCS retries, queue bounds, concurrent flushing, and health reporting.

Confidence Score: 5/5

The PR appears safe to merge, with no outstanding findings in the current review state.

No new changes exist since the previous review SHA, both previous threads are resolved, and the current diff contains no confirmed rule violation or actionable regression.

Important Files Changed

Filename	Overview
litellm/integrations/gcs_bucket/gcs_bucket.py	Adds locked flush reporting and retains failed GCS events for later retry while preserving bounded-queue behavior.
litellm/proxy/litellm_pre_call_utils.py	Treats empty key logging metadata as unset while retaining team-level empty logging as an explicit disable state.
litellm/proxy/management_endpoints/key_management_endpoints.py	Resolves effective request callbacks during key health checks and reports callback validation or GCS upload failures.
litellm/types/integrations/gcs_bucket.py	Introduces an immutable result type containing sent and failed GCS event identifiers.
tests/test_litellm/integrations/gcs_bucket/test_gcs_bucket.py	Covers failed-upload retention, retry behavior, multi-batch flushing, individual uploads, and queue overflow.
tests/test_litellm/proxy/management_endpoints/test_key_management_endpoints.py	Covers effective callback selection, invalid metadata, GCS health attribution, and in-flight flush synchronization.
tests/test_litellm/proxy/test_litellm_pre_call_utils.py	Verifies key/team callback precedence and explicit team logging disable behavior.
tests/test_litellm/integrations/otel/test_otel_v2_destinations.py	Updates OTel destination expectations to match empty-key callback inheritance.
litellm/proxy/_types.py	Generalizes health callback names to a read-only sequence.
ui/litellm-dashboard/src/lib/http/schema.d.ts	Synchronizes generated key-health API documentation with the expanded callback checks.

_{Reviews (2): Last reviewed commit: "refactor(proxy): read ModelResponse off ..." | Re-trigger Greptile}

[devin-ai-integration]

@greptileai

[yucheng-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit cd322fa. Configure here.}

[devin-ai-integration]

Devin Review found 2 potential issues.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

[codecov]

Codecov Report

❌ Patch coverage is 97.47899% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/integrations/gcs_bucket/gcs_bucket.py	94.64%	3 Missing ⚠️

📢 Thoughts on this report? Let us know!