Skip to content

test(logging): pin routing_decision and internal_call_origin in the gcs pubsub spend log fixture - #35506

Merged
yuneng-berri merged 1 commit into
litellm_internal_stagingfrom
litellm_/gcs-pubsub-test-metadata-4685a9
Aug 1, 2026
Merged

test(logging): pin routing_decision and internal_call_origin in the gcs pubsub spend log fixture#35506
yuneng-berri merged 1 commit into
litellm_internal_stagingfrom
litellm_/gcs-pubsub-test-metadata-4685a9

Conversation

@yuneng-berri

Copy link
Copy Markdown
Collaborator

TLDR

Problem this solves:

  • test_async_gcs_pub_sub_v1 fails on litellm_internal_staging
  • Two new SpendLogsMetadata fields never reached the golden fixture
  • Failure is stale test data, no runtime behavior changed

How it solves it:

  • Adds routing_decision and internal_call_origin to the fixture
  • Values pinned to null, matching a real proxy request
  • Keeps them out of ignored_keys so drift still fails

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • My PR passes all CI/CD checks (e.g., lint, format, unit tests)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

This PR changes a golden test fixture only; there is no runtime diff, so there is no before/after behavior to exercise on a proxy. What is worth proving is that the two keys the fixture now pins are genuinely what a live proxy emits, and that the fixture still catches drift rather than tolerating it. Both are shown below

1. A real request through a live proxy emits both keys

Proxy started against this branch at aaf619c270, real OpenAI call, real spend

curl -s -X POST http://localhost:4001/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o","messages":[{"role":"user","content":"Say the single word: fixture"}]}'
{
  "id": "chatcmpl-E8Bgj253c9SgKKWgFfRorp1COZ2GE",
  "model": "gpt-4o",
  "choices": [{"finish_reason": "stop", "index": 0,
               "message": {"content": "Fixture.", "role": "assistant"}}],
  "usage": {"completion_tokens": 2, "prompt_tokens": 13, "total_tokens": 15}
}

That request's spend log row, read straight out of Postgres:

psql "$DATABASE_URL" -At -c \
  "SELECT request_id, model, spend FROM \"LiteLLM_SpendLogs\" WHERE request_id='chatcmpl-E8Bgj253c9SgKKWgFfRorp1COZ2GE';"
chatcmpl-E8Bgj253c9SgKKWgFfRorp1COZ2GE|openai/gpt-4o|5.25e-05
psql "$DATABASE_URL" -At -c \
  "SELECT metadata::text FROM \"LiteLLM_SpendLogs\" WHERE request_id='chatcmpl-E8Bgj253c9SgKKWgFfRorp1COZ2GE';" \
| python -c "
import sys, json
md = json.loads(sys.stdin.read())
print('routing_decision     present:', 'routing_decision' in md, '| value:', md.get('routing_decision'))
print('internal_call_origin present:', 'internal_call_origin' in md, '| value:', md.get('internal_call_origin'))"
routing_decision     present: True | value: None
internal_call_origin present: True | value: None

Both keys are present and null on a plain non-router call, which is exactly what the fixture now asserts

2. The fixture still fails on drift

Temporarily setting the fixture's routing_decision to a non-null value at aaf619c270 reproduces a real failure, so the added keys are being compared rather than waved through:

AssertionError: Dictionary mismatch: ['Value mismatch at metadata.routing_decision:
  expected: mutated
  got:      None']

3. Before and after on the failing test

At b1fd20f4cd (staging HEAD, before this commit):

Dictionary mismatch: ['Extra key in actual: metadata.routing_decision',
                      'Extra key in actual: metadata.internal_call_origin']

At aaf619c270, both tests in the file pass

Type

✅ Test

Changes

_get_spend_logs_metadata builds the spend log metadata blob by iterating SpendLogsMetadata.__annotations__, so every field added to that TypedDict automatically lands in the emitted payload. Two feature PRs added a field without updating the golden fixture that test_async_gcs_pub_sub_v1 diffs against: #35016 added routing_decision and #35300 added internal_call_origin. The fixture has been failing on litellm_internal_staging ever since

The fix adds both keys to the metadata blob in tests/logging_callback_tests/gcs_pub_sub_body/spend_logs_payload.json with null values. They are deliberately not added to ignored_keys; that list exists for values that genuinely vary per run, and both of these are stable null for a request that goes through neither the auto-router nor an internal classifier call. Putting them in the ignore list would silence the fixture on two fields that consumers of the spend log schema depend on

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

@greptile-apps

greptile-apps Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR refreshes the GCS Pub/Sub spend-log golden fixture to match the current metadata schema without changing runtime behavior.

  • Adds routing_decision with a pinned null value.
  • Adds internal_call_origin with a pinned null value.
  • Keeps both fields under strict fixture comparison so future schema drift remains detectable.

Confidence Score: 5/5

The PR appears safe to merge because it only synchronizes a strictly compared test fixture with the demonstrated spend-log payload.

The added fields match the current payload for the fixture’s request scenario, remain subject to strict comparison, and do not alter production code.

Important Files Changed

Filename Overview
tests/logging_callback_tests/gcs_pub_sub_body/spend_logs_payload.json Updates the serialized metadata fixture with two stable null fields now emitted for the tested non-router, non-internal request.

Reviews (1): Last reviewed commit: "test(logging): pin routing_decision and ..." | Re-trigger Greptile

@codecov

codecov Bot commented Aug 1, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@yuneng-berri
yuneng-berri merged commit 22c3aae into litellm_internal_staging Aug 1, 2026
76 checks passed
@yuneng-berri
yuneng-berri deleted the litellm_/gcs-pubsub-test-metadata-4685a9 branch August 1, 2026 22:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants