test(pass-through): de-flake vertex spend-log test by routing through the proxy

[mateo-berri]

Relevant issues

tests/pass_through_tests/test_vertex_ai.py::test_basic_vertex_ai_pass_through_with_spendlog is one of the most flaky tests in the suite; CircleCI's flaky-test insights show it flaking dozens of times in the recent window in proxy_pass_through_endpoint_tests

Linear ticket

N/A

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

The root cause is visible in the proxy container logs of a failing proxy_pass_through_endpoint_tests run. Across the whole window the Vertex test was executing, the proxy served all 46 of the test's own GET /global/spend/logs polls but not a single generateContent, and its only upstream forwards were to AssemblyAI and OpenAI

$ grep -oE '"(POST|GET) /[^"]*"' proxy.log | sort | uniq -c | sort -rn
     46 "GET /global/spend/logs?api_key=best-api-key-ever HTTP/1.1"
      7 "GET /assemblyai/v2/transcript/... HTTP/1.1"
      2 "POST /openai/assistants HTTP/1.1"
      ...assemblyai / openai only, no vertex...

$ grep -ciE 'vertex|gemini|generateContent|aiplatform' proxy.log
0

The 46 polls match the test's exact count (1 baseline read plus 5 attempts of 9 polls), so the window covers the full test, yet the billed model.generate_content("hi") calls never reached LiteLLM. The vertexai SDK sent them straight to the public Vertex endpoint, which is why no spend was ever recorded and why re-billing through the SDK could not help

To confirm the rewrite is no longer flaky, the exact CI job that owns this test (proxy_pass_through_endpoint_tests) was rerun until it passed five times in a row on the final commit, with test_basic_vertex_ai_pass_through_with_spendlog green in every run (CircleCI jobs 1956777, 1957800, 1957849, 1957898, 1957899)

Type

✅ Test

Changes

The test configured the vertexai SDK with location="global" and an http api_endpoint pointing at the proxy, then billed generateContent and waited for the global spend aggregate to move. With that configuration the SDK intermittently ignores the endpoint override and calls the public Vertex endpoint directly, so the request bypasses the proxy and no LiteLLM_SpendLogs row is ever written; that bypass, not logging lag, is the flake

This change drives the pass-through over HTTP instead of through the SDK. It mints a Google access token from the same service-account credentials the SDK would use, posts the Vertex-shaped generateContent request straight to /vertex_ai/... on the proxy, and asserts that the specific call's own spend log lands with spend > 0, a gemini model, and a custom_llm_provider of vertex_ai, keyed on the x-litellm-call-id the proxy returns. Because the request now always reaches the proxy, the only residual flake is the best-effort background logging worker occasionally dropping a single event, so the test re-bills a few times and still fails hard if no call is ever tracked. The SDK-based client path stays covered by the sibling jest test (test_vertex_with_spend.test.js)

A small follow-up commit addresses the Greptile review: the get_tracked_spend helper used by the skipped streaming test now reuses the shared LITE_LLM_ENDPOINT constant instead of a hard-coded 0.0.0.0:4000, and drops its needless async since it only performs a blocking requests call

The change is test-only; no production behavior, performance, or contract changes

---

Note

Low Risk
Test-only changes in test_vertex_ai.py; no production proxy, billing, or API behavior is modified.

Overview
test_basic_vertex_ai_pass_through_with_spendlog no longer bills through the vertexai SDK. It POSTs generateContent to the LiteLLM /vertex_ai/... URL with a Google access token, then polls /spend/logs?request_id= using x-litellm-call-id until that call has spend > 0, gemini model, and vertex_ai provider—retrying up to three billed calls if background logging drops an event.

Shared helpers replace the old async global-spend polling: get_tracked_spend() sums all rows from /global/spend/logs (avoids UTC “today” mismatches) and uses LITE_LLM_ENDPOINT. The skipped streaming test switches to that helper instead of await call_spend_logs_endpoint().

^{Reviewed by Cursor Bugbot for commit 65e29fc. Bugbot is set up for automated code reviews on this repo. Configure here.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

Rewrites test_basic_vertex_ai_pass_through_with_spendlog to drive the Vertex pass-through over a direct HTTP request instead of the vertexai SDK, eliminating the root cause of the flake (the SDK silently bypassing the proxy endpoint override). The test now asserts on a per-call spend log keyed by x-litellm-call-id and retries up to three times to tolerate the background logging worker dropping a single event.

• Replaces the global-spend-delta assertion with a precise per-call spend log lookup using x-litellm-call-id, reducing false negatives from unrelated concurrent traffic.
• Extracts get_tracked_spend, _vertex_access_token, _spend_log_for_request, and shared constants (LITE_LLM_ENDPOINT, SPEND_LOG_API_KEY) to clean up the previously tangled helpers and fix the async/hardcoded-address issues flagged in earlier review rounds.
• Updates the skipped streaming test to use the shared get_tracked_spend helper for consistency.

Confidence Score: 5/5

Test-only change with no production code modified; the rewrite is strictly additive and the previous review comments have both been addressed.

The change is confined to a single integration test file. The root cause of the flake is well-documented in the PR, the fix is mechanically sound (direct HTTP avoids the SDK endpoint-override bug), the per-call x-litellm-call-id assertion is stronger than the previous global-spend-delta check, and earlier review concerns about the async helper and hard-coded endpoint have been resolved.

No files require special attention.

Important Files Changed

Filename	Overview
tests/pass_through_tests/test_vertex_ai.py	Rewrites the spend-log test to use direct HTTP instead of the vertexai SDK, adding precise per-call assertion via x-litellm-call-id and fixing the previously flagged async/hardcoded-endpoint issues in the helper functions.

_{Reviews (2): Last reviewed commit: "test(pass-through): reuse LITE_LLM_ENDPO..." | Re-trigger Greptile}

[codspeed-hq]

Merging this PR will not alter performance

✅ 30 untouched benchmarks

---

_{Comparing litellm_vertex_passthrough_spendlog_deflake (65e29fc) with litellm_internal_staging (8beb68a)}

[mateo-berri]

@greptileai

---

Generated by Claude Code

[mateo-berri]

bugbot run

---

Generated by Claude Code

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 65e29fc. Configure here.}