test(ci): repair stale tests and flaky CI infrastructure

[yuneng-berri]

TLDR

Problem this solves:

• Tests red on main across CircleCI, GHA and Buildkite after intended product changes
• A few are flaky because they race the UI or share database state
• GHA installs time out and Codecov uploads fail, so jobs go red with every test passing
• Integration and unit harnesses race on ports, keep-alive sockets, worker threads and outage timing

How it solves it:

• GCS pub/sub spend-log golden gains used_client_oauth_token from feat(proxy): record in spend logs whether a request used a client-forwarded Anthropic OAuth token #43063
• Auto-router edit test waits for the availability check before clicking Save
• Streamed service-tier e2e expects a default-served call to bill base rates, per fix(streaming): keep the served service_tier on streamed chunks and spend rows #42870
• Logs call-id e2e waits for the search response before hovering the row
• Together structured-output e2e moves off a reasoning-only model to a hybrid one with reasoning off
• Agent 365 MCP integration test reads its own guardrail's status, not whichever is first
• Budget limiter push-failure test ignores asyncio's reports about tasks other tests leaked
• Tag management e2e fills the create form inside its dialog, clear of feat(ui): filter tags by name and description on the Tag Management page #42949's new filter inputs
• Integration proxies read the CI license from the environment, so every worker of a multi-worker proxy has it
• GHA caches save only from main, so pull requests restore warm caches instead of downloading everything
• The unit and Redis workflows move to codecov-action 5.5.5, which imports Codecov's signing key from its current keybase account; main already moved the Postgres workflow to 7.1.1
• Owned integration proxies relaunch on a fresh port when another process takes theirs
• Integration gateway clients drop idle connections before uvicorn's 5 second keep-alive closes them
• The budget-reset database outage lasts 10 seconds instead of 5 refused connections
• The HTTP handler finalizer test joins its worker thread before collecting the handler
• The base SDK wheel build gets the 30 minute no-output window the Windows build already has

User Flow

Before: a contributor's PR inherits red CI jobs that have nothing to do with their change

1. They open a PR against main and add the run-ci label
2. CircleCI unit, logging_testing, integration-mcp, integration-extensions and e2e_ui_testing, GHA UI unit tests, and the scheduled Buildkite litellm-e2e build fail on tests their diff never touched

After: those jobs only go red when LiteLLM behavior actually changes

1. They open a PR against main and add the run-ci label
2. The repaired tests pass, and still fail when the behavior they guard is broken

Screenshots / Proof of Fix

Tests-only PR, so there is no customer-facing request to curl. Product code is identical in Before and After; the proof is that each repaired test fails the way CI does on the old test, passes on the new one, and fails again when the guarded behavior is mutated

Before (6d7d183)

GCS pub/sub golden

1. pytest tests/logging_callback_tests/test_gcs_pub_sub.py::test_async_gcs_pub_sub_v1
2. Failed: Dictionary mismatch: ['Extra key in actual: metadata.used_client_oauth_token']

Auto-router threshold save

1. npx vitest run src/components/edit_auto_router/edit_auto_router_modal.integration.test.tsx with the availability debounce raised to 1000ms to make the CI race deterministic
2. The threshold save cases fail because Save is still disabled while the page shows "Checking availability"

Streamed service tier

1. pytest tests/e2e/quota_management/spend_tracking/test_service_tier_pricing_e2e.py against a local proxy and live OpenAI
2. 3 failed, matching Buildkite: the chat, responses and messages streams expect the bill to record default, but a default-served call is now billed at base rates with no tier

Logs call-id tooltip

1. A Playwright script against the local dashboard fills the call-id search and hovers the row right away
2. No tooltip appears, because the search response re-renders the table under the mouse

Together structured output

1. Buildkite litellm-e2e 336 to 338 fail test_response_format_json_schema_shapes_the_reply
2. The cheapest-model picker moved to GLM-5.3-Flash after DeepSeek-V4-Flash-0731 hit its 2026-09-29 deprecation, and GLM-5.3-Flash is reasoning-only, so the reply content is None

Agent 365 guardrail status

1. Local integration rig, with a default_on pre_mcp_call content filter in the shared database, as test_mcp_accounting_guardrails.py creates it under xdist
2. pytest "tests/integration/mcp/test_mcp_agent_365_guardrail.py::test_a_missing_or_malformed_caller_bearer_blocks_on_every_entry_point_whatever_the_fallback[fail_closed]"
3. assert ['success', ...] == ['guardrail_intervened', ...], the same failure as CircleCI integration-mcp

Budget limiter push failure

1. A preceding test leaves a pending task on a closed event loop, as some earlier test does on CircleCI unit
2. pytest tests/unit/router_strategy/test_budget_limiter_hotpath.py::test_push_task_failure_is_logged_once_and_not_leaked
3. Fails with an extra Task was destroyed but it is pending! record from asyncio, the same failure as CircleCI unit

Tag management create

1. ./run_e2e.sh tests/tagManagement/tagManagement.spec.ts from tests/e2e/ui, which builds the dashboard from source and serves it from a local proxy
2. Fails with AggregateError: Action and cleanup failed, matching CircleCI e2e_ui_testing, whose trace names the cause: strict mode violation: getByLabel('Tag Name') resolved to 2 elements, the second being the new Filter by tag name input

Azure Storage on multi-worker proxies

1. CircleCI integration-extensions on 2684cd6 runs the Azure Storage files from fix(azure_storage): keep the DataLakeServiceClient alive until its TTL elapses #43082 and fix(azure_storage): name Data Lake objects without base64 padding or slashes #43914
2. Every 2-worker case fails waiting for an upload, and the proxy logs AzureBlobStorageLogger is only available for premium users about 100 times; the single-worker case passes

CI infrastructure

1. GHA auth-checks, proxy-server-core, proxy-utils and proxy-behavior time out in "Install dependencies": the uv cache misses even on its prefix, only pull-request-scoped caches exist (about 4.5 GB per PR in a 10 GB budget), and 18 of 55 packages finish downloading in 8 minutes
2. proxy-behavior uploads fail with gpg: Can't check signature: No public key after all 929 tests pass; https://keybase.io/codecovsecurity/pgp_keys.asc now returns SELF-SIGNED PUBLIC KEY NOT FOUND
3. CircleCI base_sdk_install is killed at "Build the wheel" with Too long with no output (exceeded 10m0s) in about 30% of recent runs

Integration and unit harness races

1. integration-mcp: an owned proxy logs [Errno 98] ... address already in use right after startup completes, and the test fails with Owned proxy exited before readiness
2. integration-management: httpx.ReadError: [Errno 104] Connection reset by peer on the first request after the test waits on Redis reconnects; reproduced with a bare uvicorn server as 9 resets in 450 requests reused 4.99 to 5.02 seconds after the last one
3. integration-accounting: the budget reset never lands within 80 seconds; a relay probe shows recovery needs 3 to 4 proxy retries because each retry uses about one of the 5 refused connections
4. unit: test_finalizer_on_live_loop_disposes_foreign_loop_session_without_scheduling fails assert session.closed; a probe that makes the pool worker linger 0.5 seconds after setting its result reproduces it every time
5. unit: the rust_bridge o200k_base tokenizer cases fail with SocketConnectBlockedError when their xdist worker has not imported default_encoding yet, so tiktoken skips the bundled cache; running the file alone with no temp cache fails all 8 cases
6. integration-providers: test_only_string_reasoning_content_is_forwarded_to_hosted_vllm[null-dropped] sees ('GET', '/v1/models') ahead of the chat call. The router's periodic upstream model info refresh probes hosted_vllm deployments, and with the refresh interval forced to 1 second locally the original file fails 3 of 21

After (8325201)

GCS pub/sub golden

1. Same command: 1 passed
2. Mutation, restored after the run: dropping used_client_oauth_token from the stamped spend-log metadata keys makes it fail

Auto-router threshold save

1. Same command with the 1000ms debounce: all 64 tests pass
2. Mutation: forcing Save to stay blocked makes the threshold save cases fail

Streamed service tier

1. Same command against live OpenAI: 4 passed
2. Mutation: billing a default-served call at the priority tier fails the chat and responses stream cases

Logs call-id tooltip

1. Same script, waiting for the search response first: the tooltip shows 3 of 3 runs

Together structured output

1. The case now runs on together_ai/Qwen/Qwen3.5-9B with reasoning_effort="none", which Together lists as hybrid with structured outputs

Agent 365 guardrail status

1. Same polluter and command: 1 passed, and the whole file passes 5 of 5
2. Mutation, restored after the run: agent 365 recording success on a rejected call fails the test with the polluter still present
3. Polluter removed: still passes

Budget limiter push failure

1. Same polluter and command: passes, and the whole file passes 25 of 25
2. Mutation, restored after the run: logging the Redis sync error a second time through another LiteLLM logger makes it fail

Tag management create

1. Same command: 1 passed
2. Only the create step's locators changed; every assertion on the create, edit and read-back requests is untouched

Azure Storage on multi-worker proxies

1. Local integration rig with LITELLM_LICENSE set: pytest tests/integration/observability/test_azure_storage_client_ttl.py tests/integration/observability/test_azure_storage_chaos.py tests/integration/observability/test_azure_storage_file_names.py
2. 16 passed, and no proxy logs the premium error

CI infrastructure

1. Pull request jobs restore caches and never save them; main saves them. zizmor reports no findings, and check_workflow_startup_safety still sees a 35 minute setup ceiling
2. The codecovsecops key imports as 806BB28AED779869, the key ID on the current CLI's SHA256SUM.sig
3. circleci config validate passes with the 30 minute window on the wheel build

Integration and unit harness races

1. With the first port handed out already held, the agent 365 test passes: the first proxy log shows address already in use and the second launch serves the test
2. With a 2 second client idle limit, the same bare uvicorn sweep gives 0 resets in 450; the wedged Redis and agent 365 files pass 6 of 6
3. The budget-reset test passes 3 of 3 in 74 to 98 seconds
4. The finalizer test passes under the same lingering-worker probe and the whole file passes 90 of 90; removing the finalizer's session dispose makes it fail
5. With the bundled cache fixture in the root unit conftest, the same isolated run passes 8 of 8 and the neighbors pass 152 of 152; pointing the native o200k_base at cl100k_base still fails 4 cases
6. With discovery probes answered with an empty model list and left out of the provider-call assertions, the same forced-refresh run passes 21 of 21 three times, and making non-string reasoning_content pass through still fails 3 cases

Type

✅ Test
🚄 Infrastructure

Caveats (if any)

Medium

• Cache savings start only after this merges and main's next push saves warm caches
• The budget-reset test still passes when LiteLLM's own reconnect never succeeds
  • Prisma's engine reconnects by itself once the database returns, so the test cannot tell them apart
• Together fix is unverified live, since no Together key exists outside CI
• The messages-stream service-tier case only checks the bill is self-consistent
  • It still passes when a default-served call is billed at the priority tier

QA runbook

• tests/e2e/quota_management/spend_tracking/test_service_tier_pricing_e2e.py::TestServiceTierPricing::test_streamed_call_records_and_bills_the_served_tier - a streamed chat call OpenAI served at default records no tier and bills base input rates

  • Register an OpenAI model with custom input, output and priority input rates (needs OPENAI_API_KEY)
  • POST /v1/chat/completions with "stream": true and no service_tier
  • Read the spend log row: no service_tier in the cost breakdown, and fresh input tokens billed at the base input rate
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/quota_management/spend_tracking/test_service_tier_pricing_e2e.py::TestServiceTierPricing::test_responses_stream_records_the_served_tier - the same for a streamed /v1/responses call

  • POST /v1/responses with "stream": true against the same custom-priced model
  • Expect the spend log to record no tier and bill fresh input tokens at the base rate
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/quota_management/spend_tracking/test_service_tier_pricing_e2e.py::TestServiceTierPricing::test_messages_stream_records_the_served_tier - a streamed /v1/messages call bills at the rate matching the tier it recorded

  • POST /v1/messages with "stream": true against the same custom-priced model
  • Expect the recorded tier to be none or priority, and fresh input tokens billed at that tier's rate
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/ui/tests/logs/logs.spec.ts "a served request's Logs row and drawer show its x-litellm-call-id" - the Logs page finds a request by call id and shows it in the row tooltip

  • Send one /v1/chat/completions request and note its x-litellm-call-id response header
  • Open http://localhost:4000/ui/?page=logs, paste the call id in the search box, and wait for the table to refresh
  • Hover the row's Request ID cell and expect a tooltip reading x-litellm-call-id: <call id>
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/llm_translation/test_together_ai_e2e.py::test_response_format_json_schema_shapes_the_reply - a json_schema response format on Together returns JSON matching the schema

  • POST /v1/chat/completions with model: together_ai/Qwen/Qwen3.5-9B, reasoning_effort: "none", and a json_schema response format (needs TOGETHERAI_API_KEY)
  • Expect the reply content to parse as JSON with the schema's required fields
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

• tests/e2e/ui/tests/tagManagement/tagManagement.spec.ts "creates, edits, reopens, and reads back a tag" - a tag created and edited on the Tag Management page reads back with the edited description

  • Open http://localhost:4000/ui/tag-management/ and click "+ Create New Tag"
  • In the dialog, fill Tag Name and Description, click "Create Tag", and expect the tag in GET /tag/list
  • Open the tag, click "Edit Tag", change Description, and expect POST /tag/update to carry the new description
  • Expect POST /tag/info to return the updated description
  • Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

[greptile-apps]

Confidence Score: 5/5

[Low risk] Updates test fixture data for logging tests.

The PR appears safe to merge.

Summary

Updates the GCS pub/sub spend-log golden fixture to include used_client_oauth_token: null, matching the metadata produced for the existing non-OAuth test case.

_{Reviews (1) · Last reviewed commit: "test(ci): add used_client_oauth_token to..."}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_ci_stale_tests_1001 (8325201) with main (d9f7324)¹}

Footnotes

1. No successful run was found on main (91ff045) during the generation of this report, so d9f7324 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩