Skip to content

test: unstick the suites CircleCI is failing on - #37059

Merged
yuneng-berri merged 3 commits into
litellm_internal_stagingfrom
litellm_/circleci-pipeline-triage-9b92e5
Aug 16, 2026
Merged

test: unstick the suites CircleCI is failing on#37059
yuneng-berri merged 3 commits into
litellm_internal_stagingfrom
litellm_/circleci-pipeline-triage-9b92e5

Conversation

@yuneng-berri

@yuneng-berri yuneng-berri commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • Three CI suites assert against code that has since changed
  • UI e2e specs target antd classes the dashboard no longer renders
  • Every one is red on staging today, hiding real failures

How it solves it:

  • Re-point the stale assertions at current behavior
  • Address migrated controls by role instead of antd class
  • Drop the AssemblyAI EU case whose credential no longer resolves

User Flow

No user-facing change: this PR only touches tests.

Relevant issues

Linear ticket

Pre-Submission checklist

  • I have added meaningful tests
  • My PR passes all CI/CD checks (e.g., lint, format, unit tests)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Not applicable: no product code changes, so the pipeline on this branch is the proof.

Type

✅ Test

Caveats (if any)

QA runbook

  • tests/e2e/ui/tests/playground/playground.spec.ts - the playground still chats and still switches key source after the dashboard's move off antd

    • Open http://localhost:4000/ui/playground and confirm "Virtual Key Source" reads "Current UI Session"
    • Pick a model from the "Select a Model" box, type a prompt, press the send arrow, and expect the reply to stream in
    • Switch "Virtual Key Source" to "Virtual Key", paste a key from POST /key/generate, send again, and expect the same reply
    • Sanity check: this test makes sense to add and is not hand-wavey or potentially flaky
  • tests/e2e/ui/tests/logs/logs.spec.ts - the log detail drawer still opens, copies, and switches to the JSON view

    • Send a chat completion, then open http://localhost:4000/ui/?page=logs and click its row
    • Expect a drawer showing "Request & Response", and confirm the copy buttons put the prompt and response on the clipboard
    • Click the "JSON" toggle in the drawer and expect the Request and Response tabs to render raw JSON
    • Sanity check: this test makes sense to add and is not hand-wavey or potentially flaky
  • tests/e2e/ui/tests/modelHub/modelHub.spec.ts - the make-models-public modal still completes its two steps

    • Open http://localhost:4000/ui/?page=model-hub-table and click "Select Models to Make Public"
    • In the dialog, click "Select All", then "Next", and expect "Confirm Making Models Public"
    • Click "Make Public" and expect a toast saying the model groups were made public
    • Sanity check: this test makes sense to add and is not hand-wavey or potentially flaky

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

…td classes

The playground, logs drawer and AI Hub modal moved off antd, so the specs
that reached for .ant-select, .ant-drawer-content, .ant-modal and
.ant-radio-button-wrapper no longer match anything and time out.

Address the same controls through their accessible role and name instead,
which holds across the component library swap and reads closer to what a
user does.
The google ai studio responses test still asserted tools == [], but the
transformation now pops empty tools and tool_choice before calling
completion, so assert the keys are absent.

test_openai_endpoints pinned claude-3-sonnet-20240229, which Bedrock has
retired; move it to us.anthropic.claude-sonnet-4-5-20250929-v1:0.

The AssemblyAI EU passthrough test depended on a credential that no longer
resolves in CI, and the US path plus the bad-key case already cover the
route; drop it rather than keep a permanently red case.
@yuneng-berri
yuneng-berri force-pushed the litellm_/circleci-pipeline-triage-9b92e5 branch from 232a317 to 6de68f1 Compare August 15, 2026 22:37
@greptile-apps

greptile-apps Bot commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR prevents client Accept-Encoding values from being forwarded to passthrough providers and refreshes several tests to match current API and dashboard behavior.

  • Protects accept-encoding in both ordinary forwarding and x-pass- header handling.
  • Adds focused passthrough-header regression tests.
  • Replaces stale UI implementation-class selectors with accessible locators.
  • Updates provider assertions and integration-test fixtures, while removing successful EU AssemblyAI coverage.

Confidence Score: 4/5

The production fix appears safe to merge, but the removed successful EU AssemblyAI scenario should be replaced with equivalent coverage to preserve regression detection.

Client compression negotiation is filtered on the passthrough path and covered by focused tests; the remaining concern is non-blocking test-coverage loss for a still-supported EU route.

Files Needing Attention: tests/store_model_in_db_tests/test_adding_passthrough_model.py

Important Files Changed

Filename Overview
litellm/passthrough/utils.py Removes normalized client accept-encoding from forwarded headers and blocks reintroduction through the x-pass- mechanism.
tests/test_litellm/passthrough/test_passthrough_utils.py Adds focused tests for ordinary and prefixed Accept-Encoding filtering without making network calls.
tests/e2e/ui/helpers/playground.ts Migrates playground helpers from antd implementation classes to current accessible labels, roles, and placeholders.
tests/e2e/ui/tests/logs/logs.spec.ts Updates drawer and JSON-view selectors to use dialog and tab semantics.
tests/e2e/ui/tests/modelHub/modelHub.spec.ts Addresses the make-models-public modal by its accessible dialog name.
tests/store_model_in_db_tests/test_adding_passthrough_model.py Removes the only successful end-to-end EU AssemblyAI passthrough scenario while retaining only unauthorized EU-route coverage.
tests/llm_responses_api_testing/test_google_ai_studio_responses_api.py Updates the mock assertion to require converted web-search fields to omit tools and tool_choice.
tests/test_openai_endpoints.py Refreshes the Bedrock Anthropic structured-output test model identifier.

Reviews (1): Last reviewed commit: 232a317 | Re-trigger Greptile

2. user points api base to <proxy-base>/asssemblyai/us
3. user points api base to <proxy-base>/assemblyai/eu
4. Bad API Key / credential - 401
3. Bad API Key / credential - 401

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Successful EU coverage removed

This removes the only successful end-to-end test of the supported /eu.assemblyai route; the remaining tests cover isolated credential matching or unauthorized requests, so CI no longer detects regressions in EU upstream or credential selection.

Rule Used: What: Flag any modifications to existing tests and... (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

@yuneng-berri yuneng-berri changed the title fix(passthrough): stop forwarding the client's accept-encoding upstream test: unstick the suites CircleCI is failing on Aug 15, 2026
@codecov

codecov Bot commented Aug 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@codspeed-hq

codspeed-hq Bot commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing litellm_/circleci-pipeline-triage-9b92e5 (232a317) with litellm_internal_staging (abddd64)1

Open in CodSpeed

Footnotes

  1. No successful run was found on litellm_internal_staging (d7d10be) during the generation of this report, so abddd64 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

@yuneng-berri
yuneng-berri merged commit 1968562 into litellm_internal_staging Aug 16, 2026
115 of 116 checks passed
@yuneng-berri
yuneng-berri deleted the litellm_/circleci-pipeline-triage-9b92e5 branch August 16, 2026 00:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants