Skip to content

test(e2e): drop the auto-router select "opens below" spec - #41568

Merged
yuneng-berri merged 1 commit into
mainfrom
litellm_/ci-test-failures-investigation-99785b
Sep 17, 2026
Merged

yuneng-berri merged 1 commit into
mainfrom
litellm_/ci-test-failures-investigation-99785b

Conversation

@yuneng-berri

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

How it solves it:

User Flow

Before: a contributor opens any PR and CircleCI e2e_ui_testing fails on a spec unrelated to their change

  1. They push a branch and add the run-ci label
  2. https://app.circleci.com/pipelines/github/BerriAI/litellm shows e2e_ui_testing failed after 3 attempts of tests/modelsPage/autoRouterTemplateSelect.spec.ts "opens the options below the trigger when there is room below it", with Expected: < 450, Received: 487
  3. They open the Admin UI at http://localhost:4000/ui/?page=models, Auto-Routers, Add Auto Router, and click the Template select in a 900px tall window: the list opens above the field and the field stays visible, which is what a user expects

After: the same PR's e2e_ui_testing is green and the select still behaves the same way

  1. They push a branch and add the run-ci label
  2. e2e_ui_testing passes; autoRouterTemplateSelect.spec.ts now runs one test, "keeps the trigger uncovered when the options open with no room below it"
  3. The Template select behaves exactly as in step 3 above

Relevant issues

Affected release

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests (this PR removes one; no new tests)
  • The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
  • My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Both runs use the CircleCI harness locally: tests/e2e/ui/run_e2e.sh boots a fresh Postgres, builds the dashboard from source, starts the mock LLM server and the proxy on tests/e2e/ui/fixtures/config.yml, then runs Playwright with the CI settings (--workers=1 --retries=2)

Before (5ef40a6)

  1. bash tests/e2e/ui/run_e2e.sh --workers=1 --retries=2 --reporter=list
  2. Observed:
✘   89 [chromium] › tests/modelsPage/autoRouterTemplateSelect.spec.ts:49:7 › Auto Router template select anchoring › opens the options below the trigger when there is room below it (10.6s)
✘   90 [chromium] › tests/modelsPage/autoRouterTemplateSelect.spec.ts:49:7 › Auto Router template select anchoring › opens the options below the trigger when there is room below it (retry #1) (11.3s)
✘   91 [chromium] › tests/modelsPage/autoRouterTemplateSelect.spec.ts:49:7 › Auto Router template select anchoring › opens the options below the trigger when there is room below it (retry #2) (11.3s)
✓   92 [chromium] › tests/modelsPage/autoRouterTemplateSelect.spec.ts:62:7 › Auto Router template select anchoring › keeps the trigger uncovered when the options open with no room below it (1.8s)

    Error: expect(received).toBeLessThan(expected)

    Expected: < 450
    Received:   487
  1. Same result on CircleCI for main's scheduled run at 43713f7 and for test: fix seven tests left stale by #41311, #41337, #39996, #41310, #41289 and #41315 #41527's own run: https://app.circleci.com/workflow/3c3e854b-3fed-4249-b1e0-2064d055edcf/job/1c645a5e-0a32-4e6a-9426-dadfeb5a7dc9

After (25445e8)

  1. bash tests/e2e/ui/run_e2e.sh tests/modelsPage/autoRouterTemplateSelect.spec.ts --reporter=list
  2. Observed:
✓  1 [chromium] › tests/modelsPage/autoRouterTemplateSelect.spec.ts:42:7 › Auto Router template select anchoring › keeps the trigger uncovered when the options open with no room below it (1.4s)

  1 passed (5.3s)

Type

✅ Test

Caveats (if any)

Low

  • Which side the template list opens on is no longer asserted anywhere; Base UI picks it from the space available

QA runbook

  • tests/e2e/ui/tests/modelsPage/autoRouterTemplateSelect.spec.ts::Auto Router template select anchoring::opens the options below the trigger when there is room below it (removed) - it asserted the list opens under the field whenever the field sits in the upper half of a 900px window, which stopped being true once the form grew
    • Open http://localhost:4000/ui/?page=models in a 1280x900 window, click Auto-Routers, then Add Auto Router
    • Click the Template select
    • Expect the list to open above the field and end just above it, with the field still fully visible; that is the behaviour the surviving "keeps the trigger uncovered" test pins at a 560px window
    • Sanity check: removing this test loses no guarantee our code makes; the side the list opens on is decided by the select library from the space left in the window

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

The spec pinned Base UI's collision behaviour, not our code: it only passes
while the template popup happens to fit under the trigger at 1280x900, and
#41315's taller Add Auto Router form broke that premise for the second time
in three weeks. #41527 tried to scroll the trigger into the upper half, but
the dialog content is shorter than its max height, so nothing scrolls and CI
still fails 3/3 with the trigger at y=487

The guarantee #38554 introduced is that the popup never covers the trigger,
and the sibling spec keeps asserting that at a viewport with no room below
@yuneng-berri
yuneng-berri requested a review from a team September 17, 2026 05:45
@greptile-apps

greptile-apps Bot commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

The PR appears safe to merge because it removes an assertion against library-controlled collision placement without removing coverage of the intended user-facing behavior.

Summary

This PR removes a flaky end-to-end assertion that depended on the Auto Router template popup fitting below its trigger.

  • Deletes the geometry-specific placement test and its now-unused polling helper.
  • Retains coverage ensuring the options popup does not cover the trigger when available space causes collision handling to reposition it.

Reviews (1) · Last reviewed commit: "test(e2e): drop the auto-router select "..."

@yuneng-berri
yuneng-berri merged commit 5c48612 into main Sep 17, 2026
133 of 134 checks passed
@yuneng-berri
yuneng-berri deleted the litellm_/ci-test-failures-investigation-99785b branch September 17, 2026 06:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants