Skip to content

fix(ui): read the preset catalog at runtime in the dashboard tests - #39478

Merged
mateo-berri merged 3 commits into
litellm_internal_stagingfrom
litellm_ui_presets_mock_build_ctx
Sep 3, 2026
Merged

fix(ui): read the preset catalog at runtime in the dashboard tests#39478
mateo-berri merged 3 commits into
litellm_internal_stagingfrom
litellm_ui_presets_mock_build_ctx

Conversation

@tin-berri

@tin-berri tin-berri commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • The UI container image no longer builds from source
  • Two dashboard tests import a catalog outside the image's build context
  • Image Scan is red on staging and on unrelated PRs

How it solves it:

  • Both tests read the catalog from disk at test time
  • No module import crosses out of ui/litellm-dashboard/ any more
  • Test expectations still come from the real shipped catalog

User Flow

Before: an operator building the UI container from source never gets an image, because the build stops on a type error

  1. They check out litellm_internal_staging and run docker build -f ui/Dockerfile -t litellm-ui . from the repo root
  2. The build installs the dashboard dependencies, then during npm run build prints ✓ Compiled successfully followed by Running TypeScript ...
  3. Half a minute later it prints Failed to type check. and Type error: Cannot find module '../../../../litellm/proxy/public_endpoints/autorouter_presets.json' or its corresponding type declarations.
  4. docker build exits 1 with process "/bin/sh -c npm run build" did not complete successfully, and no litellm-ui image is tagged
  5. With no image to run, http://localhost:3000/ui/ refuses the connection

After: the same build finishes and the container serves the Admin UI

  1. They check out this branch and run the same docker build -f ui/Dockerfile -t litellm-ui . from the repo root
  2. The build installs the dashboard dependencies, then during npm run build prints ✓ Compiled successfully followed by Running TypeScript ...
  3. The type check passes and the build writes the static export
  4. docker build exits 0 and tags litellm-ui
  5. They run docker run -p 3000:3000 litellm-ui and http://localhost:3000/ui/ serves the Admin UI

Relevant issues

Linear ticket

Resolves LIT-6783

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
  • My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

The failing job is ui-image in the Image Scan workflow, which runs the same docker build -f ui/Dockerfile . an operator would. That workflow is path-filtered, so it does not run on this PR automatically; both runs below were started with gh workflow run image-scan.yml --ref <ref> on the exact commits named.

Before (ff17e8b, litellm_internal_staging)

  1. gh workflow run image-scan.yml --repo BerriAI/litellm --ref litellm_internal_staging started run 33715609093
  2. Three of the six jobs failed, and they are exactly the three that build the dashboard: ui-image, runtime-image and image-scan. migrations-image, gateway-image and backend-image passed
  3. The ui-image log shows the build dying in the type check:
#18 19.61 ✓ Compiled successfully in 18.5s
#18 19.62   Running TypeScript ...
#18 55.74 Failed to type check.
#18 55.74 ./tests/mocks/autoRouterPresets.ts:2:28
#18 55.74 Type error: Cannot find module '../../../../litellm/proxy/public_endpoints/autorouter_presets.json' or its corresponding type declarations.
#18 55.86 Next.js build worker exited with code: 1 and signal: null
#18 ERROR: process "/bin/sh -c npm run build" did not complete successfully: exit code: 1

After (f62e87d)

  1. gh workflow run image-scan.yml --repo BerriAI/litellm --ref litellm_ui_presets_mock_build_ctx started run 33716064367
  2. All six jobs passed. The ui-image log now walks straight past the type check into the static export:
#18 20.61 ✓ Compiled successfully in 19.5s
#18 20.62   Running TypeScript ...
#18 58.59   Generating static pages using 3 workers (0/51) ...
#18 59.88 ✓ Generating static pages using 3 workers (51/51) in 1284ms
#18 59.88   Finalizing page optimization ...
#18 DONE 60.1s
  1. The job then ran the image as an arbitrary uid with a read-only root filesystem and got the UI back over HTTP:
tests/proxy_migration_tests/test_ui_image_serves_offline.py::test_ui_serves_as_arbitrary_uid_read_only PASSED [100%]
======================== 1 passed, 2 warnings in 2.75s =========================
  1. The dashboard suites that touch the catalog still pass locally:
$ npx vitest run src/components/add_model/add_auto_router_tab.test.tsx src/lib/autorouter_presets.test.ts
 Test Files  2 passed (2)
      Tests  143 passed (143)
 Type Errors  no errors
   Duration  23.56s

add_auto_router_tab.test.tsx asserts the exact preset labels the shipped catalog carries (["Anthropic Family", "Gemini Family", "Lite", "OpenAI Family", "Custom Configuration"]), so a green run is also evidence the tests still read the real catalog rather than a stale copy.

  • f62e87d passes /live-pr-risk
    • Two test files changed, no production symbol
    • The mock has exactly two consumers, both test files, both green above
    • All four of its existing exports are preserved and one is added
    • No module specifier anywhere under ui/litellm-dashboard/ still resolves outside the dashboard root
    • Production reaches the catalog over GET /public/autorouter_presets, which this diff does not touch

Type

🐛 Bug Fix

Caveats (if any)

Low

  • Nothing stops a future dashboard test from importing outside ui/litellm-dashboard/
  • autorouter_presets.test.ts is a plain unit test but now imports the vitest mock module for its data
    • That module calls vi.fn() at import time, so the unit test pulls in a test double it never uses
    • The clean shape is a third fixture module holding only the parsed catalog, imported by both
    • Not worth a third file and more churn on a hotfix unblocking a red staging fleet
  • next build drops type errors coming from *.test.ts and *.spec.ts files
    • So a bad import only reds the build once a non-test file has it
    • Here the mock under tests/mocks/ is the file that surfaced it
  • Image Scan is path-filtered and does not run on dashboard source changes
    • The nightly 41 6 * * * run catches a break within a day
    • Widening the filter touches .github/**, which needs two approvals, so it belongs in its own PR
    • Tracked as LIT-6805

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

The autoRouterPresets mock imported litellm/proxy/public_endpoints/autorouter_presets.json
as a module. That path sits outside ui/litellm-dashboard, the only directory the UI
Dockerfile copies, so `next build` type-checking inside the image failed with
"Cannot find module" and the ui-image job went red on every PR that touched an
image-scan path. Read the file with fs at runtime instead; vitest still derives
expectations from the real bundled catalog.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HqEPCNLDrssxsuezhAaL4j
@greptile-apps

greptile-apps Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR replaces a cross-package JSON module import with a test-runtime filesystem read anchored to the mock file, avoiding UI Docker type-check resolution failures while preserving catalog-derived expectations.

  • Exports the parsed catalog response and hydrated presets from the shared Vitest mock.
  • Updates the catalog tests to consume the raw response while independently exercising hydration.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the runtime path is independent of the caller's working directory, the mock is not evaluated by the production build, and the existing catalog assertions remain intact.

Important Files Changed

Filename Overview
ui/litellm-dashboard/tests/mocks/autoRouterPresets.ts Loads the real preset catalog at test runtime through a cwd-independent path; both previously reported concerns are resolved.
ui/litellm-dashboard/src/lib/autorouter_presets.test.ts Reuses the mock's raw parsed catalog without changing assertions or weakening independent hydration coverage.

Reviews (3): Last reviewed commit: "fix(ui): read the preset catalog through..." | Re-trigger Greptile

Comment thread ui/litellm-dashboard/tests/mocks/autoRouterPresets.ts Outdated
Comment thread ui/litellm-dashboard/tests/mocks/autoRouterPresets.ts Outdated
@tin-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

@tin-berri

Copy link
Copy Markdown
Contributor Author

@greptileai review

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit fcc9b81. Configure here.

@tin-berri
tin-berri enabled auto-merge (squash) September 3, 2026 03:07
@tin-berri
tin-berri disabled auto-merge September 3, 2026 03:11
@tin-berri
tin-berri enabled auto-merge (squash) September 3, 2026 03:12

@mateo-berri mateo-berri left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks!

One tiny non-blocking nit

// Derived from the real bundled catalog so a preset edit there flows into test expectations
// instead of redding on a stale copy. Exported as vi.fn so a test can override the query state.
export const BUNDLED_PRESETS = hydratePresets(bundledPresets as AutoRouterPresetsResponse);
const CATALOG_PATH = resolve(__dirname, "../../../../litellm/proxy/public_endpoints/autorouter_presets.json");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: hmm I never really liked this drilling. Maybe it could be absolute...?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed it's not pretty, but I'd keep it: anchor-on-own-file-and-walk-up is the existing pattern on both sides of the repo, and there is no "absolute" alternative here that isn't machine-specific.

  • TS: scripts/gen-api-types.mjs computes dashboardDir and repoRoot by walking up from import.meta.url with .. hops; vitest.config.ts anchors its alias with resolve(__dirname, "src").
  • Python: 37 tests use Path(__file__).resolve().parents[2], 18 more use Path(__file__).parents[2] / "model_prices_and_context_window.json", and a handful go parents[3..5]. No shared repo-root helper exists.
  • process.cwd() was the previous version of this line; Greptile flagged it (vitest from the repo root fails at import time), which is why fcc9b81 moved to __dirname.

The only dot-free option is a repo-root constant defined in vitest.config.ts plus a global .d.ts so next build still type-checks. Happy to add that once a second test needs a repo-root path, but for one consumer it's more machinery than the line it replaces.

@mateo-berri

Copy link
Copy Markdown
Contributor

bugbot run

@mateo-berri

Copy link
Copy Markdown
Contributor

@greptileai

@mateo-berri mateo-berri changed the title fix(ui): read the preset catalog at runtime in the vitest mock fix(ui): read the preset catalog at runtime in the dashboard tests Sep 3, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit f62e87d. Configure here.

@mateo-berri
mateo-berri merged commit 62ed7e1 into litellm_internal_staging Sep 3, 2026
133 checks passed
@mateo-berri
mateo-berri deleted the litellm_ui_presets_mock_build_ctx branch September 3, 2026 05:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants