Skip to content

test(ui): derive reasoning-effort assertion from the anthropic preset - #40456

Merged
tin-berri merged 1 commit into
litellm_internal_stagingfrom
litellm_fix_add_auto_router_tab_reasoning_test
Sep 9, 2026
Merged

tin-berri merged 1 commit into
litellm_internal_stagingfrom
litellm_fix_add_auto_router_tab_reasoning_test

Conversation

@cursor

@cursor cursor Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

How it solves it:

  • Match sibling tests: derive the expected tier_model_configs from ANTHROPIC_PRESET
  • Future preset refreshes flow through instead of redding this suite

User Flow

Test-only change, no user-visible behaviour changes

Relevant issues

Linear ticket

Pre-Submission checklist

  • I have added meaningful tests
  • The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
  • My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

UI-test-only fix, so the proof is the same vitest file CI runs, on the same head as this PR

Before (f529d6d, the staging tip that first failed)

  1. Run npx vitest run src/components/add_model/add_auto_router_tab.test.tsx --pool forks under ui/litellm-dashboard
  2. See FAIL src/components/add_model/add_auto_router_tab.test.tsx > AddAutoRouterTab > template presets > carries a preset's per-tier reasoning effort through to the create payload, AssertionError: expected { …(5) } to match object { complexity_router_config: { …(1) } }, diff shows - "model_name": "claude-opus-5" vs + "model_name": "claude-fable-5-1"
  3. Full CI run: https://github.com/BerriAI/litellm/actions/runs/34404056649

After (da4d010)

  1. Same command on this branch
  2. Output
 Test Files  1 passed (1)
      Tests  68 passed (68)
Type Errors  no errors

Type

✅ Test

Caveats (if any)

Low

  • Assertion now trusts hydratePresets and the JSON preset catalog; a bug there would silently satisfy this one test, but the dedicated preset tests in src/lib/autorouter_presets.test.ts still guard that surface
Open in Web View Automation 

Preset #40341 pointed the Anthropic family REASONING tier at claude-fable-5-1,
but this test still hardcoded claude-opus-5, so the payload it saw no longer
matched. Rebase the assertion on ANTHROPIC_PRESET.complexity_router_config.tier_model_configs
so a preset refresh flows through instead of redding the suite on staging.

Co-authored-by: Krrish Dholakia <krrish-berri-2@users.noreply.github.com>
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@tin-berri
tin-berri marked this pull request as ready for review September 9, 2026 21:09
@tin-berri
tin-berri requested a review from a team September 9, 2026 21:09
@tin-berri
tin-berri enabled auto-merge (squash) September 9, 2026 21:09
@greptile-apps

greptile-apps Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This test-only change updates the auto-router assertion to follow the current Anthropic preset

  • Removes the stale hardcoded model name
  • Compares the submitted tier configuration with the bundled preset
  • Weakens the independent regression check for the required reasoning effort

Confidence Score: 4/5

The production code is unaffected, but the test should independently retain its reasoning-effort assertion before merging

The dynamic model name fixes preset-refresh failures, while deriving the entire expected configuration allows an accidental preset removal of reasoning effort to pass

Files Needing Attention: ui/litellm-dashboard/src/components/add_model/add_auto_router_tab.test.tsx

Important Files Changed

Filename Overview
ui/litellm-dashboard/src/components/add_model/add_auto_router_tab.test.tsx Makes the expected model preset-driven, but also derives the behavior under test from the input fixture

Reviews (1): Last reviewed commit: "test(ui): derive reasoning-effort assert..." | Re-trigger Greptile

tier_model_configs: {
REASONING: [{ model_name: "claude-opus-5", litellm_params: { reasoning_effort: "high" } }],
},
tier_model_configs: ANTHROPIC_PRESET.complexity_router_config.tier_model_configs,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Assertion tracks fixture

This derives expected reasoning effort from the input fixture, weakening regression coverage. Repository rules require modified tests to preserve coverage before merge

Rule Used: What: Flag any modifications to existing tests and... (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

@tin-berri
tin-berri merged commit 2c836f4 into litellm_internal_staging Sep 9, 2026
77 of 78 checks passed
@tin-berri
tin-berri deleted the litellm_fix_add_auto_router_tab_reasoning_test branch September 9, 2026 21:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants