Skip to content

test(e2e): source the cheap Anthropic model from e2e_config across suites - #32964

Open
mateo-berri wants to merge 4 commits into
litellm_internal_stagingfrom
litellm_e2e_haiku_model_refs
Open

test(e2e): source the cheap Anthropic model from e2e_config across suites#32964
mateo-berri wants to merge 4 commits into
litellm_internal_stagingfrom
litellm_e2e_haiku_model_refs

Conversation

@mateo-berri

@mateo-berri mateo-berri commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • My PR passes all CI/CD checks (e.g., lint, format, unit tests)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

This PR is stacked on #32914, which introduces the CHEAP_ANTHROPIC_MODEL constant in tests/e2e/e2e_config.py; merge that first

This is a pure test-code refactor with no proxy behavior change, so the proof is the touched suites still passing against a live proxy plus the type and registry gates, all captured at commit e44aa07 against the tests/e2e docker compose stack (ghcr.io/berriai/litellm:main-latest with the suite's inline config, mapped to host port 4010 here because 4000 was taken by another local proxy) with real Anthropic and OpenAI traffic:

$ cd tests/e2e && LITELLM_PROXY_URL=http://localhost:4010 uv run pytest \
    llm_translation/test_chat_completions_regression_e2e.py llm_translation/test_passthrough_e2e.py \
    quota_management/budgets/test_soft_budget_e2e.py quota_management/spend_tracking/test_spend_tracking_e2e.py \
    -k "claude or anthropic_passthrough or soft_budget_does_not_block or spend_calculate_returns_nonzero" -v
llm_translation/test_chat_completions_regression_e2e.py::TestChatCompletionsRegression::test_chat_returns_real_completion[claude-haiku-4-5-anthropic] PASSED
llm_translation/test_passthrough_e2e.py::test_anthropic_passthrough_nonstreaming_logs_cost PASSED
llm_translation/test_passthrough_e2e.py::test_anthropic_passthrough_streaming_logs_cost PASSED
llm_translation/test_passthrough_e2e.py::test_anthropic_passthrough_tool_call_logs_cost PASSED
quota_management/budgets/test_soft_budget_e2e.py::test_soft_budget_does_not_block PASSED
quota_management/spend_tracking/test_spend_tracking_e2e.py::test_spend_calculate_returns_nonzero_cost PASSED
====================== 6 passed, 18 deselected in 40.04s =======================

$ uv run --no-sync basedpyright tests/e2e
0 errors, 0 warnings, 0 notes

$ cd tests/e2e && uv run python -m coverage_registry.collector --strict
Headline coverage: 53/313  (16.9%)

The --strict collector run performs a full pytest collect-only pass over tests/e2e, so it proves every touched module still imports and every covers marker still resolves to a registry cell

Type

🧹 Refactoring

Changes

Follow-up to #32914, which added CHEAP_ANTHROPIC_MODEL to tests/e2e/e2e_config.py (env E2E_CHEAP_ANTHROPIC_MODEL, default claude-haiku-4-5) after review feedback that a hardcoded model name is hard to maintain, but only used it in the new ratelimit suite; the older suites still hardcode the alias, so retiring or swapping the cheap Anthropic model would mean touching dozens of call sites

This replaces the 22 remaining hardcoded "claude-haiku-4-5" proxy-alias literals across 16 files with the shared constant, each with the matching e2e_config import: 14 occurrences across 12 files in quota_management/budgets/, the DRIVER_MODELS proxy alias in quota_management/spend_tracking/conftest.py plus a call and two row-matching assertions in test_spend_tracking_e2e.py, the 3 Anthropic passthrough calls in llm_translation/test_passthrough_e2e.py, and the claude row of CHAT_MODELS in llm_translation/test_chat_completions_regression_e2e.py. Every occurrence was a standalone literal, so no f-strings were needed and no behavior changes

Following review feedback (commit 65e90c2, gates rerun clean: basedpyright 0 errors, collector --strict passes, the spend_calculate test passes live), the spend_tracking DRIVER_MODELS row now derives its provider route as f"anthropic/{CHEAP_ANTHROPIC_MODEL}", so overriding the env var on a proxy where the alias is not pre-registered can no longer silently register the new alias against the old upstream model, and the passthrough module docstring references CHEAP_ANTHROPIC_MODEL instead of naming the model

Deliberately left alone: the upstream provider route id "anthropic/claude-haiku-4-5" in llm_translation/test_messages_e2e.py (that test registers its own e2e-messages-* alias, so its alias and route stay consistent regardless of the env var), dated bedrock model ids, and the docker-compose.yml inline config, which defines the alias the stack actually serves and matches the constant's default

@greptile-apps

greptile-apps Bot commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Pure test-code refactor that replaces 22 hardcoded "claude-haiku-4-5" proxy-alias literals across 16 e2e test files with the CHEAP_ANTHROPIC_MODEL constant imported from e2e_config.py (env-backed via E2E_CHEAP_ANTHROPIC_MODEL, default "claude-haiku-4-5"). No proxy behaviour changes; all assertions and model registration logic remain functionally identical to the previous state.

  • All 16 changed files receive a matching from e2e_config import CHEAP_ANTHROPIC_MODEL import and substitute every standalone literal occurrence with the constant.
  • quota_management/spend_tracking/conftest.py derives the provider route dynamically as f"anthropic/{CHEAP_ANTHROPIC_MODEL}", so overriding the env var no longer silently registers the new alias against a mismatched upstream model.
  • Row-matching predicates in test_spend_tracking_e2e.py use CHEAP_ANTHROPIC_MODEL in m, which is correct because the logged proxy alias will equal the constant value at runtime.

Confidence Score: 5/5

Safe to merge — test-only refactor with no proxy code touched and verified clean against a live stack.

Every change is a mechanical substitution of a string literal with a shared constant; no test logic, assertions, or fixtures were weakened. The provider-route derivation in conftest.py was tightened to use the same constant, which is a strict improvement. All touched modules were verified with basedpyright (0 errors) and the coverage registry collector, and the affected test functions were run against a real Anthropic-backed proxy.

No files require special attention.

Important Files Changed

Filename Overview
tests/e2e/e2e_config.py Defines CHEAP_ANTHROPIC_MODEL constant (env-backed, default "claude-haiku-4-5") that all changed files now import — the source of truth for this refactor
tests/e2e/quota_management/spend_tracking/conftest.py DRIVER_MODELS now derives the provider route dynamically as f"anthropic/{CHEAP_ANTHROPIC_MODEL}", addressing the previous review concern about alias/route mismatch
tests/e2e/quota_management/spend_tracking/test_spend_tracking_e2e.py Row-matching assertions updated to use CHEAP_ANTHROPIC_MODEL; substring matching is correct since the logged proxy alias will equal the constant value
tests/e2e/llm_translation/test_passthrough_e2e.py Three call sites updated to CHEAP_ANTHROPIC_MODEL; docstring updated from the literal model name to the constant name
tests/e2e/llm_translation/test_chat_completions_regression_e2e.py CHAT_MODELS tuple anthropic entry updated to CHEAP_ANTHROPIC_MODEL — mechanical substitution, no logic change

Reviews (2): Last reviewed commit: "fix(e2e): derive the anthropic driver ro..." | Re-trigger Greptile

Comment thread tests/e2e/quota_management/spend_tracking/conftest.py Outdated
@codecov

codecov Bot commented Jul 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@mateo-berri

Copy link
Copy Markdown
Contributor Author

@greptileai

Base automatically changed from litellm_e2e_key_rate_limit_coverage to litellm_internal_staging July 14, 2026 21:12
@mateo-berri
mateo-berri enabled auto-merge July 14, 2026 21:47
@mateo-berri
mateo-berri disabled auto-merge July 16, 2026 18:40
@codspeed-hq

codspeed-hq Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing litellm_e2e_haiku_model_refs (318c5c0) with litellm_internal_staging (582907d)

Open in CodSpeed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants