test(integration): exact four-part translation cases on a shared fake provider and shared YAML deployment by devin-ai-integration[bot] · Pull Request #44451 · BerriAI/litellm

devin-ai-integration · 2026-10-03T22:51:27Z

TLDR

Problem this solves:

Translation regressions slip through because tests check only a few fields
Each wire test builds its own deployment and fake server

How it solves it:

TranslationTestCase holds client request, provider request, provider reply, client response
The runner compares the provider request body, client response body and non-transport provider headers in full
Shared deployments for anthropic/claude-opus-5-5 and anthropic/claude-sonnet-4-6 in tests/integration/proxy_config.yaml, pointing at 127.0.0.1:8191
One shared fake provider fixture on :8191, checked between tests for stray or unused calls
First cases: Anthropic /v1/messages basic on Opus 5.5 and Sonnet 4.6, and the LIT-4957 thinking budget on Sonnet 4.6

User Flow

This PR adds tests only, so there is no end-user change. The flow below is a developer adding a translation case

Before: a developer covering a translation bug writes a new fake server, a new deployment and hand-picked field asserts

They start a wire_server on a random port and register a deployment for it with POST /model/new under a random name
They send the client request and assert two or three provider fields, such as thinking
A field that LiteLLM adds, drops or renames elsewhere in the body goes unnoticed

After: the developer writes one literal and the whole round trip is pinned

They add a case under tests/integration/translation/<endpoint>/<feature>/ with dataclasses.replace(BASE, ...) listing only the fields that change
The client request names the shared deployment, for example "model": "anthropic/claude-sonnet-4-6"
The runner sends it through the proxy and diffs the whole provider body and the whole client body against the literals
Any added, removed, renamed or moved field fails the test with a full dict diff

Linear ticket

Covers LIT-4957 (the existing LIT-4957 wire test stays)

Pre-Submission checklist

I have added meaningful tests
The handful of test files covering my change pass locally
My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Run against a local proxy started from tests/integration/proxy_config.yaml with Postgres and Redis, as the providers CircleCI job does

Before (`fd14e51`, with the LIT-4957 fix in `litellm/llms/anthropic/common_utils.py` reverted)

thinking budget on /v1/messages

pytest tests/integration/translation tests/integration/messages_endpoint/providers/anthropic/test_anthropic_legacy_thinking_budget_wire.py against that proxy
test_messages_basic_anthropic[anthropic/claude-sonnet-4-6-basic] passes and test_messages_reasoning_anthropic[anthropic/claude-sonnet-4-6-thinking_budget] fails: the provider body has 'thinking': {'type': 'adaptive'} and an extra 'output_config': {'effort': 'low'} instead of {'type': 'enabled', 'budget_tokens': 1024}

shared provider between tests

Not present: each wire test starts its own fake server on a random port

After (`502f51a`)

thinking budget on /v1/messages

Same command
Both cases pass, as does the existing messages_endpoint/providers/anthropic/test_anthropic_legacy_thinking_budget_wire.py

shared provider between tests

Shared-provider checks, each tried with a throwaway test: a queued reply nobody requested fails with 1 queued provider replies were never requested, an unread request fails with the test never read the provider requests ['/x'], and a request landing after a test ends fails the next test with the shared fake provider got ['/late'] after ...::test_c_late finished

Type

✅ Test

Caveats (if any)

Low

Captured Anthropic replies are verbatim, including the opaque thinking signature
Every group's proxy now loads the shared Anthropic deployments; existing tests filter models by their own names
The shared provider refuses pytest-xdist, so mcp and cost cannot use it
Moving the existing per-test /model/new deployments to the YAML is a follow-up PR

Note

Low Risk
Test-only changes with no production code paths; shared YAML deployments are additive and the provider fixture is serial-only.

Overview
Adds a translation/ integration shard that pins full proxy round-trips via literal TranslationTestCase records and a runner that diffs the entire provider body, non-transport provider headers, and LiteLLM response.

Shared fake provider: one in-process wire_server on :8191 with provider / shared_provider_server fixtures that queue replies with expect, drain with received, and fail on stray traffic, unread requests, or unused queued replies; it refuses pytest-xdist. proxy_config.yaml now declares shared Anthropic deployments (anthropic/claude-opus-5-5, anthropic/claude-sonnet-4-6) aimed at that URL. The providers group and manifest own the new directory.

Initial cases: Anthropic /v1/messages basic for both models plus a thinking budget variant on Sonnet 4.6 (LIT-4957). Docs (translation/README.md, integration README) describe case layout and capture rules.

^{Reviewed by Cursor Bugbot for commit 502f51a. Bugbot is set up for automated code reviews on this repo. Configure here.}

… provider and shared YAML deployment Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration · 2026-10-03T22:51:31Z

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

Disable automatic comment, CI, and merge conflict monitoring

CLAassistant · 2026-10-03T22:51:35Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

litellm-stampbot · 2026-10-03T22:52:01Z

@greptileai

litellm-stampbot · 2026-10-03T22:52:02Z

bugbot run

cursor

Stale Bugbot comment from a previous run.

greptile-apps · 2026-10-03T22:59:27Z

Confidence Score: 5/5

[Medium risk] Adds integration test framework for provider request translation.

The PR appears safe to merge based on the reviewed changes and previous threads.

Findings

Inline fallback sentinel ▶

Summary

The PR adds exact request-and-response translation tests for Anthropic Messages using shared YAML deployments and a shared fake provider.

The changes since the previous review move translation-specific contributor guidance into the translation directory without changing its requirements.
No new actionable issue was identified.

_{Reviews (7) · Last reviewed commit: "docs(integration): move translation test..."}

codecov · 2026-10-03T23:05:55Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

…eck for late provider requests at session end Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

litellm-stampbot · 2026-10-03T23:26:28Z

@greptileai

litellm-stampbot · 2026-10-03T23:26:29Z

bugbot run

cursor

Stale Bugbot comment from a previous run.

…provider sides and drop regressions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

litellm-stampbot · 2026-10-03T23:34:25Z

@greptileai

litellm-stampbot · 2026-10-03T23:34:26Z

bugbot run

cursor

Stale Bugbot comment from a previous run.

…ected_ and name the fake reply mock_provider_response Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

… README Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

litellm-stampbot · 2026-10-03T23:36:58Z

@greptileai

litellm-stampbot · 2026-10-03T23:36:58Z

bugbot run

cursor

Stale Bugbot comment from a previous run.

greptile-apps · 2026-10-03T23:39:11Z

+
+PROVIDER_PORT: Final = 8191
+PROVIDER_URL: Final = f"http://127.0.0.1:{PROVIDER_PORT}"
+_UNQUEUED: Final = Reply(status=500, body=b'{"error": "the shared fake provider has no reply queued for this request"}')


Inline fallback sentinel

_UNQUEUED is defined in this module and returned when no provider reply is queued. The repository requires sentinel-like variables to be defined in constants.py instead. Please satisfy that requirement before merging.

Rule Used: What: Require sentinel-like variables (e.g., _NEGATIVE_TEAM_SENTINEL) to be defined in constants.py instead of inline or in other modules. Why: Centralizing shared constants prevents duplication, inconsistencies, and makes future changes safer ... (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

This is a test-only fake reply, so litellm/constants.py would ship it in the package; tests/integration/_support keeps private constants local, like _NO_MARKER

You're right. _UNQUEUED is private state for the integration-only fake provider, and moving it into litellm/constants.py would unnecessarily ship a test sentinel with the package. The existing _NO_MARKER pattern in tests/integration/_support supports keeping these test-only constants local, so this comment can be disregarded.

…t to claude-sonnet-4-6 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

litellm-stampbot · 2026-10-03T23:45:09Z

@greptileai

litellm-stampbot · 2026-10-03T23:45:10Z

bugbot run

cursor

Stale Bugbot comment from a previous run.

…ent the naming rule Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

litellm-stampbot · 2026-10-03T23:54:30Z

@greptileai

litellm-stampbot · 2026-10-03T23:54:31Z

bugbot run

cursor

Stale Bugbot comment from a previous run.

…/translation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

litellm-stampbot · 2026-10-04T00:01:20Z

@greptileai

litellm-stampbot · 2026-10-04T00:01:21Z

bugbot run

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 502f51a. Configure here.}

test(integration): exact four-part translation cases on a shared fake…

12a4817

… provider and shared YAML deployment Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration Bot requested a review from a team October 3, 2026 22:51

devin-ai-integration Bot assigned litellm-stampbot Oct 3, 2026

cursor Bot reviewed Oct 3, 2026

View reviewed changes

greptile-apps Bot reviewed Oct 3, 2026

View reviewed changes

Comment thread tests/integration/translation/runner.py Outdated

Comment thread tests/integration/conftest.py

test(integration): compare every non-transport provider header and ch…

ae6ed76

…eck for late provider requests at session end Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

cursor Bot reviewed Oct 3, 2026

View reviewed changes

test(integration): name TranslationTestCase fields after litellm and …

b1c46c2

…provider sides and drop regressions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

cursor Bot reviewed Oct 3, 2026

View reviewed changes

litellm-stampbot and others added 2 commits October 3, 2026 23:36

test(integration): prefix checked TranslationTestCase fields with exp…

6e866cc

…ected_ and name the fake reply mock_provider_response Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

docs(integration): name TranslationTestCase fields in the translation…

e323739

… README Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

cursor Bot reviewed Oct 3, 2026

View reviewed changes

greptile-apps Bot reviewed Oct 3, 2026

View reviewed changes

test(integration): add a claude-opus-5-5 base case and deployment nex…

fffdcb1

…t to claude-sonnet-4-6 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

cursor Bot reviewed Oct 3, 2026

View reviewed changes

test(integration): name translation cases <MODEL>_TEST_CASE and docum…

9804606

…ent the naming rule Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

cursor Bot reviewed Oct 3, 2026

View reviewed changes

docs(integration): move translation test rules into tests/integration…

502f51a

…/translation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

cursor Bot reviewed Oct 4, 2026

View reviewed changes

litellm-stampbot approved these changes Oct 4, 2026

View reviewed changes

litellm-stampbot enabled auto-merge (squash) October 4, 2026 00:03

litellm-stampbot merged commit f850b2c into main Oct 4, 2026
91 of 92 checks passed

litellm-stampbot deleted the litellm_translation_integ_example branch October 4, 2026 00:10

devin-ai-integration Bot mentioned this pull request Oct 5, 2026

fix(caching): key response cache by router model group in litellm_metadata #44542

Merged

5 of 6 tasks

litellm-stampbot mentioned this pull request Oct 5, 2026

test(integration): anthropic-route basic translation cases for six Claude models on messages, chat completions and responses #44607

Merged

4 of 6 tasks

Conversation

devin-ai-integration Bot commented Oct 3, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

User Flow

Linear ticket

Pre-Submission checklist

Delays in PR merge?

Screenshots / Proof of Fix

Before (fd14e51, with the LIT-4957 fix in litellm/llms/anthropic/common_utils.py reverted)

thinking budget on /v1/messages

shared provider between tests

After (502f51a)

thinking budget on /v1/messages

shared provider between tests

Type

Caveats (if any)

Low

Uh oh!

devin-ai-integration Bot commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CLAassistant commented Oct 3, 2026

Uh oh!

litellm-stampbot commented Oct 3, 2026

Uh oh!

litellm-stampbot commented Oct 3, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot commented Oct 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Findings

Summary

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Oct 3, 2026

Codecov Report

Uh oh!

litellm-stampbot commented Oct 3, 2026

Uh oh!

litellm-stampbot commented Oct 3, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

litellm-stampbot commented Oct 3, 2026

Uh oh!

litellm-stampbot commented Oct 3, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

litellm-stampbot commented Oct 3, 2026

Uh oh!

litellm-stampbot commented Oct 3, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 3, 2026

Choose a reason for hiding this comment

Uh oh!

litellm-stampbot Oct 3, 2026

Choose a reason for hiding this comment

Uh oh!

greptile-apps Bot Oct 3, 2026

Choose a reason for hiding this comment

Uh oh!

litellm-stampbot commented Oct 3, 2026

Uh oh!

litellm-stampbot commented Oct 3, 2026

Uh oh!

devin-ai-integration Bot commented Oct 3, 2026 •

edited by cursor Bot

Loading

Before (`fd14e51`, with the LIT-4957 fix in `litellm/llms/anthropic/common_utils.py` reverted)

After (`502f51a`)

devin-ai-integration Bot commented Oct 3, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

greptile-apps Bot commented Oct 3, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading

cursor Bot left a comment •

edited

Loading