feat: shadow eval samples /v1/messages and /v1/responses traffic

[tin-berri]

TLDR

Problem this solves:

• Shadow evals sample only /v1/chat/completions traffic; /v1/messages and /v1/responses requests are silently skipped by the call_type allowlist, so a key whose clients speak those surfaces (Claude Code and Desktop, Responses API apps) evaluates as zero judged turns
• Live-proven before this change: one request per surface on a running job left the attempt count unchanged, with spend rows logged as anthropic_messages and aresponses

How it solves it:

• A per-surface dispatch table (_SURFACE_OPS) replaces the allowlist frozenset and the inline chat-shape coercion: membership in the table is the allowlist, and each row says how that surface's logged request becomes a chat-shaped request, messages and generation params both, and how its response yields the judgeable final text

• Normalization reuses litellm's own owners end to end, no new format logic: LiteLLMAnthropicMessagesAdapter.translate_anthropic_to_openai for Anthropic requests (block messages, top-level system, and params: tools, tool_choice, thinking, stop_sequences, output_format translate, sampling params copy through), LiteLLMCompletionResponsesConfig.transform_responses_api_request_to_chat_completion_request for Responses requests (input plus instructions, max_output_tokens to max_tokens, Responses tools to chat tools, reasoning to reasoning_effort, text to response_format), and ResponsesAPIResponse.output_text for response text

• Param values come from the proxy's wire-body snapshot rather than the logged optional_params, because the logged params switch dialect per provider path: the openai-compatible /v1/messages bridge rides the Responses API and logs Responses-shaped params for an Anthropic request (live-probed), while the wire body is always what the client sent. Each surface filters the body to its own request schema, so surface-only keys like previous_response_id never reach the shadow call, and one shared strip removes model, messages, stream, stream_options, and metadata from every translated request before it forwards

• litellm's logging layer already normalizes the response half per surface (Messages responses arrive as chat-shaped ModelResponse on both provider paths, Responses requests keep their input in kwargs["messages"]), so each table row composes existing transformations only

• A uniform text-final gate skips turns whose real response carries tool calls (tool_calls/function_call on chat-shaped responses, function_call output items on Responses), formalizing what chat sampling already did implicitly; request-side tool history stays allowed on every surface, matching chat behavior today

• Rebased onto reverse-direction jobs (feat(shadow_eval): add reverse-direction shadow eval jobs #36865): per-job gates (direction, turn budget, sampling) run first and the request normalizes once, only when at least one job sampled it. Dict-shaped Responses payloads validate into ResponsesAPIResponse, so the derived output_text property applies to both payload shapes

• Requests a pre_call guardrail rewrote are skipped on the two wire-body-sourced surfaces, detected through the standard_logging_guardrail_information entries spend logging already records: the wire-body snapshot is taken before the guardrail pre-call hook, so replaying it would resurrect content the guardrail stripped or masked. Chat sampling is unaffected since it sources the dispatched call

User Flow

1. Admin starts a shadow eval on a key whose traffic arrives via /v1/messages (for example Claude Code pointed at the gateway) or /v1/responses
2. Sampled requests on all three surfaces produce judged attempt rows; tool-call turns are skipped as unjudgeable, text turns are normalized to chat shape, params included, and re-driven through the auto-router
3. Win rates stratify by tier and incumbent model exactly as before, now fed by every surface

Relevant issues

• Follow-up to feat: pre-adoption shadow eval for the auto-router (blind pairwise judge, derived state) #36587 (pre-adoption shadow eval), which shipped with the chat-only allowlist as a declared caveat

Linear ticket

Pre-Submission checklist

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review

Screenshots / Proof of Fix

Live proxy on this branch (port 4216, same rig and database as #36587's proof; stub upstream serving all three surfaces). Fresh job on the shadowed key, then one request per arm, each carrying surface-native params that would break or skew a chat call if forwarded raw:

$ curl -s localhost:4216/v1/messages -H "Authorization: Bearer $KEY" \
    -d '{"model":"gpt-5","max_tokens":180,"system":"you are terse","stop_sequences":["STOPWORD"],
         "tools":[{"name":"get_weather","description":"weather lookup","input_schema":{...}}],
         "messages":[{"role":"user","content":"params probe via messages bridge: what is 7+7?"}]}'
200   # openai-provider bridge path

$ curl -s localhost:4216/v1/messages -H "Authorization: Bearer $KEY" \
    -d '{"model":"claude-stub","max_tokens":150,"system":"you are terse","top_k":5,"stop_sequences":["STOPWORD"],
         "messages":[{"role":"user","content":[{"type":"text","text":"params probe via native messages: what is 6+6?"}]}]}'
200   # native anthropic-provider path

$ curl -s localhost:4216/v1/responses -H "Authorization: Bearer $KEY" \
    -d '{"model":"gpt-5","input":"params probe via responses: what is 8+8?","instructions":"you are terse",
         "max_output_tokens":222,"temperature":0.3,"previous_response_id":"resp_probe_0",
         "tools":[{"type":"function","name":"get_weather","description":"weather lookup","parameters":{...}}]}'
200

$ curl -s localhost:4216/v1/chat/completions -H "Authorization: Bearer $KEY" \
    -d '{"model":"gpt-5","temperature":0.4,"messages":[{"role":"user","content":"chat regression probe: what is 9+9?"}]}'
200

job detail: judged_count 4, error_count 0, last_error null
by_tier MEDIUM/COMPLEX/SIMPLE, by_current_model openai/gpt-5 and anthropic/claude-sonnet-5

A logging probe on the shadow calls confirms the translations (optional_params as the shadow acompletion saw them):

bridge messages arm  -> {"stop": ["STOPWORD"], "max_tokens": 180, "tools": [{"type": "function", "function": {...}}]}, system message first
native messages arm  -> {"stop": ["STOPWORD"], "max_tokens": 150, "extra_body": {"top_k": 5}}, system message first
responses arm        -> {"temperature": 0.3, "max_tokens": 222, "tools": [{"type": "function", "function": {...}}]},
                        system message first, no previous_response_id, no max_output_tokens, no instructions param
chat arm             -> {"temperature": 0.4}, unchanged

The same probe showed the bridge messages arm logs model_parameters as {"instructions": ..., "max_output_tokens": 180, "tools": [Responses-shaped]} for an Anthropic-shaped client request, which is why params source from the wire body and not the logged snapshot

Type

🆕 New Feature

Caveats (if any)

• Turns whose real response contains tool calls are skipped on every surface: a tool invocation has no text answer to judge blind. Text turns inside agentic conversations still sample, with the tool history normalized into the chat context
• The /anthropic/v1/messages passthrough route logs a different call_type with httpx payloads and stays excluded
• Streamed turns on the two new surfaces are not sampled yet (live-probed): litellm's logging hands the success hook an empty-content placeholder for streamed Messages turns and fires no sampled call_type for streamed Responses turns, so both skip fail-closed with zero cost. Streamed chat sampling is unchanged (live-verified judging a streamed chat turn). Closing the gap needs the assembled streamed text upstream and is follow-up scope
• Live proof used the stub upstream, same as feat: pre-adoption shadow eval for the auto-router (blind pairwise judge, derived state) #36587; the judge and shadow mechanics against live providers are covered by that PR's owed staging pass

QA runbook

1. Start a shadow eval on a key, then send traffic through /v1/messages (both an openai-backed and an anthropic-backed model) and /v1/responses, including surface-native params (stop_sequences, top_k, anthropic tools, max_output_tokens, Responses tools, previous_response_id)
2. Expect judged attempts for each surface on the job detail endpoint within seconds, stratified by tier and incumbent model, with zero error rows
3. Send a tool-use request on any surface; expect no attempt row for it
4. Confirm /v1/chat/completions sampling is unchanged, streamed and non-streamed
5. Send streamed requests on /v1/messages and /v1/responses; expect them skipped without error rows (declared caveat)

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

^{Cursor Bugbot is generating a summary for commit e413a2f. Configure here.}

[tin-berri]

@greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

Score: 4/5

What's done well:

The dispatch table design (_SURFACE_OPS) is clean — membership in the table is the allowlist, with each row encoding both the normalization and text-extraction logic for a surface. Reusing LiteLLMAnthropicMessagesAdapter and LiteLLMCompletionResponsesConfig means no new format logic, just composition of existing owners. The fail-closed default (unknown call type → skip) is correct. The uniform tool-call gate across surfaces is a necessary formalization of what chat sampling did implicitly. Test coverage is solid: each surface, the bridge path's system recovery, and both tool-call shapes.

Why not 5:

1. kwargs["messages"] alias for responses input is undocumented in code. The PR description explains it ("litellm's logging layer... keeps their input in kwargs["messages"]"), but the function's docstring doesn't, and the pyright-ignore comment only explains the type issue, not why messages is the right key for responses input. A reader encountering _responses_input_as_chat cold would have to trace the logging layer to understand the alias.

2. No test for the sync responses call_type. _SURFACE_OPS includes "responses" mapped to _RESPONSES_OPS but all tests use "aresponses". They're identical ops so it's low-risk, but it's an uncovered path.

3. dict(item) in _responses_final_text is fragile for the tool-call gate. The gate does dict(item) if not isinstance(item, Mapping) else item then calls .get("type"). For Pydantic v2 models, dict(model) uses __iter__ and works shallowly, so type will be present — but this is relying on undocumented Pydantic behavior rather than item.model_dump(). If a Pydantic model's __iter__ behavior changes, the gate silently stops detecting function-call items (missing tool-call turns would get sampled instead of skipped).

[greptile-apps]

Greptile Summary

The PR extends shadow evaluation sampling to Anthropic Messages and Responses API traffic while preserving chat-completion behavior

• Adds per-surface request normalization and response-text extraction
• Prevents replay of wire-body requests when request-mutating guardrails ran
• Adds coverage for request translation, tool-call skipping, guardrail handling, and surface dispatch

Confidence Score: 5/5

The PR appears safe to merge

No blocking failure remains

Important Files Changed

Filename	Overview
litellm/integrations/shadow_eval_logger.py	Adds surface-specific normalization, judgeability checks, guardrail-aware replay prevention, and shared dispatch for chat, Messages, and Responses traffic
litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py	Tightens the return annotation for the existing Anthropic translatable-parameter list
tests/test_litellm/integrations/test_shadow_eval_logger.py	Expands regression coverage across new surfaces, translations, tool calls, malformed payloads, and guardrail modes
basedpyright-code-budget.json	Ratchets down two basedpyright diagnostic budgets after the typing improvements

_{Reviews (3): Last reviewed commit: "feat: shadow eval samples /v1/messages a..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 99.01961% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/integrations/shadow_eval_logger.py	99.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_shadoweval_surfaces (e413a2f) with litellm_internal_staging (87abb87)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (592564d) during the generation of this report, so 87abb87 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[tin-berri]

All four findings addressed in one round: bridge system recovery via the wire body, messages alias documented, both responses call types tested, model_dump in the tool gate. @greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

Score: 5/5

All four findings are cleanly closed:

1. Bridge system recovery — _anthropic_messages_as_chat now tries kwargs.get("system") first, then falls through to _proxy_wire_body(kwargs).get("system"). The test test_anthropic_bridge_path_recovers_system_from_proxy_wire_body exercises it with no system in kwargs, only in the wire body snapshot, and asserts the system message lands first in the shadow call's message list.

2. Messages alias documented — _responses_input_as_chat's docstring explicitly names the function_setup alias and the three input shapes (bare string, chat dicts, item dicts), making the indirection findable in the future.

3. Both responses call types tested — test_responses_arms_normalize_bare_string_input_and_instructions is @pytest.mark.parametrized over ["aresponses", "responses"], so both entries in _SURFACE_OPS get an attempt row asserted.

4. model_dump in the tool gate — _responses_final_text calls item.model_dump() before the isinstance(item, Mapping) and .get("type") checks, so Pydantic output items are unwrapped before the gate runs. The "responses-function-call" arm in test_unjudgeable_turns_are_skipped_without_consuming_budget validates this path end to end.

The only remaining micro-nit is that _responses_final_text uses model_dump() without exclude_none=True while _responses_input_as_chat uses exclude_none=True, but for a gate that only inspects "type" this has zero behavioral effect. Not a concern.

[tin-berri]

Params finding fixed: each surface row translates params via its owner transformation, live-proven. @greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

Score: 4/5

The params concern is resolved — wire body sourcing for both surfaces is correct and live-proven, and the tests confirm the right params reach the shadow call. Here's the full picture:

What's working well:

• The dispatch table design is clean and the "membership = allowlist" invariant is clear and maintainable
• Reusing LiteLLMAnthropicMessagesAdapter.translate_anthropic_to_openai and LiteLLMCompletionResponsesConfig.transform_responses_api_request_to_chat_completion_request end-to-end means the normalization inherits correctness from the owners rather than duplicating format logic
• Wire body sourcing is correctly motivated — live-probed dialect switching on the bridge path makes the logged optional_params unreliable for Anthropic surface params
• Tool-final gate is now uniform and formalized across all surfaces (previously implicit for chat)
• Test coverage is solid: wire body recovery, param translation, bridge vs native paths, bare string input, tool-final skipping on both chat and Responses shapes

Remaining concerns (minor):

1. ResponsesAPIOptionalRequestParams.__annotations__ (shadow_eval_logger.py ~line 130): For a TypedDict, __annotations__ only captures the class's own annotations — not those inherited from parent TypedDicts. If ResponsesAPIOptionalRequestParams inherits fields from a base TypedDict (common in litellm's type hierarchy), inherited params like temperature or top_p would be silently dropped from the filter set and never reach the transformer. Worth checking the inheritance chain; get_type_hints() is the safer accessor for TypedDicts.

2. custom_tool_call in _responses_final_text: The check skips turns where any output item has type == "custom_tool_call". This isn't a type in the OpenAI Responses API spec — it appears to be a litellm-internal type. If it never appears in real Responses output, this is dead code; if it does, it should be documented. Either way, a comment explaining its source would help.

3. {"messages": _chat_messages(kwargs), **model_parameters} in _chat_request_from_chat: If model_parameters somehow contains a messages key, it silently overrides the extracted messages. Unlikely given what gets logged, but a model_parameters snapshot minus messages would be safer.

None of these block the feature — they're edge cases on the correctness margin. The core behavior is right, the params translation is now solid, and the test suite covers the important scenarios.

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit cc32851. Configure here.}

[tin-berri]

Point 3 hardened in 6ef599d. Point 1: TypedDict is flat and 3.13 merges inherited annotations. Point 2: custom_tool_call is OpenAI's custom-tools output type. @greptileai rescore?

[tin-berri]

bugbot run

[tin-berri]

CI fully green on 2611f41, codecov patch covered. @greptileai can you reply here with a score and why that score?

[greptile-apps]

4/5.

The implementation is well-engineered. The _SURFACE_OPS dispatch table is a clean replacement for the frozenset allowlist — membership implies support, and each entry fully encapsulates what's needed for that surface. Sourcing params from the wire body rather than model_parameters is correctly motivated and live-probed. Reusing LiteLLMAnthropicMessagesAdapter and LiteLLMCompletionResponsesConfig rather than writing custom format logic is the right call. The fail-closed pattern is consistent throughout, and the test suite covers the normalization paths, wire-body param correctness, tool-call skipping, and unsampleable shape failures.

The one thing I'd flag:

*((k, v) for k, v in wire_body.items() if k in ResponsesAPIOptionalRequestParams.__annotations__),

Using __annotations__ directly is fragile. In Pydantic v2, __annotations__ only reflects fields declared on that exact class — fields inherited from a parent class are not included. If ResponsesAPIOptionalRequestParams inherits any fields (e.g., from a shared params base), they'll be silently dropped from the wire body filter, meaning valid generation params like temperature or top_p won't be forwarded to the shadow call. The correct accessor is ResponsesAPIOptionalRequestParams.model_fields.keys(), which reflects the full validated field set regardless of where each field is defined.

                *((k, v) for k, v in wire_body.items() if k in ResponsesAPIOptionalRequestParams.model_fields),

Everything else is solid. The _forwards_nothing guard, the _UNFORWARDED_REQUEST_KEYS strip, the per-surface text extraction, and the type adapter validation boundaries are all well-placed. Good PR.

[tin-berri]

ResponsesAPIOptionalRequestParams is a TypedDict, not a BaseModel: model_fields doesn't exist on it, and TypedDicts merge inherited annotations. Keeping annotations

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

[tin-berri]

@greptileai can you reply here with score and why

[tin-berri]

bugbot run

[greptile-apps]

Overall, this is a well-engineered feature with excellent test coverage and thoughtful security design. Here's my assessment:

Score: 4/5

---

What's done well:

• Dispatch table design (_SURFACE_OPS) is clean — membership IS the allowlist, each row is self-contained, and unknown call types fail closed. This is a better abstraction than the old frozenset + inline coercion.
• Guardrail safety (_request_mutating_guardrail_ran) correctly gates wire-body surfaces, fails closed on unrecognized modes, and correctly excludes chat (which sources the post-guardrail dispatched call). The test coverage for all four combinations is solid.
• Normalization once, only when eligible — the eligible check before _judgeable_sample is an efficient ordering: turn budget, sampling, and direction checks filter cheaply before any transformation work happens.
• Tool-call gating is now formalized across all surfaces rather than implicit on chat.
• Tests are comprehensive: per-surface normalization, wire body vs. logged param precedence, tool-call skipping, guardrail bypasses, fail-closed shapes.

---

One real concern — ResponsesAPIOptionalRequestParams.__annotations__ misses inherited keys:

In _chat_request_from_responses, you filter the wire body through:

k in ResponsesAPIOptionalRequestParams.__annotations__

For a TypedDict, __annotations__ only contains keys declared directly on that class, not keys inherited from parent TypedDicts. If ResponsesAPIOptionalRequestParams inherits from any parent TypedDict, those parent keys would silently be excluded from the wire-body extraction. The safe alternative is get_type_hints(ResponsesAPIOptionalRequestParams), which follows the MRO:

from typing import get_type_hints
# ...
*((k, v) for k, v in wire_body.items() if k in get_type_hints(ResponsesAPIOptionalRequestParams)),

This matters because if a future parent TypedDict addition carries a generation param like top_logprobs, it would silently drop from the shadow call on the Responses surface without any error.

---

Minor:

• LiteLLMAnthropicMessagesAdapter() is instantiated on every sampled Anthropic request. Not a hot path, but a module-level singleton would be slightly cleaner.
• Streaming declared as a caveat is appropriate; the fail-closed behavior (skips without error rows) is the right choice.

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit e413a2f. Configure here.}

[tin-berri]

Demoed on 3.13: TypedDict merges inherited annotations, a subclass sees top_logprobs in annotations, and the class is flat. Adapter init is pass. @greptileai score?