Skip to content

Fix/tool call anthropic transformation for vllm - #28549

Merged
Sameerlite merged 4 commits into
BerriAI:litellm_oss_staging_2from
FelipeRodriguesGare:fix/tool-call-anthropic
May 22, 2026
Merged

Fix/tool call anthropic transformation for vllm#28549
Sameerlite merged 4 commits into
BerriAI:litellm_oss_staging_2from
FelipeRodriguesGare:fix/tool-call-anthropic

Conversation

@FelipeRodriguesGare

@FelipeRodriguesGare FelipeRodriguesGare commented May 22, 2026

Copy link
Copy Markdown
Contributor

Relevant issues

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have Added testing in the tests/test_litellm/ directory, Adding at least 1 test is a hard requirement - see details
  • My PR passes all unit tests on make test-unit
  • My PR's scope is as isolated as possible, it only solves 1 specific problem
  • I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

CI status guideline:

  • 50-55 passing tests: main is stable with minor issues.
  • 45-49 passing tests: acceptable but needs attention
  • <= 40 passing tests: unstable; be careful with your merges and assess the risk.
  • Branch creation CI run
    Link:

  • CI run for the last commit
    Link:

  • Merge / cherry-pick CI run
    Links:

Screenshots / Proof of Fix

Before:
Captura de Tela 2026-05-22 às 01 06 09

After:
Captura de Tela 2026-05-22 às 01 04 19

Type

🆕 New Feature
🐛 Bug Fix
✅ Test

Changes

Changed tool_calls validation to accept falsy values as well. Hosted_vllm can send the attribute as an empty list (tool_call: []) and this condition was breaking claude code integration with hosted_vllm models. The agent simply did not return any message due to interpreting everything as tool calls without any tool being called.

@CLAassistant

CLAassistant commented May 22, 2026

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
2 out of 3 committers have signed the CLA.

✅ yuneng-berri
✅ FelipeRodriguesGare
❌ shin-berri
You have signed the CLA already but the status is still pending? Let us recheck it.

@greptile-apps

greptile-apps Bot commented May 22, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Fixes a streaming adaptation bug where OpenAI-compatible providers (e.g. hosted vLLM) emit tool_calls: [] on ordinary text chunks, causing the Anthropic adapter to return input_json_delta instead of text_delta and silently drop all streamed text.

  • transformation.py: Tightens the tool-call guard from is not None to a truthiness check (if choice.delta.tool_calls:), so an empty list is treated the same as None.
  • test_…_transformation.py: Adds a focused mock test verifying both _translate_streaming_openai_chunk_to_anthropic and _translate_streaming_openai_chunk_to_anthropic_content_block return text-type results when tool_calls=[] is present alongside real text content.

Confidence Score: 5/5

Safe to merge — the change is a single-line guard tightening in one streaming helper, with a direct reproduction test, and no impact on existing non-empty tool-call paths.

The fix is minimal and surgical: replacing is not None with a truthiness check on a list that vLLM can return as empty. The companion test directly exercises both affected methods with tool_calls=[] and verifies the correct text_delta output. The parallel guard in _translate_streaming_openai_chunk_to_anthropic_content_block already used len(...) > 0, so this change brings the two helpers into alignment.

No files require special attention.

Important Files Changed

Filename Overview
litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py Single-line fix: changes tool_calls is not None to a truthiness check so an empty list from vLLM no longer triggers the input_json_delta path
tests/test_litellm/llms/anthropic/experimental_pass_through/adapters/test_anthropic_experimental_pass_through_adapters_transformation.py Adds a new mock-only test covering the empty tool_calls case for both streaming translation helpers; no real network calls, no weakened assertions

Reviews (1): Last reviewed commit: "fix(anthropic): handle empty streaming t..." | Re-trigger Greptile

@oss-pr-review-agent-shin

Copy link
Copy Markdown
Contributor

🤖 litellm-agent: This PR is currently BLOCKED from merge.

Score: 4/5

Why blocked:

  • 1 unresolved reviewer concern (cla-assistant) (unresolved_concern, -1 pts)

Details: Score docked for: 1 unresolved reviewer concern (cla-assistant).

Fix the issues above and push an update — the bot will re-review automatically.

Note: This bot is still in beta and might not always work as expected. Please share any feedback via Slack.

@codecov

codecov Bot commented May 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@Sameerlite Sameerlite left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Sameerlite
Sameerlite changed the base branch from litellm_internal_staging to litellm_oss_staging_2 May 22, 2026 11:49
@Sameerlite
Sameerlite merged commit 0168f0f into BerriAI:litellm_oss_staging_2 May 22, 2026
44 checks passed
mateo-berri pushed a commit that referenced this pull request May 22, 2026
* fix(anthropic): handle empty streaming tool calls (#28549)

Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>

* [Feature][Bug Fix] Decouple Azure OpenAI Deployment ID from model name via base_model to fix gpt5 model routing (#28490)

* feat(azure): decouple deployment ID from model name via base_model

Azure OpenAI deployments have arbitrary names (deployment IDs) that may
not match the underlying model. Previously, model-type detection
(o-series, gpt-5, etc.) relied on substring matching against the
deployment name, causing misrouted configs and rejected params when
deployment names were non-standard (e.g. 'my-deployment-id' for gpt-5.2).

This change extends the existing base_model field to drive model-type
detection, config selection, supported param resolution, and param
mapping throughout the Azure call path:

- _get_azure_config() uses base_model for is_o_series/is_gpt_5 checks
- get_provider_chat_config() threads base_model for Azure
- get_supported_openai_params() accepts and uses base_model
- get_optional_params() accepts base_model and passes it to all Azure
  config method calls (get_supported_openai_params, map_openai_params)
- azure.py completion handler uses base_model for GPT-5 detection
- Config internal methods (e.g. is_model_gpt_5_2_model) now receive
  base_model so features like logprobs are correctly enabled

Fully backward compatible - when base_model is unset, behavior is
identical. Existing o_series/ and gpt5_series/ prefix workarounds
continue to work.

Usage in proxy config:
  model_list:
    - model_name: my-gpt5
      litellm_params:
        model: azure/my-deployment-id
      model_info:
        base_model: azure/gpt-5.2

Fixes: non-standard deployment names like 'prefix-gpt-5.2' rejecting
logprobs/top_logprobs despite the underlying model supporting them.

* Addressing Greptile comments.

* gemini-3.1-flash-lite pricing (#27933)

* feat(model_prices): add gemini-3.1-flash-lite pricing with standard/batch/flex/priority tiers

* fix pricing

* add service tier

---------

Co-authored-by: shin-berri <shin-laptop@berri.ai>

* fix(openai-responses): strip Anthropic cache_control from Responses API requests (#28431)

Squash-merged by litellm-agent from cwang-otto's PR.

* Treat None litellm_provider as wildcard in _check_provider_match (#28523)

Squash-merged by litellm-agent from adityasingh2400's PR.

* fix greptile

* fix: use _azure_detection_model in default Azure branch of get_supported_openai_params

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(openai-responses): strip cache_control on compact endpoint as well

Co-authored-by: Yassin Kortam <yassin@berri.ai>

---------

Co-authored-by: Felipe Garé <90070734+FelipeRodriguesGare@users.noreply.github.com>
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: withomasmicrosoft <withomas@microsoft.com>
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Co-authored-by: cwang-otto <chengxuan.wang@ottotheagent.com>
Co-authored-by: Aditya Singh <60082699+adityasingh2400@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
fzowl pushed a commit to fzowl/litellm that referenced this pull request Jun 24, 2026
* fix(anthropic): handle empty streaming tool calls (BerriAI#28549)

Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>

* [Feature][Bug Fix] Decouple Azure OpenAI Deployment ID from model name via base_model to fix gpt5 model routing (BerriAI#28490)

* feat(azure): decouple deployment ID from model name via base_model

Azure OpenAI deployments have arbitrary names (deployment IDs) that may
not match the underlying model. Previously, model-type detection
(o-series, gpt-5, etc.) relied on substring matching against the
deployment name, causing misrouted configs and rejected params when
deployment names were non-standard (e.g. 'my-deployment-id' for gpt-5.2).

This change extends the existing base_model field to drive model-type
detection, config selection, supported param resolution, and param
mapping throughout the Azure call path:

- _get_azure_config() uses base_model for is_o_series/is_gpt_5 checks
- get_provider_chat_config() threads base_model for Azure
- get_supported_openai_params() accepts and uses base_model
- get_optional_params() accepts base_model and passes it to all Azure
  config method calls (get_supported_openai_params, map_openai_params)
- azure.py completion handler uses base_model for GPT-5 detection
- Config internal methods (e.g. is_model_gpt_5_2_model) now receive
  base_model so features like logprobs are correctly enabled

Fully backward compatible - when base_model is unset, behavior is
identical. Existing o_series/ and gpt5_series/ prefix workarounds
continue to work.

Usage in proxy config:
  model_list:
    - model_name: my-gpt5
      litellm_params:
        model: azure/my-deployment-id
      model_info:
        base_model: azure/gpt-5.2

Fixes: non-standard deployment names like 'prefix-gpt-5.2' rejecting
logprobs/top_logprobs despite the underlying model supporting them.

* Addressing Greptile comments.

* gemini-3.1-flash-lite pricing (BerriAI#27933)

* feat(model_prices): add gemini-3.1-flash-lite pricing with standard/batch/flex/priority tiers

* fix pricing

* add service tier

---------

Co-authored-by: shin-berri <shin-laptop@berri.ai>

* fix(openai-responses): strip Anthropic cache_control from Responses API requests (BerriAI#28431)

Squash-merged by litellm-agent from cwang-otto's PR.

* Treat None litellm_provider as wildcard in _check_provider_match (BerriAI#28523)

Squash-merged by litellm-agent from adityasingh2400's PR.

* fix greptile

* fix: use _azure_detection_model in default Azure branch of get_supported_openai_params

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(openai-responses): strip cache_control on compact endpoint as well

Co-authored-by: Yassin Kortam <yassin@berri.ai>

---------

Co-authored-by: Felipe Garé <90070734+FelipeRodriguesGare@users.noreply.github.com>
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: withomasmicrosoft <withomas@microsoft.com>
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Co-authored-by: cwang-otto <chengxuan.wang@ottotheagent.com>
Co-authored-by: Aditya Singh <60082699+adityasingh2400@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants