Skip to content

fix(hosted-vllm): preserve reasoning_content in assistant history - #41396

Closed
RishikeshVepura wants to merge 9 commits into
BerriAI:mainfrom
RishikeshVepura:fix/hosted-vllm-preserve-reasoning-content
Closed

RishikeshVepura wants to merge 9 commits into
BerriAI:mainfrom
RishikeshVepura:fix/hosted-vllm-preserve-reasoning-content

Conversation

@RishikeshVepura

@RishikeshVepura RishikeshVepura commented Sep 16, 2026 •

Copy link
Copy Markdown

TLDR

Problem this solves:

How it solves it:

  • Preserve reasoning_content in assistant message history
  • Add regression coverage for the transformation

User Flow

Before: a developer’s tool-calling agent loses prior reasoning on its next turn

  1. Their application calls a hosted vLLM reasoning model through LiteLLM
  2. The assistant responds with reasoning content and a tool call
  3. The application saves that assistant response and appends the tool result
  4. The application makes its next call with the complete conversation history
  5. The hosted vLLM model does not receive the assistant’s earlier reasoning content

After: the same agent retains prior reasoning on its next turn

  1. Their application calls a hosted vLLM reasoning model through LiteLLM
  2. The assistant responds with reasoning content and a tool call
  3. The application saves that assistant response and appends the tool result
  4. The application makes its next call with the complete conversation history
  5. The hosted vLLM model receives the assistant’s earlier reasoning content

Relevant issues

Related to the Fireworks reasoning-content regression, but this affects the separate hosted_vllm provider transformation.

Affected release

Regression in v1.100.0

Linear ticket

Pre-Submission checklist

  • I have added meaningful tests
  • The focused test file passes locally: 11 passed
  • make lint passes locally
  • TODO: Confirm GitHub Actions CI passes after pushing the PR
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

TODO: After committing this branch, run the local vLLM reproduction again and add the live before/after output here

Before

TODO: Add the output from LiteLLM v1.102.0 showing reasoning_content reached vLLM: False

After

TODO: Add the output from this PR branch showing reasoning_content reached vLLM: True

Type

🐛 Bug Fix

Caveats (if any)

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

milan-berri and others added 9 commits July 28, 2026 14:14
…ticated requests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…stency

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…nd Model Hub UI

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…mutation

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…ow read fails

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…iption

feat(model_hub): surface model_info.description in Model Hub
…ip_save_on_failed_read

fix(health): skip background health check DB writes when the latest-row read fails
feat(dd_span_tagger): emit litellm.user_email span tag for JWT-authenticated requests
@codspeed

codspeed Bot commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing RishikeshVepura:fix/hosted-vllm-preserve-reasoning-content (5e08695) with litellm_internal_staging (252c71c)

Open in CodSpeed

@greptile-apps

greptile-apps Bot commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Preserves reasoning_content in replayed hosted vLLM assistant messages while continuing to remove thinking_blocks and normalize assistant content. The updated regression assertion verifies the field survives the production request transformation

Confidence Score: 5/5

The PR appears safe to merge, with the targeted behavior covered through the production transformation path

No actionable failures remain. The change directly removes the field deletion responsible for lost reasoning history, and the updated assertion would fail on the prior implementation

Important Files Changed
Filename Overview
litellm/llms/hosted_vllm/chat/transformation.py Stops removing assistant reasoning_content during hosted vLLM request transformation
tests/test_litellm/llms/hosted_vllm/chat/test_hosted_vllm_chat_transformation.py Updates the mapped transformation test to verify that assistant reasoning history is preserved

Reviews (1): Last reviewed commit: "fix(hosted-vllm): preserve reasoning_con..." | Re-trigger Greptile

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@codecov

codecov Bot commented Sep 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@mateo-berri

Copy link
Copy Markdown
Contributor

Closing as a duplicate of #43599, which carries the same hosted_vllm reasoning_content fix and merges cleanly against main. Thanks for the fix

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants