Bugfix/responses streaming by Ri0S · Pull Request #1 · Ri0S/vllm

Ri0S · 2026-01-09T02:01:44Z

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

Test Plan

Test Result

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
(Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

BEFORE SUBMITTING, PLEASE READ https://docs.vllm.ai/en/latest/contributing (anything written below this line will be removed by GitHub Actions)

…e decoding in `eagle.py` (vllm-project#31714) Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>

Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn> Signed-off-by: Roger Wang <hey@rogerw.io> Signed-off-by: h100 <h100@inferact.ai> Co-authored-by: Roger Wang <hey@rogerw.io> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Co-authored-by: h100 <h100@inferact.ai>

…el accuracy diff (vllm-project#31776) Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Signed-off-by: Fadi Arafeh <fadi.arafeh@arm.com>

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>

Signed-off-by: BlankR <hjyblanche@gmail.com>

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

…omputed_tokens_cpu` CommonAttentionMetadata properties (vllm-project#31773) Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>

…oject#31650) Signed-off-by: kunzh <zhikun.wu@outlook.com>

…31796) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

…ct#31790) Signed-off-by: Zhongze Jiang <jiangzhongze.jzz@ant-intl.com>

…m-project#31788) Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

…project#31806) Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

…ect#31759) Signed-off-by: Robert Shaw <robshaw@redhat.com> Co-authored-by: Robert Shaw <robshaw@redhat.com>

…#31593) Signed-off-by: Robert Shaw <robshaw@redhat.com> Co-authored-by: Robert Shaw <robshaw@redhat.com>

) Signed-off-by: Roi Koren <roik@nvidia.com>

…ect#31571) Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com> Co-authored-by: Michael Goin <mgoin64@gmail.com>

…omputed_tokens_cpu` CommonAttentionMetadata properties (vllm-project#31774) Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

…ct#31722) Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>

…lm-project#29493) Signed-off-by: Andy Xie <andy.xning@gmail.com>

Signed-off-by: Masataro Asai <guicho2.71828@gmail.com>

…vllm-project#31055) Signed-off-by: ReinforcedKnowledge <reinforced.knowledge@gmail.com>

…llm-project#29821) Signed-off-by: Benjamin Chislett <bchislett@nvidia.com> Signed-off-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: Nick Hill <nickhill123@gmail.com>

Signed-off-by: charlifu <charlifu@amd.com> Signed-off-by: Micah Williamson <micah.williamson@amd.com> Signed-off-by: Charlie Fu <Charlie.Fu@amd.com> Co-authored-by: Micah Williamson <micah.williamson@amd.com> Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>

…oject#31801) Signed-off-by: jiang1.li <jiang1.li@intel.com> Signed-off-by: Li, Jiang <bigpyj64@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Signed-off-by: Nikhil Ghosh <nikhil@anyscale.com>

Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com> Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Signed-off-by: bk-201 <joy25810@foxmail.com> Signed-off-by: Jee Jee Li <pandaleefree@gmail.com> Signed-off-by: prashanth058 <prashanth.dannamaneni@uipath.com> Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com> Co-authored-by: Jee Jee Li <pandaleefree@gmail.com> Co-authored-by: bk-201 <joy25810@foxmail.com> Co-authored-by: prashanth058 <prashanth.dannamaneni@uipath.com> Co-authored-by: Anexdeus <5142168@mail.ru> Co-authored-by: Julien Denize <40604584+juliendenize@users.noreply.github.com> Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk> Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@gmail.com>

Signed-off-by: dangoldbj <dangoldbj23@gmail.com>

Signed-off-by: Tianshu Yu <tianshuyu.formal@gmail.com> Signed-off-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Co-authored-by: Roger Wang <hey@rogerw.io> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

Signed-off-by: <> Signed-off-by: 赵策 <alcor@zhaocedeMacBook-Air.local> Signed-off-by: 赵策 <alcor@mac.mynetworksettings.com> Co-authored-by: 赵策 <alcor@mac.mynetworksettings.com>

…extra data (vllm-project#31610) Signed-off-by: maylikenoother <ogedengbemary19@gmail.com> Co-authored-by: Chauncey <chaunceyjiang@gmail.com>

…m-project#31967) Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

Signed-off-by: yxing <yxing@iquestlab.com>

…vllm-project#31969) Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

…ct#31960) Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

…#31591) Signed-off-by: Nick Hill <nickhill123@gmail.com>

Signed-off-by: Nishidha Panpaliya <nishidha.panpaliya@partner.ibm.com>

Signed-off-by: NickLucche <nlucches@redhat.com>

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

…rnels" (vllm-project#31978)

…multimodal encoders (vllm-project#31627) Signed-off-by: Lucas Kabela <lucaskabela@meta.com>

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

Signed-off-by: Bill Nell <bnell@redhat.com>

… for NVFP4A16 MoEs (vllm-project#30881)

…d issues in the future (vllm-project#31747) Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com> Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com> Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>

…compressed tensor kernel selection (vllm-project#31752) Signed-off-by: Robert Shaw <robshaw@redhat.com> Signed-off-by: Yongye Zhu <zyy1102000@gmail.com> Co-authored-by: Robert Shaw <robshaw@redhat.com>

Signed-off-by: Robert Shaw <robshaw@redhat.com> Signed-off-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com> Co-authored-by: Robert Shaw <robshaw@redhat.com> Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>

…project#31982) Signed-off-by: Nick Hill <nickhill123@gmail.com>

…roject#31193) Signed-off-by: Max Hu <maxhu@nvidia.com> Signed-off-by: Max Hu <hyoung2991@gmail.com> Co-authored-by: Max Hu <maxhu@nvidia.com>

Signed-off-by: Robert Shaw <robshaw@redhat.com> Co-authored-by: Robert Shaw <robshaw@redhat.com>

…ect#31761) Signed-off-by: Daniel Salib <danielsalib@meta.com>

Signed-off-by: Ri0S <aa248424@gmail.com>

…aming # Conflicts: # vllm/entrypoints/openai/serving_responses.py Signed-off-by: Ri0S <aa248424@gmail.com>

vllmellm and others added 30 commits January 6, 2026 07:53

[Bugfix][ROCm] Fix Unsupported attention metadata type for speculativ…

e971780

…e decoding in `eagle.py` (vllm-project#31714) Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>

[Bugfix][CI/Build] Fix failing pooling models test due to Triton kern…

ee2e69d

…el accuracy diff (vllm-project#31776) Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

[Chore] Remove more V0 dead code from sequence.py (vllm-project#31783)

97ca4c3

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

[cpu][bench] Add CPU paged attention benchmarks (vllm-project#31720)

799b572

Signed-off-by: Fadi Arafeh <fadi.arafeh@arm.com>

[Misc] Use deprecated for seed_everything (vllm-project#31780)

db31832

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

[CI] Increase the MTEB_EMBED_TOL threshold to 5e-4. (vllm-project#31797)

43d384b

Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>

[Doc] Fix format of multimodal_inputs.md (vllm-project#31800)

6ebb66c

Signed-off-by: BlankR <hjyblanche@gmail.com>

[Chore] Cleanup mem_utils.py (vllm-project#31793)

14df02b

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

[Attention][1/n] Remove usage of deprecated seq_lens_cpu and `num_c…

e0327c9

…omputed_tokens_cpu` CommonAttentionMetadata properties (vllm-project#31773) Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>

[Bugfix] Fix torch.compile error for DP + MoE on CPU Backend (vllm-pr…

bf0f3a4

…oject#31650) Signed-off-by: kunzh <zhikun.wu@outlook.com>

[Misc] Implement TokenizerLike.convert_tokens_to_ids (vllm-project#…

6444824

…31796) Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

[Bugfix]: avoid overriding audio/text kwargs (Qwen3-Omni) (vllm-proje…

2c1a4f2

…ct#31790) Signed-off-by: Zhongze Jiang <jiangzhongze.jzz@ant-intl.com>

[Frontend] Support GLM-4.5 / GLM-4.7 with enable_thinking: false (vll…

0202971

…m-project#31788) Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

[Model] rename use_pad_token to use_sep_token (vllm-project#31784)

96860af

Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>

[LoRA]Disable linear LoRA kernel PDL (vllm-project#31777)

cbd4690

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

[Bugfix]: Fix cross attention backend selection for Turing GPU (vllm-…

02809af

…project#31806) Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>

[MoE Refactor] Add Temporary Integration Tests - H100/B200 (vllm-proj…

d3e477c

…ect#31759) Signed-off-by: Robert Shaw <robshaw@redhat.com> Co-authored-by: Robert Shaw <robshaw@redhat.com>

[MoE Refactor][14/N] Clean Up FI Quant Config Smuggling (vllm-project…

af8fd73

…#31593) Signed-off-by: Robert Shaw <robshaw@redhat.com> Co-authored-by: Robert Shaw <robshaw@redhat.com>

[NemotronH] Use ReplicatedLinear for fc1_latent_proj (vllm-project#31807

28c9477

) Signed-off-by: Roi Koren <roik@nvidia.com>

[Quantization][MoE] remove unused ep logic from moe marlin (vllm-proj…

2f4bdee

…ect#31571) Signed-off-by: Jinzhen Lin <jinzhen.ljz@antgroup.com> Co-authored-by: Michael Goin <mgoin64@gmail.com>

[PERF] Speed-up of GDN attention decode part (Qwen3-Next) (vllm-proje…

22dffca

…ct#31722) Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>

[Log] add log about gpu worker init snapshot and requested memory (vl…

6f5e653

…lm-project#29493) Signed-off-by: Andy Xie <andy.xning@gmail.com>

make 500: InternalServerError more informative (vllm-project#20610)

142c4d1

Signed-off-by: Masataro Asai <guicho2.71828@gmail.com>

[Bugfix] Fix GLM-4 MoE router logits dtype for data parallel chunking (…

4e67a8f

…vllm-project#31055) Signed-off-by: ReinforcedKnowledge <reinforced.knowledge@gmail.com>

[Perf] Async Scheduling + Speculative Decoding + Structured Outputs (v…

f7008ce

…llm-project#29821) Signed-off-by: Benjamin Chislett <bchislett@nvidia.com> Signed-off-by: Nick Hill <nickhill123@gmail.com> Co-authored-by: Nick Hill <nickhill123@gmail.com>

Fix RecursionError in MediaWithBytes unpickling (vllm-project#31191)

ada6f91

Signed-off-by: Nikhil Ghosh <nikhil@anyscale.com>

patrickvonplaten and others added 29 commits January 8, 2026 18:34

[Model] Add Grok-2 (vllm-project#31847)

59d260f

Signed-off-by: dangoldbj <dangoldbj23@gmail.com>

[Model] Add LFM2-VL model support (vllm-project#31758)

03fd76c

Signed-off-by: Tianshu Yu <tianshuyu.formal@gmail.com> Signed-off-by: Cyrus Leung <cyrus.tl.leung@gmail.com> Co-authored-by: Roger Wang <hey@rogerw.io> Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>

[Model] Enable LoRA support for Pixtral (vllm-project#31724)

1123a87

Signed-off-by: <> Signed-off-by: 赵策 <alcor@zhaocedeMacBook-Air.local> Signed-off-by: 赵策 <alcor@mac.mynetworksettings.com> Co-authored-by: 赵策 <alcor@mac.mynetworksettings.com>

[OpenAI] Fix tool_choice=required streaming when output has trailing …

7645bc5

…extra data (vllm-project#31610) Signed-off-by: maylikenoother <ogedengbemary19@gmail.com> Co-authored-by: Chauncey <chaunceyjiang@gmail.com>

[CI] [Bugfix] Fix unbounded variable in run-multi-node-test.sh (vll…

72c068b

…m-project#31967) Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>

[Docs]: update claude code url (vllm-project#31971)

1da3a54

Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

[Model] Support IQuestCoder model (vllm-project#31575)

fe86be6

Signed-off-by: yxing <yxing@iquestlab.com>

[Bugfix]: Fix Step3ReasoningParser missing is_reasoning_end_streaming (…

eaba8ec

…vllm-project#31969) Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>

[Bugfix] Fix vllm serve failure with Nemotron Nano V3 FP8 (vllm-proje…

b8112c1

…ct#31960) Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com>

[Doc] Improve MM models LoRA notes (vllm-project#31979)

49568d5

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>

[Misc] Tidy up some spec decode logic in GPUModelRunner (vllm-project…

a3d909a

…#31591) Signed-off-by: Nick Hill <nickhill123@gmail.com>

Fix ijson build for Power. (vllm-project#31702)

a563866

Signed-off-by: Nishidha Panpaliya <nishidha.panpaliya@partner.ibm.com>

[CI][ROCm] Fix NIXL tests on ROCm (vllm-project#31728)

83e1c76

Signed-off-by: NickLucche <nlucches@redhat.com>

[Model Runner V2] Simplify BlockTables with UVA (vllm-project#31965)

7508243

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>

Revert "feat(moe): Add is_act_and_mul=False support for Triton MoE ke…

87e07a6

…rnels" (vllm-project#31978)

[Documentation][torch.compile] Add documentation for torch.compile + …

f16bfbe

…multimodal encoders (vllm-project#31627) Signed-off-by: Lucas Kabela <lucaskabela@meta.com>

[Frontend] Improve error message (vllm-project#31987)

aa125ec

Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>

[Misc][Refactor] Add FusedMoERouter object (vllm-project#30519)

e74698c

Signed-off-by: Bill Nell <bnell@redhat.com>

[Compressed-Tensors] Simplify NVFP4 Conditions, enable marlin support…

5d3b609

… for NVFP4A16 MoEs (vllm-project#30881)

[MoE Refactoring][Bugfix]Wrap WNA16 Triton kernel into mk and change …

d62cfe5

…compressed tensor kernel selection (vllm-project#31752) Signed-off-by: Robert Shaw <robshaw@redhat.com> Signed-off-by: Yongye Zhu <zyy1102000@gmail.com> Co-authored-by: Robert Shaw <robshaw@redhat.com>

[BugFix] Add spec-decode-incompatible request param validation (vllm-…

11cec29

…project#31982) Signed-off-by: Nick Hill <nickhill123@gmail.com>

[Feature] Add iteration level logging and enhance nvtx marker (vllm-p…

6ebe34d

…roject#31193) Signed-off-by: Max Hu <maxhu@nvidia.com> Signed-off-by: Max Hu <hyoung2991@gmail.com> Co-authored-by: Max Hu <maxhu@nvidia.com>

[Bugfix] Fix Typo from NVFP4 Refactor (vllm-project#31977)

0fa8dd2

Signed-off-by: Robert Shaw <robshaw@redhat.com> Co-authored-by: Robert Shaw <robshaw@redhat.com>

[Frontend] Add MCP tool streaming support to Responses API (vllm-proj…

a4ec0c5

…ect#31761) Signed-off-by: Daniel Salib <danielsalib@meta.com>

Resolve merge conflict

6e41966

Signed-off-by: Ri0S <aa248424@gmail.com>

Merge remote-tracking branch 'origin/main' into bugfix/responses_stre…

a35d727

…aming # Conflicts: # vllm/entrypoints/openai/serving_responses.py Signed-off-by: Ri0S <aa248424@gmail.com>

Ri0S merged commit f8d2831 into main Jan 9, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bugfix/responses streaming#1

Bugfix/responses streaming#1
Ri0S merged 136 commits intomainfrom
bugfix/responses_streaming

Ri0S commented Jan 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

Ri0S commented Jan 9, 2026

Purpose

Test Plan

Test Result

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants