Skip to content

[Fix] add streaming support in process_vllm_two_stage_request_discovered#139

Closed
simondanielsson wants to merge 1 commit into
vllm-project:mainfrom
simondanielsson:feature/streaming-support-srv-discovery
Closed

[Fix] add streaming support in process_vllm_two_stage_request_discovered#139
simondanielsson wants to merge 1 commit into
vllm-project:mainfrom
simondanielsson:feature/streaming-support-srv-discovery

Conversation

@simondanielsson

@simondanielsson simondanielsson commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

We might close this PR in favor of #114

Closes #113.

The service discovery path for vllm PD disaggregation does not support streaming, as opposed to the regular process_vllm_two_stage_request path (without service discovery) which does support streaming.

This PR adds support for response streaming also in the service discovery path. Without this, you cannot use the router with service discovery together with vllm bench serve --backend vllm which requires that streaming works.

Test Plan

Reproducer:

Test Result

Reproducer:


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results

Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>
@simondanielsson

Copy link
Copy Markdown
Contributor Author

Closed in favor of this: #114

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature]: Support stream in the vllm PD router

1 participant