Skip to content

feat(vllm): forward routed experts data - #13531

Draft
biswapanda wants to merge 10 commits into
ai-dynamo:mainfrom
biswapanda:feat/vllm-sidecar-routed-output
Draft

feat(vllm): forward routed experts data#13531
biswapanda wants to merge 10 commits into
ai-dynamo:mainfrom
biswapanda:feat/vllm-sidecar-routed-output

Conversation

@biswapanda

@biswapanda biswapanda commented Aug 19, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

  • forward routed_experts_prompt_start through the native Generate request and vLLM gRPC response options
  • carry terminal routed-expert bytes opaquely through the sidecar without interpreting NumPy shape or dtype
  • base64-encode the payload only at Dynamo's JSON response boundary
  • reject encoder-side and nonterminal routed output and preserve multimodal-feature coexistence
  • cover large payloads above the default gRPC message size

This rebuilds #13531 on the current #14244 head and consolidates the request-control behavior previously isolated in #13535. The branch is cumulative until #14244 merges and will remain draft until the routed-experts Kubernetes gate passes.

Validation

  • cargo fmt --all -- --check
  • cargo test -p dynamo-vllm-sidecar: 50 library tests and 1 executable integration test passed
  • cargo test -p dynamo-mocker: 205 passed
  • byte-identical protobuf contract with vLLM, SHA-256 522085198ae5d60256261542cd667285c426fa8de3c5eafc0beaea3fa79e2a2d
  • reviewed terminal-only behavior, prompt bounds, large opaque payloads, base64 fidelity, and VLM coexistence
  • Kubernetes binaries were built from the frozen routed source; the native preflight and exact five-step routed-expert gate are still pending and are not claimed as passed

Dependencies

@biswapanda
biswapanda temporarily deployed to external_collaborator August 19, 2026 10:10 — with GitHub Actions Inactive
@biswapanda
biswapanda temporarily deployed to external_collaborator August 19, 2026 10:10 — with GitHub Actions Inactive
@github-actions github-actions Bot added the feat label Aug 19, 2026
@github-actions

github-actions Bot commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

👋 Hi biswapanda! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@dynamo-ops

dynamo-ops commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

/ok to test 0fae23b

@github-actions github-actions Bot added external-contribution Pull request is from an external contributor documentation Improvements or additions to documentation labels Aug 19, 2026
This was referenced Aug 19, 2026
Open
Open
@biswapanda
biswapanda force-pushed the feat/vllm-sidecar-routed-output branch from 0fae23b to 5265f8c Compare August 20, 2026 07:53
@biswapanda
biswapanda temporarily deployed to external_collaborator August 20, 2026 07:53 — with GitHub Actions Inactive
@dynamo-ops

dynamo-ops commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

/ok to test 5265f8c

@biswapanda biswapanda changed the title feat(vllm): preserve routed expert output feat(vllm): preserve native routed expert output Aug 20, 2026
@biswapanda
biswapanda temporarily deployed to external_collaborator August 20, 2026 10:26 — with GitHub Actions Inactive
@dynamo-ops

dynamo-ops commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

/ok to test ff5197b

@pull-request-size pull-request-size Bot added size/M and removed size/L labels Aug 26, 2026
@biswapanda biswapanda changed the title feat(vllm): preserve native routed expert output feat(vllm): forward opaque routed expert output Aug 26, 2026
@biswapanda
biswapanda temporarily deployed to external_collaborator August 26, 2026 16:36 — with GitHub Actions Inactive
@dynamo-ops

dynamo-ops commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

/ok to test b7ff8d3

@dynamo-ops

dynamo-ops commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

/ok to test 1196a35

@biswapanda
biswapanda temporarily deployed to external_collaborator August 26, 2026 17:34 — with GitHub Actions Inactive
@dynamo-ops

dynamo-ops commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

/ok to test 060741b

@knarangN knarangN mentioned this pull request Aug 27, 2026
Closed
1 task
@biswapanda biswapanda mentioned this pull request Sep 4, 2026
Open
@biswapanda
biswapanda temporarily deployed to external_collaborator September 4, 2026 15:36 — with GitHub Actions Inactive
@dynamo-ops

dynamo-ops commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

/ok to test e1e4a6a

@biswapanda biswapanda changed the title feat(vllm): forward opaque routed expert output feat(vllm): forward routed experts data Sep 4, 2026
@github-actions github-actions Bot added the frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` label Sep 4, 2026
@biswapanda
biswapanda temporarily deployed to external_collaborator September 4, 2026 17:35 — with GitHub Actions Inactive
@dynamo-ops

dynamo-ops commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

/ok to test d43ce7d

@biswapanda
biswapanda temporarily deployed to external_collaborator September 4, 2026 18:34 — with GitHub Actions Inactive
@dynamo-ops

dynamo-ops commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

/ok to test f11e5a1

@biswapanda
fix(vllm): project generate request controls
61c818a 
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
@biswapanda
test(vllm): move sidecar regression module
d65e80c 
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
@biswapanda
fix(vllm): complete generate sidecar controls
3643d5b 
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
@biswapanda
biswapanda force-pushed the feat/vllm-sidecar-routed-output branch from f11e5a1 to 83eaadd Compare September 7, 2026 06:41
@biswapanda
biswapanda temporarily deployed to external_collaborator September 7, 2026 06:41 — with GitHub Actions Inactive
@dynamo-ops

dynamo-ops commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

/ok to test 83eaadd

@github-actions github-actions Bot added the backend::vllm Relates to the vllm backend label Sep 7, 2026
@biswapanda
fix(vllm): reconcile sidecar request controls
e341582 
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
@biswapanda
refactor(vllm): minimize request reconciliation
852677e 
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
@biswapanda
biswapanda force-pushed the feat/vllm-sidecar-routed-output branch from 83eaadd to 9360b85 Compare September 8, 2026 07:15
@biswapanda
biswapanda temporarily deployed to external_collaborator September 8, 2026 07:15 — with GitHub Actions Inactive
@dynamo-ops

dynamo-ops commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

/ok to test 9360b85

@biswapanda
fix(vllm): reject unsupported thinking budgets
6c04037 
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
@biswapanda
feat(vllm): forward preprocessed multimodal features
8256deb 
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
@biswapanda
feat(vllm): forward routed experts data
d3c86f0 
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
@biswapanda
feat(vllm): forward sampling masks
2987b3d 
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
@biswapanda
fix(vllm): initialize mock sampling masks
6463eaa 
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
@biswapanda
biswapanda force-pushed the feat/vllm-sidecar-routed-output branch from 9360b85 to 6463eaa Compare September 8, 2026 07:40
@biswapanda
biswapanda temporarily deployed to external_collaborator September 8, 2026 07:41 — with GitHub Actions Inactive
@dynamo-ops

dynamo-ops commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

/ok to test 6463eaa

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

backend::vllm Relates to the vllm backend documentation Improvements or additions to documentation external-contribution Pull request is from an external contributor feat frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` size/XXL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@biswapanda @dynamo-ops