[Model Runner V2] Support multi-modal embeddings for spec decode model - #36097
Conversation
|
This pull request has merge conflicts that must be resolved before it can be |
There was a problem hiding this comment.
Code Review
This pull request adds support for multi-modal embeddings in the speculative decoding model runner. The changes involve propagating scheduler_output to access encoder inputs and preparing multi-modal embeddings for the speculative step. The logic appears correct, but I've identified an opportunity to improve code maintainability by refactoring duplicated code.
|
Can you please rebase? |
df1a75e to
244b038
Compare
bc2e42e to
8ffc9b2
Compare
e9dadcd to
e95f68f
Compare
|
This pull request has merge conflicts that must be resolved before it can be |
b3fdf54 to
245e44f
Compare
4e6dfc4 to
c251158
Compare
|
This pull request has merge conflicts that must be resolved before it can be |
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
WoosukKwon
left a comment
There was a problem hiding this comment.
LGTM. Thanks for the PR! I've made small edits in the PR for minor code cleanup.
vllm-project#36097) Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai> Signed-off-by: Woosuk Kwon <woosuk@inferact.ai> Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
vllm-project#36097) Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai> Signed-off-by: Woosuk Kwon <woosuk@inferact.ai> Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
vllm-project#36097) Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai> Signed-off-by: Woosuk Kwon <woosuk@inferact.ai> Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
vllm-project#36097) Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai> Signed-off-by: Woosuk Kwon <woosuk@inferact.ai> Co-authored-by: Woosuk Kwon <woosuk@inferact.ai> Signed-off-by: Nithin Chalapathi <nithin.ch10@gmail.com>
vllm-project#36097) Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai> Signed-off-by: Woosuk Kwon <woosuk@inferact.ai> Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
vllm-project#36097) Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai> Signed-off-by: Woosuk Kwon <woosuk@inferact.ai> Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
vllm-project#36097) Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai> Signed-off-by: Woosuk Kwon <woosuk@inferact.ai> Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
vllm-project#36097) Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai> Signed-off-by: Woosuk Kwon <woosuk@inferact.ai> Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
Purpose
Currently the multimodal inputs from the target model vision encoder are not passed to the draft model. Draft models trained on vision embeddings need to merge these inputs with the input token ids to function properly. This behavior is supported in MRV1, but not yet in MRV2. This PR addresses that gap.
Manual Testing
Verified no regression in acceptance rate for the following vision models:
Unfortunately, I couldn't find any target/draft model pairs that are both trained on vision embeddings and also currently work on MRV2 to test on and observe an increase in acceptance rate. So I instead tested the scenario where the draft model receives vision embeddings by forcing
self.supports_mm_inputsto True, and checking that the correct number of embeddings are passed and merged with the input ids during draft model prefill.