[PluggableLayer][MM] Add PluggableLayer for CustomQwen2Decoder#37293
[PluggableLayer][MM] Add PluggableLayer for CustomQwen2Decoder#37293ProExpertProg merged 6 commits intovllm-project:mainfrom
Conversation
Signed-off-by: Wangbei25 <wangbei41@huawie.com>
Signed-off-by: Wangbei25 <wangbei41@huawie.com>
|
Documentation preview: https://vllm--37293.org.readthedocs.build/en/37293/ |
There was a problem hiding this comment.
Code Review
This pull request correctly refactors CustomQwen2Decoder to be a PluggableLayer, which is a good step for extensibility. The documentation is also updated to reflect this and other related changes. My review includes a couple of suggestions to fix path separators in the documentation, which will ensure the documentation builds correctly across different operating systems.
docs/design/custom_op.md
Outdated
| **12. Encoder:** | ||
|
|
||
| ```python | ||
| --8<-- "vllm\model_executor\models\deepencoder2.py:qwen2_decoder" |
There was a problem hiding this comment.
The file path here uses a backslash (\) as a separator. For cross-platform compatibility and consistency with other paths in this document, please use a forward slash (/).
| --8<-- "vllm\model_executor\models\deepencoder2.py:qwen2_decoder" | |
| --8<-- "vllm/model_executor/models/deepencoder2.py:qwen2_decoder" |
Signed-off-by: Wangbei25 <wangbei41@huawei.com>
|
@ProExpertProg Would you mind take a look at this PR? Just a minor refactor for pluggable layer. |
…project#37293) Signed-off-by: Wangbei25 <wangbei41@huawie.com> Signed-off-by: Wangbei25 <wangbei41@huawei.com> Co-authored-by: Wangbei25 <wangbei41@huawie.com>
Purpose
Add PluggableLayer for CustomQwen2Decoder. As a result, this kind of encoder can be replaced with custom OOT implementations.
Test Plan
vllm serve /weights/DeepSeek-OCR-2
--served-model-name deepseekocr2
--trust-remote-code
curl http://localhost:1055/v1/chat/completions
-H "Content-Type: application/json"
-d '{"model": "deepseekocr2","messages": [{"role": "user","content": [
{"type": "image_url","image_url": {"url": "file:///datasets/textvqa_val/train_images/1080p.jpg","detail": "high"}},
{"type": "text","text": "Explain the contents of the picture with more than 500 words"}]}],
"max_tokens": 300
}'
Test Result
{"id":"chatcmpl-9be9f1663df4218e","object":"chat.completion","created":1773060077,"model":"deepseekocr2","choices":[{"index":0,"message":{"role":"assistant","content":"Ixar sampled a moment on the edge of a sandy beach. The white dog, sitting on its hind legs, looks directly at the camera. The person, dressed in a blue and white plaid shirt, sits on the sand with their legs crossed. In the background, the vast expanse of the ocean stretches out. The sun is setting, adding a warm glow to the scene. The image captures a peaceful moment between a dog and a person on a beach at sunset.","refusal":null,"annotations":null,"audio":null,"function_call":null,"tool_calls":[],"reasoning":null},"logprobs":null,"finish_reason":"stop","stop_reason":null,"token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":559,"total_tokens":656,"completion_tokens":97,"prompt_tokens_details":null},"prompt_logprobs":null,"prompt_token_ids":null,"kv_transfer_params":null}
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.