Skip to content

[Core] enable out-of-tree model register - #3871

Merged
youkaichao merged 14 commits into
vllm-project:mainfrom
youkaichao:oot_models
Apr 7, 2024
Merged

[Core] enable out-of-tree model register#3871
youkaichao merged 14 commits into
vllm-project:mainfrom
youkaichao:oot_models

Conversation

@youkaichao

@youkaichao youkaichao commented Apr 5, 2024

Copy link
Copy Markdown
Member

Usage: add three lines:

from vllm.model_executor.models import ModelRegistry
from your_code import YourModelForCausalLM
ModelRegistry.register_out_of_tree_model("YourModelForCausalLM", YourModelForCausalLM)

@youkaichao
youkaichao requested a review from WoosukKwon April 5, 2024 16:45
@njhill

njhill commented Apr 5, 2024

Copy link
Copy Markdown
Member

See also: #3820. I guess this registration approach is probably better.

@simon-mo

simon-mo commented Apr 5, 2024

Copy link
Copy Markdown
Collaborator

anyway to test this in CI?

@youkaichao

youkaichao commented Apr 6, 2024

Copy link
Copy Markdown
Member Author

@simon-mo added tests in ci. PTAL.

WoosukKwon
WoosukKwon reviewed Apr 6, 2024
View reviewed changes
Comment thread vllm/model_executor/models/__init__.py Outdated
WoosukKwon
WoosukKwon reviewed Apr 6, 2024
View reviewed changes

@WoosukKwon WoosukKwon left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the work! Left some comments.

Comment thread vllm/model_executor/models/__init__.py
Comment thread docs/source/models/adding_model.rst Outdated
Comment thread tests/models/test_oot_registration.py
@youkaichao youkaichao mentioned this pull request Apr 6, 2024
Closed
@youkaichao
youkaichao requested a review from WoosukKwon April 6, 2024 17:29
WoosukKwon
WoosukKwon approved these changes Apr 6, 2024
View reviewed changes

@WoosukKwon WoosukKwon left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! Thanks for the great work!

@youkaichao
youkaichao merged commit 95baec8 into vllm-project:main Apr 7, 2024
@youkaichao
youkaichao deleted the oot_models branch April 7, 2024 00:11
z103cb pushed a commit to z103cb/opendatahub_vllm that referenced this pull request Apr 22, 2024
@youkaichao @joerunde
[Core] enable out-of-tree model register (vllm-project#3871)
d0bc197
@dtrifiro dtrifiro mentioned this pull request May 15, 2024
Merged
@xwjiang2010 xwjiang2010 mentioned this pull request Jun 28, 2024
Merged
@youkaichao youkaichao mentioned this pull request Aug 4, 2024
Closed
@voipmonitor voipmonitor mentioned this pull request Jul 11, 2026
Closed
wenhow pushed a commit to wenhow/vllm that referenced this pull request Aug 5, 2026
@menogrey
[Doc] Refactor the DeepSeek-V3.2-Exp tutorial. (vllm-project#3871)
5f08e07 
### What this PR does / why we need it?
Refactor the DeepSeek-V3.2-Exp tutorial.

- vLLM version: v0.11.0
- vLLM main:
vllm-project@83f478b

---------

Signed-off-by: menogrey <1299267905@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@WoosukKwon WoosukKwon WoosukKwon approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@youkaichao @njhill @simon-mo @WoosukKwon