[2/N][torch.compile] make compilation cfg part of vllm cfg #10383

youkaichao · 2024-11-15T23:36:38Z

continue of #10237

move vllm.compilation.config into vllm.config

it is still controlled by the env var, but in the core code, no one should read the env var.

TODO:

move the env var to cli arg.
remove compilation context, initialize all config fields during init, rather than during model forward time

Signed-off-by: youkaichao <[email protected]>

github-actions · 2024-11-15T23:36:52Z

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

Add ready label to the PR
Enable auto-merge.

🚀

Signed-off-by: youkaichao <[email protected]>

youkaichao · 2024-11-16T00:55:39Z

vllm/config.py

+    capture_sizes: List[int] = PrivateAttr
+
+    def model_post_init(self, __context: Any) -> None:
+        self.level = envs.VLLM_TORCH_COMPILE_LEVEL


currently it is still read from env var so that api server can set it. later we will move it to the cli args.

Signed-off-by: youkaichao <[email protected]>

WoosukKwon · 2024-11-16T23:48:53Z

vllm/plugins/__init__.py

A dumb question: How does it work with TP? It seems we relying on more global variables that are used by the model executor. Does it add any new assumption or restriction to the TP implementation?

It works with TP naturally, because every TP worker (process) has its own model executor and plugins module.

WoosukKwon

LGTM. Please fix the CI error before merge.

youkaichao · 2024-11-17T02:02:33Z

LGTM. Please fix the CI error before merge.

the ci error is huggingface timeout. now it passes.

…ect#10383) Signed-off-by: youkaichao <[email protected]> Signed-off-by: Linkun Chen <[email protected]>

ProExpertProg

I know this was merged but I had a few questions

vllm/model_executor/custom_op.py

vllm/compilation/backends.py

vllm/v1/worker/gpu_model_runner.py

…ect#10383) Signed-off-by: youkaichao <[email protected]>

…ect#10383) Signed-off-by: youkaichao <[email protected]> Signed-off-by: Maxime Fournioux <[email protected]>

…ect#10383) Signed-off-by: youkaichao <[email protected]> Signed-off-by: rickyx <[email protected]>

…ect#10383) Signed-off-by: youkaichao <[email protected]> Signed-off-by: Tyler Michael Smith <[email protected]>

…ect#10383) Signed-off-by: youkaichao <[email protected]>

youkaichao added 4 commits November 15, 2024 14:49

move levels to config

b05329b

Signed-off-by: youkaichao <[email protected]>

remove compilation.config

9a07bc6

Signed-off-by: youkaichao <[email protected]>

add check_and_update_config

4e14ef9

Signed-off-by: youkaichao <[email protected]>

rename to level

21417e5

Signed-off-by: youkaichao <[email protected]>

youkaichao marked this pull request as draft November 15, 2024 23:39

youkaichao added 13 commits November 15, 2024 15:58

move to config

3fb9cad

Signed-off-by: youkaichao <[email protected]>

move custom op to config

8382a27

Signed-off-by: youkaichao <[email protected]>

fix circular import

8e26087

Signed-off-by: youkaichao <[email protected]>

fix

5aa5c34

Signed-off-by: youkaichao <[email protected]>

fix

d4278cf

Signed-off-by: youkaichao <[email protected]>

fix

656cabd

Signed-off-by: youkaichao <[email protected]>

fix

c93ba92

Signed-off-by: youkaichao <[email protected]>

fix level

16c430c

Signed-off-by: youkaichao <[email protected]>

fix tests

186d562

Signed-off-by: youkaichao <[email protected]>

fix

aa88b3e

Signed-off-by: youkaichao <[email protected]>

fix

d2bdcbc

Signed-off-by: youkaichao <[email protected]>

fix

3e29fea

Signed-off-by: youkaichao <[email protected]>

update

49907fe

Signed-off-by: youkaichao <[email protected]>

youkaichao commented Nov 16, 2024

View reviewed changes

youkaichao added 3 commits November 15, 2024 17:05

fix init

0d3b058

Signed-off-by: youkaichao <[email protected]>

fix init

f176234

Signed-off-by: youkaichao <[email protected]>

fix

3a1a3fd

Signed-off-by: youkaichao <[email protected]>

youkaichao marked this pull request as ready for review November 16, 2024 01:11

youkaichao changed the title ~~[2/N][torch.compile] move config out of compilation~~ [2/N][torch.compile] make compilation cfg part of vllm cfg Nov 16, 2024

youkaichao added 3 commits November 15, 2024 17:17

fix

b48ee55

Signed-off-by: youkaichao <[email protected]>

fix import

fec2702

Signed-off-by: youkaichao <[email protected]>

fix tests

99cfa24

Signed-off-by: youkaichao <[email protected]>

WoosukKwon reviewed Nov 16, 2024

View reviewed changes

WoosukKwon approved these changes Nov 17, 2024

View reviewed changes

youkaichao added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 17, 2024

youkaichao merged commit 4fd9375 into vllm-project:main Nov 17, 2024
60 of 62 checks passed

youkaichao deleted the compile_rollout branch November 17, 2024 02:02

This was referenced Nov 17, 2024

[3/N][torch.compile] consolidate custom op logging #10399

Merged

[4/N][torch.compile] clean up set_torch_compile_backend #10401

Merged

lk-chen pushed a commit to lk-chen/vllm that referenced this pull request Nov 18, 2024

[2/N][torch.compile] make compilation cfg part of vllm cfg (vllm-proj…

578e482

…ect#10383) Signed-off-by: youkaichao <[email protected]> Signed-off-by: Linkun Chen <[email protected]>

youkaichao mentioned this pull request Nov 18, 2024

[ci][bugfix] fix kernel tests #10431

Merged

ProExpertProg reviewed Nov 18, 2024

View reviewed changes

vllm/model_executor/custom_op.py Show resolved Hide resolved

vllm/compilation/backends.py Show resolved Hide resolved

vllm/v1/worker/gpu_model_runner.py Show resolved Hide resolved

coolkp pushed a commit to coolkp/vllm that referenced this pull request Nov 20, 2024

[2/N][torch.compile] make compilation cfg part of vllm cfg (vllm-proj…

4575dec

…ect#10383) Signed-off-by: youkaichao <[email protected]>

KuntaiDu pushed a commit to KuntaiDu/vllm that referenced this pull request Nov 20, 2024

[2/N][torch.compile] make compilation cfg part of vllm cfg (vllm-proj…

d611f96

…ect#10383) Signed-off-by: youkaichao <[email protected]>

mfournioux pushed a commit to mfournioux/vllm that referenced this pull request Nov 20, 2024

[2/N][torch.compile] make compilation cfg part of vllm cfg (vllm-proj…

3f24b9f

…ect#10383) Signed-off-by: youkaichao <[email protected]> Signed-off-by: Maxime Fournioux <[email protected]>

rickyyx pushed a commit to rickyyx/vllm that referenced this pull request Nov 20, 2024

[2/N][torch.compile] make compilation cfg part of vllm cfg (vllm-proj…

30c9dcc

…ect#10383) Signed-off-by: youkaichao <[email protected]> Signed-off-by: rickyx <[email protected]>

tlrmchlsmth pushed a commit to neuralmagic/vllm that referenced this pull request Nov 23, 2024

[2/N][torch.compile] make compilation cfg part of vllm cfg (vllm-proj…

127c5b9

…ect#10383) Signed-off-by: youkaichao <[email protected]> Signed-off-by: Tyler Michael Smith <[email protected]>

prashantgupta24 pushed a commit to opendatahub-io/vllm that referenced this pull request Dec 3, 2024

[2/N][torch.compile] make compilation cfg part of vllm cfg (vllm-proj…

11d2bbc

…ect#10383) Signed-off-by: youkaichao <[email protected]>

sleepwalker2017 pushed a commit to sleepwalker2017/vllm that referenced this pull request Dec 13, 2024

[2/N][torch.compile] make compilation cfg part of vllm cfg (vllm-proj…

53e2c0d

…ect#10383) Signed-off-by: youkaichao <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[2/N][torch.compile] make compilation cfg part of vllm cfg #10383

[2/N][torch.compile] make compilation cfg part of vllm cfg #10383

youkaichao commented Nov 15, 2024 •

edited by github-actions bot

Loading

github-actions bot commented Nov 15, 2024

youkaichao Nov 16, 2024

WoosukKwon Nov 16, 2024

youkaichao Nov 17, 2024

WoosukKwon left a comment

youkaichao commented Nov 17, 2024

ProExpertProg left a comment

[2/N][torch.compile] make compilation cfg part of vllm cfg #10383

[2/N][torch.compile] make compilation cfg part of vllm cfg #10383

Conversation

youkaichao commented Nov 15, 2024 • edited by github-actions bot Loading

github-actions bot commented Nov 15, 2024

youkaichao Nov 16, 2024

Choose a reason for hiding this comment

WoosukKwon Nov 16, 2024

Choose a reason for hiding this comment

youkaichao Nov 17, 2024

Choose a reason for hiding this comment

WoosukKwon left a comment

Choose a reason for hiding this comment

youkaichao commented Nov 17, 2024

ProExpertProg left a comment

Choose a reason for hiding this comment

youkaichao commented Nov 15, 2024 •

edited by github-actions bot

Loading