Skip to content

Use standalone_compile by default in torch >= 2.8.0#18846

Merged
houseroad merged 1 commit intovllm-project:mainfrom
zou3519:default_standalone_compile
May 29, 2025
Merged

Use standalone_compile by default in torch >= 2.8.0#18846
houseroad merged 1 commit intovllm-project:mainfrom
zou3519:default_standalone_compile

Conversation

@zou3519
Copy link
Copy Markdown
Collaborator

@zou3519 zou3519 commented May 28, 2025 •
edited by github-actions bot
Loading

This includes the current PyTorch nightlies.

Test Plan:

@github-actions
Copy link
Copy Markdown

github-actions bot commented May 28, 2025

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

@zou3519 zou3519 added the ready ONLY add when PR is ready to merge/full CI is needed label May 28, 2025
@zou3519 zou3519 marked this pull request as ready for review May 28, 2025 17:11
@zou3519 zou3519 requested review from houseroad and youkaichao May 29, 2025 00:02
houseroad
houseroad reviewed May 29, 2025
View reviewed changes
vllm/envs.py Outdated
Copy link
Copy Markdown
Collaborator

@houseroad houseroad May 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering why there is TEST in the name?

Also do we want to turn on the flag by default?

Copy link
Copy Markdown
Collaborator Author

@zou3519 zou3519 May 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The flag was already named VLLM_TEST_STANDALONE_COMPILE in a previous PR. I can rename it if you want, to something like VLLM_USE_STANDALONE_COMPILE, but I was following the naming of VLLM_TEST_DYNAMO_FULLGRAPH_CAPTURE.

This PR turns the flag on by default.

Copy link
Copy Markdown
Collaborator

@houseroad houseroad May 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, let's use something like VLLM_USE_STANDALONE_COMPILE? I feel VLLM_TEST_STANDALONE_COMPILE is misleading, like some flag used in the test only.

Copy link
Copy Markdown
Collaborator Author

@zou3519 zou3519 May 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

@zou3519 zou3519 force-pushed the default_standalone_compile branch from 0c6e52b to d8e7241 Compare May 29, 2025 12:05
@zou3519 zou3519 requested a review from houseroad May 29, 2025 12:15
houseroad
houseroad approved these changes May 29, 2025
View reviewed changes
Copy link
Copy Markdown
Collaborator

@houseroad houseroad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks for updating the env var name.

vllm/envs.py Outdated
Copy link
Copy Markdown
Collaborator

@houseroad houseroad May 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does internal mean meta internal? If so, it's not internal anymore :-)

Copy link
Copy Markdown
Collaborator Author

@zou3519 zou3519 May 29, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

internal does not mean "meta internal", it means "not public API". I'll drop the "internal" word.

@zou3519 zou3519 force-pushed the default_standalone_compile branch from d8e7241 to e239a1a Compare May 29, 2025 12:31
@zou3519
Use standalone_compile by default in torch >= 2.8.0
ec26191 
This includes the current PyTorch nightlies.

It also renames the VLLM_TEST_STANDALONE_COMPILE envvar to
VLLM_USE_STANDALONE_COMPILE to make it clearer.

Test Plan:
- in vllm-project#17057, I verified that running
  https://gist.github.com/zou3519/aebb622714e80f4cd4c369472f2372cd with
  or without VLLM_TEST_STANDALONE_COMPILE resulted in Inductor producing
  the same exact output code (via tlparse). I did this for the
  cold-start case and the warm start case.
- there are vllm x torch nightly tests in CI that I will trigger on this
  PR.

Signed-off-by: rzou <zou3519@gmail.com>
@zou3519 zou3519 force-pushed the default_standalone_compile branch from e239a1a to ec26191 Compare May 29, 2025 12:31
@zou3519 zou3519 mentioned this pull request May 29, 2025
Closed
1 task
@houseroad houseroad merged commit a521ef0 into vllm-project:main May 29, 2025
63 checks passed
@drisspg
Copy link
Copy Markdown
Contributor

drisspg commented May 30, 2025

Cc @jerryzh168 we shud make sure that we don't land the change disabling compile cache for 2.8+

@zou3519
Copy link
Copy Markdown
Collaborator Author

zou3519 commented May 30, 2025

Btw, I thought I changed the commit message but looks like I forgot to or it didn't update the PR body. The envvar was renamed to VLLM_USE_STANDALONE_COMPILE in this PR.

amitm02 pushed a commit to amitm02/vllm that referenced this pull request Jun 1, 2025
@zou3519 @amitm02
Use standalone_compile by default in torch >= 2.8.0 (vllm-project#18846)
ec3b0f4 
Signed-off-by: rzou <zou3519@gmail.com>
Signed-off-by: amit <amit.man@gmail.com>
amitm02 pushed a commit to amitm02/vllm that referenced this pull request Jun 1, 2025
@zou3519 @amitm02
Use standalone_compile by default in torch >= 2.8.0 (vllm-project#18846)
e5ae9ce 
Signed-off-by: rzou <zou3519@gmail.com>
Signed-off-by: amit <amit.man@gmail.com>
jikunshang pushed a commit to jikunshang/vllm that referenced this pull request Jun 6, 2025
@zou3519 @jikunshang
Use standalone_compile by default in torch >= 2.8.0 (vllm-project#18846)
50fdcbc 
Signed-off-by: rzou <zou3519@gmail.com>
@BoyuanFeng BoyuanFeng mentioned this pull request Jun 13, 2025
Merged
chaojun-zhang pushed a commit to chaojun-zhang/vllm that referenced this pull request Jun 17, 2025
@jikunshang @zou3519
fix torch.compile issues (vllm-project#204)
59509de 
* use 2025.1.1 instead (vllm-project#196)

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>

* Use standalone_compile by default in torch >= 2.8.0 (vllm-project#18846)

Signed-off-by: rzou <zou3519@gmail.com>

* fix xpu compile issue

---------

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Signed-off-by: rzou <zou3519@gmail.com>
Co-authored-by: Richard Zou <zou3519@users.noreply.github.com>
chaojun-zhang pushed a commit to chaojun-zhang/vllm that referenced this pull request Jun 17, 2025
@jikunshang @zou3519 @chaojun-zhang
fix torch.compile issues (vllm-project#204)
117a2ed 
* use 2025.1.1 instead (vllm-project#196)

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>

* Use standalone_compile by default in torch >= 2.8.0 (vllm-project#18846)

Signed-off-by: rzou <zou3519@gmail.com>

* fix xpu compile issue

---------

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Signed-off-by: rzou <zou3519@gmail.com>
Co-authored-by: Richard Zou <zou3519@users.noreply.github.com>
jikunshang added a commit to jikunshang/vllm that referenced this pull request Jun 18, 2025
@jikunshang @zou3519 @yma11
fix torch.compile issues (vllm-project#204)
d3f788c 
* use 2025.1.1 instead (vllm-project#196)

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>

* Use standalone_compile by default in torch >= 2.8.0 (vllm-project#18846)

Signed-off-by: rzou <zou3519@gmail.com>

* fix xpu compile issue

---------

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Signed-off-by: rzou <zou3519@gmail.com>
Co-authored-by: Richard Zou <zou3519@users.noreply.github.com>
zhenwei-intel pushed a commit to zhenwei-intel/vllm that referenced this pull request Jun 23, 2025
@jikunshang @zou3519 @yma11
fix torch.compile issues (vllm-project#204)
f56825e 
* use 2025.1.1 instead (vllm-project#196)

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>

* Use standalone_compile by default in torch >= 2.8.0 (vllm-project#18846)

Signed-off-by: rzou <zou3519@gmail.com>

* fix xpu compile issue

---------

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Signed-off-by: rzou <zou3519@gmail.com>
Co-authored-by: Richard Zou <zou3519@users.noreply.github.com>
jikunshang added a commit to jikunshang/vllm that referenced this pull request Jun 24, 2025
@jikunshang @zou3519 @yma11
fix torch.compile issues (vllm-project#204)
8915100 
* use 2025.1.1 instead (vllm-project#196)

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>

* Use standalone_compile by default in torch >= 2.8.0 (vllm-project#18846)

Signed-off-by: rzou <zou3519@gmail.com>

* fix xpu compile issue

---------

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Signed-off-by: rzou <zou3519@gmail.com>
Co-authored-by: Richard Zou <zou3519@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@houseroad houseroad houseroad approved these changes

@youkaichao youkaichao Awaiting requested review from youkaichao youkaichao is a code owner

Assignees

No one assigned

Labels

ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@zou3519 @drisspg @houseroad