Skip to content

Ignore '<string>' filepath#17330

Merged
vllm-bot merged 1 commit intovllm-project:mainfrom
zou3519:fix_string
Apr 29, 2025
Merged

Ignore '<string>' filepath#17330
vllm-bot merged 1 commit intovllm-project:mainfrom
zou3519:fix_string

Conversation

@zou3519
Copy link
Collaborator

@zou3519 zou3519 commented Apr 28, 2025 •
edited by github-actions bot
Loading

vLLM tries to get a list of files that Dynamo traced through. Some of these files can be equal to '<string>'. This may happen if the code object was dynamically generated, which does happen in some dataclasses cases.

This is one of the issues that was blocking the upgrade of vLLM from PyTorch 2.6 to PyTorch 2.7 (#16859). The reason why this happened was because PyTorch 2.7 now traces through more HuggingFace code, including some custom dataclasses.

Test Plan:

  • ran pytest -v tests/models/test_transformers.py -k test_models[meta-llama/Llama-3.2-1B-Instruct-transformers] with PyTorch >= 2.7

@zou3519
Ignore '<string>' filepath
ff78182 
vLLM tries to get a list of files that Dynamo traced through. Some of
these files can be equal to `'<string>'`. This may happen if the code
object was dynamically generated, which does happen in some dataclasses
cases.

This is one of the issues that was blocking the upgrade of vLLM from
PyTorch 2.6 to PyTorch 2.7 (vllm-project#16859). The reason why this happened was because
PyTorch 2.7 now traces through more HuggingFace code, including some
custom dataclasses.

Test Plan:
- ran `pytest -v tests/models/test_transformers.py -k test_models[meta-llama/Llama-3.2-1B-Instruct-transformers]` with PyTorch >= 2.7

Signed-off-by: rzou <zou3519@gmail.com>
@github-actions
Copy link

github-actions bot commented Apr 28, 2025

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

@zou3519 zou3519 marked this pull request as ready for review April 28, 2025 23:56
houseroad
houseroad approved these changes Apr 29, 2025
View reviewed changes
Copy link
Collaborator

@houseroad houseroad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wondering if we can add some unittest?

@houseroad houseroad added the ready ONLY add when PR is ready to merge/full CI is needed label Apr 29, 2025
@zou3519 zou3519 mentioned this pull request Apr 29, 2025
Merged
@vllm-bot vllm-bot merged commit 165cb56 into vllm-project:main Apr 29, 2025
56 of 61 checks passed
@zou3519
Copy link
Collaborator Author

zou3519 commented Apr 29, 2025 •
edited
Loading

Wondering if we can add some unittest?

I'm not sure how to actually reproduce this on PyTorch 2.6, this would involve getting Dynamo to run on a function with "<string>" co_filename. The test would be #16859

lk-chen pushed a commit to lk-chen/vllm that referenced this pull request Apr 29, 2025
@zou3519 @lk-chen
Ignore '<string>' filepath (vllm-project#17330)
5795b8f 
Signed-off-by: rzou <zou3519@gmail.com>
RichardoMrMu pushed a commit to RichardoMrMu/vllm that referenced this pull request May 12, 2025
@zou3519
Ignore '<string>' filepath (vllm-project#17330)
d3e56f0 
Signed-off-by: rzou <zou3519@gmail.com>
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
zzzyq pushed a commit to zzzyq/vllm that referenced this pull request May 24, 2025
@zou3519
Ignore '<string>' filepath (vllm-project#17330)
886c70e 
Signed-off-by: rzou <zou3519@gmail.com>
Signed-off-by: Yuqi Zhang <yuqizhang@google.com>
zou3519 added a commit to zou3519/vllm that referenced this pull request May 27, 2025
@zou3519
Fix PiecewiseCompileInterpreter
a23052b 
This PR fixes the other issue discovered in vllm-project#16859 when upgrading from
PyTorch 2.6 to PyTorch 2.7. I don't know why the code used to work in
PyTorch 2.6, but the explanation is:
- when we are running PiecewiseCompileInterpreter, we end up doing
  FakeTensor propagation
- FakeTensor propagation requires `enable_python_dispatcher` to work.
  The mechanism is that some of our "C++ implementations" for
  operations, like matmul, force specialization of dynamic shapes.
  torch.compile works around this by replacing PyTorch's "C++
  implementation" for matmul with a python-based implementation for
  matmul that does not force specialization.

Test Plan:
- Ran `pytest -v tests/models/test_transformers.py -k test_models[meta-llama/Llama-3.2-1B-Instruct-transformers]`
  with PyTorch >= 2.7 and vllm-project#17330, verified that the test passes.

Signed-off-by: rzou <zou3519@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tlrmchlsmth tlrmchlsmth tlrmchlsmth approved these changes

@houseroad houseroad houseroad approved these changes

@mgoin mgoin Awaiting requested review from mgoin

@youkaichao youkaichao Awaiting requested review from youkaichao youkaichao is a code owner

+1 more reviewer

@huydhn huydhn huydhn approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@zou3519 @huydhn @tlrmchlsmth @houseroad @vllm-bot