[Bugfix] Fix mock.patch resolution failure for standalone_compile.FakeTensorMode on Python <= 3.10#37158
Conversation
dbari
requested review from
BoyuanFeng,
ProExpertProg,
youkaichao and
zou3519
as code owners
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request effectively resolves a critical bug that caused mock.patch to fail when resolving standalone_compile.FakeTensorMode on Python versions 3.10 and older. The issue stemmed from mock.patch incorrectly interpreting torch._inductor.standalone_compile as a wrapper function rather than the intended module. The fix, which involves using patch.object with an explicit reference to the module from sys.modules, correctly targets the FakeTensorMode attribute within the standalone_compile module. This ensures compatibility and prevents crashes in older Python environments, significantly improving the robustness of the compilation process. The added comments provide clear context for the change, enhancing code clarity and maintainability.
dbari
force-pushed
the
dbariamis/fix-faketensors-bug
branch
from
3a5c132 to
c68b827
Compare
|
Hi @dbari, the pre-commit checks have failed. Please run: uv pip install pre-commit
pre-commit install
pre-commit run --all-filesThen, commit the changes and push to your branch. For future commits, Tip Is
|
Signed-off-by: Dimitrios Bariamis <12195802+dbari@users.noreply.github.com>
dbari
force-pushed
the
dbariamis/fix-faketensors-bug
branch
from
c68b827 to
59a769f
Compare
|
Hi, could I please get a review for this one? Thanks! |
zou3519
left a comment
zou3519
left a comment
There was a problem hiding this comment.
lgtm, though I'm not sure how to prevent future regressions
…eTensorMode on Python <= 3.10 (vllm-project#37158) Signed-off-by: Dimitrios Bariamis <12195802+dbari@users.noreply.github.com> Co-authored-by: Dimitrios Bariamis <12195802+dbari@users.noreply.github.com>
I would consider this kind of collision to be a code smell in Pytorch, but I couldn't find any linting rule that would ensure it does not happen. While it's not a problem in the 99% of cases when working through standard python import machinery, it can cause problems with manual traversing of modules, or in this case patching. In any case, for vLLM it seems to be a one-off problem that is solved now, and Python 3.10 will be EOL in 2026-10, so in a few months patching will work for all supported python versions. Thanks for the review and merge! |
…eTensorMode on Python <= 3.10 (vllm-project#37158) Signed-off-by: Dimitrios Bariamis <12195802+dbari@users.noreply.github.com> Co-authored-by: Dimitrios Bariamis <12195802+dbari@users.noreply.github.com>
|
I still encounter this issue: I use python 3.10.12, installing vLLM by this: pip install vllm --extra-index-url https://download.pytorch.org/whl/cu129 |
|
@duongck this had fixed the problem for me. Can you post a reproducer, i.e. which vLLM git hash you're using and what command line / model? You could check if there are other instances of patching the same function, which would need to be fixed. |
|
I think the problem is the vLLM v0.18.0 release didn't include this patch |
…eTensorMode on Python <= 3.10 (vllm-project#37158) Signed-off-by: Dimitrios Bariamis <12195802+dbari@users.noreply.github.com> Co-authored-by: Dimitrios Bariamis <12195802+dbari@users.noreply.github.com>
…eTensorMode on Python <= 3.10 (vllm-project#37158) Signed-off-by: Dimitrios Bariamis <12195802+dbari@users.noreply.github.com> Co-authored-by: Dimitrios Bariamis <12195802+dbari@users.noreply.github.com>
…eTensorMode on Python <= 3.10 (vllm-project#37158) Signed-off-by: Dimitrios Bariamis <12195802+dbari@users.noreply.github.com> Co-authored-by: Dimitrios Bariamis <12195802+dbari@users.noreply.github.com>
…eTensorMode on Python <= 3.10 (vllm-project#37158) Signed-off-by: Dimitrios Bariamis <12195802+dbari@users.noreply.github.com> Co-authored-by: Dimitrios Bariamis <12195802+dbari@users.noreply.github.com>
3 participants
Purpose
This PR fixes an
AttributeError: <function standalone_compile at 0x7b8cf83c40d0> does not have the attribute 'FakeTensorMode'crash introduced by #36093 when running with Python <= 3.10.Stacktrace
The string-based
patch("torch._inductor.standalone_compile.FakeTensorMode", ...)resolvestorch._inductor.standalone_compileto the wrapper function defined intorch/_inductor/__init__.pyrather than the module, because in Python 3.10 themock._importerusesgetattr. In Python 3.11+,mock.patchswitched topkgutil.resolve_name(cpython#18544) which correctly resolves to the module.Test Plan
Test by loading a model before and after the fix in a Python 3.10 environment.
Test Result
Before:
After:
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.