-
-
Notifications
You must be signed in to change notification settings - Fork 11.8k
[torch.compile] Passing only necessary compilation config to inductor pass config #27041
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request refactors how compilation configuration is passed to the VllmInductorPass. Instead of passing a weak reference to the entire CompilationConfig, it now passes a new SimplifiedCompilationConfig object containing only the necessary fields. This is a great improvement for safety and decoupling, as it avoids passing potentially dangerous pointers (like static_forward_context) into the torch.compile process. However, I've identified a critical type mismatch in the new SimplifiedCompilationConfig dataclass that needs to be addressed.
…lm-project#27041) Summary: Pull Request resolved: vllm-project#27041 we should not pass the weakref to compilation_config, which include static_forward_context that will holds the pointers to the model layers (e.g. moe, attention), which is dangerous, as this will be passed as config to torch.compile Test Plan: local tests Differential Revision: D84790018
d161999 to
170f61e
Compare
…lm-project#27041) Summary: Pull Request resolved: vllm-project#27041 we should not pass the weakref to compilation_config, which include static_forward_context that will holds the pointers to the model layers (e.g. moe, attention), which is dangerous, as this will be passed as config to torch.compile Test Plan: local tests Differential Revision: D84790018 Signed-off-by: Lu Fang <[email protected]>
170f61e to
b8b8c71
Compare
…lm-project#27041) Summary: Pull Request resolved: vllm-project#27041 we should not pass the weakref to compilation_config, which include static_forward_context that will holds the pointers to the model layers (e.g. moe, attention), which is dangerous, as this will be passed as config to torch.compile Test Plan: local tests Differential Revision: D84790018 Signed-off-by: Lu Fang <[email protected]>
b8b8c71 to
31ef04c
Compare
…lm-project#27041) Summary: Pull Request resolved: vllm-project#27041 we should not pass the weakref to compilation_config, which include static_forward_context that will holds the pointers to the model layers (e.g. moe, attention), which is dangerous, as this will be passed as config to torch.compile Test Plan: local tests Differential Revision: D84790018 Signed-off-by: Lu Fang <[email protected]>
31ef04c to
cdb1bfd
Compare
…lm-project#27041) Summary: Pull Request resolved: vllm-project#27041 we should not pass the weakref to compilation_config, which include static_forward_context that will holds the pointers to the model layers (e.g. moe, attention), which is dangerous, as this will be passed as config to torch.compile Test Plan: local tests Differential Revision: D84790018 Signed-off-by: Lu Fang <[email protected]>
cdb1bfd to
948efa4
Compare
…lm-project#27041) Summary: Pull Request resolved: vllm-project#27041 we should not pass the weakref to compilation_config, which include static_forward_context that will holds the pointers to the model layers (e.g. moe, attention), which is dangerous, as this will be passed as config to torch.compile Test Plan: local tests Differential Revision: D84790018
948efa4 to
5493d3c
Compare
Signed-off-by: Lu Fang <[email protected]>
5493d3c to
1fd5983
Compare
| use_inductor_graph_partition=config.use_inductor_graph_partition, | ||
| compile_sizes=config.compile_sizes, | ||
| ) | ||
| self.pass_config = config.compilation_config.pass_config |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
later pass_config can be also moved.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can utilize it, but it will introduce duplicated attribute in config level, we can think of how to organize these config better in following PR. @zou3519
Signed-off-by: Lu Fang <[email protected]>
Signed-off-by: Lu Fang <[email protected]>
houseroad
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me
Signed-off-by: Lu Fang <[email protected]>
… pass config (vllm-project#27041) Signed-off-by: Lu Fang <[email protected]> Co-authored-by: Lucia (Lu) Fang <[email protected]>
… pass config (vllm-project#27041) Signed-off-by: Lu Fang <[email protected]> Co-authored-by: Lucia (Lu) Fang <[email protected]>
… pass config (vllm-project#27041) Signed-off-by: Lu Fang <[email protected]> Co-authored-by: Lucia (Lu) Fang <[email protected]> Signed-off-by: Alberto Perdomo <[email protected]>
… pass config (vllm-project#27041) Signed-off-by: Lu Fang <[email protected]> Co-authored-by: Lucia (Lu) Fang <[email protected]>
… pass config (vllm-project#27041) Signed-off-by: Lu Fang <[email protected]> Co-authored-by: Lucia (Lu) Fang <[email protected]> Signed-off-by: xuebwang-amd <[email protected]>
… pass config (vllm-project#27041) Signed-off-by: Lu Fang <[email protected]> Co-authored-by: Lucia (Lu) Fang <[email protected]> Signed-off-by: xuebwang-amd <[email protected]>
… pass config (vllm-project#27041) Signed-off-by: Lu Fang <[email protected]> Co-authored-by: Lucia (Lu) Fang <[email protected]> Signed-off-by: 0xrushi <[email protected]>
… pass config (vllm-project#27041) Signed-off-by: Lu Fang <[email protected]> Co-authored-by: Lucia (Lu) Fang <[email protected]> Signed-off-by: 0xrushi <[email protected]>
… pass config (vllm-project#27041) Signed-off-by: Lu Fang <[email protected]> Co-authored-by: Lucia (Lu) Fang <[email protected]>
… pass config (vllm-project#27041) Signed-off-by: Lu Fang <[email protected]> Co-authored-by: Lucia (Lu) Fang <[email protected]>
… pass config (vllm-project#27041) Signed-off-by: Lu Fang <[email protected]> Co-authored-by: Lucia (Lu) Fang <[email protected]>
Summary: we should not pass the weakref to compilation_config, which include static_forward_context that will holds the pointers to the model layers (e.g. moe, attention), which is dangerous, as this will be passed as config to torch.compile
Differential Revision: D84790018