Skip to content

[revert] [Build] Fix QuTLASS compilation with PyTorch 2.13 #61844

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cleonard530 wants to merge 8 commits into
base: main
Choose a base branch
from
+8 −1
Open

[revert] [Build] Fix QuTLASS compilation with PyTorch 2.13 #61844

Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
a0fd642
revert qutlass to use TORCH_TARGET_VERSION again
cleonard530 Oct 15, 2026
aa0fac9
add check for qutlass stable ABI support
cleonard530 Oct 15, 2026
967ea9f
Merge branch 'main' into add_torch_target_version_tag_back_to_qutlass
cleonard530 Oct 15, 2026
3189a2c
Merge branch 'main' into add_torch_target_version_tag_back_to_qutlass
Harry-Chen Oct 16, 2026
4e19ec8
Merge branch 'main' into add_torch_target_version_tag_back_to_qutlass
Harry-Chen Oct 16, 2026
bef0d64
Merge branch 'main' into add_torch_target_version_tag_back_to_qutlass
Harry-Chen Oct 18, 2026
bccef1a
Merge branch 'main' into add_torch_target_version_tag_back_to_qutlass
mergify[bot] Oct 19, 2026
88384b4
Merge branch 'main' into add_torch_target_version_tag_back_to_qutlass
mergify[bot] Oct 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion cmake/external_projects/qutlass.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,13 @@ endif()
if(NOT qutlass_SOURCE_DIR)
message(FATAL_ERROR "[QUTLASS] source directory could not be resolved.")
endif()

if(NOT EXISTS "${qutlass_SOURCE_DIR}/qutlass/csrc/include/registration.h")
message(FATAL_ERROR
"[QUTLASS] Stable ABI support was not found in the Qutlass source. "
"Please update your Qutlass package.")
endif()

message(STATUS "[QUTLASS] QuTLASS is available at ${qutlass_SOURCE_DIR}")

if(${CMAKE_CUDA_COMPILER_VERSION} VERSION_GREATER_EQUAL 13.0)
Expand Down Expand Up @@ -142,9 +149,9 @@ if(${CMAKE_CUDA_COMPILER_VERSION} VERSION_GREATER_EQUAL 12.8 AND QUTLASS_ARCHS)

target_compile_definitions(_qutlass_C PRIVATE
QUTLASS_MINIMAL_BUILD=1
QUTLASS_DISABLE_PYBIND=1
TARGET_CUDA_ARCH=${QUTLASS_TARGET_CC}
CUTLASS_ENABLE_DIRECT_CUDA_DRIVER_CALL=1
TORCH_TARGET_VERSION=0x020B000000000000ULL

@yewentao256 yewentao256 Oct 19, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
TORCH_TARGET_VERSION=0x020B000000000000ULL
TORCH_TARGET_VERSION=0x020D000000000000ULL

@Harry-Chen I think we should pin 2.13 now to avoid a breaking change based on current situation
Fully agree that should have some more checks here to avoid changes of the code by accident

@Harry-Chen Harry-Chen Oct 19, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still don't think setting to 2.11 will be a breaking change, because as long as the code compiles with 2.11, they are compatible with libtorch 2.11+, given the nature of how libtorch stable ABI works. Nothing changes for the user IMHO, until they decide that they will try to reuse a .so file from different commits / release with build system changes, which is not desired to work by design.

@yewentao256 yewentao256 Oct 19, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might still break some user specific env, eg. that's why we have #58173 to fix for our customer.

@yewentao256 yewentao256 Oct 19, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't know if the changes will bring more bugs here as current main built based on 2.13 close to one month

@Harry-Chen Harry-Chen Oct 19, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I and @cleonard530 think #58173 was because a stale version of cutlass was checked out in the source tree. And this PR exactly adds a check against this situation, not to work around it by disabling stable ABI checks: https://github.com/vllm-project/vllm/pull/61844/changes#diff-cc0576febbd89620d5e7a9b3dc8ff346e44df0b4e2c8a6ecaba019668c05fe8cR55-R60

I did not know about the specific environment, but the release pipeline never failed before or after #58173:

image

We don't know if the changes will bring more bugs here as current main built based on 2.13 close to one month

That flag does not change the version qutlass is built against, but just works as a feature gate. The APIs you can use with 2.11 is a strict subset of what you can use with 2.13. But it is always built against the latest pytorch headers, 2.14 today.

@yewentao256 yewentao256 Oct 19, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feature gate doesn't mean the code is exactly same https://github.com/pytorch/pytorch/blob/2b3ec34829036a65cd9d1398ea72a0167dc37470/torch/csrc/stable/macros.h#L114, for instance

My concern is about the change in compiled behavior, while targeting 2.11 should theoretically be compatible, it doesn't necessarily produce the same binary behavior as targeting 2.13.

Since the lower target hasn't been well tested in the current setup, I'd prefer either keeping 2.13 for this fix.
(I don't think just standard CI runs are enough, as users and customers have many specific envs, you never know what will happen in their situation)

USE_CUDA)

set_property(SOURCE ${QUTLASS_SOURCES} APPEND PROPERTY COMPILE_OPTIONS
Expand Down
Loading