[CI/Build][Docker] Bump nvidia-cutlass-dsl to 4.6.0 and drop packaging workarounds by arpera · Pull Request #47442 · vllm-project/vllm

arpera · 2026-07-02T13:33:43Z

Purpose

In nvidia-cutlass-dsl wheel package there was a bug in using the same paths in filesystem by two different sub-wheels that resulted in race conditions during uv pip install process in vLLM. Original bug reports NVIDIA/cutlass#3259, NVIDIA/cutlass#3170 in cutlass repository. To overcome this issue vLLM used some temporary workarounds #43427, #45204.

The bug was resolved in a new release 4.6.0. Now it's time to rollback all of those temporary solutions in vLLM.

Test Plan

4.6.0 wheel does not use the same paths in filesystem compared to 4.5.2. You can check it this way:

# Verify that nvidia-cutlass-dsl 4.6.0 sub-wheels no longer collide on any file.
# This check downloads all wheels pulled in by `nvidia-cutlass-dsl[cu13]` WITHOUT
# installing them, and asserts that no file path appears in more than one wheel.
# It is deterministic: independent of install order, pip version, or installer.

check_version() {
  local version="$1"
  local dir="/tmp/cutlass_check_${version}"
  rm -rf "$dir" && mkdir -p "$dir" && cd "$dir" || return 1

  # 1. Download the full [cu13] closure (with deps), no install.
  pip download "nvidia-cutlass-dsl[cu13]==${version}" -d . >/dev/null

  # 2. List every file shipped by each wheel (skip directory entries and the
  #    per-wheel .dist-info/ metadata), tagged with the owning wheel name.
  for whl in *.whl; do
    unzip -Z1 "$whl" | grep -v '/$' | grep -v '\.dist-info/' | sed "s|^|$whl\t|"
  done > wheel_paths.txt

  # 3. A "collision" is any path owned by more than one wheel.
  cut -f2 wheel_paths.txt | sort | uniq -d > collisions.txt

  echo "=== nvidia-cutlass-dsl ${version}: $(wc -l < collisions.txt) colliding path(s) ==="
  if [ -s collisions.txt ]; then
    echo "FAIL: the following files are shipped by multiple wheels:"
    cat collisions.txt
    return 1
  else
    echo "PASS: no file is shipped by more than one wheel."
    return 0
  fi
}

# New version - PASS
check_version 4.6.0

# Old version - FAIL
check_version 4.5.2

Test Result

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Harry-Chen

I think many CI errors are due to changes in cutlass APIs. For example:

https://buildkite.com/vllm/ci/builds/76127/canvas?jid=019f2634-a182-4c11-b299-66551b1b903a&tab=output

(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231] EngineCore failed to start.
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231] Traceback (most recent call last):
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1200, in run_engine_core
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return func(*args, **kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 966, in __init__
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     super().__init__(
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 133, in __init__
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return func(*args, **kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 283, in _initialize_kv_caches
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 147, in determine_available_memory
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return self.collective_rpc("determine_available_memory")
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/uniproc_executor.py", line 92, in collective_rpc
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     result = run_method(self.driver_worker, method, args, kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/serial_utils.py", line 510, in run_method
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return func(*args, **kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return func(*args, **kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 468, in determine_available_memory
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     self.model_runner.profile_run()
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 6311, in profile_run
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     hidden_states, last_hidden_states = self._dummy_run(
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]                                         ^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return func(*args, **kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5971, in _dummy_run
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     outputs = self.model(
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]               ^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/cuda_graph.py", line 254, in __call__
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return self.runnable(*args, **kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return self._call_impl(*args, **kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return forward_call(*args, **kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/nemotron_h.py", line 936, in forward
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     hidden_states = self.model(
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]                     ^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 670, in __call__
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     output = self.aot_compiled_fn(self, *args, **kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/aot_compile.py", line 224, in __call__
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return self.fn(*args, **kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/nemotron_h.py", line 592, in forward
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     def forward(
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/caching.py", line 217, in __call__
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return self.optimized_call(*args, **kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "<string>", line 127, in execution_fn
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/cuda_graph.py", line 254, in __call__
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return self.runnable(*args, **kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/piecewise_backend.py", line 380, in __call__
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return range_entry.runnable(*args)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/standalone_compile.py", line 122, in __call__
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return self._compiled_fn(*args)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 1263, in _fn
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return fn(*args, **kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/aot_autograd.py", line 1200, in forward
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return compiled_fn(full_args)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 580, in runtime_wrapper
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     all_outs = call_func_at_runtime_with_args(
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/utils.py", line 138, in call_func_at_runtime_with_args
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     out = normalize_as_list(f(args))
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]                             ^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 2298, in __call__
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return self.compiled_fn(*args, **kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 783, in wrapper
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return compiled_fn(runtime_args)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 1011, in inner_fn
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     outs = compiled_fn(args)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/output_code.py", line 656, in __call__
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return self.current_callable(inputs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/torch/_inductor/utils.py", line 3401, in run
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     out = model(new_inputs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]           ^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/root/.cache/vllm/torch_compile_cache/torch_aot_compile/315878d4854b18a075e15f4d30c52675ce5d5c1c0897e85c8f092c6aebc70167/inductor_cache/dv/cdvvfdmmlb4wvn4mqboll2iuovdow4viys46mkd6bbaf7hyojg33.py", line 325, in call
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     buf11 = torch.ops.vllm.flashinfer_mm_fp4.default(buf2, reinterpret_tensor(arg5_1, (1344, 10304), (1, 1344), 0), buf10, reinterpret_tensor(arg6_1, (168, 10368), (1, 168), 0), arg7_1, torch.bfloat16, False, 'cute-dsl')
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 865, in __call__
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return self._op(*args, **kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/torch/_compile.py", line 54, in inner
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return disable_fn(*args, **kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 1263, in _fn
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return fn(*args, **kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 409, in __torch_dispatch__
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     res = func(*args, **kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]           ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 865, in __call__
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return self._op(*args, **kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py", line 347, in backend_impl
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     result = self._backend_fns[device_type](*args, **kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/torch/_compile.py", line 54, in inner
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return disable_fn(*args, **kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 1263, in _fn
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return fn(*args, **kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py", line 382, in wrapped_fn
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return fn(*args, **kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/flashinfer.py", line 582, in flashinfer_mm_fp4
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return flashinfer_mm_fp4_(
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/flashinfer/utils.py", line 1254, in wrapper
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return func(*args, **kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/flashinfer/api_logging.py", line 2333, in _auto_dump_wrapper
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return _inner(*args, **kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/flashinfer/gemm/gemm_base.py", line 6310, in mm_fp4
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     runner(inputs=inputs, tactic=tactic)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/flashinfer/autotuner.py", line 419, in __call__
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return self.forward(inputs, **kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/flashinfer/gemm/gemm_base.py", line 5751, in forward
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     compiled_gemm, _ = _compile_block_scaled_gemm(
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/flashinfer/gemm/gemm_base.py", line 4525, in _compile_block_scaled_gemm
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     compiled_gemm = cute.compile(
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]                     ^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/nvidia_cutlass_dsl/dsl_packages/cutlass/base_dsl/compiler.py", line 1250, in __call__
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return self._compile(*args, **kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/nvidia_cutlass_dsl/dsl_packages/cutlass/base_dsl/compiler.py", line 1400, in _compile
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return func._dsl_object._func(func, *args, **kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/nvidia_cutlass_dsl/dsl_packages/cutlass/base_dsl/dsl.py", line 2680, in _func
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return self._func_impl(funcBody, *args, **kwargs)
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/nvidia_cutlass_dsl/dsl_packages/cutlass/base_dsl/dsl.py", line 2693, in _func_impl
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     result = self.generate_mlir(
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]              ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/nvidia_cutlass_dsl/dsl_packages/cutlass/base_dsl/dsl.py", line 2301, in generate_mlir
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     jit_function = self.compile_and_cache(
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]                    ^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/nvidia_cutlass_dsl/dsl_packages/cutlass/cutlass_dsl/cutlass.py", line 909, in compile_and_cache
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return super().compile_and_cache(
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/nvidia_cutlass_dsl/dsl_packages/cutlass/base_dsl/dsl.py", line 2080, in compile_and_cache
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     fn = func_type(
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]          ^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/nvidia_cutlass_dsl/dsl_packages/cutlass/cutlass_dsl/cutlass.py", line 895, in _make_compiled_func
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     return TVMFFIJitCompiledFunctionWithKwargs(
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]   File "/usr/local/lib/python3.12/dist-packages/nvidia_cutlass_dsl/dsl_packages/cutlass/cutlass_dsl/tvm_ffi_provider.py", line 659, in __init__
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]     self._kwargs_wrapper = kwargs_wrapper.make_kwargs_wrapper(
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=17469) ERROR 07-03 05:13:37 [core.py:1231] TypeError: make_kwargs_wrapper() got an unexpected keyword argument 'map_dataclass_to_tuple'

Harry-Chen · 2026-07-06T04:46:58Z

There are still API compatibility issues:

(EngineCore pid=7858)   File "/usr/local/lib/python3.12/dist-packages/vllm/vllm_flash_attn/cute/utils.py", line 279, in <module>
(EngineCore pid=7858)     smem: cute.Tensor, thr_mma: cute.core.ThrMma, swapAB: cutlass.Constexpr[bool] = False
(EngineCore pid=7858)                                 ^^^^^^^^^^^^^^^^
(EngineCore pid=7858) AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma'

arpera · 2026-07-06T06:17:09Z

Yes, I know about these issues. They require some changes in flash_attn, so firstly I need to make a PR in flash_attn to remove deprecated cutlass API calls from flash_attn.

MatthewBonanni · 2026-07-07T14:56:32Z

Hi @arpera, please point your PR branch's vllm_flash_attn.cmake to your PR branch from vllm-project/flash-attention#157

Once that passes CI, I'll land vllm-project/flash-attention#157, then you can update vllm_flash_attn.cmake in this PR again

arpera · 2026-07-08T06:50:34Z

I tried to test this PR against updated flash-attn vllm-project/flash-attention#157 to see if flash-attn update to support cutlass 4.6.0 solves issues with failed CI jobs in this PR. The new CI run results of this PR shows that previous CI failures were resolved. Currently I have the following failed CI jobs but they are all not related to the change:

buildkite/ci/pr/amd-multi-modal-models-standard-4-other-plus-whis failed due to infrastructure error (503 from download.pytorch.org)
buildkite/ci/pr/amd-multi-modal-models-standard-4-other-plus-whis was cancelled not failed
buildkite/ci/pr/arm-cpu-test failed on main today, see CI run 76847
buildkite/ci/pr/e2e-core-1-gpu failed on main today, see CI run 76887
buildkite/ci/pr/extract-hidden-states-integration-2-gpus failed on main today, see CI run 76847

arpera · 2026-07-08T11:29:13Z

buildkite/ci/pr/arm-cpu-test -- failed on main, see CI run 76847
buildkite/ci/pr/docker-build-arm64-image -- infrastructure error
buildkite/ci/pr/docker-build-image -- infrastructure error

@MatthewBonanni, there is no more issues with CI tests in this PR after temporarily updating flash-attn with this patch vllm-project/flash-attention#157

MatthewBonanni · 2026-07-08T14:56:11Z

The docker-build-image failure means that none of the tests actually ran. Retrying that

mergify · 2026-07-08T19:09:56Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @arpera.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>

…for cutlass 4.6.0 CI validation Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>

…based on vllm-project#149) for cutlass 4.6.0 validation Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>

Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>

Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>

lengrongfu · 2026-07-19T10:34:26Z

RuntimeError: Worker failed with error 'module 'cutlass.cute.core' has no attribute 'ThrMma'', please check the stack trace above for the root cause

main branch have this error.

arpera · 2026-07-19T12:40:41Z

@lengrongfu, could you please create a bug report then?

arpera requested review from Harry-Chen, ZJY0516, khluu, tdoublep and vadiklyutiy as code owners July 2, 2026 13:33

claude Bot reviewed Jul 2, 2026

View reviewed changes

mergify Bot added ci/build nvidia labels Jul 2, 2026

github-project-automation Bot added this to NVIDIA Jul 2, 2026

vadiklyutiy approved these changes Jul 2, 2026

View reviewed changes

github-project-automation Bot moved this to Ready in NVIDIA Jul 2, 2026

vadiklyutiy added the ready ONLY add when PR is ready to merge/full CI is needed label Jul 2, 2026

vadiklyutiy enabled auto-merge (squash) July 2, 2026 20:58

Harry-Chen mentioned this pull request Jul 3, 2026

[Bugfix][SM120] Enable CUTLASS grouped GEMM (MoE) for SM_120/SM_121 consumer Blackwell #43814

Open

Harry-Chen requested changes Jul 3, 2026

View reviewed changes

github-project-automation Bot moved this from Ready to In review in NVIDIA Jul 3, 2026

vadiklyutiy disabled auto-merge July 3, 2026 10:51

arpera mentioned this pull request Jul 7, 2026

Fix compatibility with nvidia-cutlass-dsl 4.6.0 vllm-project/flash-attention#157

Merged

arpera requested review from LucasWilkinson and tlrmchlsmth as code owners July 7, 2026 15:28

arpera force-pushed the bump-cutlass-dsl-4.6.0 branch from 9f8169a to ce63a95 Compare July 7, 2026 15:33

depthfirst-app Bot reviewed Jul 7, 2026

View reviewed changes

Comment thread cmake/external_projects/vllm_flash_attn.cmake Outdated

mergify Bot added the needs-rebase label Jul 8, 2026

mergify Bot added the needs-rebase label Jul 16, 2026

arpera and others added 7 commits July 16, 2026 11:42

Bump nvidia-cutlass-dsl to 4.6.0 and drop packaging workarounds

a4c4233

Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>

bump apache-tvm-ffi

8777a64

Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>

[CI] Temporarily point vllm-flash-attn at arpera/flash-attention#157 …

616da36

…for cutlass 4.6.0 CI validation Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>

[CI] Point vllm-flash-attn at arpera/flash-attention#157 (68dfa7c, re…

2f19367

…based on vllm-project#149) for cutlass 4.6.0 validation Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>

Fix FA4 _flash_attn_fwd return value unpacking

7680ee0

Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>

Update GIT_TAG for vllm-flash-attn repository

29c7501

Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>

Add auto tolerance

e97052a

Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>

arpera force-pushed the bump-cutlass-dsl-4.6.0 branch from 937978e to e97052a Compare July 16, 2026 08:55

mergify Bot removed the needs-rebase label Jul 16, 2026

Harry-Chen approved these changes Jul 16, 2026

View reviewed changes

github-project-automation Bot moved this from In review to Ready in NVIDIA Jul 16, 2026

MatthewBonanni enabled auto-merge (squash) July 16, 2026 12:52

MatthewBonanni merged commit 7cd1d57 into vllm-project:main Jul 16, 2026
240 of 241 checks passed

github-project-automation Bot moved this from Ready to Done in NVIDIA Jul 16, 2026

MatthewBonanni mentioned this pull request Jul 17, 2026

[CI Failure]: test_score_api_queries_str_documents_text_content CI test fail on GB200 #48742

Closed

3 tasks

SmartAI mentioned this pull request Jul 17, 2026

[Bugfix] Require quack-kernels>=0.4.1 for cutlass-dsl 4.5+ compatibility #47571

Open

wzhao18 mentioned this pull request Jul 18, 2026

[Bug]: Minimax MSA - AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' #49005

Open

1 task

gau-nernst mentioned this pull request Jul 19, 2026

[Bugfix] Fix broken NVVM caused by CuteDSL 4.6.0 #49108

Merged

4 tasks

gau-nernst mentioned this pull request Jul 20, 2026

[CI] Add M3 MSA tests to CI #49143

Merged

4 tasks

volkanncicek mentioned this pull request Jul 26, 2026

[Bug]: v0.26.0 EngineCore fails to start on SM90+ — _warmup_ll_bf16_router_gemm hits AttributeError: module 'cutlass.cute.core' has no attribute 'ThrMma' (nvidia-cutlass-dsl 4.6.0) #49859

Open

andrewor14 mentioned this pull request Aug 3, 2026

Add _MXFP8GroupedMM variant with bf16 backward pytorch/ao#4649

Draft

This was referenced Aug 6, 2026

fix: unpack FA4 auxiliary outputs TomerBN-Nvidia/vllm#36

Closed

fix: unpack FA4 auxiliary outputs TomerBN-Nvidia/vllm#37

Merged

Navigation Menu

Uh oh!

Uh oh!

[CI/Build][Docker] Bump nvidia-cutlass-dsl to 4.6.0 and drop packaging workarounds - #47442

[CI/Build][Docker] Bump nvidia-cutlass-dsl to 4.6.0 and drop packaging workarounds#47442
MatthewBonanni merged 7 commits into
vllm-project:mainfrom
arpera:bump-cutlass-dsl-4.6.0

arpera commented Jul 2, 2026

Uh oh!

claude Bot left a comment

Uh oh!

Harry-Chen left a comment

Uh oh!

Harry-Chen commented Jul 6, 2026

Uh oh!

arpera commented Jul 6, 2026

Uh oh!

MatthewBonanni commented Jul 7, 2026

Uh oh!

Uh oh!

arpera commented Jul 8, 2026

Uh oh!

arpera commented Jul 8, 2026

Uh oh!

MatthewBonanni commented Jul 8, 2026

Uh oh!

mergify Bot commented Jul 8, 2026

Uh oh!

Uh oh!

lengrongfu commented Jul 19, 2026

Uh oh!

arpera commented Jul 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Uh oh!

Conversation

arpera commented Jul 2, 2026

Purpose

Test Plan

Test Result

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

Harry-Chen left a comment

Choose a reason for hiding this comment

Uh oh!

Harry-Chen commented Jul 6, 2026

Uh oh!

arpera commented Jul 6, 2026

Uh oh!

MatthewBonanni commented Jul 7, 2026

Uh oh!

Uh oh!

arpera commented Jul 8, 2026

Uh oh!

arpera commented Jul 8, 2026

Uh oh!

MatthewBonanni commented Jul 8, 2026

Uh oh!

mergify Bot commented Jul 8, 2026

Uh oh!

Uh oh!

lengrongfu commented Jul 19, 2026

Uh oh!

arpera commented Jul 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants