Fix undefined symbol: cutlass_moe_mm_sm100

[jasl]

Purpose

The fix is a little bit hacky, but it can make vLLM run on Thor

jasl@jasl-thor:~/Workspace$ uv run vllm
INFO 10-02 20:17:26 [__init__.py:216] Automatically detected platform cuda.
Traceback (most recent call last):
  File "/home/jasl/.venv/bin/vllm", line 4, in <module>
    from vllm.entrypoints.cli.main import main
  File "/home/jasl/Workspace/vllm/vllm/entrypoints/cli/__init__.py", line 3, in <module>
    from vllm.entrypoints.cli.benchmark.latency import BenchmarkLatencySubcommand
  File "/home/jasl/Workspace/vllm/vllm/entrypoints/cli/benchmark/latency.py", line 5, in <module>
    from vllm.benchmarks.latency import add_cli_args, main
  File "/home/jasl/Workspace/vllm/vllm/benchmarks/latency.py", line 18, in <module>
    from vllm.engine.arg_utils import EngineArgs
  File "/home/jasl/Workspace/vllm/vllm/engine/arg_utils.py", line 24, in <module>
    from vllm.config import (BlockSize, CacheConfig, CacheDType, CompilationConfig,
  File "/home/jasl/Workspace/vllm/vllm/config/__init__.py", line 12, in <module>
    from vllm.config.lora import LoRAConfig
  File "/home/jasl/Workspace/vllm/vllm/config/lora.py", line 14, in <module>
    from vllm.platforms import current_platform
  File "/home/jasl/Workspace/vllm/vllm/platforms/__init__.py", line 248, in __getattr__
    _current_platform = resolve_obj_by_qualname(
                        ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jasl/Workspace/vllm/vllm/utils/__init__.py", line 2680, in resolve_obj_by_qualname
    module = importlib.import_module(module_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jasl/Workspace/vllm/vllm/platforms/cuda.py", line 18, in <module>
    import vllm._C  # noqa
    ^^^^^^^^^^^^^^
ImportError: /home/jasl/Workspace/vllm/vllm/_C.abi3.so: undefined symbol: _Z20cutlass_moe_mm_sm100RN2at6TensorERKS0_S3_S3_S3_S3_S3_S3_S3_S3_bb

In addition, as @johnnynunez suggests, I add an essential patch to calculate free memory correctly for Nvidia UMA hardwares

Test Plan

Test Result

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

[jasl]

cc @johnnynunez
If you've worked a proper fix, please let me know

[gemini-code-assist]

Code Review

This pull request aims to fix a build issue for new hardware by enabling compilation of cutlass_moe_mm_sm100 for newer GPU architectures. While the change to CMakeLists.txt correctly adds the new architectures to the build, a related change in csrc/quantization/cutlass_w8a8/scaled_mm_entry.cu introduces a bug in the dispatch logic. It prevents the newly compiled kernel from being used on sm_120 and newer architectures, which would lead to a runtime error. My review identifies this critical contradiction and suggests reverting the problematic condition in the C++ code to align with the build configuration and ensure the fix works as intended.

[johnnynunez]

@mgoin @Aidyn-A LGTM!

[ProExpertProg]

Is this not fixed by #26077?

[johnnynunez]

Is this not fixed by #26077?

i didn't saw that PR, thank you...! Checking

[jasl]

OK, so I will test it later, and make a pure patch for the memory issue
@ProExpertProg is the memory fix commit look good to you?

[ProExpertProg]

I do actually have a question about the fix

[jasl]

Is this not fixed by #26077?

I have tested it, and that's not working for my case
( My testing branch: https://github.com/jasl/vllm/tree/jetson )

[ProExpertProg]

@jasl that fixes the issue for pre-10.0 platforms that don't support the function and adds a dummy implementation that just throws an error. You probably still need some of your fixes for the function to actually run on 10.0+ platforms. Or am I missing something?

[jasl]

@jasl that fixes the issue for pre-10.0 platforms that don't support the function and adds a dummy implementation that just throws an error. You probably still need some of your fixes for the function to actually run on 10.0+ platforms. Or am I missing something?

I don't know about how to write kernels. In my analysis, the root cause is somewhere leaking SM100 flag to SM110, yet SM110 shares many features with SM100.
Or we have to distinguish SM110 and change everywhere, I think that's make the work super complicated.

So I choose a simplest way to workaround it. compile the file, but restrict the code path

[ProExpertProg]

@jasl let's wait for the other PR to land and you can rebase on top and see what you still need to resolve your issue.

[jasl]

@jasl let's wait for the other PR to land and you can rebase on top and see what you still need to resolve your issue.

I guess the trouble is the 80 words per line rule. Could you force it to pass? Or should I break the URL into two lines?

[johnnynunez]

@jasl let's wait for the other PR to land and you can rebase on top and see what you still need to resolve your issue.

I guess the trouble is the 80 words per line rule. Could you force it to pass? Or should I break the URL into two lines?

Run: pre-commir run -a

[jasl]

@ProExpertProg Could you review again?

[ProExpertProg]

LGTM