an illegal instruction was encountered when run moe fp4 on spark

[qiyuxinlin]

I am attempting to run FP4 MoE on spark using vLLM and SGLang, with flashinfer_cutlass selected as the MoE operator. No errors occur during the capture phase, but the following error occurs randomly during replay:

CUDA error: an illegal instruction was encountered
Search for `cudaErrorIllegalInstruction' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.

[yzh119]

Hi @qiyuxinlin would you mind trying #2020 and see if the error still exists?

[qiyuxinlin]

After reinstalling flashinfer inside the Spark container provided by sglang, I encountered the following issue:

[2025-11-10 03:11:34] Prefill batch, #new-seq: 1, #new-token: 6, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, 
[2025-11-10 03:11:34] Scheduler hit an exception: Traceback (most recent call last):
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2815, in run_scheduler_process
    scheduler.event_loop_overlap()
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 995, in event_loop_overlap
    batch_result = self.run_batch(batch)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 1986, in run_batch
    batch_result = self.model_worker.forward_batch_generation(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker.py", line 371, in forward_batch_generation
    logits_output, can_run_cuda_graph = self.model_runner.forward(
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 2147, in forward
    output = self._forward_raw(
             ^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 2204, in _forward_raw
    ret = self.forward_extend(
          ^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 2092, in forward_extend
    return self.model.forward(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/models/glm4_moe.py", line 847, in forward
    hidden_states = self.model(
                    ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/models/glm4_moe.py", line 772, in forward
    hidden_states, residual = layer(
                              ^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/models/glm4_moe.py", line 677, in forward
    hidden_states = self.self_attn(
                    ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/models/glm4_moe.py", line 312, in forward
    s = self.forward_prepare(
        ^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/models/glm4_moe.py", line 290, in forward_prepare
    qkv, _ = self.qkv_proj(hidden_states)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/layers/linear.py", line 428, in forward
    output_parallel = self.quant_method.apply(self, input_, bias)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/layers/quantization/modelopt_quant.py", line 1059, in apply
    out = fp4_gemm(
          ^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/flashinfer/utils.py", line 1123, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/flashinfer/gemm/gemm_base.py", line 1994, in mm_fp4
    gemm_module.cutlass_fp4_gemm(
  File "/usr/local/lib/python3.12/dist-packages/flashinfer/gemm/gemm_base.py", line 486, in cutlass_fp4_gemm
    fp4_runner(inputs=inputs, tactic=tactic)
  File "/usr/local/lib/python3.12/dist-packages/flashinfer/autotuner.py", line 217, in __call__
    return self.forward(inputs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/flashinfer/gemm/gemm_base.py", line 428, in forward
    module.fp4_gemm(
  File "python/tvm_ffi/cython/function.pxi", line 814, in core.Function.__call__
RuntimeError: [FP4 gemm Runner] Failed to run cutlass FP4 gemm on sm120. Error: Error Internal

I noticed that sglang's Spark Dockerfile doesn't perform any special operations on flashinfer (https://github.com/yvbbrjdr/sglang/blob/spark/docker/Dockerfile.spark). Should I set some environment variables?

[qiyuxinlin]

I have tried by vllm:

TORCH_USE_CUDA_DSA=1  VLLM_USE_FLASHINFER_MOE_FP4=1 vllm serve /mnt/data/models/GLM-4.5-Air-nvfp4 --gpu-memory-utilization 0.75 --max-num-batched-tokens 4096 --max-cudagraph-capture-size 4 --enable-chunked-prefill -O3 --max-model-len 80000 --safetensors-load-strategy eager

(APIServer pid=107197) INFO 11-10 16:51:40 [api_server.py:1967] vLLM API server version 0.11.1rc6.dev86+g4022a9d27.d20251104
(APIServer pid=107197) INFO 11-10 16:51:40 [utils.py:253] non-default args: {'model_tag': '/mnt/data/models/GLM-4.5-Air-nvfp4', 'model': '/mnt/data/models/GLM-4.5-Air-nvfp4', 'max_model_len': 80000, 'safetensors_load_strategy': 'eager', 'gpu_memory_utilization': 0.75, 'max_num_batched_tokens': 4096, 'enable_chunked_prefill': True, 'max_cudagraph_capture_size': 4, 'compilation_config': {'level': None, 'mode': 3, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': [], 'splitting_ops': None, 'use_inductor': None, 'compile_sizes': None, 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': None, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': None, 'cudagraph_copy_inputs': False, 'full_cuda_graph': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_cudagraph_capture_size': None, 'local_cache_dir': None}}
(APIServer pid=107197) INFO 11-10 16:51:40 [model.py:657] Resolved architecture: Glm4MoeForCausalLM
(APIServer pid=107197) INFO 11-10 16:51:40 [model.py:1752] Using max model len 80000
(APIServer pid=107197) INFO 11-10 16:51:40 [scheduler.py:226] Chunked prefill is enabled with max_num_batched_tokens=4096.
(APIServer pid=107197) WARNING 11-10 16:51:40 [modelopt.py:707] Detected ModelOpt NVFP4 checkpoint. Please note that the format is experimental and could change in future.
(EngineCore_DP0 pid=107470) INFO 11-10 16:51:44 [core.py:93] Initializing a V1 LLM engine (v0.11.1rc6.dev86+g4022a9d27.d20251104) with config: model='/mnt/data/models/GLM-4.5-Air-nvfp4', speculative_config=None, tokenizer='/mnt/data/models/GLM-4.5-Air-nvfp4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=80000, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=modelopt_fp4, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=/mnt/data/models/GLM-4.5-Air-nvfp4, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={'level': None, 'mode': 3, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'use_inductor': None, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_cudagraph_capture_size': 4, 'local_cache_dir': None}
(EngineCore_DP0 pid=107470) /home/azure/venvs/vllm/lib/python3.12/site-packages/torch/cuda/__init__.py:283: UserWarning: 
(EngineCore_DP0 pid=107470)     Found GPU0 NVIDIA GB10 which is of cuda capability 12.1.
(EngineCore_DP0 pid=107470)     Minimum and Maximum cuda capability supported by this version of PyTorch is
(EngineCore_DP0 pid=107470)     (8.0) - (12.0)
(EngineCore_DP0 pid=107470)     
(EngineCore_DP0 pid=107470)   warnings.warn(
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
(EngineCore_DP0 pid=107470) INFO 11-10 16:51:44 [parallel_state.py:1325] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
(EngineCore_DP0 pid=107470) INFO 11-10 16:51:44 [gpu_model_runner.py:2932] Starting to load model /mnt/data/models/GLM-4.5-Air-nvfp4...
(EngineCore_DP0 pid=107470) INFO 11-10 16:51:45 [modelopt.py:968] Using flashinfer-cutlass for NVFP4 GEMM
(EngineCore_DP0 pid=107470) INFO 11-10 16:51:45 [cuda.py:409] Using Flash Attention backend.
(EngineCore_DP0 pid=107470) INFO 11-10 16:51:45 [nvfp4_moe_support.py:38] Using FlashInfer kernels for ModelOptNvFp4FusedMoE.
(EngineCore_DP0 pid=107470) INFO 11-10 16:51:45 [modelopt.py:1177] Using FlashInfer CUTLASS kernels for ModelOptNvFp4FusedMoE.
Loading safetensors checkpoint shards (eager):   0% Completed | 0/13 [00:00<?, ?it/s]
Loading safetensors checkpoint shards (eager):   8% Completed | 1/13 [00:07<01:25,  7.14s/it]
Loading safetensors checkpoint shards (eager):  15% Completed | 2/13 [00:15<01:23,  7.57s/it]
Loading safetensors checkpoint shards (eager):  23% Completed | 3/13 [00:16<00:49,  5.00s/it]
Loading safetensors checkpoint shards (eager):  31% Completed | 4/13 [00:23<00:51,  5.71s/it]
Loading safetensors checkpoint shards (eager):  38% Completed | 5/13 [00:29<00:46,  5.87s/it]
Loading safetensors checkpoint shards (eager):  46% Completed | 6/13 [00:36<00:42,  6.00s/it]
Loading safetensors checkpoint shards (eager):  54% Completed | 7/13 [00:43<00:38,  6.44s/it]
Loading safetensors checkpoint shards (eager):  62% Completed | 8/13 [00:50<00:33,  6.62s/it]
Loading safetensors checkpoint shards (eager):  69% Completed | 9/13 [00:57<00:26,  6.69s/it]
Loading safetensors checkpoint shards (eager):  77% Completed | 10/13 [01:03<00:19,  6.57s/it]
Loading safetensors checkpoint shards (eager):  85% Completed | 11/13 [01:10<00:13,  6.76s/it]
Loading safetensors checkpoint shards (eager):  92% Completed | 12/13 [01:18<00:07,  7.04s/it]
Loading safetensors checkpoint shards (eager): 100% Completed | 13/13 [01:24<00:00,  6.66s/it]
Loading safetensors checkpoint shards (eager): 100% Completed | 13/13 [01:24<00:00,  6.49s/it]
(EngineCore_DP0 pid=107470) 
(EngineCore_DP0 pid=107470) INFO 11-10 16:53:11 [default_loader.py:314] Loading weights took 84.34 seconds
(EngineCore_DP0 pid=107470) INFO 11-10 16:53:15 [gpu_model_runner.py:2997] Model loading took 57.7519 GiB and 89.609242 seconds
(EngineCore_DP0 pid=107470) INFO 11-10 16:53:20 [backends.py:620] Using cache directory: /home/azure/.cache/vllm/torch_compile_cache/9ebe60dec2/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=107470) INFO 11-10 16:53:20 [backends.py:636] Dynamo bytecode transform time: 4.87 s
(EngineCore_DP0 pid=107470) INFO 11-10 16:53:26 [backends.py:209] Directly load the compiled graph(s) for dynamic shape from the cache, took 5.714 s
(EngineCore_DP0 pid=107470) INFO 11-10 16:53:28 [monitor.py:34] torch.compile takes 10.58 s in total
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843] EngineCore failed to start.
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843] Traceback (most recent call last):
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 834, in run_engine_core
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 602, in __init__
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]     super().__init__(
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 109, in __init__
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]     num_gpu_blocks, num_cpu_blocks, kv_cache_config = self._initialize_kv_caches(
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]                                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 239, in _initialize_kv_caches
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 126, in determine_available_memory
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]     return self.collective_rpc("determine_available_memory")
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 73, in collective_rpc
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]     return [run_method(self.driver_worker, method, args, kwargs)]
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/v1/serial_utils.py", line 459, in run_method
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]     return func(*args, **kwargs)
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]     return func(*args, **kwargs)
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 313, in determine_available_memory
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]     self.model_runner.profile_run()
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 3816, in profile_run
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]     hidden_states, last_hidden_states = self._dummy_run(
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]                                         ^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]     return func(*args, **kwargs)
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 3582, in _dummy_run
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]     return hidden_states, hidden_states[logit_indices_device]
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843]                           ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843] torch.AcceleratorError: CUDA error: an illegal instruction was encountered
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843] Search for `cudaErrorIllegalInstruction' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843] Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(EngineCore_DP0 pid=107470) ERROR 11-10 16:53:29 [core.py:843] 
(EngineCore_DP0 pid=107470) Process EngineCore_DP0:
(EngineCore_DP0 pid=107470) Traceback (most recent call last):
(EngineCore_DP0 pid=107470)   File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore_DP0 pid=107470)     self.run()
(EngineCore_DP0 pid=107470)   File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=107470)     self._target(*self._args, **self._kwargs)
(EngineCore_DP0 pid=107470)   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 847, in run_engine_core
(EngineCore_DP0 pid=107470)     raise e
(EngineCore_DP0 pid=107470)   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 834, in run_engine_core
(EngineCore_DP0 pid=107470)     engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=107470)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107470)   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 602, in __init__
(EngineCore_DP0 pid=107470)     super().__init__(
(EngineCore_DP0 pid=107470)   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 109, in __init__
(EngineCore_DP0 pid=107470)     num_gpu_blocks, num_cpu_blocks, kv_cache_config = self._initialize_kv_caches(
(EngineCore_DP0 pid=107470)                                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107470)   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 239, in _initialize_kv_caches
(EngineCore_DP0 pid=107470)     available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore_DP0 pid=107470)                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107470)   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 126, in determine_available_memory
(EngineCore_DP0 pid=107470)     return self.collective_rpc("determine_available_memory")
(EngineCore_DP0 pid=107470)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107470)   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 73, in collective_rpc
(EngineCore_DP0 pid=107470)     return [run_method(self.driver_worker, method, args, kwargs)]
(EngineCore_DP0 pid=107470)             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107470)   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/v1/serial_utils.py", line 459, in run_method
(EngineCore_DP0 pid=107470)     return func(*args, **kwargs)
(EngineCore_DP0 pid=107470)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107470)   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_DP0 pid=107470)     return func(*args, **kwargs)
(EngineCore_DP0 pid=107470)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107470)   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 313, in determine_available_memory
(EngineCore_DP0 pid=107470)     self.model_runner.profile_run()
(EngineCore_DP0 pid=107470)   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 3816, in profile_run
(EngineCore_DP0 pid=107470)     hidden_states, last_hidden_states = self._dummy_run(
(EngineCore_DP0 pid=107470)                                         ^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107470)   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
(EngineCore_DP0 pid=107470)     return func(*args, **kwargs)
(EngineCore_DP0 pid=107470)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107470)   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 3582, in _dummy_run
(EngineCore_DP0 pid=107470)     return hidden_states, hidden_states[logit_indices_device]
(EngineCore_DP0 pid=107470)                           ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=107470) torch.AcceleratorError: CUDA error: an illegal instruction was encountered
(EngineCore_DP0 pid=107470) Search for `cudaErrorIllegalInstruction' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
(EngineCore_DP0 pid=107470) CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(EngineCore_DP0 pid=107470) For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(EngineCore_DP0 pid=107470) Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
(EngineCore_DP0 pid=107470) 
[rank0]:[W1110 16:53:29.562667993 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(APIServer pid=107197) Traceback (most recent call last):
(APIServer pid=107197)   File "/home/azure/venvs/vllm/bin/vllm", line 10, in <module>
(APIServer pid=107197)     sys.exit(main())
(APIServer pid=107197)              ^^^^^^
(APIServer pid=107197)   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py", line 73, in main
(APIServer pid=107197)     args.dispatch_function(args)
(APIServer pid=107197)   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/entrypoints/cli/serve.py", line 59, in cmd
(APIServer pid=107197)     uvloop.run(run_server(args))
(APIServer pid=107197)   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/uvloop/__init__.py", line 96, in run
(APIServer pid=107197)     return __asyncio.run(
(APIServer pid=107197)            ^^^^^^^^^^^^^^
(APIServer pid=107197)   File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
(APIServer pid=107197)     return runner.run(main)
(APIServer pid=107197)            ^^^^^^^^^^^^^^^^
(APIServer pid=107197)   File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=107197)     return self._loop.run_until_complete(task)
(APIServer pid=107197)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=107197)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=107197)   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=107197)     return await main
(APIServer pid=107197)            ^^^^^^^^^^
(APIServer pid=107197)   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 2011, in run_server
(APIServer pid=107197)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=107197)   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 2027, in run_server_worker
(APIServer pid=107197)     async with build_async_engine_client(
(APIServer pid=107197)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=107197)     return await anext(self.gen)
(APIServer pid=107197)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=107197)   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 195, in build_async_engine_client
(APIServer pid=107197)     async with build_async_engine_client_from_engine_args(
(APIServer pid=107197)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=107197)     return await anext(self.gen)
(APIServer pid=107197)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=107197)   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 242, in build_async_engine_client_from_engine_args
(APIServer pid=107197)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=107197)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=107197)   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/utils/func_utils.py", line 116, in inner
(APIServer pid=107197)     return fn(*args, **kwargs)
(APIServer pid=107197)            ^^^^^^^^^^^^^^^^^^^
(APIServer pid=107197)   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 202, in from_vllm_config
(APIServer pid=107197)     return cls(
(APIServer pid=107197)            ^^^^
(APIServer pid=107197)   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 132, in __init__
(APIServer pid=107197)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=107197)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=107197)   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 121, in make_async_mp_client
(APIServer pid=107197)     return AsyncMPClient(*client_args)
(APIServer pid=107197)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=107197)   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 808, in __init__
(APIServer pid=107197)     super().__init__(
(APIServer pid=107197)   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 469, in __init__
(APIServer pid=107197)     with launch_core_engines(vllm_config, executor_class, log_stats) as (
(APIServer pid=107197)   File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
(APIServer pid=107197)     next(self.gen)
(APIServer pid=107197)   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 898, in launch_core_engines
(APIServer pid=107197)     wait_for_engine_startup(
(APIServer pid=107197)   File "/home/azure/venvs/vllm/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 955, in wait_for_engine_startup
(APIServer pid=107197)     raise RuntimeError(
(APIServer pid=107197) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
(vllm) azure@spark-0809:~/wjh$ pip show flashinfer-python
Name: flashinfer-python
Version: 0.5.2.dev20251110
Summary: FlashInfer: Kernel Library for LLM Serving
Home-page: https://github.com/flashinfer-ai/flashinfer
Author: FlashInfer team
Author-email: 
License-Expression: Apache-2.0
Location: /home/azure/venvs/vllm/lib/python3.12/site-packages
Requires: apache-tvm-ffi, click, einops, ninja, numpy, nvidia-cudnn-frontend, nvidia-cutlass-dsl, nvidia-ml-py, packaging, requests, tabulate, torch, tqdm
Required-by: vllm

[qiyuxinlin]

The issue has been resolved by disabling PDL in FlashInfer (in the device_support_pdl function located in flashinfer/utils.py).