[MatthewBonanni]

Overview

#65201 forces the trtllm FlashInfer all-reduce backend under HiSparse to avoid the MNNVL fused AR+RMSNorm race (#64327). The override was skipped when the shared workspace was created lazily outside the vLLM config context. HiSparse servers then still came up on mnnvl and crashed with CUDA error: unspecified launch failure. Related: #56699.

Claims

• HiSparse servers on main now get the trtllm all-reduce workspace without needing VLLM_FLASHINFER_ALLREDUCE_BACKEND.
• Fixes the asynchronous CUDA error: unspecified launch failure (driver 719) under HiSparse multi-turn agentic load. It surfaces at the next event sync, e.g. _finish_mirror_phase or async_utils.get_output.

Validation

Hardware: 4×B200, TP4. Main = 70eaf45. PR = 70eaf45 + this diff (since merged with current main; the touched file is unchanged there). Both arms ran an identical environment shim on PYTHONPATH: the kernel fix from #61463 (merged since), an os.memfd_create polyfill for the container's Python build, and logging probes. Nothing else differs.

Server:

export VLLM_USE_V2_MODEL_RUNNER=1 VLLM_WORKER_MULTIPROC_METHOD=spawn VLLM_DEEP_GEMM_WARMUP=skip
vllm serve RedHatAI/GLM-5.3-NVFP4 --revision c8917e4258572c405575855ff53effe58c17a38e \
  --tokenizer-revision c8917e4258572c405575855ff53effe58c17a38e --trust-remote-code \
  --tensor-parallel-size 4 --kv-cache-dtype fp8_ds_mla --block-size 64 \
  --max-model-len 98304 --max-num-batched-tokens 16384 --max-num-seqs 256 --seed 0 \
  --enable-prompt-tokens-details --no-enable-extensible-kv-cache --served-model-name zai-org/GLM-5.3 \
  --port 8000 --attention-config '{"hisparse_config":{}}' \
  --kv-transfer-config '{"kv_connector":"HiSparseConnector","kv_role":"kv_both","kv_connector_extra_config":{"host_pool_gib":160}}'

Client (evalscope 1.14.1, swe_smith multi-turn dataset JSON):

evalscope perf --url http://127.0.0.1:8000/v1/chat/completions --api openai --model zai-org/GLM-5.3 \
  --dataset swe_smith --dataset-path agentic_dataset.json --multi-turn --dataset-offset 0 \
  --parallel 32 --number 64 --max-tokens 220 --min-tokens 220 \
  --tokenizer-path <GLM-5.3-NVFP4 snapshot> --extra-args '{"ignore_eos": true}' --stream --no-test-connection --seed 0

main: crash

$ grep -m1 "norm fusion workspace" server.log
(Worker_TP0 pid=703) INFO 11-03 22:10:15 [flashinfer_all_reduce.py:252] Initialized FlashInfer Allreduce norm fusion workspace with backend=mnnvl
$ # server.log around the crash
(APIServer pid=588) INFO 11-03 22:13:52 [metrics.py:108] KV Transfer metrics: HiSparse hot-buffer hits=0, HiSparse hot-buffer misses=0, HiSparse host-to-device bytes=0, HiSparse host KV cache usage %=70.8
(APIServer pid=588) INFO 11-03 22:14:02 [loggers.py:335] Engine 000: Avg prompt throughput: 16025.3 tokens/s, Avg generation throughput: 33.2 tokens/s, Running: 32 reqs, Waiting: 0 reqs, GPU KV cache usage: 37.0%, Prefix cache hit rate: 0.0%, External prefix cache hit rate: 0.0%
(APIServer pid=588) INFO 11-03 22:14:02 [metrics.py:108] KV Transfer metrics: HiSparse hot-buffer hits=0, HiSparse hot-buffer misses=0, HiSparse host-to-device bytes=0, HiSparse host KV cache usage %=76.2
(Worker_TP2 pid=705) ERROR 11-03 22:14:04 [multiproc_executor.py:1105] Error getting async model runner output
(Worker_TP2 pid=705) ERROR 11-03 22:14:04 [multiproc_executor.py:1105] Traceback (most recent call last):
(Worker_TP2 pid=705) ERROR 11-03 22:14:04 [multiproc_executor.py:1105]   File "/data/envs/nonpd-main/vllm/v1/executor/multiproc_executor.py", line 1103, in enqueue_output
(Worker_TP2 pid=705) ERROR 11-03 22:14:04 [multiproc_executor.py:1105]     output = output.get_output()
(Worker_TP2 pid=705) ERROR 11-03 22:14:04 [multiproc_executor.py:1105]              ^^^^^^^^^^^^^^^^^^^
(Worker_TP2 pid=705) ERROR 11-03 22:14:04 [multiproc_executor.py:1105]   File "/data/envs/nonpd-main/vllm/v1/worker/gpu/async_utils.py", line 175, in get_output
(Worker_TP2 pid=705) ERROR 11-03 22:14:04 [multiproc_executor.py:1105]     self.copy_event.synchronize()
(Worker_TP2 pid=705) ERROR 11-03 22:14:04 [multiproc_executor.py:1105]   File "/data/envs/nonpd-main/.venv/lib/python3.12/site-packages/torch/cuda/streams.py", line 254, in synchronize
(Worker_TP2 pid=705) ERROR 11-03 22:14:04 [multiproc_executor.py:1105]     super().synchronize()
(Worker_TP2 pid=705) ERROR 11-03 22:14:04 [multiproc_executor.py:1105] torch.AcceleratorError: CUDA error: unspecified launch failure
(Worker_TP2 pid=705) ERROR 11-03 22:14:04 [multiproc_executor.py:1105] Search for `cudaErrorLaunchFailure' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
(Worker_TP2 pid=705) ERROR 11-03 22:14:04 [multiproc_executor.py:1105] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
(Worker_TP2 pid=705) ERROR 11-03 22:14:04 [multiproc_executor.py:1105] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
(Worker_TP2 pid=705) ERROR 11-03 22:14:04 [multiproc_executor.py:1105] The CUDA driver logged these messages, which may provide useful details:
(Worker_TP2 pid=705) ERROR 11-03 22:14:04 [multiproc_executor.py:1105] Returning 719 (CUDA_ERROR_LAUNCH_FAILED) from cuEventSynchronize

evalscope exited after ~3 min (killed when the engine died). The same crash reproduced on every unpatched main run.

This PR: 832/832 turns, no errors (2 runs)

Run 1:

$ grep -m1 "norm fusion workspace" server.log
(Worker_TP0 pid=18587) INFO 11-03 23:57:03 [flashinfer_all_reduce.py:257] Initialized FlashInfer Allreduce norm fusion workspace with backend=trtllm
$ grep -c "CUDA error" server.log
0
$ # evalscope stdout
Benchmarking summary:
┌───────────────────────────────┬───────────────┐
│ Metric                        │         Value │
├───────────────────────────────┼───────────────┤
│ ── General ──                 │               │
│ Test Duration (s)             │        627.39 │
│ Concurrency                   │            32 │
│ Request Rate (req/s)          │            -1 │
│ Total / Success / Failed      │ 832 / 832 / 0 │
│ Req Throughput (req/s)        │          1.33 │
│ ── Latency ──                 │               │
│ Avg Latency (s)               │        24.081 │
│ Avg TTFT (ms)                 │       4390.15 │
│ Avg TPOT (ms)                 │         89.91 │
│ Avg ITL (ms)                  │         91.94 │
│ ── Tokens ──                  │               │
│ Avg Input Tokens              │       84147.3 │
│ Avg Output Tokens             │           220 │
│ Output Throughput (tok/s)     │        291.75 │
│ Total Throughput (tok/s)      │     111882.08 │
│ ── Multi-turn ──              │               │
│ Avg Turns/Request             │             7 │
│ KV Cache Hit Rate (%)         │            92 │
│ Avg First-Turn TTFT (ms)      │      46364.51 │
│ Avg Subsequent-Turn TTFT (ms) │        892.29 │
│ ── Speculative Decoding ──    │               │
│ Avg Decoded Tok/Iter          │          1.02 │
│ Spec. Accept Rate             │           2.4 │
└───────────────────────────────┴───────────────┘

Run 2:

$ grep -m1 "norm fusion workspace" server.log
(Worker_TP0 pid=19928) INFO 11-04 00:03:50 [flashinfer_all_reduce.py:257] Initialized FlashInfer Allreduce norm fusion workspace with backend=trtllm
$ grep -c "CUDA error" server.log
0
$ # evalscope stdout
Benchmarking summary:
┌───────────────────────────────┬───────────────┐
│ Metric                        │         Value │
├───────────────────────────────┼───────────────┤
│ ── General ──                 │               │
│ Test Duration (s)             │        607.44 │
│ Concurrency                   │            32 │
│ Request Rate (req/s)          │            -1 │
│ Total / Success / Failed      │ 832 / 832 / 0 │
│ Req Throughput (req/s)        │          1.37 │
│ ── Latency ──                 │               │
│ Avg Latency (s)               │        23.322 │
│ Avg TTFT (ms)                 │       4178.47 │
│ Avg TPOT (ms)                 │         87.41 │
│ Avg ITL (ms)                  │         89.31 │
│ ── Tokens ──                  │               │
│ Avg Input Tokens              │       84136.1 │
│ Avg Output Tokens             │           220 │
│ Output Throughput (tok/s)     │        301.33 │
│ Total Throughput (tok/s)      │     115540.23 │
│ ── Multi-turn ──              │               │
│ Avg Turns/Request             │             7 │
│ KV Cache Hit Rate (%)         │            92 │
│ Avg First-Turn TTFT (ms)      │       44984.6 │
│ Avg Subsequent-Turn TTFT (ms) │        777.96 │
│ ── Speculative Decoding ──    │               │
│ Avg Decoded Tok/Iter          │          1.02 │
│ Spec. Accept Rate             │           2.3 │
└───────────────────────────────┴───────────────┘

Details

FlashInferAllReduce.__init__ resolves its backend while the config is current, which yields trtllm under HiSparse. It then called _resolve_fi_ar_backend again when it created the global workspace on the first all-reduce. With the V2 runner, that first all-reduce happens in _profile_cudagraph_memory → capture_model(profile_only=True), which runs outside set_current_vllm_config. get_current_vllm_config_or_none() returns None there, so the HiSparse guard is skipped and mnnvl is auto-selected. The workspace is a process-wide singleton, so the fused AR+RMSNorm ops reuse the mnnvl workspace. Instrumentation confirmed config is None at workspace creation, with that stack.

The fix passes the backend resolved at init into get_fi_ar_workspace. The fusion pass already resolves within the config context and is unchanged.

The HiSparse guard can be dropped once FlashInfer ships flashinfer-ai/flashinfer#7782; this fix goes with it.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

🤖 Generated with Claude Code

[MatthewBonanni]

/ci run

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[github-actions]

✅ Triggered Buildkite CI #99988 for commit c59a5a628067.

[yewentao256]

Thanks for the work!

Please do the following:

• full reproduce command line, error log in main and the same command line works in current PR

When doing reproduce / e2e test, please attach with raw output log in case some agents fake the data. DO NOT use AI generated Table here

For example:

pytest tests/v1/kv_connector/extract_hidden_states_integration/test_extraction.py::test_extract_hidden_states_tp2
# main
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1070, in __init__
  | (EngineCore pid=1580)     super().__init__(
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 160, in __init__
  | (EngineCore pid=1580)     self.scheduler: SchedulerInterface = Scheduler(
  | (EngineCore pid=1580)                                          ^^^^^^^^^^
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/sched/async_scheduler.py", line 14, in __init__
  | (EngineCore pid=1580)     super().__init__(*args, **kwargs)
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/sched/scheduler.py", line 277, in __init__
  | (EngineCore pid=1580)     self.kv_cache_manager = KVCacheManager(
  | (EngineCore pid=1580)                             ^^^^^^^^^^^^^^^
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_manager.py", line 151, in __init__
  | (EngineCore pid=1580)     self.coordinator = get_kv_cache_coordinator(
  | (EngineCore pid=1580)                        ^^^^^^^^^^^^^^^^^^^^^^^^^
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_coordinator.py", line 891, in get_kv_cache_coordinator
  | (EngineCore pid=1580)     return HybridKVCacheCoordinator(
  | (EngineCore pid=1580)            ^^^^^^^^^^^^^^^^^^^^^^^^^
  | (EngineCore pid=1580)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_coordinator.py", line 563, in __init__
  | (EngineCore pid=1580)     assert all(
  | (EngineCore pid=1580)            ^^^^
  | (EngineCore pid=1580) AssertionError: Each KV cache group's real block_size must be divisible by hash_block_size. block_sizes=[544, 544, 544, 544, 181], hash_block_size=98464
  | FAILED
  | v1/kv_connector/extract_hidden_states_integration/test_extraction.py::test_extract_hidden_states_tp2

# Now
================= 1 passed, 23 warnings in 174.57s (0:02:54) ==================

[MatthewBonanni]

/ci run

[github-actions]

✅ Triggered Buildkite CI #99998 for commit ee943277b67b.

ℹ️ This PR is 1 commit behind upstream main. Merged upstream main into this branch as ee943277b67b before starting CI. Pull the branch before pushing more changes.

[yewentao256]

LGTM, thanks for the work!