[Bug]: EXAONE 4.0 with VSWA trtllm-serve Segmentation fault during inference

[lkm2835]

System Info

NVIDIA H200
NVIDIA A100

[TensorRT-LLM] TensorRT LLM version: 1.1.0rc6

1. TensorRT-LLM Sep 27, 2025 Version: https://github.com/NVIDIA/TensorRT-LLM/tree/a36b48bcab30dfbb64a98c299718aa64d06359d9
   Related PRs:
   [TRTLLM-6341][feature] Support SWA KV cache #6768 merged
   [None][feature] Add environment variable to adjust block pool allocation ration under kv cache manager #7923 merged
   [TLLM-6777][feature] Support SWA KV cache reuse OOW block detach #7922 not merged yet

2. TensorRT-LLM Sep 24, 2025 Version: https://github.com/NVIDIA/TensorRT-LLM/tree/b890d7fea40667fc7131bcdc26da3cc8f91e3bde (similar error)
   Related PR:
   [TRTLLM-6341][feature] Support SWA KV cache #6768 merged

Who can help?

@qixiang-99
@eopXD

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

trtllm-serve command

CUDA_VISIBLE_DEVICES=0,1,2,3 trtllm-serve EXAONE-4.0-32B --backend pytorch --tp_size 1 --extra_llm_api_options config.yml --kv_cache_free_gpu_memory_fraction 0.9

config.yml

kv_cache_config:
  enable_block_reuse: false
  max_attention_window: [4096,4096,4096,131072]
enable_chunked_prefill: true

client with openai-python API

Input Sequence Length : 14336
Output Sequence Length : 2048
Number of Concurrent Client : 16

Expected behavior

The model with VSWA should be served successfully through trtllm-serve.

actual behavior

1. TensorRT-LLM Sep 27, 2025 Version (H200)

Caught signal 11 (Segmentation fault: address not mapped to object at address 0x8)
==== backtrace (tid:    676) ====
 0  /opt/hpcx/ompi/lib/openmpi/../../../ucx/lib/libucs.so.0(ucs_handle_error+0x2e4) [0x7551f1c93774]
 1  /opt/hpcx/ompi/lib/openmpi/../../../ucx/lib/libucs.so.0(+0x3796a) [0x7551f1c9396a]
 2  /opt/hpcx/ompi/lib/openmpi/../../../ucx/lib/libucs.so.0(+0x37ba8) [0x7551f1c93ba8]
 3  /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZNK12tensorrt_llm13batch_manager16kv_cache_manager12KVCacheBlock7hasRefsEv+0x4) [0x754eb804d2f4]
 4  /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm13batch_manager16kv_cache_manager18WindowBlockManager13releaseBlocksERNS1_17GenerationRequestENS_6common11OptionalRefIKNS0_10LlmRequestEEE+0x3ee) [0x754eb805d54e]
 5  /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm13batch_manager16kv_cache_manager12BlockManager13releaseBlocksERNS1_17GenerationRequestENS_6common11OptionalRefIKNS0_10LlmRequestEEE+0xa3) [0x754eb805dd13]
 6  /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm13batch_manager16kv_cache_manager14KVCacheManager14removeSequenceEmNS_6common11OptionalRefIKNS0_10LlmRequestEEE+0x333) [0x754eb805e073]
 7  /usr/local/lib/python3.12/dist-packages/tensorrt_llm/bindings.cpython-312-x86_64-linux-gnu.so(+0x101d5b) [0x754eeb798d5b]
 8  /usr/local/lib/python3.12/dist-packages/tensorrt_llm/bindings.cpython-312-x86_64-linux-gnu.so(+0x2cae8c) [0x754eeb961e8c]
 9  /usr/bin/python(PyObject_Vectorcall+0x35) [0x549985]
10  /usr/bin/python(_PyEval_EvalFrameDefault+0xadf) [0x5d6b2f]
11  /usr/bin/python() [0x54cb32]
12  /usr/bin/python(_PyEval_EvalFrameDefault+0x4cc6) [0x5dad16]
13  /usr/bin/python() [0x54cb32]
14  /usr/bin/python() [0x6f80dc]
15  /usr/bin/python() [0x6b931c]
16  /usr/lib/x86_64-linux-gnu/libc.so.6(+0x9caa4) [0x7551f463baa4]
17  /usr/lib/x86_64-linux-gnu/libc.so.6(__clone+0x44) [0x7551f46c8a64]
=================================
 *** Process received signal ***
 Signal: Segmentation fault (11)
 Signal code:  (-6)
 Failing at address: 0x230
 [ 0] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x45330)[0x7551f45e4330]
 [ 1] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZNK12tensorrt_llm13batch_manager16kv_cache_manager12KVCacheBlock7hasRefsEv+0x4)[0x754eb804d2f4]
 [ 2] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm13batch_manager16kv_cache_manager18WindowBlockManager13releaseBlocksERNS1_17GenerationRequestENS_6common11OptionalRefIKNS0_10LlmRequestEEE+0x3ee)[0x754eb805d54e]
 [ 3] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm13batch_manager16kv_cache_manager12BlockManager13releaseBlocksERNS1_17GenerationRequestENS_6common11OptionalRefIKNS0_10LlmRequestEEE+0xa3)[0x754eb805dd13]
 [ 4] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm13batch_manager16kv_cache_manager14KVCacheManager14removeSequenceEmNS_6common11OptionalRefIKNS0_10LlmRequestEEE+0x333)[0x754eb805e073]
 [ 5] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/bindings.cpython-312-x86_64-linux-gnu.so(+0x101d5b)[0x754eeb798d5b]
 [ 6] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/bindings.cpython-312-x86_64-linux-gnu.so(+0x2cae8c)[0x754eeb961e8c]
 [ 7] /usr/bin/python(PyObject_Vectorcall+0x35)[0x549985]
 [ 8] /usr/bin/python(_PyEval_EvalFrameDefault+0xadf)[0x5d6b2f]
 [ 9] /usr/bin/python[0x54cb32]
 [10] /usr/bin/python(_PyEval_EvalFrameDefault+0x4cc6)[0x5dad16]
 [11] /usr/bin/python[0x54cb32]
 [12] /usr/bin/python[0x6f80dc]
 [13] /usr/bin/python[0x6b931c]
 [14] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x9caa4)[0x7551f463baa4]
 [15] /usr/lib/x86_64-linux-gnu/libc.so.6(__clone+0x44)[0x7551f46c8a64]
 *** End of error message ***

2. TensorRT-LLM Sep 24, 2025 Version (A100)

Caught signal 11 (Segmentation fault: Sent by the kernel at address (nil))
==== backtrace (tid:   1728) ====
 0  /opt/hpcx/ompi/lib/openmpi/../../../ucx/lib/libucs.so.0(ucs_handle_error+0x2e4) [0x7effbd6dd774]
 1  /opt/hpcx/ompi/lib/openmpi/../../../ucx/lib/libucs.so.0(+0x3796a) [0x7effbd6dd96a]
 2  /opt/hpcx/ompi/lib/openmpi/../../../ucx/lib/libucs.so.0(+0x37ba8) [0x7effbd6ddba8]
 3  /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZNK12tensorrt_llm13batch_manager16kv_cache_manager12KVCacheBlock7hasRefsEv+0x4) [0x7efdf321ce14]
 4  /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm13batch_manager16kv_cache_manager18WindowBlockManager13releaseBlocksERNS1_17GenerationRequestENS_6common11OptionalRefIKNS0_10LlmRequestEEE+0x3ee) [0x7efdf322ca5e]
 5  /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm13batch_manager16kv_cache_manager12BlockManager13releaseBlocksERNS1_17GenerationRequestENS_6common11OptionalRefIKNS0_10LlmRequestEEE+0xa3) [0x7efdf322d223]
 6  /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm13batch_manager16kv_cache_manager14KVCacheManager14removeSequenceEmNS_6common11OptionalRefIKNS0_10LlmRequestEEE+0x353) [0x7efdf322d5a3]
 7  /usr/local/lib/python3.12/dist-packages/tensorrt_llm/bindings.cpython-312-x86_64-linux-gnu.so(+0x101d5b) [0x7efe31da2d5b]
 8  /usr/local/lib/python3.12/dist-packages/tensorrt_llm/bindings.cpython-312-x86_64-linux-gnu.so(+0x2cae8c) [0x7efe31f6be8c]
 9  /usr/bin/python(PyObject_Vectorcall+0x35) [0x549985]
10  /usr/bin/python(_PyEval_EvalFrameDefault+0xadf) [0x5d6b2f]
11  /usr/bin/python() [0x54cb32]
12  /usr/bin/python(_PyEval_EvalFrameDefault+0x4cc6) [0x5dad16]
13  /usr/bin/python() [0x54cb32]
14  /usr/bin/python() [0x6f80dc]
15  /usr/bin/python() [0x6b931c]
16  /usr/lib/x86_64-linux-gnu/libc.so.6(+0x9caa4) [0x7effc0494aa4]
17  /usr/lib/x86_64-linux-gnu/libc.so.6(__clone+0x44) [0x7effc0521a64]

additional notes

This is a re-test after resolving the issue. #7741

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

[yechank-nvidia]

Hi @lkm2835, we don't have 1.1.0rc6 version, can you re-check the version you encountered this issue and update one, please?

[lkm2835]

Hi @yechank-nvidia, I tested both versions with the specific commits.

1. TensorRT-LLM Sep 27, 2025 Version: https://github.com/NVIDIA/TensorRT-LLM/tree/a36b48bcab30dfbb64a98c299718aa64d06359d9 (commit for 1.1.0rc6)

2. TensorRT-LLM Sep 24, 2025 Version: https://github.com/NVIDIA/TensorRT-LLM/tree/b890d7fea40667fc7131bcdc26da3cc8f91e3bde (similar error) (commit for 1.1.0rc6)

It looks like the 1.1.0rc6 tag wasn’t created, and it jumped directly to 1.2.0rc0.

[lkm2835]

same error

NVIDIA B200
Ubuntu 24.04
NVIDIA-driver 580.65.06

3. TensorRT-LLM Oct 6, 2025 Version: https://github.com/NVIDIA/TensorRT-LLM/tree/fba351a211021e345ef0e76a9439a81af0e7c785 (commit for 1.2.0rc1)

[:2481 :0:2606] Caught signal 11 (Segmentation fault: Sent by the kernel at address (nil))
==== backtrace (tid:   2606) ====
 0  /opt/hpcx/ompi/lib/openmpi/../../../ucx/lib/libucs.so.0(ucs_handle_error+0x2e4) [0x785f0f6cd774]
 1  /opt/hpcx/ompi/lib/openmpi/../../../ucx/lib/libucs.so.0(+0x3796a) [0x785f0f6cd96a]
 2  /opt/hpcx/ompi/lib/openmpi/../../../ucx/lib/libucs.so.0(+0x37ba8) [0x785f0f6cdba8]
 3  /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZNK12tensorrt_llm13batch_manager16kv_cache_manager12KVCacheBlock7hasRefsEv+0x4) [0x785be27ce394]
 4  /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm13batch_manager16kv_cache_manager18WindowBlockManager13releaseBlocksERNS1_17GenerationRequestENS_6common11OptionalRefIKNS0_10LlmRequestEEE+0x3d6) [0x785be27e0226]
 5  /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm13batch_manager16kv_cache_manager12BlockManager13releaseBlocksERNS1_17GenerationRequestENS_6common11OptionalRefIKNS0_10LlmRequestEEEb+0xb3) [0x785be27e0923]
 6  /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm13batch_manager16kv_cache_manager14KVCacheManager14removeSequenceEmNS_6common11OptionalRefIKNS0_10LlmRequestEEEb+0x34b) [0x785be27e0cab]
 7  /usr/local/lib/python3.12/dist-packages/tensorrt_llm/bindings.cpython-312-x86_64-linux-gnu.so(+0x746c3) [0x785c089ff6c3]
 8  /usr/local/lib/python3.12/dist-packages/tensorrt_llm/bindings.cpython-312-x86_64-linux-gnu.so(+0x112b3c) [0x785c08a9db3c]
 9  /usr/bin/python(PyObject_Vectorcall+0x35) [0x549985]
10  /usr/bin/python(_PyEval_EvalFrameDefault+0xadf) [0x5d6b2f]
11  /usr/bin/python() [0x54cb32]
12  /usr/bin/python(_PyEval_EvalFrameDefault+0x4cc6) [0x5dad16]
13  /usr/bin/python() [0x54cb32]
14  /usr/bin/python() [0x6f80dc]
15  /usr/bin/python() [0x6b931c]
16  /usr/lib/x86_64-linux-gnu/libc.so.6(+0x9caa4) [0x785f16477aa4]
17  /usr/lib/x86_64-linux-gnu/libc.so.6(__clone+0x44) [0x785f16504a64]
=================================
[:02481] *** Process received signal ***
[:02481] Signal: Segmentation fault (11)
[:02481] Signal code:  (-6)
[:02481] Failing at address: 0x9b1
[:02481] [ 0] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x45330)[0x785f16420330]
[:02481] [ 1] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZNK12tensorrt_llm13batch_manager16kv_cache_manager12KVCacheBlock7hasRefsEv+0x4)[0x785be27ce394]
[:02481] [ 2] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm13batch_manager16kv_cache_manager18WindowBlockManager13releaseBlocksERNS1_17GenerationRequestENS_6common11OptionalRefIKNS0_10LlmRequestEEE+0x3d6)[0x785be27e0226]
[:02481] [ 3] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm13batch_manager16kv_cache_manager12BlockManager13releaseBlocksERNS1_17GenerationRequestENS_6common11OptionalRefIKNS0_10LlmRequestEEEb+0xb3)[0x785be27e0923]
[:02481] [ 4] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm13batch_manager16kv_cache_manager14KVCacheManager14removeSequenceEmNS_6common11OptionalRefIKNS0_10LlmRequestEEEb+0x34b)[0x785be27e0cab]
[:02481] [ 5] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/bindings.cpython-312-x86_64-linux-gnu.so(+0x746c3)[0x785c089ff6c3]
[:02481] [ 6] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/bindings.cpython-312-x86_64-linux-gnu.so(+0x112b3c)[0x785c08a9db3c]
[:02481] [ 7] /usr/bin/python(PyObject_Vectorcall+0x35)[0x549985]
[:02481] [ 8] /usr/bin/python(_PyEval_EvalFrameDefault+0xadf)[0x5d6b2f]
[:02481] [ 9] /usr/bin/python[0x54cb32]
[:02481] [10] /usr/bin/python(_PyEval_EvalFrameDefault+0x4cc6)[0x5dad16]
[:02481] [11] /usr/bin/python[0x54cb32]
[:02481] [12] /usr/bin/python[0x6f80dc]
[:02481] [13] /usr/bin/python[0x6b931c]
[:02481] [14] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x9caa4)[0x785f16477aa4]
[:02481] [15] /usr/lib/x86_64-linux-gnu/libc.so.6(__clone+0x44)[0x785f16504a64]
[:02481] *** End of error message ***
[:2480 :0:2607] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x145d)
==== backtrace (tid:   2607) ====
 0  /opt/hpcx/ompi/lib/openmpi/../../../ucx/lib/libucs.so.0(ucs_handle_error+0x2e4) [0x7387a5664774]
 1  /opt/hpcx/ompi/lib/openmpi/../../../ucx/lib/libucs.so.0(+0x3796a) [0x7387a566496a]
 2  /opt/hpcx/ompi/lib/openmpi/../../../ucx/lib/libucs.so.0(+0x37ba8) [0x7387a5664ba8]
 3  /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm13batch_manager16kv_cache_manager18WindowBlockManager13releaseBlocksERNS1_17GenerationRequestENS_6common11OptionalRefIKNS0_10LlmRequestEEE+0x510) [0x73846c7e0360]
 4  /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm13batch_manager16kv_cache_manager12BlockManager13releaseBlocksERNS1_17GenerationRequestENS_6common11OptionalRefIKNS0_10LlmRequestEEEb+0xb3) [0x73846c7e0923]
 5  /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm13batch_manager16kv_cache_manager14KVCacheManager14removeSequenceEmNS_6common11OptionalRefIKNS0_10LlmRequestEEEb+0x34b) [0x73846c7e0cab]
 6  /usr/local/lib/python3.12/dist-packages/tensorrt_llm/bindings.cpython-312-x86_64-linux-gnu.so(+0x746c3) [0x7384929ff6c3]
 7  /usr/local/lib/python3.12/dist-packages/tensorrt_llm/bindings.cpython-312-x86_64-linux-gnu.so(+0x112b3c) [0x738492a9db3c]
 8  /usr/bin/python(PyObject_Vectorcall+0x35) [0x549985]
 9  /usr/bin/python(_PyEval_EvalFrameDefault+0xadf) [0x5d6b2f]
10  /usr/bin/python() [0x54cb32]
11  /usr/bin/python(_PyEval_EvalFrameDefault+0x4cc6) [0x5dad16]
12  /usr/bin/python() [0x54cb32]
13  /usr/bin/python() [0x6f80dc]
14  /usr/bin/python() [0x6b931c]
15  /usr/lib/x86_64-linux-gnu/libc.so.6(+0x9caa4) [0x7387a83f1aa4]
16  /usr/lib/x86_64-linux-gnu/libc.so.6(__clone+0x44) [0x7387a847ea64]
=================================
[:02480] *** Process received signal ***
[:02480] Signal: Segmentation fault (11)
[:02480] Signal code:  (-6)
[:02480] Failing at address: 0x9b0
[:02480] [ 0] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x45330)[0x7387a839a330]
[:02480] [ 1] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm13batch_manager16kv_cache_manager18WindowBlockManager13releaseBlocksERNS1_17GenerationRequestENS_6common11OptionalRefIKNS0_10LlmRequestEEE+0x510)[0x73846c7e0360]
[:02480] [ 2] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm13batch_manager16kv_cache_manager12BlockManager13releaseBlocksERNS1_17GenerationRequestENS_6common11OptionalRefIKNS0_10LlmRequestEEEb+0xb3)[0x73846c7e0923]
[:02480] [ 3] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/libs/libtensorrt_llm.so(_ZN12tensorrt_llm13batch_manager16kv_cache_manager14KVCacheManager14removeSequenceEmNS_6common11OptionalRefIKNS0_10LlmRequestEEEb+0x34b)[0x73846c7e0cab]
[:02480] [ 4] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/bindings.cpython-312-x86_64-linux-gnu.so(+0x746c3)[0x7384929ff6c3]
[:02480] [ 5] /usr/local/lib/python3.12/dist-packages/tensorrt_llm/bindings.cpython-312-x86_64-linux-gnu.so(+0x112b3c)[0x738492a9db3c]
[:02480] [ 6] /usr/bin/python(PyObject_Vectorcall+0x35)[0x549985]
[:02480] [ 7] /usr/bin/python(_PyEval_EvalFrameDefault+0xadf)[0x5d6b2f]
[:02480] [ 8] /usr/bin/python[0x54cb32]
[:02480] [ 9] /usr/bin/python(_PyEval_EvalFrameDefault+0x4cc6)[0x5dad16]
[:02480] [10] /usr/bin/python[0x54cb32]
[:02480] [11] /usr/bin/python[0x6f80dc]
[:02480] [12] /usr/bin/python[0x6b931c]
[:02480] [13] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x9caa4)[0x7387a83f1aa4]
[:02480] [14] /usr/lib/x86_64-linux-gnu/libc.so.6(__clone+0x44)[0x7387a847ea64]
[:02480] *** End of error message ***

[yechank-nvidia]

Do you still face the problem even after #7922?

[lkm2835]

The problem cannot be reproduced in this TRT-LLM version (Commits on Oct 15, 2025) after merging #7922 .
It seems that the problem has been resolved.

Test environment

• NVIDIA B200
• Ubuntu 24.04
• NVIDIA-driver 580.65.06

[yechank-nvidia]

Good to know. Thx for the confirmation. I will close this now.