Error in the Disaggregated Serving example

[pandalee99]

System Info

tensorrt-llm:release-0.20.0

Who can help?

Error in the Disaggregated Serving example

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

in https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/disaggregated

I fully reproduced the requirements of this example, using the same model. My GPU is L20 46G

Expected behavior

Successful execution

actual behavior

my cmd：

curl http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{
"model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
"prompt": "NVIDIA is a great company because",
"max_tokens": 16,
"temperature": 0
}' -w "\n"

the result history：

root@56acd428936d:/app/models# jobs -l
[1] 64414 Running CUDA_VISIBLE_DEVICES=0 trtllm-serve TinyLlama-1.1B-Chat-v1.0 --host localhost --port 8001 --backend pytorch --extra_llm_api_options config/context_extra-llm-api-config.yml &> log_ctx_0 &
[2]- 64415 Running CUDA_VISIBLE_DEVICES=1 trtllm-serve TinyLlama-1.1B-Chat-v1.0 --host localhost --port 8002 --backend pytorch --extra_llm_api_options config/context_extra-llm-api-config.yml &> log_ctx_1 &
[4]+ 66403 Running CUDA_VISIBLE_DEVICES=2 trtllm-serve TinyLlama-1.1B-Chat-v1.0 --host localhost --port 8003 --backend pytorch --extra_llm_api_options config/gen_extra-llm-api-config.yml &> log_gen_0 &
root@56acd428936d:/app/models# trtllm-serve disaggregated -c config/disagg_config.yaml
2025-07-07 09:03:34,710 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
[TensorRT-LLM] TensorRT-LLM version: 0.20.0
INFO: Started server process [66764]
INFO: Waiting for application startup.
INFO:root:Waiting for context and generation servers to be ready
INFO: Application startup complete.
INFO: Uvicorn running on http://localhost:8000 (Press CTRL+C to quit)
INFO:root:Sending request to ctx server: http://localhost:8001
INFO:root:Sending request to gen server: http://localhost:8003
ERROR:root:Received failed response {'object': 'error', 'message': 'Failed during generation: init(): incompatible constructor arguments. The following argument types are supported:\n 1. tensorrt_llm.bindings.executor.ContextPhaseParams(arg0: list[int], arg1: int, arg2: Optional[bytes], arg3: Optional[list[int]])\n\nInvoked with: None, None, None, None', 'type': 'BadRequestError', 'param': None, 'code': 400}
ERROR:root:400, message='Bad Request', url='http://localhost:8003/v1/completions'
Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/serve/openai_disagg_server.py", line 140, in openai_completion
return await self._process_generation_server_request(gen_req, ctx_response)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/serve/openai_disagg_server.py", line 214, in _process_generation_server_request
gen_response = await self.send_completion_request(gen_server, gen_req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/serve/openai_disagg_server.py", line 281, in send_completion_request
return await self.send_request(url, request, "/v1/completions", CompletionResponse, self.create_completion_generator)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/serve/openai_disagg_server.py", line 277, in send_request
response.raise_for_status()
File "/usr/local/lib/python3.12/dist-packages/aiohttp/client_reqrep.py", line 1161, in raise_for_status
raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 400, message='Bad Request', url='http://localhost:8003/v1/completions'
INFO: 127.0.0.1:56126 - "POST /v1/completions HTTP/1.1" 500 Internal Server Error

additional notes

my log about ：

"detail":"Internal server error 400, message='Bad Request', url='http://localhost:8003/v1/completions'"}
root@56acd428936d:/app/models# ls
InternVL2_5-8B Qwen3-0.6B Qwen3-30B-A3B Qwen3-4B TinyLlama-1.1B-Chat-v1.0 config log_ctx_0 log_ctx_1 log_gen_0 qwen
root@56acd428936d:/app/models# tail -f log_gen_0
Encountered an exception: Failed during generation: init(): incompatible constructor arguments. The following argument types are supported:
1. tensorrt_llm.bindings.executor.ContextPhaseParams(arg0: list[int], arg1: int, arg2: Optional[bytes], arg3: Optional[list[int]])

Invoked with: None, None, None, None
INFO: 127.0.0.1:37166 - "POST /v1/completions HTTP/1.1" 400 Bad Request
[07/07/2025-09:03:46] [TRT-LLM] [I] Thread await_response_thread stopped.
[07/07/2025-09:03:46] [TRT-LLM] [I] Thread proxy_dispatch_stats_thread stopped.
[07/07/2025-09:03:46] [TRT-LLM] [I] Thread proxy_dispatch_kv_cache_events_thread stopped.
[07/07/2025-09:03:46] [TRT-LLM] [I] Thread dispatch_stats_thread stopped.
[07/07/2025-09:03:46] [TRT-LLM] [I] Thread dispatch_kv_cache_events_thread stopped.

[ZhangGe6]

@pandalee99 FYI. The root cause may be the None transceiver. "export TRTLLM_USE_UCX_KVCACHE=1" helps me get rid of the error, and get expected generated tokens.

BTW. For TensorRT-LLM team, it is suggested to add the essential env var setting in this doc, especially when there are no errors/warnings when this env is not set.

# tensorrt_llm/_torch/pyexecutor/kv_cache_transceiver.py
def create_kv_cache_transceiver(
        mapping: Mapping, kv_cache_manager: KVCacheManager,
        attention_type: AttentionTypeCpp,
        cache_transceiver_config: CacheTransceiverConfig):

    comm_type = None
    if getenv("TRTLLM_USE_UCX_KVCACHE"):
        comm_type = CommTypeCpp.UCX
    elif getenv("TRTLLM_USE_NIXL_KVCACHE"):
        comm_type = CommTypeCpp.NIXL
    elif getenv("TRTLLM_USE_MPI_KVCACHE"):
        comm_type = CommTypeCpp.MPI

    # <---- No errors/warnings here

    cache_transceiver = None
    if comm_type is not None:
        cache_transceiver = BindKvCacheTransceiver(mapping, comm_type,
                                                   kv_cache_manager,
                                                   attention_type,
                                                   cache_transceiver_config)

    return cache_transceiver

[pandalee99]

@pandalee99 FYI. The root cause may be the None transceiver. "export TRTLLM_USE_UCX_KVCACHE=1" helps me get rid of the error, and get expected generated tokens.

BTW. For TensorRT-LLM team, it is suggested to add the essential env var setting in this doc, especially when there are no errors/warnings when this env is not set.

tensorrt_llm/_torch/pyexecutor/kv_cache_transceiver.py
def create_kv_cache_transceiver(
mapping: Mapping, kv_cache_manager: KVCacheManager,
attention_type: AttentionTypeCpp,
cache_transceiver_config: CacheTransceiverConfig):

comm_type = None
if getenv("TRTLLM_USE_UCX_KVCACHE"):
    comm_type = CommTypeCpp.UCX
elif getenv("TRTLLM_USE_NIXL_KVCACHE"):
    comm_type = CommTypeCpp.NIXL
elif getenv("TRTLLM_USE_MPI_KVCACHE"):
    comm_type = CommTypeCpp.MPI

# <---- No errors/warnings here

cache_transceiver = None
if comm_type is not None:
    cache_transceiver = BindKvCacheTransceiver(mapping, comm_type,
                                               kv_cache_manager,
                                               attention_type,
                                               cache_transceiver_config)

return cache_transceiver

I don't think it was caused by this reason. I operated completely in accordance with the official documentation. I'm sure I used export TRTLLM_USE_UCX_KVCACHE=1 or an error occurred.

But I'm still very grateful for your reply. @ZhangGe6

best regards

[pcastonguay]

In #5234, we changed the way the communication backend is selected, to avoid needing to set environment variables. Could you try following the latest instructions in https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/disaggregated#launching-disaggregated-servers-locally-on-single-node

Thanks

[pandalee99]

@pcastonguay

This problem still hasn't been solved when I use curl:

curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
        "prompt": "NVIDIA is a great company because",
        "max_tokens": 16,
        "temperature": 0
    }' -w "\n"

When making the request, the logs of ctx0 and ctx1 both show success, but the log of gen0 reports an error:

 File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/worker.py", line 733, in worker_main
    worker.submit(req)
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/worker.py", line 500, in submit
    request_id = self._enqueue_request(request)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/worker.py", line 384, in _enqueue_request
    context_phase_params = request.disaggregated_params.get_context_phase_params(
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/disaggregated_params.py", line 27, in get_context_phase_params
    return tllme.ContextPhaseParams(self.first_gen_tokens,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: __init__(): incompatible constructor arguments. The following argument types are supported:
    1. tensorrt_llm.bindings.executor.ContextPhaseParams(arg0: list[int], arg1: int, arg2: Optional[bytes], arg3: Optional[list[int]])

Invoked with: None, None, None, None

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/serve/openai_server.py", line 328, in openai_completion
    promise = self.llm.generate_async(
              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py", line 341, in generate_async
    result = self._executor.generate_async(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/executor.py", line 136, in generate_async
    result = self.submit(
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/proxy.py", line 411, in submit
    raise error
Exception: Failed during generation: __init__(): incompatible constructor arguments. The following argument types are supported:
    1. tensorrt_llm.bindings.executor.ContextPhaseParams(arg0: list[int], arg1: int, arg2: Optional[bytes], arg3: Optional[list[int]])

Invoked with: None, None, None, None
Encountered an exception: Failed during generation: __init__(): incompatible constructor arguments. The following argument types are supported:
    1. tensorrt_llm.bindings.executor.ContextPhaseParams(arg0: list[int], arg1: int, arg2: Optional[bytes], arg3: Optional[list[int]])

Invoked with: None, None, None, None
INFO:     127.0.0.1:61212 - "POST /v1/completions HTTP/1.1" 400 Bad Request

The log of dis0 also reports an error:

ERROR:root:Received failed response {'object': 'error', 'message': 'Failed during generation: __init__(): incompatible constructor arguments. The following argument types are supported:\n    1. tensorrt_llm.bindings.executor.ContextPhaseParams(arg0: list[int], arg1: int, arg2: Optional[bytes], arg3: Optional[list[int]])\n\nInvoked with: None, None, None, None', 'type': 'BadRequestError', 'param': None, 'code': 400}
ERROR:root:400, message='Bad Request', url='http://localhost:8003/v1/completions'
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/serve/openai_disagg_server.py", line 140, in openai_completion
    return await self._process_generation_server_request(gen_req, ctx_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/serve/openai_disagg_server.py", line 214, in _process_generation_server_request
    gen_response = await self.send_completion_request(gen_server, gen_req)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/serve/openai_disagg_server.py", line 281, in send_completion_request
    return await self.send_request(url, request, "/v1/completions", CompletionResponse, self.create_completion_generator)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/serve/openai_disagg_server.py", line 277, in send_request
    response.raise_for_status()
  File "/usr/local/lib/python3.12/dist-packages/aiohttp/client_reqrep.py", line 1161, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 400, message='Bad Request', url='http://localhost:8003/v1/completions'
INFO:     127.0.0.1:24876 - "POST /v1/completions HTTP/1.1" 500 Internal Server Error

[pcastonguay]

Which version of TRT-LLM are you using?

That error:

TypeError: __init__(): incompatible constructor arguments. The following argument types are supported:
    1. tensorrt_llm.bindings.executor.ContextPhaseParams(arg0: list[int], arg1: int, arg2: Optional[bytes], arg3: Optional[list[int]])

Invoked with: None, None, None, None

typically happens with versions of trt-llm prior to the PR above when export TRTLLM_USE_UCX_KVCACHE=1 is not set.

[pandalee99]

Which version of TRT-LLM are you using?

That error:

TypeError: __init__(): incompatible constructor arguments. The following argument types are supported:
    1. tensorrt_llm.bindings.executor.ContextPhaseParams(arg0: list[int], arg1: int, arg2: Optional[bytes], arg3: Optional[list[int]])

Invoked with: None, None, None, None

typically happens with versions of trt-llm prior to the PR above when export TRTLLM_USE_UCX_KVCACHE=1 is not set.

release-0.20
Yes, when I used export TRTLLM_USE_UCX_KVCACHE=1, this error did not occur, but a new one did.

Among them, ctx1 and gen0 did not report any errors.

However, ctx0 reports an error as:

  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/worker.py", line 733, in worker_main
    worker.submit(req)
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/worker.py", line 500, in submit
    request_id = self._enqueue_request(request)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/worker.py", line 392, in _enqueue_request
    raise ValueError(
ValueError: Context only requests are not supported in pytorch backend when overlap is enabled.

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/serve/openai_server.py", line 328, in openai_completion
    promise = self.llm.generate_async(
              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/llmapi/llm.py", line 341, in generate_async
    result = self._executor.generate_async(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/executor.py", line 136, in generate_async
    result = self.submit(
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/executor/proxy.py", line 411, in submit
    raise error
Exception: Failed during generation: Context only requests are not supported in pytorch backend when overlap is enabled.
Encountered an exception: Failed during generation: Context only requests are not supported in pytorch backend when overlap is enabled.
INFO:     127.0.0.1:41704 - "POST /v1/completions HTTP/1.1" 400 Bad Request

And the disaggregated log error reads:

ERROR:root:Received failed response {'object': 'error', 'message': 'Failed during generation: Context only requests are not supported in pytorch backend when overlap is enabled.', 'type': 'BadRequestError', 'param': None, 'code': 400}
ERROR:root:400, message='Bad Request', url='http://localhost:8001/v1/completions'
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/serve/openai_disagg_server.py", line 138, in openai_completion
    ctx_response = await self._process_context_server_request(req, "completion")
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/serve/openai_disagg_server.py", line 185, in _process_context_server_request
    response = await self.send_completion_request(ctx_server, ctx_req)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/serve/openai_disagg_server.py", line 281, in send_completion_request
    return await self.send_request(url, request, "/v1/completions", CompletionResponse, self.create_completion_generator)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/tensorrt_llm/serve/openai_disagg_server.py", line 277, in send_request
    response.raise_for_status()
  File "/usr/local/lib/python3.12/dist-packages/aiohttp/client_reqrep.py", line 1161, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 400, message='Bad Request', url='http://localhost:8001/v1/completions'
INFO:     127.0.0.1:39770 - "POST /v1/completions HTTP/1.1" 500 Internal Server Error

[pcastonguay]

The error:

ValueError: Context only requests are not supported in pytorch backend when overlap is enabled.

means you will need to disable overlap scheduler for the context server with:

pytorch_backend_config:
  disable_overlap_scheduler: True

[pandalee99]

The error:

ValueError: Context only requests are not supported in pytorch backend when overlap is enabled.

means you will need to disable overlap scheduler for the context server with:

pytorch_backend_config:
  disable_overlap_scheduler: True

Thank you very much, my friend. Now the experiment has been successful.

I think the problem lies in:

export TRTLLM_USE_UCX_KVCACHE=1
and

pytorch_backend_config:
  disable_overlap_scheduler: True

So in this example, if the pytorch backend is used, it is necessary to specify:
https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/disaggregated#launching-disaggregated-servers-locally-on-single-node

The configuration file of the original example is incomplete.

Thank you again.

[pcastonguay]

The disable_overlap_scheduler option is added to the context yaml file in the example. See:

echo -e "pytorch_backend_config:\n  disable_overlap_scheduler: True\ncache_transceiver_config:\n  max_num_tokens: 2048" > context_extra-llm-api-config.yml

Closing.