[TRTLLM-17242][feat] merge two renders into a standalone serve/render for dynamo

docs/source/commands/trtllm-serve/index.rst

@@ -6,4 +6,5 @@ trtllm-serve
    :maxdepth: 1
 
    trtllm-serve
+   render-endpoints
    run-benchmark-with-trtllm-serve

docs/source/commands/trtllm-serve/render-endpoints.md

@@ -0,0 +1,138 @@
+# Render endpoints
+
+The render endpoints turn an OpenAI-style request into the prompt token ids a worker would use, without running
+the model. A router or gateway can use the ids to choose a worker, and a client can send them to a worker to skip
+its tokenization and chat templating.
+
+- `POST /v1/chat/completions/render`
+- `POST /v1/completions/render`
+
+Both take the same body as the inference route of the same name. They are a prototype and are not registered with
+`trtllm-serve` by default. In this page, a *worker* is a `trtllm-serve` server that runs the model.
+
+## Start the endpoints
+
+On a worker, set `TRTLLM_ENABLE_RENDER_ENDPOINTS=1` before starting `trtllm-serve`.
+
+On a CPU-only host, `trtllm-render` loads only the tokenizer, configuration and processor files of a checkpoint:
+
+```bash
+trtllm-render --model <checkpoint> --port 8100
+```
+
+Give it the same model, tokenizer, chat template and parser options as the workers it fronts. Both processes publish
+a rendering fingerprint on `GET /server_info`. Equal `render_fingerprint.digest` values mean the ids are valid for
+that worker, so compare them once when a worker is added.
+
+## The response
+
+Chat returns one rendered request. This is the response for a chat request with a system and a user message:
+
+```json
+{
+  "schema_version": 1,
+  "kind": "chat",
+  "request_id": "chatcmpl-9651e194058642d9b538b695bb179c68",
+  "token_ids": [151644, 8948, 198, 2610, 525, 50537, "..."],
+  "model": "Qwen3-0.6B",
+  "reasoning_ended": null,
+  "reasoning_parser_kwargs": {"chat_template_kwargs": {}},
+  "cache_salt": null,
+  "request": {
+    "model": "Qwen3-0.6B",
+    "messages": [
+      {"role": "system", "content": "You are terse."},
+      {"role": "user", "content": "Name three primary colors."}
+    ],
+    "max_tokens": 24,
+    "temperature": 0
+  }
+}
+```
+
+Completions return a list with one rendered request per prompt. In each, `request` is the original body with `prompt`
+set to that item's own prompt:
+
+```json
+{
+  "schema_version": 1,
+  "kind": "completion",
+  "request_id": "cmpl-2eb4d58b52984d25916f70687f565577-0",
+  "token_ids": [785, 6722, 315, 9625, 374],
+  "model": "Qwen3-0.6B",
+  "reasoning_ended": null,
+  "reasoning_parser_kwargs": null,
+  "cache_salt": null,
+  "request": {"model": "Qwen3-0.6B", "prompt": "The capital of France is", "max_tokens": 8, "temperature": 0}
+}
+```
+
+| Field | Meaning |
+|---|---|
+| `kind` | `"chat"` or `"completion"`: which route `request` belongs to. |
+| `request_id` | Identifies this rendered request. |
+| `token_ids` | The rendered prompt. |
+| `model`, `cache_salt` | Copied from the request. |
+| `reasoning_ended` | Whether reasoning had already ended before the first generated token, read off the prompt for the reasoning parser the renderer is started with. `null` when there is no such parser or the prompt does not say. |
+| `reasoning_parser_kwargs` | The `chat_template_kwargs` the prompt was rendered with. Set for chat when the renderer has a reasoning parser, otherwise `null`. |
+| `request` | The original request body. For a completions request with several prompts, each item carries its own prompt. |
+
+## Send the result to the inference endpoints
+
+The original request is always valid to send to a worker. The ids only let the worker skip rendering.
+
+### Completions
+
+Add the ids as the `prompt` of each item's `request`:
+
+```bash
+curl -s $RENDERER/v1/completions/render -H 'Content-Type: application/json' -d @request.json \
+  | jq -c '.[] | .request + {prompt: .token_ids}' \
+  | while read -r body; do
+      curl -s $WORKER/v1/completions -H 'Content-Type: application/json' -d "$body"
+    done
+```
+
+### Chat
+
+Add the ids as `prompt_token_ids`:
+
+```bash
+curl -s $RENDERER/v1/chat/completions/render -H 'Content-Type: application/json' -d @request.json \
+  | jq '.request + {prompt_token_ids: .token_ids}' \
+  | curl -s $WORKER/v1/chat/completions -H 'Content-Type: application/json' -d @-
+```
+
+```python
+import requests
+
+rendered = requests.post(f"{RENDERER}/v1/chat/completions/render", json=body).json()
+send = {**rendered["request"], "prompt_token_ids": rendered["token_ids"]}
+reply = requests.post(f"{WORKER}/v1/chat/completions", json=send).json()
+```
+
+A request with a named `tool_choice` (the model must call one specific function) is the exception. The worker
+appends a forced tool-call prefix when it renders, and rejects the request together with `prompt_token_ids`. Send
+`rendered["request"]` as it is for those.
+
+## Limits
+
+Multimodal requests (images, video, audio) are not yet supported. The endpoints return `400` for them.
+
+## When the KV-aware router forwards its ids
+
+The `kv_cache_aware` router renders each chat request with the same code to choose a worker. It also sends the ids it
+rendered to the worker, so the worker does not render again, only when every active worker reports the same rendering
+fingerprint as the router in `/server_info`. Otherwise the worker renders the prompt itself and the router only uses
+its ids to route. The router logs a warning that names each worker that blocks forwarding and why, and logs again
+only when that changes. The causes:
+
+- A worker reports no fingerprint (an older version, or its `/server_info` could not be fetched).
+- A worker renders differently: another chat template (`--chat_template`), tokenizer, model type, or a model whose
+  processor builds the prompt (vision models).
+- A tokenizer that cannot be fully identified.
+- The router has no model path to resolve the model type from, so it renders as a generic text model.
+
+Some requests are never forwarded, whatever the workers report: a named `tool_choice`; a model whose usage count
+excludes tokens of the rendered prompt (Kimi-K3); and a prompt that carries a reasoning mode (`<think>` prefilled) when a
+worker runs a reasoning parser that reads the mode from the prompt text (`poolside_v1`, `laguna`, `minimax_m3`).

docs/source/commands/trtllm-serve/trtllm-serve.rst

@@ -22,8 +22,9 @@ The server also supports the following endpoints:
 - ``/version``
 - ``/start_profile`` (prototype)
 - ``/stop_profile`` (prototype)
+- ``/v1/chat/completions/render`` and ``/v1/completions/render`` (prototype, enabled with ``TRTLLM_ENABLE_RENDER_ENDPOINTS=1``)
 
-The ``metrics`` endpoint provides runtime-iteration statistics such as GPU memory use and inflight-batching details. The ``start_profile`` and ``stop_profile`` endpoints control iteration-scoped profiling of the backend engine; see :ref:`runtime-profiling-endpoints` below.
+The ``metrics`` endpoint provides runtime-iteration statistics such as GPU memory use and inflight-batching details. The ``start_profile`` and ``stop_profile`` endpoints control iteration-scoped profiling of the backend engine; see :ref:`runtime-profiling-endpoints` below. The render endpoints return the prompt token ids that the server would use for a request, without running the model; see :doc:`render-endpoints`.
 
 For encoder-only models (BERT-style classifiers, reward models, text-embedding models), the ``trtllm-serve embeddings`` subcommand starts a server that exposes an OpenAI-compatible ``/v1/embeddings`` endpoint with native dynamic batching. See :doc:`Embeddings <../../features/embeddings>` for details.
 
@@ -299,6 +300,27 @@ For checkpoints that support both LLM and Visual Generation, such as Cosmos3, pa
 
 For full details, see the :doc:`../../models/visual-generation.md` feature documentation. Example client scripts are available in the `examples/visual_gen/serve/ <https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/visual_gen/serve>`_ directory.
 
+Render Endpoints
+----------------
+
+The render endpoints accept the same request body as ``/v1/chat/completions`` and ``/v1/completions`` and return the
+prompt token ids that a ``trtllm-serve`` server would build from it, including its chat template, tool and reasoning
+handling, and tokenization. They do not run the model. The response is a *rendered request*: the ``token_ids`` plus
+the original request body.
+
+Typical uses:
+
+- **Routing.** A KV-cache-aware router or gateway, such as Dynamo's endpoint picker, calls a render endpoint to get
+  exactly the token ids the server will see. It hashes them to find the server with the longest cached prefix, then
+  forwards the original request to that server.
+- **Skipping rendering on the server.** A client can send the ids to a server's chat or completions endpoint, so the
+  server does not apply the chat template and tokenize again.
+
+The render endpoints are a prototype and are not registered with ``trtllm-serve`` by default. Set
+``TRTLLM_ENABLE_RENDER_ENDPOINTS=1`` to register them on a ``trtllm-serve`` server, or run the standalone, CPU-only
+``trtllm-render`` next to your servers. See :doc:`render-endpoints` for the response format and for how to send a
+rendered request to the chat and completions endpoints.
+
 Multi-node Serving with Slurm
 -----------------------------
 

jenkins/L0_MergeRequest.groovy

@@ -1778,6 +1778,8 @@ def getMultiGpuFileChanged(pipeline, testFilter, globalVars)
         "tensorrt_llm/evaluate/mmlu.py",
         "tensorrt_llm/executor/",
         "tensorrt_llm/functional.py",
+        "tensorrt_llm/inputs/registry.py",
+        "tensorrt_llm/inputs/tokenization.py",
         "tensorrt_llm/llmapi/disagg_utils.py",
         "tensorrt_llm/llmapi/mgmn_leader_node.py",
         "tensorrt_llm/llmapi/mgmn_worker_node.py",
@@ -1786,14 +1788,23 @@ def getMultiGpuFileChanged(pipeline, testFilter, globalVars)
         "tensorrt_llm/mapping.py",
         "tensorrt_llm/models/llama/",
         "tensorrt_llm/parameter.py",
+        "tensorrt_llm/serve/__init__.py",
         "tensorrt_llm/serve/cluster_storage.py",
         "tensorrt_llm/serve/disagg_auto_scaling.py",
+        "tensorrt_llm/serve/extensions/",
         "tensorrt_llm/serve/metadata_server.py",
         "tensorrt_llm/serve/openai_client.py",
         "tensorrt_llm/serve/openai_disagg_server.py",
         "tensorrt_llm/serve/openai_disagg_service.py",
         "tensorrt_llm/serve/openai_server.py",
+        // The shared prompt-preparation core and its callers (chat, completions,
+        // multimodal-encoder routes, count_tokens, KV-cache governor, Responses, router).
+        "tensorrt_llm/serve/render/",
+        "tensorrt_llm/serve/resource_governor.py",
+        "tensorrt_llm/serve/responses_utils.py",
         "tensorrt_llm/serve/router.py",
+        "tensorrt_llm/serve/router_utils.py",
+        "tensorrt_llm/serve/serving_extensions.py",
         "tests/integration/defs/cpp/test_multi_gpu.py",
         "tests/integration/defs/model_express/",
         "tests/integration/test_lists/test-db/l0_b200_multi_gpus_perf_sanity.yml",

setup.py

@@ -715,7 +715,8 @@ def get_build_state_options():
         'console_scripts': [
             'trtllm-bench=tensorrt_llm.commands.bench:main',
             'trtllm-serve=tensorrt_llm.commands.serve:main',
-            'trtllm-eval=tensorrt_llm.commands.eval:main'
+            'trtllm-eval=tensorrt_llm.commands.eval:main',
+            'trtllm-render=tensorrt_llm.serve.render.__main__:main'
         ],
     },
     scripts=['tensorrt_llm/llmapi/trtllm-llmapi-launch'],

tensorrt_llm/inputs/registry.py

@@ -30,7 +30,6 @@
 from transformers import (AutoProcessor, PretrainedConfig,
                           PreTrainedTokenizerBase)
 
-from .._utils import nvtx_range_debug
 from ..logger import logger
 from ..sampling_params import SamplingParams
 from .content_format import ContentFormat
@@ -44,6 +43,7 @@
                          hexdigest_to_int32, validate_mm_inputs)
 from .multimodal_data import serialize_item
 from .prefix_token_cache import create_prefix_token_cache
+from .tokenization import tokenize_prompt
 
 N = TypeVar("N", bound=Type[nn.Module])
 
@@ -175,49 +175,12 @@ def __call__(
         self, inputs: TextPrompt, sampling_params: SamplingParams
     ) -> Tuple[List[int], Optional[ExtraProcessedInputs]]:
         """The default input processor handles only tokenization."""
-        if self.tokenizer is None:
-            raise ValueError("tokenizer is required to tokenize string prompt")
-        # Only when the tokenizer would be called exactly as the cache calls it.
-        if (self._prefix_token_cache is not None
-                and not sampling_params.add_special_tokens
-                and sampling_params.truncate_prompt_tokens is None):
-            with nvtx_range_debug("tokenize prompt"), nvtx_range_debug(
-                    "prefix cache"):
-                return self._prefix_token_cache.encode(self.tokenizer,
-                                                       inputs["prompt"]), None
-        kwargs = {}
-        if sampling_params.truncate_prompt_tokens is not None:
-            kwargs = dict(truncation=True,
-                          max_length=sampling_params.truncate_prompt_tokens)
-        toktoken_special_tokens = {
-            "<|startoftext|>",
-            "<|endoftext|>",
-            "<|reserved_200000|>",
-            "<|reserved_200001|>",
-            "<|return|>",
-            "<|constrain|>",
-            "<|reserved_200004|>",
-            "<|channel|>",
-            "<|start|>",
-            "<|end|>",
-            "<|message|>",
-            "<|reserved_200009|>",
-            "<|reserved_200010|>",
-            "<|reserved_200011|>",
-            "<|call|>",
-            "<|reserved_200013|>",
-        }
-        with nvtx_range_debug("tokenize prompt"):
-            try:
-                token_ids = self.tokenizer.encode(
-                    inputs["prompt"],
-                    add_special_tokens=sampling_params.add_special_tokens,
-                    **kwargs)
-            except:
-                # Tiktoken path
-                token_ids = self.tokenizer.encode(
-                    inputs["prompt"], allowed_special=toktoken_special_tokens)
-
+        token_ids = tokenize_prompt(
+            self.tokenizer,
+            inputs["prompt"],
+            add_special_tokens=sampling_params.add_special_tokens,
+            truncate_prompt_tokens=sampling_params.truncate_prompt_tokens,
+            prefix_token_cache=self._prefix_token_cache)
         return token_ids, None