feat: add native TileRT backend and router support

CLAUDE.md

@@ -289,11 +289,28 @@ with patch.dict(os.environ, H100Rack.slurm_env()):
    - `build_worker_command(process, runtime)` - Command construction
 3. Export from `backends/__init__.py`
 4. Add polymorphic deserialization in `BackendConfigField` in `schema.py`
+5. Register the per-role argument key in `core/roles.py` and backend class in
+   `core/schema_docs.py`, then regenerate schema docs. Recipes use `engine:` and
+   `roles.<role>.args` / `.env`.
+
+Optional role-image and finite preparation hooks are described in
+[TileRT's extension-point reference](docs/tilert.md#backend-extension-points).
+Keep engine-specific commands in the backend; use the shared worker launcher
+and process registry for lifecycle handling.
 
 **Current backends:**
 - **SGLang**: Per-process srun launching, supports prefill/decode/aggregated modes
 - **TRTLLM**: MPI-style launching (one srun per endpoint with all nodes), prefill/decode only
 
+### Adding a New Frontend
+
+Implement `FrontendProtocol` (or extend `StaticRouterFrontend` for a router with
+explicit worker URLs), export it from `frontends/__init__.py`, and add its factory
+entry in `frontends/base.py`. Validate backend pairing in `SrtConfig`. Supply the
+frontend health parser and direct backend health URLs when router health alone
+is insufficient. Use `start_srun_process` and `ManagedProcess` with matching
+`step_name` values so normal signal cleanup reaches the router.
+
 ### Adding a New Benchmark
 
 1. Create `benchmarks/mybench.py` inheriting from `BenchmarkRunner`

configs/tilert_setup.sh

@@ -0,0 +1,77 @@
+#!/usr/bin/env bash
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+set -euo pipefail
+
+readonly TILERT_VERSION="${TILERT_VERSION:-0.1.5.post3}"
+readonly NIXL_VERSION="${TILERT_NIXL_VERSION:-1.3.1}"
+readonly FASTAPI_VERSION="${TILERT_FASTAPI_VERSION:-0.141.1}"
+readonly UVICORN_VERSION="${TILERT_UVICORN_VERSION:-0.52.4}"
+readonly TRANSFORMERS_VERSION="${TILERT_TRANSFORMERS_VERSION:-5.16.1}"
+
+if [[ -d /opt/conda/envs/tilert/bin ]]; then
+    export PATH="/opt/conda/envs/tilert/bin:${PATH}"
+fi
+
+PYTHON="$(command -v python || command -v python3)"
+readonly PYTHON
+
+installed_version() {
+    "${PYTHON}" - "$1" <<'PY'
+import sys
+from importlib.metadata import PackageNotFoundError, version
+
+try:
+    print(version(sys.argv[1]))
+except PackageNotFoundError:
+    pass
+PY
+}
+
+install_exact() {
+    local package="$1"
+    local version="$2"
+    shift 2
+    if [[ "$(installed_version "${package}")" == "${version}" ]]; then
+        return
+    fi
+    "${PYTHON}" -m pip install --no-cache-dir "$@" "${package}==${version}"
+}
+
+case "${TILERT_ROLE:-}" in
+    decode)
+        install_exact tilert "${TILERT_VERSION}"
+        install_exact nixl "${NIXL_VERSION}"
+        # TileRT's wheel does not declare the HTTP service dependencies used
+        # by its native decode server.  Install them explicitly before the
+        # import preflight so the image is made runnable deterministically.
+        install_exact fastapi "${FASTAPI_VERSION}"
+        install_exact uvicorn "${UVICORN_VERSION}"
+        "${PYTHON}" -c 'import tilert.pd_vllm.decode_server'
+        ;;
+    prefill)
+        "${PYTHON}" -c 'import vllm'
+        install_exact tilert "${TILERT_VERSION}" --no-deps
+        install_exact nixl "${NIXL_VERSION}"
+        "${PYTHON}" -c 'import tilert.pd_vllm.prefill_connector'
+        ;;
+    router)
+        install_exact tilert "${TILERT_VERSION}"
+        # GLM-5.1 tokenizer metadata names the Transformers v5
+        # TokenizersBackend class.  TileRT's broad >=4.46.3 dependency leaves
+        # its native router on the image's incompatible 4.x release unless we
+        # make the router-side tokenizer contract explicit.
+        install_exact transformers "${TRANSFORMERS_VERSION}"
+        # The native router uses the same undeclared FastAPI/Uvicorn stack as
+        # the decode server and must be self-contained on the frontend host.
+        install_exact fastapi "${FASTAPI_VERSION}"
+        install_exact uvicorn "${UVICORN_VERSION}"
+        "${PYTHON}" -c 'from transformers import TokenizersBackend'
+        "${PYTHON}" -c 'import tilert.pd_vllm.pd_router'
+        ;;
+    *)
+        echo "TILERT_ROLE must be prefill, decode, or router (got ${TILERT_ROLE:-unset})" >&2
+        exit 1
+        ;;
+esac

docs/legacy-v1.md

@@ -40,6 +40,7 @@ Recipes without `schema: 2` (or with `schema: 1`) use the pre-2.0 layout: worker
 | `backend.trtllm_config` | `roles.<role>.args` (one mapping per role; the `prefill` / `decode` / `aggregated` keys) |
 | `backend.vllm_config` | `roles.<role>.args` (one mapping per role; the `prefill` / `decode` / `aggregated` keys) |
 | `backend.mocker_config` | `roles.<role>.args` (one mapping per role; the `prefill` / `decode` / `aggregated` keys) |
+| `backend.tilert_config` | `roles.<role>.args` (one mapping per role; the `prefill` / `decode` / `aggregated` keys) |
 
 ## v1 values that changed meaning
 
@@ -144,6 +145,16 @@ These keys exist in both layouts, but the value means something else in 2.0. A s
 | `aggregated_environment` | dict[str, str] | `{}` |  |
 | `mocker_config` | [MockerServerConfig](#mockerserverconfig) \| None | `None` | Per-mode CLI overrides |
 
+### TileRTProtocol
+
+`backend.type: tilert`
+
+| Key | Type | Default | Description |
+|---|---|---|---|
+| `prefill_environment` | dict[str, str] | `{}` |  |
+| `decode_environment` | dict[str, str] | `{}` |  |
+| `tilert_config` | [TileRTServerConfig](#tilertserverconfig) \| None | `None` |  |
+
 ### SGLangServerConfig
 
 SGLang server CLI configuration per mode (prefill/decode/aggregated).
@@ -205,6 +216,15 @@ Mocker CLI configuration per mode (prefill/decode/aggregated).
 | `decode` | dict[str, Any] \| None | `None` |  |
 | `aggregated` | dict[str, Any] \| None | `None` |  |
 
+### TileRTServerConfig
+
+Role-specific CLI arguments not owned by the topology adapter.
+
+| Key | Type | Default | Description |
+|---|---|---|---|
+| `prefill` | dict[str, Any] \| None | `None` |  |
+| `decode` | dict[str, Any] \| None | `None` |  |
+
 ## infra
 
 ### InfraConfig

docs/schema-reference.md

@@ -13,7 +13,7 @@ Top-level keys of a recipe YAML.
 | `name` | str | required |  |
 | `model` | [ModelConfig](#modelconfig) | required |  |
 | `resources` | [ResourceConfig](#resourceconfig) | required |  |
-| `engine` | str \| mapping | required | The engine type (`sglang`, `trtllm`, `vllm`, `mocker`) as a string, or a mapping with `type` plus the engine-wide knobs listed under [Engine types](#engine-types). |
+| `engine` | str \| mapping | required | The engine type (`sglang`, `trtllm`, `vllm`, `mocker`, `tilert`) as a string, or a mapping with `type` plus the engine-wide knobs listed under [Engine types](#engine-types). |
 | `roles` | mapping of role -> [Role](#roles) | required | One block per worker role (`prefill`, `decode`, `agg`): topology, env, and engine args. |
 | `schema` | int | `2` | Recipe schema version. Write `schema: 2` for this layout. |
 | `slurm` | [SlurmConfig](#slurmconfig) | `SlurmConfig()` |  |
@@ -44,7 +44,7 @@ Three vocabularies are specific to the 2.0 layout. They are normalized into the
 
 ### engine
 
-`engine: <type>` or `engine: {type: <type>, ...}`. `type` is one of `sglang`, `trtllm`, `vllm`, `mocker`; the remaining keys are that engine's knobs, listed under [Engine types](#engine-types).
+`engine: <type>` or `engine: {type: <type>, ...}`. `type` is one of `sglang`, `trtllm`, `vllm`, `mocker`, `tilert`; the remaining keys are that engine's knobs, listed under [Engine types](#engine-types).
 
 ### roles
 
@@ -112,7 +112,7 @@ Frontend/router configuration.
 
 | Key | Type | Default | Description |
 |---|---|---|---|
-| `type` | str | `'dynamo'` | Frontend type - "dynamo" (default); "sglang-router" (SGLang Model Gateway) and "vllm-router" (static routers); "sglang", "vllm", and "trtllm_serve" (direct: the single aggregate worker binds the public port, no router process). In schema 1 recipes "sglang" still means the router and loads as "sglang-router". |
+| `type` | str | `'dynamo'` | Frontend type - "dynamo" (default); "sglang-router" (SGLang Model Gateway), "vllm-router", and "tilert-router" (static routers); "sglang", "vllm", and "trtllm_serve" (direct: the single aggregate worker binds the public port, no router process). In schema 1 recipes "sglang" still means the router and loads as "sglang-router". |
 | `enable_multiple_frontends` | bool | `True` | Scale with nginx + multiple routers. When ``True`` (default), srtctl stands up nginx and fans out to ``num_additional_frontends + 1`` router replicas. When ``False``, there is NO nginx proxy — the benchmark must target the single master router (or a worker) directly at ``http://localhost:<port>``. ``benchmark.command`` has no placeholder substitution, so write the URL out literally. |
 | `num_additional_frontends` | int | `9` | Additional routers beyond master (default: 9) |
 | `nginx_container` | str | `'nginx:1.27.4'` | Custom nginx container image (default: nginx:1.27.4) |
@@ -595,6 +595,27 @@ Dynamo Mocker protocol - implements BackendProtocol.
 | `enable_chunked_prefill` | bool | `True` |  |
 | `preemption_mode` | str \| None | `None` |  |
 
+### TileRTProtocol
+
+`engine.type: tilert`
+
+Run vLLM prefill workers and TileRT decode workers without Dynamo.
+
+| Key | Type | Default | Description |
+|---|---|---|---|
+| `type` | one of `'tilert'` | `'tilert'` |  |
+| `served_model_name` | str \| None | `None` | OpenAI model name; defaults to the final component of model.path. |
+| `prefill_container` | str \| None | `None` | vLLM image or cluster container alias for prefill workers only. |
+| `model_profile` | str | `'glm5'` | TileRT P/D model profile (for example glm5). |
+| `weight_model_type` | str | `'glm-5'` | Model type passed to the TileRT weight converter. |
+| `weights_dir` | str | `'/tilert_weights/converted'` | Dedicated shared converted-weight directory, mounted at this path in all decode containers. |
+| `max_seq_len` | int | `202752` | Maximum sequence length shared by prefill and decode. |
+| `kv_cache_dtype` | str | `'fp8'` | TileRT decode KV cache dtype. |
+| `prefill_kv_cache_dtype` | str | `'fp8_ds_mla'` | vLLM KV cache dtype; must be wire-compatible with the decode dtype. |
+| `transport` | one of `'nixl'` | `'nixl'` | KV transfer transport (the adapter supports NIXL). |
+| `with_mtp` | bool | `True` | Enable MTP on both sides of the P/D transfer. |
+| `speculative_tokens` | int | `1` | Number of speculative tokens advertised by vLLM prefill. |
+
 ## Cluster config
 
 Top-level keys of `srtslurm.yaml`. Recipes inherit these defaults and resolve aliases through them.

docs/tilert.md

@@ -0,0 +1,95 @@
+# TileRT with vLLM prefill
+
+`engine.type: tilert` pairs a native vLLM prefill server with native TileRT decode
+servers. `frontend.type: tilert-router` runs TileRT's own P/D router. It does not
+start Dynamo, etcd, or NATS. The adapter implements `BackendProtocol`; the router
+extends `StaticRouterFrontend` and uses its worker discovery, Slurm launch, and
+process cleanup.
+
+See [`examples/tilert/glm5-disagg.yaml`](https://github.com/SemiAnalysisAI/srt-slurm/blob/main/examples/tilert/glm5-disagg.yaml)
+for a schema 2 recipe. This is a launch example, not a claim of a hardware-validated
+benchmark result. The native interfaces follow
+[TileRT's P/D implementation](https://github.com/tile-ai/TileRT/tree/main/tilert/pd_vllm).
+
+## Images and setup
+
+- `model.container` selects the TileRT decode image.
+- `engine.prefill_container` selects the vLLM prefill image. It accepts the same
+  cluster `containers:` aliases as `model.container`.
+- `frontend.container_image` optionally selects a router image, otherwise it
+  uses the decode image.
+
+The example uses `ghcr.io/tile-ai/tilert:0.1.5` and
+`vllm/vllm-openai:v0.26.0`. `setup_script: tilert_setup.sh` installs
+TileRT `0.1.5.post3` and NIXL `1.3.1`; it adds the HTTP dependencies to decode and
+router containers and Transformers 5 to the router for GLM-5.1 tokenizers.
+The prefill install uses `--no-deps` for TileRT to preserve vLLM's dependencies.
+The script accepts `TILERT_VERSION`, `TILERT_NIXL_VERSION`,
+`TILERT_FASTAPI_VERSION`, `TILERT_UVICORN_VERSION`, and
+`TILERT_TRANSFORMERS_VERSION` overrides. Omit setup when using prepared images.
+The adapter supplies `TILERT_ROLE` to each worker and to the router setup.
+
+Use `roles.prefill.args` / `roles.decode.args` and `roles.<role>.env` for normal
+engine arguments and environment. srtctl owns ports, model paths, prefill TP,
+KV transfer configuration, and MTP flags. Attempts to override those arguments
+are rejected. `engine.served_model_name` overrides the OpenAI model name;
+otherwise the final component of `model.path` is used.
+
+## Weight conversion and mounts
+
+TileRT requires converted weights. Before workers launch, the backend requests
+one GPU on the first decode worker's node and runs TileRT's weight converter in
+the decode image. The step is tracked in the normal process registry, including
+signal cleanup, failure propagation, and a six-hour timeout. Its log is
+`tilert_weight_conversion.out`.
+
+Set `engine.weights_dir` to a dedicated directory under a shared writable mount,
+visible at the same container path to decode workers **and the router**. For
+example, add `/shared/tilert:/tilert_weights` to the cluster's mounts. Use a
+separate directory for every model, revision, and converter version; the adapter
+does not infer cache identity from model contents. Do not point it at source
+weights or a directory with unrelated files.
+
+Conversion uses a filesystem lock shared by concurrent jobs, writes to a sibling
+temporary directory, copies tokenizer metadata and the chat template, and
+publishes only after checking the index and every referenced shard. Complete
+caches are reused. A cache with missing shards or tokenizer metadata is rebuilt.
+Failed conversions leave the previous target untouched. The router loads its
+tokenizer from this prepared directory, including for local model paths.
+Hugging Face sources are completed through `snapshot_download` before conversion;
+local sources use the worker's mounted model path.
+
+## Topology, readiness, and metrics
+
+The native router supports exactly one prefill worker and one or more decode
+workers. Every worker must fit on one node. The GLM-5 decode kernels require
+exactly eight GPUs per decode worker. Use one frontend
+(`enable_multiple_frontends: false`); aggregate mode, Dynamo sidecars, and Torch
+profiling are rejected. Nsys profiling remains available.
+
+The backend configures the TileRT NIXL connector on vLLM, derives TP from the
+allocated prefill GPUs, and supplies matching context length and transport to
+both roles. `engine.with_mtp` controls MTP on both sides. Choose compatible
+`kv_cache_dtype` and `prefill_kv_cache_dtype` values for the model profile.
+
+The router is started after every worker `/health` endpoint is ready. Before
+benchmark traffic, srtctl checks the router's `/health` plus all worker endpoints;
+a healthy router alone does not prove the workers are ready. AIPerf and Tachometer receive
+only vLLM prefill Prometheus URLs because TileRT decode does not expose `/metrics`.
+
+## Backend extension points
+
+TileRT uses two optional backend hooks alongside the required `BackendProtocol`
+methods:
+
+- `get_container_image_for_mode(mode, default)` selects a role image for worker
+  and preparation launches. Backends without it keep `model.container`.
+- `get_preparation(runtime, processes)` returns a `BackendPreparation` describing
+  a finite pre-worker command, selected node/mode, log, timeout, and optional GPU
+  request. The orchestrator runs it after HF download/model staging and before
+  workers, using that role's image, setup, mounts, and environment.
+
+`get_metrics_port(process)` optionally limits AIPerf discovery to real metrics
+endpoints for both AIPerf and Tachometer. Returning `None` omits that process from metrics, without changing
+its serving or profiling endpoint. Backends without it retain existing discovery.
+A frontend may set `has_metrics = False` to omit its own Tachometer target.

examples/README.md

@@ -32,6 +32,12 @@ Every example is written in the 2.0 layout: `engine:` names the engine (a string
 | `features/infra-services.yaml` | etcd and NATS as declared services on a dedicated node with a NATS payload limit; the implied exporters overridden or switched off |
 | `features/dynamo-source.yaml` | `dynamo.source:` building Dynamo from a git tag (or a PR head via `--set dynamo.source.rev=refs/pull/<n>/head`), pinned to a commit at submit |
 
+## TileRT
+
+`tilert/glm5-disagg.yaml` runs vLLM prefill and TileRT decode with a native
+TileRT router on two B200 nodes. See [the TileRT guide](../docs/tilert.md) for
+image and shared converted-weight mount requirements.
+
 ## Cluster aliases
 
 The examples reference two kinds of aliases that `srtslurm.yaml` resolves:

examples/tilert/glm5-disagg.yaml

@@ -0,0 +1,65 @@
+# Native vLLM prefill + TileRT decode, one 8-GPU B200 node per role.
+# Mount a shared writable directory at /tilert_weights on every node.
+# See docs/tilert.md for images, setup, and cache requirements.
+schema: 2
+name: glm5-tilert-disagg
+
+model:
+  path: hf:zai-org/GLM-5.1-FP8
+  container: ghcr.io/tile-ai/tilert:0.1.5
+  precision: fp8
+
+identity:
+  model:
+    repo: zai-org/GLM-5.1-FP8
+    revision: f396cf805182f4ca10fa675e1a99815b3ca384db
+  container:
+    image: ghcr.io/tile-ai/tilert:0.1.5
+  frameworks:
+    tilert: 0.1.5.post3
+    vllm: 0.26.0
+    nixl: 1.3.1
+
+resources:
+  gpu_type: b200
+  gpus_per_node: 8
+
+engine:
+  type: tilert
+  prefill_container: vllm/vllm-openai:v0.26.0
+  model_profile: glm5
+  weight_model_type: glm-5
+  weights_dir: /tilert_weights/glm5.1-fp8
+  max_seq_len: 202752
+  with_mtp: true
+
+roles:
+  prefill:
+    nodes: 1
+    workers: 1
+    gpus: 8
+    args:
+      enforce-eager: true
+      trust-remote-code: true
+      gpu-memory-utilization: 0.75
+  decode:
+    nodes: 1
+    workers: 1
+    gpus: 8
+
+frontend:
+  type: tilert-router
+  enable_multiple_frontends: false
+  args:
+    parser: none
+    queue-timeout: 1800
+
+setup_script: tilert_setup.sh
+srun_options:
+  container-writable: ""
+health_check:
+  max_attempts: 720
+  interval_seconds: 5
+
+benchmark:
+  type: manual

mkdocs.yml

@@ -59,6 +59,7 @@ nav:
       - Architecture: architecture.md
       - SGLang Router: sglang-router.md
       - vLLM Router: vllm-router.md
+      - TileRT: tilert.md
       - Mooncake KV Store: mooncake-kv-store.md
       - Services: services.md
   - Benchmarking: