[Klaud Cold] [TileRT] GLM-5.3 FP8 MI355X AgentX: tilert 0.1.6.post3 + vLLM 0.28/ATOM prefill / [Klaud Cold] [TileRT] GLM-5.3 FP8 MI355X AgentX：tilert 0.1.6.post3 + vLLM 0.28/ATOM prefill

inferencex-e2e/benchmarks/multi_node/srt-slurm-recipes/glm5.3/tilert/mi355x-fp8/agentx/disagg-1p1d-tp8-mtp.yaml

@@ -11,7 +11,7 @@ resources:
   gpus_per_node: 8
 setup_script: glm5.3-tilert-rocm.sh
 environment:
-  TILERT_VERSION: "0.1.6.post2"
+  TILERT_VERSION: "0.1.6.post3"
   RDMAV_FORK_SAFE: "1"
   PYTHONDONTWRITEBYTECODE: "1"
   NCCL_SOCKET_IFNAME: eno0
@@ -22,7 +22,7 @@ roles:
     engine:
       type: vllm
       set_visible_devices: true
-    container: ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6
+    container: ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6.post1
     nodes: 1
     workers: 1
     gpus: 8
@@ -32,16 +32,22 @@ roles:
       VLLM_ROCM_USE_AITER_MOE: "1"
       VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS: "1"
       VLLM_ENGINE_READY_TIMEOUT_S: "10800"
+      AITER_QUICK_REDUCE_QUANTIZATION: INT4
+      AITER_USE_FLYDSL_MOE_SORTING: "1"
     args:
       served-model-name: glm5_2
       tensor-parallel-size: 8
       max-model-len: 1048576
-      enforce-eager: true
       trust-remote-code: true
       return-tokens-as-token-ids: true
       gpu-memory-utilization: 0.85
       kv-cache-dtype: bfloat16
       block-size: 64
+      async-scheduling: true
+      load-format: fastsafetensors
+      compilation-config: '{"cudagraph_mode":"FULL_AND_PIECEWISE"}'
+      max-num-batched-tokens: 16384
+      enable-prefix-caching: true
       speculative-config: '{"method":"mtp","num_speculative_tokens":1}'
       kv-transfer-config: >-
         {"kv_connector":"TileRTConnector","kv_connector_module_path":"tilert.pd_vllm.prefill_connector",

inferencex-e2e/configs/amd-master.yaml

@@ -1486,7 +1486,7 @@ glm5.3-fp8-mi355x-tilert-agentic:
   runner: cluster:mi355x-amds
   precision: fp8
   framework: tilert
-  router: { name: tilert-pd-router, version: "0.1.6.post2" }
+  router: { name: tilert-pd-router, version: "0.1.6.post3" }
   multinode: true
   disagg: true
   kv-p2p-transfer: mooncake
@@ -1501,7 +1501,7 @@ glm5.3-fp8-mi355x-tilert-agentic:
           ep: 1
           dp-attn: false
           additional-settings:
-          - "PREFILL_IMAGE=ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6"
+          - "PREFILL_IMAGE=ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6.post1"
           - "CONFIG_FILE=recipes/glm5.3/tilert/mi355x-fp8/agentx/disagg-1p1d-tp8-mtp.yaml"
         decode:
           num-worker: 1

inferencex-e2e/perf-changelog.yaml

@@ -9168,3 +9168,12 @@
     - "Relevant ATOM changes in the range: V4 decode reuses the sparse prefill ASM (ROCm/ATOM#2271), greedy sampler picks via aiter.topk_select (ROCm/ATOM#2244), and FP8 block scales declared scale_fmt ue8m0 are now stored as E8M0 on gfx950 by default (ROCm/ATOM#2419; previously FP32 unless ATOM_FP8_BLOCKSCALE_USE_E8M0_SCALE=1). The upstream DeepSeek-V4 recipes are unchanged across the range, and every recipe flag and choice (all2all-backend rccl, dp-load-balance least_tokens, moe-backend standard) remains valid."
     - "No data-type or precision change to the DeepSeek-V4-Pro-0813 DSpark draft: no online quantization is configured, so it keeps its checkpoint precision; the E8M0 scale storage represents the checkpoint's power-of-two block scales exactly. kv-cache-dtype and index-cache-dtype touch cache storage only."
   pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3605
+
+- config-keys:
+    - glm5.3-fp8-mi355x-tilert-agentic
+  scenario-type:
+    - agentic-coding
+  description:
+    - "Bump tilert 0.1.6.post2 -> 0.1.6.post3 (PyPI, 2026-09-28) on both TileRT ranks; router metadata follows. post3 turns on three prefill-to-decode latency features by default: multi-sender staging (TILERT_PD_SENDERS=8, each TP rank extracts and sends its own layers), per-layer pipelined KV send (TILERT_PD_PIPELINE=1, a layer is extracted and RDMA-written as soon as it is computed), and router-side incremental chat tokenization (TILERT_ROUTER_TOKENIZE=1, the router tokenizes only the new turn through a per-segment cache and hands vLLM token ids; a start-up self-check against vLLM /tokenize keeps it off on any mismatch). Move the prefill rank to a vLLM 0.28 + ATOM image (ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6.post1, built on rocm/atom-dev:vllm-v0.28.0-nightly_20260923) with CUDA graphs, async scheduling, prefix caching and the GLM-5.2 ATOM MI355X agentic recipe's AITER settings. Decode image, 1M context, bf16 KV and layer-sharded PD buffers unchanged. TileRT reports AgentX TTFT p90 4.35 s -> 1.60 s at concurrency 1 (local 2x8 MI350X against the post2 MI355X run)."
+    - "两侧 TileRT rank 上 tilert 由 0.1.6.post2 升级到 0.1.6.post3（PyPI，2026-09-28），router 元数据随之更新。post3 默认打开三项 prefill 到 decode 的时延优化：多发送端暂存（TILERT_PD_SENDERS=8，每个 TP rank 抽取并发送自己负责的层）、逐层流水线发送 KV（TILERT_PD_PIPELINE=1，每层算完即抽取并 RDMA 写出）、router 侧增量对话分词（TILERT_ROUTER_TOKENIZE=1，router 经按段缓存只对新一轮分词，把 token id 交给 vLLM；启动自检与 vLLM /tokenize 不一致即保持关闭）。prefill rank 改用 vLLM 0.28 + ATOM 镜像（ghcr.io/tile-ai/tilert-rocm-prefill:0.1.6.post1，基于 rocm/atom-dev:vllm-v0.28.0-nightly_20260923），开启 CUDA graph、异步调度、prefix caching，并沿用 GLM-5.2 ATOM MI355X agentic 配方的 AITER 设置。decode 镜像、1M 上下文、bf16 KV 与按层分片的 PD 缓冲不变。TileRT 报告并发 1 下 AgentX TTFT p90 由 4.35 秒降至 1.60 秒（本地 2x8 MI350X 对比 post2 的 MI355X 官方轮次）。"
+  pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3617