pranavraja99 · Sep 24, 2026 · Sep 24, 2026
diff --git a/docs/glm53-w4afp8-long-context-rollout.md b/docs/glm53-w4afp8-long-context-rollout.md
@@ -86,7 +86,30 @@ Never let orphan removal perform the switch. On gpu02, the only orphan relative
 - **Completions.** A real completion through the customer path and on `glm-5-3-flash-long.completions.near.ai`, then a cache-hit follow-up whose second turn reports cached prompt tokens.
 - **Host restore.** The replica's HiCache host-hit counters (engine `/metrics`, also reachable through the soak relay) increase for a follow-up whose prefix had left the device pool, with no `write_through_unbacked_eviction` drops.
 - **Registries.** Both model-proxy peers list gpu02 healthy under `glm-5-3-flash-long.completions.near.ai`, and not under the base domain (`LONG_TIER_ONLY=true`).
-- **Telemetry.** Both engines report precision `int4-weights-fp8-activations-bf16-kv`, the L2 `config_variant` and `engine_image` `fde25985aea3`.
+- **Telemetry.** Both engines report precision `int4-weights-fp8-activations-bf16-kv`, their L2 `config_variant` (`…-pool-clamp-pdi1-…` on r1, `…-pool-clamp-pdi2-…` on r2) and `engine_image` `fde25985aea3`.
+
+## Canary: `--prefill-decode-interval 2` on r2 (both replicas already on this file)
+
+r2 runs `--prefill-decode-interval 2` (two decode steps between prefill chunks). r1 stays at 1 and is the live control. Nothing else differs, so an operator can split the two replicas' TTFT and TPOT dashboards by `config_variant`.
+
+Lab evidence, gpu31 on stored long-tier traffic (2026-09-24):
+- **Paired runs, both island orders:** TPOT p50 0.53–0.72× of pdi 1, paired TTFT at most 1.02×, 100% completion.
+- **Combined qualification,** alongside the DSA indexer split and chunk 16384 (not in this file yet): no restarts, no OOMs, quality at the noise floor.
+
+Only r2's engine definition changed.
+1. `compose/down` this file with `["model-sg-glm53-w4afp8-tp4-r2"]`. r1 carries the tier while r2 is down, so pick the low-traffic window (gate 4).
+2. `compose/up` this file with the same `services`.
+3. Run the r2 startup and direct checks (steps 3–4 above).
+4. Do not recreate r1, the proxy or the registrar. A `dry_run` of step 2 must plan exactly `model-sg-glm53-w4afp8-tp4-r2`.
+
+**Readout.** Compare r2 against r1 over at least 24 h of normal traffic:
+- decode inter-token latency (p50/p90) under concurrent long prefills;
+- TTFT (p50/p95);
+- queue-full rejections, aborts, restarts, and SGLang free-memory warnings.
+
+pdi 2 should cut r2's inter-token latency during prefills without moving TTFT.
+
+**Next.** If it holds, a follow-up PR moves r1 to pdi 2. If r2 regresses, roll back: redeploy the previous tag of this file for r2 only, with the same `compose/down` then `compose/up` pair.
 
 ## Rollback
 

diff --git a/prod/GLM-5.3-Flash-SGL-TP4-W4AFP8-LongContext.yaml b/prod/GLM-5.3-Flash-SGL-TP4-W4AFP8-LongContext.yaml
@@ -3,7 +3,10 @@
 # Both replicas run the gpu31 campaign-2 arm L2 (2026-09-23): the W4AFP8 checkpoint
 # graphistry/GLM-5.3-Flash-W4AFP8@99f1fa7, 8192-token prefill chunks with
 # --max-prefill-tokens 32768, HiCache with CUDA-owned host memory and a fixed 406 GiB
-# startup host-memory budget per replica, and no admission reserve. Both pin
+# startup host-memory budget per replica, and no admission reserve. r2 is a canary at
+# --prefill-decode-interval 2 (two decode steps between prefill chunks); r1 stays at 1 as
+# the live control. On stored long-tier traffic (gpu31, 2026-09-24) pdi 2 cut TPOT p50 to
+# 0.53-0.72x of pdi 1 at a paired TTFT cost of at most 2%, with 100% completion. Both pin
 # docker.io/nearaidev/sglang@sha256:fde25985aea3ebabf1eb581ae21d53be8540e32933eef942ee8b962a1bfbea20
 # (docker/sglang-glm53-hicache-w4afp8), published by workflow run 35903077821 from recipe
 # merge commit f8106f096e9c838b283a0f79a452d6c43e470641. Its HCC/PPCIe host-memory path
@@ -12,7 +15,8 @@
 # Do not hand-edit this file.
 # This is the DEDICATED LONG-CONTEXT TIER variant of prod/GLM-5.3-Flash-SGL-TP4.yaml.
 # Generated from the long-context file. Routing remains the dedicated long-context
-# contract below; both replicas run the same W4AFP8 + HiCache engine.
+# contract below; both replicas run the same W4AFP8 + HiCache engine, r2 at
+# --prefill-decode-interval 2 as a canary.
 #
 # *** LONG-CONTEXT TIER (dedicated host, both TP4/EP4 replicas) ***
 #   Deploy this file on exactly the host(s) chosen as the GLM-5.3 Flash long-context
@@ -53,7 +57,8 @@
 #       turn is projected to double C_long; cloud-api routes per request today.
 #     - Request-level tiering trades the tail for the median: p50 TTFT -24%,
 #       p95 TTFT 10.0 s -> 27.4 s on identical replicas. Judge it on the SLO you mean.
-#   Both replicas run identical engine flags; only --dist-init-addr differs. The
+#   The replicas' engine flags differ only in --dist-init-addr and r2's canary
+#   --prefill-decode-interval 2. The
 #   8192-token chunk is the W4AFP8 setting: FP8 at 8192 OOMed in the DSA indexer under
 #   concurrent 400K+ contexts, while W4AFP8's 2.4x larger device KV pool kept 3.7 GiB
 #   free through 8 concurrent 647K-756K-token cold prefills on gpu31 (16384 fell to
@@ -96,8 +101,9 @@ x-logging-conf: &logging-conf
 # Admission reserve is deliberately disabled on both replicas: reserve v10 admitted a
 # 292K-cached waiter beside an in-flight chunked prefill on gpu02 and exhausted the
 # long-context token pool, and the qualified arm (L2) ran without it. The image's
-# reserve patch is inert while those variables are unset; --prefill-decode-interval 1
-# remains part of the serving contract. The official CUDA 13 SGLang image is the
+# reserve patch is inert while those variables are unset; --prefill-decode-interval
+# (1 on r1, 2 on the r2 canary) is part of the serving contract. The official CUDA 13
+# SGLang image is the
 # pinned build base. The serving envelope is
 # intentionally conservative: BF16 KV, 0.80 static memory, 32 running requests, a
 # bounded 8-request queue, 8192-token prefill chunks with --max-prefill-tokens 32768,
@@ -213,8 +219,8 @@ x-sg-glm53-flash-common: &sg-glm53-flash-common
     # proxy's 3 s probe times out, the replica flaps unhealthy, and the conversation
     # affinity pin is skipped (full re-prefill on the other replica). nearai/infra#234.
     - SGLANG_ENABLE_HEALTH_ENDPOINT_GENERATION=0
-    # No admission reserve on the long tier (see the header); --prefill-decode-interval 1
-    # is retained independently of the inert reserve patch.
+    # No admission reserve on the long tier (see the header); --prefill-decode-interval
+    # is set per replica, independently of the inert reserve patch.
     - SGLANG_ENABLE_STRICT_MEM_CHECK_DURING_IDLE=1
     # HiCache host tier: a fixed 406 GiB per replica across its four TP ranks (the
     # qualified L2 value). A percentage would resolve against MemAvailable at each start,
@@ -481,7 +487,7 @@ services:
         --disable-priority-preemption
         --chunked-prefill-size 8192
         --max-prefill-tokens 32768
-        --prefill-decode-interval 1
+        --prefill-decode-interval 2
         --cuda-graph-max-bs-decode 32
         --dsa-prefill-backend tilelang
         --dsa-decode-backend tilelang
@@ -521,7 +527,7 @@ services:
               device_ids: ["4","5","6","7"]
               capabilities: [gpu]
     labels:
-      com.datadoghq.ad.logs: '[{"source":"sglang","service":"sglang","tags":["model:z-ai/glm-5.3-flash","model_path:graphistry/GLM-5.3-Flash-W4AFP8","served_model:z-ai/glm-5.3-flash","precision:int4-weights-fp8-activations-bf16-kv","deployment:glm53-flash-sgl-tp4","config_variant:fc91d24-long-context-w4afp8-c8192-hicache-cuda-host-pooled-v1-admission-reserve-disabled-pool-clamp-pdi1-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192","request_logging:disabled","engine_image:fde25985aea3","env:${ENV}","host:${CVM_HOST}","ip:${HOST_IP}","port:8000","instance:2"]}]'
+      com.datadoghq.ad.logs: '[{"source":"sglang","service":"sglang","tags":["model:z-ai/glm-5.3-flash","model_path:graphistry/GLM-5.3-Flash-W4AFP8","served_model:z-ai/glm-5.3-flash","precision:int4-weights-fp8-activations-bf16-kv","deployment:glm53-flash-sgl-tp4","config_variant:fc91d24-long-context-w4afp8-c8192-hicache-cuda-host-pooled-v1-admission-reserve-disabled-pool-clamp-pdi2-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192","request_logging:disabled","engine_image:fde25985aea3","env:${ENV}","host:${CVM_HOST}","ip:${HOST_IP}","port:8000","instance:2"]}]'
       nearai.otel.scrape: "true"
       nearai.otel.job: "sglang"
       nearai.otel.service: "sglang"
@@ -533,7 +539,7 @@ services:
       nearai.otel.model_path: "graphistry/GLM-5.3-Flash-W4AFP8"
       nearai.otel.served_model: "z-ai/glm-5.3-flash"
       nearai.otel.deployment: "glm53-flash-sgl-tp4"
-      nearai.otel.config_variant: "fc91d24-long-context-w4afp8-c8192-hicache-cuda-host-pooled-v1-admission-reserve-disabled-pool-clamp-pdi1-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192"
+      nearai.otel.config_variant: "fc91d24-long-context-w4afp8-c8192-hicache-cuda-host-pooled-v1-admission-reserve-disabled-pool-clamp-pdi2-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192"
       nearai.otel.thinking_budget_policy: "default8192-public-to-native"
       nearai.otel.request_logging: "disabled"
       nearai.otel.engine_image: "fde25985aea3"
@@ -959,7 +965,7 @@ configs:
                       ip: "${HOST_IP}"
                       port: "8000"
                       instance: "2"
-                      config_variant: "fc91d24-long-context-w4afp8-c8192-hicache-cuda-host-pooled-v1-admission-reserve-disabled-pool-clamp-pdi1-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192"
+                      config_variant: "fc91d24-long-context-w4afp8-c8192-hicache-cuda-host-pooled-v1-admission-reserve-disabled-pool-clamp-pdi2-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192"
                       thinking_budget_policy: "default8192-public-to-native"
                       request_logging: "disabled"
                       engine_image: "fde25985aea3"

diff --git a/scripts/prepare_glm53_w4afp8_long_context.py b/scripts/prepare_glm53_w4afp8_long_context.py
@@ -37,10 +37,15 @@
     "fc91d24-long-context-admission-reserve-disabled-hicache-disabled-pdi1-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192",
     "fc91d24-long-context-admission-reserve-disabled-hicache-cuda-host-pooled-v1-pdi1-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192",
 )
-VARIANT: Final = (
-    "fc91d24-long-context-w4afp8-c8192-hicache-cuda-host-pooled-v1-admission-reserve-disabled"
-    "-pool-clamp-pdi1-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192"
-)
+# r1 is the unchanged #294 control; r2 is the --prefill-decode-interval 2 canary. The
+# variants differ only in pdi1/pdi2 so dashboards split the replicas by config_variant.
+VARIANTS: Final = {
+    1: "fc91d24-long-context-w4afp8-c8192-hicache-cuda-host-pooled-v1-admission-reserve-disabled"
+    "-pool-clamp-pdi1-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192",
+    2: "fc91d24-long-context-w4afp8-c8192-hicache-cuda-host-pooled-v1-admission-reserve-disabled"
+    "-pool-clamp-pdi2-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192",
+}
+PREFILL_DECODE_INTERVAL: Final = {1: 1, 2: 2}
 SOURCE_DIST_INIT: Final = "127.0.0.1:29510"
 DIST_INIT: Final = {1: "127.0.0.1:29510", 2: "127.0.0.1:29511"}
 HICACHE_FLAGS: Final = (
@@ -61,7 +66,10 @@
     "# Both replicas run the gpu31 campaign-2 arm L2 (2026-09-23): the W4AFP8 checkpoint\n"
     "# graphistry/GLM-5.3-Flash-W4AFP8@99f1fa7, 8192-token prefill chunks with\n"
     "# --max-prefill-tokens 32768, HiCache with CUDA-owned host memory and a fixed 406 GiB\n"
-    "# startup host-memory budget per replica, and no admission reserve. Both pin\n"
+    "# startup host-memory budget per replica, and no admission reserve. r2 is a canary at\n"
+    "# --prefill-decode-interval 2 (two decode steps between prefill chunks); r1 stays at 1 as\n"
+    "# the live control. On stored long-tier traffic (gpu31, 2026-09-24) pdi 2 cut TPOT p50 to\n"
+    "# 0.53-0.72x of pdi 1 at a paired TTFT cost of at most 2%, with 100% completion. Both pin\n"
     f"# {IMAGE}\n"
     "# (docker/sglang-glm53-hicache-w4afp8), published by workflow run 35903077821 from recipe\n"
     "# merge commit f8106f096e9c838b283a0f79a452d6c43e470641. Its HCC/PPCIe host-memory path\n"
@@ -76,7 +84,8 @@
         "# long-context contract below, while the engines intentionally form an r1 control / r2\n"
         "# HiCache experiment and therefore are not byte-identical to the canonical file.\n",
         "# Generated from the long-context file. Routing remains the dedicated long-context\n"
-        "# contract below; both replicas run the same W4AFP8 + HiCache engine.\n",
+        "# contract below; both replicas run the same W4AFP8 + HiCache engine, r2 at\n"
+        "# --prefill-decode-interval 2 as a canary.\n",
     ),
     (
         "#   tier; every other host keeps the canonical file. The inference-proxy and routing\n"
@@ -91,7 +100,8 @@
         "#   DSA indexer under concurrent 400K+ contexts (exactly this tier's load), chunk 2048\n"
         "#   halves prefill speed, and no other per-replica flag moved the tail. Conversation\n"
         "#   affinity stays on because the prefix cache is worth ~3x in request capacity.\n",
-        "#   Both replicas run identical engine flags; only --dist-init-addr differs. The\n"
+        "#   The replicas' engine flags differ only in --dist-init-addr and r2's canary\n"
+        "#   --prefill-decode-interval 2. The\n"
         "#   8192-token chunk is the W4AFP8 setting: FP8 at 8192 OOMed in the DSA indexer under\n"
         "#   concurrent 400K+ contexts, while W4AFP8's 2.4x larger device KV pool kept 3.7 GiB\n"
         "#   free through 8 concurrent 647K-756K-token cold prefills on gpu31 (16384 fell to\n"
@@ -133,8 +143,9 @@
         "# Admission reserve is deliberately disabled on both replicas: reserve v10 admitted a\n"
         "# 292K-cached waiter beside an in-flight chunked prefill on gpu02 and exhausted the\n"
         "# long-context token pool, and the qualified arm (L2) ran without it. The image's\n"
-        "# reserve patch is inert while those variables are unset; --prefill-decode-interval 1\n"
-        "# remains part of the serving contract. The official CUDA 13 SGLang image is the\n"
+        "# reserve patch is inert while those variables are unset; --prefill-decode-interval\n"
+        "# (1 on r1, 2 on the r2 canary) is part of the serving contract. The official CUDA 13\n"
+        "# SGLang image is the\n"
         "# pinned build base. The serving envelope is\n",
     ),
     (
@@ -158,8 +169,8 @@
     "  restart: unless-stopped\n"
 )
 ANCHOR_ENV_NEW: Final = (
-    "    # No admission reserve on the long tier (see the header); --prefill-decode-interval 1\n"
-    "    # is retained independently of the inert reserve patch.\n"
+    "    # No admission reserve on the long tier (see the header); --prefill-decode-interval\n"
+    "    # is set per replica, independently of the inert reserve patch.\n"
     "    - SGLANG_ENABLE_STRICT_MEM_CHECK_DURING_IDLE=1\n"
     "    # HiCache host tier: a fixed 406 GiB per replica across its four TP ranks (the\n"
     "    # qualified L2 value). A percentage would resolve against MemAvailable at each start,\n"
@@ -204,6 +215,7 @@ def engine_arguments(source: list[str], replica: int) -> list[str]:
         f"--revision {FP8_REVISION}",
         "--moe-runner-backend deep_gemm",
         "--chunked-prefill-size 4096",
+        "--prefill-decode-interval 1",
         f"--dist-init-addr {SOURCE_DIST_INIT}",
     }
     missing = sorted(argument for argument in expected if source.count(argument) != 1)
@@ -219,6 +231,8 @@ def engine_arguments(source: list[str], replica: int) -> list[str]:
             arguments.append(MODEL_PATH)
         elif argument == "--chunked-prefill-size 4096":
             arguments.extend(("--chunked-prefill-size 8192", "--max-prefill-tokens 32768"))
+        elif argument == "--prefill-decode-interval 1":
+            arguments.append(f"--prefill-decode-interval {PREFILL_DECODE_INTERVAL[replica]}")
         elif argument == f"--dist-init-addr {SOURCE_DIST_INIT}":
             arguments.append(f"--dist-init-addr {DIST_INIT[replica]}")
         else:
@@ -270,8 +284,8 @@ def generate(source: str) -> str:
         ('model_path: "zai-org/GLM-5.3-Flash"', f'model_path: "{CHECKPOINT}"', 6, "metric model_path"),
         ("precision:fp8-weights-bf16-kv", f"precision:{PRECISION}", 2, "log precision"),
         ('precision: "fp8-weights-bf16-kv"', f'precision: "{PRECISION}"', 2, "scrape precision"),
-        (SOURCE_VARIANTS[0], VARIANT, 3, "replica 1 config_variant"),
-        (SOURCE_VARIANTS[1], VARIANT, 3, "replica 2 config_variant"),
+        (SOURCE_VARIANTS[0], VARIANTS[1], 3, "replica 1 config_variant"),
+        (SOURCE_VARIANTS[1], VARIANTS[2], 3, "replica 2 config_variant"),
         ('"request_logging:disabled",', f'"request_logging:disabled","engine_image:{ENGINE_IMAGE_LABEL}",', 2, "log engine_image"),
         (
             '      nearai.otel.request_logging: "disabled"\n',

diff --git a/scripts/test_glm53_w4afp8_long_context.py b/scripts/test_glm53_w4afp8_long_context.py
@@ -119,6 +119,9 @@ def test_rejects_engine_argv_drift(self) -> None:
                 argv,
             ),
             ("\n      --max-queued-requests 8\n", "\n      --max-queued-requests 32\n", argv),
+            # r1 is the pdi 1 control and r2 the pdi 2 canary; neither may take the other's value.
+            ("\n      --prefill-decode-interval 1\n", "\n      --prefill-decode-interval 2\n", "--prefill-decode-interval 1"),
+            ("\n        --prefill-decode-interval 2\n", "\n        --prefill-decode-interval 1\n", "--prefill-decode-interval 2"),
         )
         for before, after, message in cases:
             with self.subTest(mutation=after.strip()[:60]):
@@ -163,9 +166,11 @@ def test_rejects_routing_drift_from_the_long_context_file(self) -> None:
 
     def test_rejects_untruthful_telemetry(self) -> None:
         cases = (
-            (f'nearai.otel.config_variant: "{generator.VARIANT}"', 'nearai.otel.config_variant: "incorrect-variant"', 1,
+            (f'nearai.otel.config_variant: "{generator.VARIANTS[1]}"', 'nearai.otel.config_variant: "incorrect-variant"', 0,
              "nearai.otel.config_variant must be"),
-            (f"config_variant:{generator.VARIANT}", "config_variant:incorrect-variant", 0, "log metadata must carry exactly config_variant:"),
+            (f'nearai.otel.config_variant: "{generator.VARIANTS[2]}"', f'nearai.otel.config_variant: "{generator.VARIANTS[1]}"', 0,
+             "nearai.otel.config_variant must be"),
+            (f"config_variant:{generator.VARIANTS[1]}", "config_variant:incorrect-variant", 0, "log metadata must carry exactly config_variant:"),
             ('      nearai.otel.engine_image: "fde25985aea3"\n', '      nearai.otel.engine_image: "e9d29a1cb1cd"\n', 1, "nearai.otel.engine_image must be"),
             ('"precision:int4-weights-fp8-activations-bf16-kv"', '"precision:fp8-weights-bf16-kv"', 0, "log metadata must carry precision:"),
             ('                      engine_image: "fde25985aea3"\n', '                      engine_image: "e9d29a1cb1cd"\n', 1, "scrape label engine_image"),