Encode long-tier prompts off the HTTP event loop (--enable-dynamic-batch-tokenizer)

docs/glm53-w4afp8-long-context-rollout.md

@@ -135,6 +135,24 @@ Deployment is the same r2-only pair as the pdi 2 canary: `compose/down` then `co
 
 **Untested before this canary:** the all-gather's cost under CC/PPCIe inside a CVM. gpu31 is bare metal with CC off, so the r2 canary is the first measurement of it. If TTFT regresses against r1 rather than improving, suspect the all-gather and roll back.
 
+## Async tokenizer: `--enable-dynamic-batch-tokenizer` on both replicas
+
+Both replicas run `--enable-dynamic-batch-tokenizer`. Prompts are encoded on one background thread instead of the tokenizer manager's asyncio loop, which also serves `/health`, `/metrics` and every in-flight stream. The flag changes when tokenization runs, not its result. Multimodal requests tokenize inside the image processor and are unaffected.
+
+Why (gpu02, 2026-09-25 13:45–14:10Z): a burst of roughly 100 requests of 1.4 MB or more per 5 minutes made both replicas miss the CVM proxy's 3 s probe every 10–60 s (about 35 unhealthy/recovered transitions in 25 minutes), the proxy failed over 600 chat requests with "Internal server error (error sending request)", and the OpenRouter gateway refused the long tier as `tier_unavailable`, while the GPUs stayed 40–90% busy. Offline, the tokenizer is linear (0.3 MB in at most 0.7 s) and its batch encode releases the GIL, so on a background thread a 2 MB prompt costs a few seconds of tokenizer time without stalling the loop.
+
+What it does not do: it adds no tokenizer throughput. Large prompts still queue behind one thread, so their own TTFT is unchanged. What changes is that probes, metrics and the other streams keep flowing, and overload surfaces as the bounded queue's 429s instead of proxy timeouts.
+
+Deployment: both engine definitions changed, so recreate one replica at a time, r2 first, then r1, each with `compose/down` then `compose/up` scoped to `["<replica>"]`, a `dry_run` that must plan exactly that one service, and the low-traffic window (gate 4). Allow about 15 minutes per replica; the other replica carries the tier meanwhile. Then recreate `otelcol-contrib` alone (`compose/up` scoped to `["otelcol-contrib"]`): both `config_variant` labels gained `dyntok`, and a replica-scoped deploy leaves the collector reporting the old variant (PR #303). Do not recreate the proxy or the registrar.
+
+Verification per replica:
+- the startup `server_args` line shows `'enable_dynamic_batch_tokenizer': True`;
+- `count by (container_name, config_variant) (sglang_num_running_reqs{host_machine="gpu02"})` shows the `dyntok` variant for that replica (the stale series lingers about 5 minutes);
+- over the first hour, proxy-glm53 `backend probe timed out` lines for that replica are rare during large-prompt bursts (before: dozens per 10 minutes) and `Backend marked unhealthy` does not appear;
+- TTFT p50/p95 and TPOT match the previous 24 h.
+
+Rollback: redeploy the previous tag of this file for the affected replica with the same `compose/down` then `compose/up` pair, then recreate `otelcol-contrib` again.
+
 ## Rollback
 
 Redeploy the previous tag and file, scoped to the same services, one replica at a time and under the same orphan rule. Keep at least one replica serving throughout.

prod/GLM-5.3-Flash-SGL-TP4-W4AFP8-LongContext.yaml

@@ -2,7 +2,11 @@
 # prod/GLM-5.3-Flash-SGL-TP4-LongContext.yaml by scripts/prepare_glm53_w4afp8_long_context.py.
 # Both replicas run the W4AFP8 checkpoint graphistry/GLM-5.3-Flash-W4AFP8@99f1fa7 with
 # --max-prefill-tokens 32768, HiCache with CUDA-owned host memory and a fixed 406 GiB
-# startup host-memory budget per replica, and no admission reserve.
+# startup host-memory budget per replica, and no admission reserve. Both also run
+# --enable-dynamic-batch-tokenizer: prompts are encoded on a background thread instead
+# of the tokenizer manager's event loop, so /health, /metrics and in-flight streams keep
+# flowing while a multi-MB prompt is tokenized (2026-09-25: both gpu02 replicas flapped
+# unhealthy under a burst of >=1.4 MB requests).
 #
 # THE REPLICAS ARE NOT IDENTICAL. r1 is the live control, r2 the canary:
 #   r1: gpu31 campaign-2 arm L2 (2026-09-23), chunk 8192, --prefill-decode-interval 1
@@ -120,8 +124,8 @@ x-logging-conf: &logging-conf
 # bounded 8-request queue, prefill chunks of 8192 (r1) / 16384 (r2) with
 # --max-prefill-tokens 32768,
 # decode graphs capped at batch 32, TileLang DSA, the CUTLASS W4A8 MoE path (hence no
-# --moe-runner-backend and no FP8 --revision), and adaptive EAGLE 5/1/6. This retains
-# the full 1,048,576-token model context.
+# --moe-runner-backend and no FP8 --revision), adaptive EAGLE 5/1/6, and the async
+# dynamic batch tokenizer. This retains the full 1,048,576-token model context.
 # Priority scheduling (--enable-priority-scheduling, --disable-priority-preemption)
 # relies on the pinned inference-proxy below (build 2834196, nearai/inference-proxy#241)
 # overwriting the `priority` field of every forwarded request, so no client-chosen
@@ -206,6 +210,7 @@ x-sg-glm53-flash-common: &sg-glm53-flash-common
       --hicache-write-policy write_through
       --hicache-io-backend direct
       --hicache-mem-layout page_first_direct
+      --enable-dynamic-batch-tokenizer
   volumes:
     - kernel_cache:/root/.cache
     - huggingface_cache:/root/.cache/huggingface
@@ -459,7 +464,7 @@ services:
               device_ids: ["0","1","2","3"]
               capabilities: [gpu]
     labels:
-      com.datadoghq.ad.logs: '[{"source":"sglang","service":"sglang","tags":["model:z-ai/glm-5.3-flash","model_path:graphistry/GLM-5.3-Flash-W4AFP8","served_model:z-ai/glm-5.3-flash","precision:int4-weights-fp8-activations-bf16-kv","deployment:glm53-flash-sgl-tp4","config_variant:fc91d24-long-context-w4afp8-c8192-hicache-cuda-host-pooled-v1-admission-reserve-disabled-pool-clamp-pdi1-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192","request_logging:disabled","engine_image:fde25985aea3","env:${ENV}","host:${CVM_HOST}","ip:${HOST_IP}","port:8000","instance:1"]}]'
+      com.datadoghq.ad.logs: '[{"source":"sglang","service":"sglang","tags":["model:z-ai/glm-5.3-flash","model_path:graphistry/GLM-5.3-Flash-W4AFP8","served_model:z-ai/glm-5.3-flash","precision:int4-weights-fp8-activations-bf16-kv","deployment:glm53-flash-sgl-tp4","config_variant:fc91d24-long-context-w4afp8-c8192-hicache-cuda-host-pooled-v1-admission-reserve-disabled-pool-clamp-pdi1-dyntok-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192","request_logging:disabled","engine_image:fde25985aea3","env:${ENV}","host:${CVM_HOST}","ip:${HOST_IP}","port:8000","instance:1"]}]'
       nearai.otel.scrape: "true"
       nearai.otel.job: "sglang"
       nearai.otel.service: "sglang"
@@ -471,7 +476,7 @@ services:
       nearai.otel.model_path: "graphistry/GLM-5.3-Flash-W4AFP8"
       nearai.otel.served_model: "z-ai/glm-5.3-flash"
       nearai.otel.deployment: "glm53-flash-sgl-tp4"
-      nearai.otel.config_variant: "fc91d24-long-context-w4afp8-c8192-hicache-cuda-host-pooled-v1-admission-reserve-disabled-pool-clamp-pdi1-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192"
+      nearai.otel.config_variant: "fc91d24-long-context-w4afp8-c8192-hicache-cuda-host-pooled-v1-admission-reserve-disabled-pool-clamp-pdi1-dyntok-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192"
       nearai.otel.thinking_budget_policy: "default8192-public-to-native"
       nearai.otel.request_logging: "disabled"
       nearai.otel.engine_image: "fde25985aea3"
@@ -529,6 +534,7 @@ services:
         --hicache-write-policy write_through
         --hicache-io-backend direct
         --hicache-mem-layout page_first_direct
+        --enable-dynamic-batch-tokenizer
     environment:
       - HF_TOKEN=${HUGGING_FACE_HUB_TOKEN}
       # Local snapshot path; nothing is fetched at engine start.
@@ -576,7 +582,7 @@ services:
               device_ids: ["4","5","6","7"]
               capabilities: [gpu]
     labels:
-      com.datadoghq.ad.logs: '[{"source":"sglang","service":"sglang","tags":["model:z-ai/glm-5.3-flash","model_path:graphistry/GLM-5.3-Flash-W4AFP8","served_model:z-ai/glm-5.3-flash","precision:int4-weights-fp8-activations-bf16-kv","deployment:glm53-flash-sgl-tp4","config_variant:fc91d24-long-context-w4afp8-c16384-qsplit-hicache-cuda-host-pooled-v1-admission-reserve-disabled-pool-clamp-pdi2-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192","request_logging:disabled","engine_image:8ff1a487b98a","env:${ENV}","host:${CVM_HOST}","ip:${HOST_IP}","port:8000","instance:2"]}]'
+      com.datadoghq.ad.logs: '[{"source":"sglang","service":"sglang","tags":["model:z-ai/glm-5.3-flash","model_path:graphistry/GLM-5.3-Flash-W4AFP8","served_model:z-ai/glm-5.3-flash","precision:int4-weights-fp8-activations-bf16-kv","deployment:glm53-flash-sgl-tp4","config_variant:fc91d24-long-context-w4afp8-c16384-qsplit-hicache-cuda-host-pooled-v1-admission-reserve-disabled-pool-clamp-pdi2-dyntok-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192","request_logging:disabled","engine_image:8ff1a487b98a","env:${ENV}","host:${CVM_HOST}","ip:${HOST_IP}","port:8000","instance:2"]}]'
       nearai.otel.scrape: "true"
       nearai.otel.job: "sglang"
       nearai.otel.service: "sglang"
@@ -588,7 +594,7 @@ services:
       nearai.otel.model_path: "graphistry/GLM-5.3-Flash-W4AFP8"
       nearai.otel.served_model: "z-ai/glm-5.3-flash"
       nearai.otel.deployment: "glm53-flash-sgl-tp4"
-      nearai.otel.config_variant: "fc91d24-long-context-w4afp8-c16384-qsplit-hicache-cuda-host-pooled-v1-admission-reserve-disabled-pool-clamp-pdi2-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192"
+      nearai.otel.config_variant: "fc91d24-long-context-w4afp8-c16384-qsplit-hicache-cuda-host-pooled-v1-admission-reserve-disabled-pool-clamp-pdi2-dyntok-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192"
       nearai.otel.thinking_budget_policy: "default8192-public-to-native"
       nearai.otel.request_logging: "disabled"
       nearai.otel.engine_image: "8ff1a487b98a"
@@ -989,7 +995,7 @@ configs:
                       ip: "${HOST_IP}"
                       port: "8000"
                       instance: "1"
-                      config_variant: "fc91d24-long-context-w4afp8-c8192-hicache-cuda-host-pooled-v1-admission-reserve-disabled-pool-clamp-pdi1-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192"
+                      config_variant: "fc91d24-long-context-w4afp8-c8192-hicache-cuda-host-pooled-v1-admission-reserve-disabled-pool-clamp-pdi1-dyntok-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192"
                       thinking_budget_policy: "default8192-public-to-native"
                       request_logging: "disabled"
                       engine_image: "fde25985aea3"
@@ -1014,7 +1020,7 @@ configs:
                       ip: "${HOST_IP}"
                       port: "8000"
                       instance: "2"
-                      config_variant: "fc91d24-long-context-w4afp8-c16384-qsplit-hicache-cuda-host-pooled-v1-admission-reserve-disabled-pool-clamp-pdi2-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192"
+                      config_variant: "fc91d24-long-context-w4afp8-c16384-qsplit-hicache-cuda-host-pooled-v1-admission-reserve-disabled-pool-clamp-pdi2-dyntok-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192"
                       thinking_budget_policy: "default8192-public-to-native"
                       request_logging: "disabled"
                       engine_image: "8ff1a487b98a"

scripts/prepare_glm53_w4afp8_long_context.py

@@ -49,13 +49,14 @@
     "fc91d24-long-context-admission-reserve-disabled-hicache-disabled-pdi1-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192",
     "fc91d24-long-context-admission-reserve-disabled-hicache-cuda-host-pooled-v1-pdi1-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192",
 )
-# r1 is the unchanged #294 control; r2 is the --prefill-decode-interval 2 canary. The
-# variants differ only in pdi1/pdi2 so dashboards split the replicas by config_variant.
+# r1 is the #294 control; r2 is the --prefill-decode-interval 2 canary. The variants differ
+# only in chunk/qsplit/pdi so dashboards split the replicas by config_variant; "dyntok"
+# marks the async dynamic batch tokenizer both replicas run (INGESTION_FLAGS).
 VARIANTS: Final = {
     1: "fc91d24-long-context-w4afp8-c8192-hicache-cuda-host-pooled-v1-admission-reserve-disabled"
-    "-pool-clamp-pdi1-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192",
+    "-pool-clamp-pdi1-dyntok-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192",
     2: "fc91d24-long-context-w4afp8-c16384-qsplit-hicache-cuda-host-pooled-v1-admission-reserve-disabled"
-    "-pool-clamp-pdi2-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192",
+    "-pool-clamp-pdi2-dyntok-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192",
 }
 PREFILL_DECODE_INTERVAL: Final = {1: 1, 2: 2}
 SOURCE_DIST_INIT: Final = "127.0.0.1:29510"
@@ -66,6 +67,17 @@
     "--hicache-io-backend direct",
     "--hicache-mem-layout page_first_direct",
 )
+# Tokenization off the tokenizer manager's event loop. By default the HF tokenizer call runs
+# synchronously on the asyncio loop that also serves /health, /metrics and every in-flight
+# stream, so encoding one multi-MB prompt (this tier's normal traffic) stalls the whole HTTP
+# front. On 2026-09-25 a burst of >=1.4 MB requests made both gpu02 replicas flap unhealthy
+# every 10-60 s (3 s probe timeouts) and the CVM proxy failed 600+ requests with "error
+# sending request" while the GPUs were busy. --enable-dynamic-batch-tokenizer encodes single-
+# string prompts on one background thread (the fast tokenizer's batch encode releases the
+# GIL), so probes and streams keep flowing while a prompt is tokenized. Multimodal requests
+# tokenize inside the processor and are unaffected. Both replicas carry it so the pdi/chunk
+# canary stays paired.
+INGESTION_FLAGS: Final = ("--enable-dynamic-batch-tokenizer",)
 FP8_MODEL_PATH: Final = f"--model-path /root/.cache/huggingface/hub/models--zai-org--GLM-5.3-Flash/snapshots/{FP8_REVISION}"
 MODEL_PATH: Final = (
     "--model-path /root/.cache/huggingface/hub/models--graphistry--GLM-5.3-Flash-W4AFP8/"
@@ -77,7 +89,11 @@
     "# prod/GLM-5.3-Flash-SGL-TP4-LongContext.yaml by scripts/prepare_glm53_w4afp8_long_context.py.\n"
     "# Both replicas run the W4AFP8 checkpoint graphistry/GLM-5.3-Flash-W4AFP8@99f1fa7 with\n"
     "# --max-prefill-tokens 32768, HiCache with CUDA-owned host memory and a fixed 406 GiB\n"
-    "# startup host-memory budget per replica, and no admission reserve.\n"
+    "# startup host-memory budget per replica, and no admission reserve. Both also run\n"
+    "# --enable-dynamic-batch-tokenizer: prompts are encoded on a background thread instead\n"
+    "# of the tokenizer manager's event loop, so /health, /metrics and in-flight streams keep\n"
+    "# flowing while a multi-MB prompt is tokenized (2026-09-25: both gpu02 replicas flapped\n"
+    "# unhealthy under a burst of >=1.4 MB requests).\n"
     "#\n"
     "# THE REPLICAS ARE NOT IDENTICAL. r1 is the live control, r2 the canary:\n"
     "#   r1: gpu31 campaign-2 arm L2 (2026-09-23), chunk 8192, --prefill-decode-interval 1\n"
@@ -181,8 +197,8 @@
         "# bounded 8-request queue, prefill chunks of 8192 (r1) / 16384 (r2) with\n"
         "# --max-prefill-tokens 32768,\n"
         "# decode graphs capped at batch 32, TileLang DSA, the CUTLASS W4A8 MoE path (hence no\n"
-        "# --moe-runner-backend and no FP8 --revision), and adaptive EAGLE 5/1/6. This retains\n"
-        "# the full 1,048,576-token model context.\n",
+        "# --moe-runner-backend and no FP8 --revision), adaptive EAGLE 5/1/6, and the async\n"
+        "# dynamic batch tokenizer. This retains the full 1,048,576-token model context.\n",
     ),
 )
 
@@ -245,8 +261,11 @@ def engine_arguments(source: list[str], replica: int) -> list[str]:
     missing = sorted(argument for argument in expected if source.count(argument) != 1)
     if missing:
         raise GenerationError(f"source engine command changed: {missing}")
-    if any(argument.startswith(("--hicache", "--enable-hierarchical-cache", "--max-prefill-tokens")) for argument in source):
-        raise GenerationError("source engine command already carries L2 options")
+    if any(
+        argument.startswith(("--hicache", "--enable-hierarchical-cache", "--max-prefill-tokens", "--enable-dynamic-batch-tokenizer"))
+        for argument in source
+    ):
+        raise GenerationError("source engine command already carries L2 or ingestion options")
     arguments: list[str] = []
     for argument in source:
         if argument in (f"--revision {FP8_REVISION}", "--moe-runner-backend deep_gemm"):
@@ -264,6 +283,7 @@ def engine_arguments(source: list[str], replica: int) -> list[str]:
         else:
             arguments.append(argument)
     arguments.extend(HICACHE_FLAGS)
+    arguments.extend(INGESTION_FLAGS)
     return arguments
 
 

scripts/test_glm53_w4afp8_long_context.py

@@ -125,6 +125,9 @@ def test_rejects_engine_argv_drift(self) -> None:
             # r1 is the c8192 control and r2 the c16384 canary; neither may take the other's chunk.
             ("\n      --chunked-prefill-size 8192\n", "\n      --chunked-prefill-size 16384\n", "model-sg-glm53-w4afp8-tp4-r1 argv must be"),
             ("\n        --chunked-prefill-size 16384\n", "\n        --chunked-prefill-size 8192\n", "model-sg-glm53-w4afp8-tp4-r2 argv must be"),
+            # Both replicas encode prompts off the event loop; dropping the flag on either is drift.
+            ("\n      --enable-dynamic-batch-tokenizer\n", "\n", "model-sg-glm53-w4afp8-tp4-r1 argv must be"),
+            ("\n        --enable-dynamic-batch-tokenizer\n", "\n", "model-sg-glm53-w4afp8-tp4-r2 argv must be"),
         )
         for before, after, message in cases:
             with self.subTest(mutation=after.strip()[:60]):

scripts/validate_glm53_prod_config.rb

@@ -634,7 +634,8 @@ def validate_long_context(errors, compose, replica_services)
 # r2 runs the v2 image (#300) with the opt-in DSA indexer query split; r1 stays on the #294
 # image as the live control, so a targeted `compose up` recreates r2 alone.
 W4AFP8_LONG_CONTEXT_R2_IMAGE = "docker.io/nearaidev/sglang@sha256:8ff1a487b98a52fe08b781715bebd7c8c445d4fe068f312f03f527d5a3c77e84"
-W4AFP8_LONG_CONTEXT_VARIANT = "fc91d24-long-context-w4afp8-cCHUNK-QSPLIThicache-cuda-host-pooled-v1-admission-reserve-disabled-pool-clamp-pdiPDI-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192"
+# "dyntok" = --enable-dynamic-batch-tokenizer on both replicas (tokenization off the HTTP event loop).
+W4AFP8_LONG_CONTEXT_VARIANT = "fc91d24-long-context-w4afp8-cCHUNK-QSPLIThicache-cuda-host-pooled-v1-admission-reserve-disabled-pool-clamp-pdiPDI-dyntok-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192"
 W4AFP8_QSPLIT_ENV = "SGLANG_DSA_INDEXER_QSPLIT"
 # c16384 is only memory-safe WITH the split: without it a concurrent long burst left 0.04-0.65 GB
 # free, the condition that preceded the gpu02 crash. Enforced below for every replica.
@@ -670,6 +671,7 @@ def validate_long_context(errors, compose, replica_services)
   --disable-fast-image-processor --limit-mm-data-per-request '{"image": 64}'
   --enable-hierarchical-cache --hicache-write-policy write_through
   --hicache-io-backend direct --hicache-mem-layout page_first_direct
+  --enable-dynamic-batch-tokenizer
 ARGV
 W4AFP8_LONG_CONTEXT_HICACHE_ENV = HICACHE_ENV.merge("SGLANG_HICACHE_RAM_BUDGET" => "${GLM53_HICACHE_RAM_BUDGET:-406GiB}").freeze