[PierreLeGuen]

Why

The long tier's tokenizer manager (the sglang serve main process) runs the HF tokenizer call synchronously on the asyncio loop that also serves /health, /metrics and every in-flight stream (TokenizerManager._tokenize_texts at the pinned fc91d24). Encoding one multi-MB prompt therefore stalls the whole HTTP front of that replica.

On 2026-09-25 13:45–14:10Z a burst of roughly 100 requests of 1.4 MB or more per 5 minutes hit gpu02:

• both replicas missed the CVM proxy's 3 s probe every 10–60 s (about 35 Backend marked unhealthy / recovered transitions in 25 minutes), while their GPUs stayed 40–90% busy and the engines kept logging;
• the proxy failed over 600 chat requests with Internal server error (error sending request) and the OpenRouter gateway refused the long tier as tier_unavailable;
• the engines never rejected anything: the requests were legitimate (bodies up to 100 MB have completed on this tier this week, e.g. 100 MB for 239k input tokens), so a size cap is not the fix.

Offline, the tokenizer is linear (0.3 MB in at most 0.7 s) and its batch encode releases the GIL, so the cost is real but only harmful because it runs on the event loop.

What changes

--enable-dynamic-batch-tokenizer on both gpu02 long-context replicas. In the pinned build this routes single-string prompts through AsyncDynamicbatchTokenizer, which encodes on one background thread; probes, metrics and other streams keep flowing while a prompt is tokenized. Multimodal requests tokenize inside the image processor and are unaffected. The flag changes when tokenization runs, not its result.

• scripts/prepare_glm53_w4afp8_long_context.py: new INGESTION_FLAGS, appended to both engines' argv; the generator refuses a source that already carries the flag; both config_variant values gain dyntok (truthful telemetry, same convention as qsplit/pdi2); header text.
• prod/GLM-5.3-Flash-SGL-TP4-W4AFP8-LongContext.yaml: regenerated (flag on r1 and r2, variants in service labels and scrape config).
• scripts/validate_glm53_prod_config.rb: argv contract and variant template updated.
• scripts/test_glm53_w4afp8_long_context.py: dropping the flag on either replica is argv drift.
• docs/glm53-w4afp8-long-context-rollout.md: new section with deployment, verification and rollback.

Both replicas carry the flag so the pdi/chunk canary (#299, #302) stays paired. r1 and r2 otherwise remain exactly as merged. Base hosts (gpu03/gpu04/gpu23) and every other file are unchanged.

What it does not do

It adds no tokenizer throughput: large prompts still queue behind one thread, so their own TTFT is unchanged. Overload now surfaces as the bounded queue's 429s instead of proxy 500s. It is also not a claimed fix for the separate multi-hour hangs seen on gpu02 this morning (root cause still open); it removes one known way for a replica to stop answering probes.

Risk

First production use of this flag in our stack. The code path is small (a queue and a single-thread executor; prompts with differing tokenizer kwargs fall back to per-prompt encodes on the same thread) and it does not touch the scheduler, KV or HiCache. Rollback is the previous tag of this file per replica.

Deployment

Both engine definitions changed, so one replica at a time, r2 first then r1: compose/down then compose/up scoped to ["<replica>"], dry_run that plans exactly that service, low-traffic window; about 15 minutes per replica with the other one carrying the tier. Then recreate otelcol-contrib alone so the new config_variant labels are reported (#303). Do not recreate the proxy or the registrar.

Verify per replica: startup server_args shows 'enable_dynamic_batch_tokenizer': True; count by (container_name, config_variant) (sglang_num_running_reqs{host_machine="gpu02"}) shows the dyntok variant; proxy-glm53 backend probe timed out lines for that replica become rare during large-prompt bursts and Backend marked unhealthy does not appear; TTFT/TPOT match the previous 24 h.

Validation

• python3 -m unittest scripts.test_glm53_w4afp8_long_context (12 tests, includes the ruby validator cases)
• ruby scripts/validate_glm53_prod_config.rb, ruby scripts/validate_glm53_dcgm_metrics.rb, ruby scripts/validate_otel_labels.rb
• prepare_glm53_w4afp8_canary.py --check, prepare_glm53_w4afp8_base.py --check and their tests, test_glm53_hicache_canary.py
• docker compose -f prod/GLM-5.3-Flash-SGL-TP4-W4AFP8-LongContext.yaml config with the CI dummy environment

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-25T15:01:18.146261Z	5c4a343	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[lloydmak99]

Config-generation change adding --enable-dynamic-batch-tokenizer to both gpu02 long-context GLM-5.3 replicas, moving HF tokenization off the tokenizer manager's asyncio event loop. The flag is appended after the HiCache flags in both engine argvs (r1 line 213, r2 line 537), and dyntok is threaded consistently through the generator, validator, tests, and all 6 config_variant telemetry strings. No blocking issues.

Optional follow-up:

• Couldn't confirm from here that --enable-dynamic-batch-tokenizer is a valid argparse flag in the pinned build (fc91d24); a bad flag would fail fast at engine startup, and the one-replica-at-a-time rollout with a server_args startup gate keeps the tier covered, so this is non-blocking.

Checks: prepare_glm53_w4afp8_long_context.py --check passed (committed YAML byte-identical to generator output), GeneratedFileTest 4/4 passed, git diff --check clean, YAML parses cleanly, and the GitHub "Validate Compose Files" check passed. Ruby validators / ValidatorContractTest and docker compose config were skipped locally (ruby and docker unavailable) but reported clean in CI, and the validator's static ARGV/variant contract matches the generated file.