[PierreLeGuen]

Summary

Adds three patches to docker/sglang-glm53-hicache-w4afp8 and makes publish-glm53-admission-reserve.yaml build them as glm53-hicache-w4afp8-v3. The v1 and v2 tags and digests are left alone. No compose file changes here: pinning v3 on a replica is a follow-up PR once the image is published (plan and pre-validation below).

v3 is v2 plus these three patches. The recipe directory is unchanged since v2 (8ff1a487…) was built from it at 556482c (#300): same base 3eccc307…, same loader, pool-clamp and DSA indexer split diffs, and no other build step. Since #307, both long-tier replicas run v2 at chunk 8192 with SGLANG_DSA_INDEXER_QSPLIT=1 (r1 at --prefill-decode-interval 1, r2 at 2). Moving one replica from v2 to v3 therefore changes only these patches, so the replica's before/after is a single-variable comparison.

patch	what it does	switch
sglang-pr30771.diff	Upstream sgl-project/sglang#30771 (open), byte-identical to its diff at head 60a56aa (re-checked today). Offloads request preprocessing: chat template rendering and tokenization in _convert_to_internal_request, and the regular-tokenizer path of TokenizerManager._tokenize_texts, run in a single-worker request-preprocessor thread instead of on the uvicorn event loop. Ships the PR's three CPU tests.	none
shm-off-loop.diff	New, applied after #30771. Moves the dispatch-time shared-memory copy of multimodal features (wrap_shm_features: posix_fallocate + copy_ into /dev/shm, about 141 MiB per large image, 8.8-8.9 GiB for a 64-image request) to its own sglang-mm-shm thread. A request cancelled during the copy still has its segments unlinked once the copy finishes.	none
event-loop-stall-dump.diff	New, diagnostic. If the event loop goes 30 s without a tick, writes every thread's stack to stderr, each line prefixed [event-loop-stall], repeats every 60 s while the stall lasts, and logs event loop recovered after N s. Output is stack frames only (file, line, function, source line) plus pid, stall age and loop-thread CPU: no local variables and no request content. faulthandler covers a loop stuck in one GIL-holding C call.	on by default; SGLANG_EVENT_LOOP_STALL_DUMP_SECS (0 disables), SGLANG_EVENT_LOOP_STALL_DUMP_REPEAT_SECS

Why

The sglang serve main process answers /health, /metrics and /v1/chat/completions from one uvicorn event loop, so anything it does synchronously also delays /health. The CVM proxy marks a replica unhealthy after 3 consecutive missed probes (3 s timeout, 5 s interval). On 2026-09-25 the gpu02 long tier flapped this way under image-heavy multi-MB traffic from one customer's agent workload. Separately, the tier has had hangs of 14-155 min in which the engine stopped answering /health and left no stack in any log.

Evidence

Lab: gpu32 (bare metal, CC off, 8x H200), 2026-09-25. The same source bytes were bind-mounted over the v1 image fde25985…; one request at a time; /health probed every 200 ms; the stock arm was measured in the same half hour. All request bodies are synthetic (seeded generators, no customer data). Internal report: ~/Documents/cloud/reports/glm53-hang-lab-2026-09-25/ (results/RESULTS-phase3a.md, results/RESULTS-phase3b.md); raw data on gpu32 under /data/glm53-dyntok-20260925.

Worst /health probe per request, stock vs all three patches (stall dump at its 30 s default):

request	stock	patched	patched probes over 3 s
4.7 MB text chat (653,502 prompt tokens)	3,033 ms	30 ms	0
64 images of 9 MP	3,098 ms	163 ms	0
64 images of 48 MP	2,976 ms	551 ms	0
64 images of 169 MP	4,062 ms	1,258 ms	0

• 0 of 3,211 probes over 3 s on the patched arm. The stock arm had one over 3 s on three of the four requests.
• Same outputs: prompt_tokens identical to stock on every request (653,502 / 604,876 / 610,730 / 610,172), all HTTP 200.
• No shared-memory leak: /dev/shm was back to the same bytes and segment names after every request. No Traceback, no Already borrowed, and no stall dump in the engine log.
• Why both preprocessing patches are needed. PR #30771 alone fixes the text case (38 ms against 3,444 ms stock), but each 64-image request still blocked the loop once at dispatch, for 3.7 s to over 5 s. py-spy put that in ShmPointerMMData.__init__ on the loop thread: 64 segments of about 141 MiB each. With shm-off-loop.diff, py-spy shows the loop idle while sglang-mm-shm_0 does the copy.
• Text flood through the pinned proxy. 2, 4 and 8 MB text prompts, each a new conversation, 0.5 rps for 10 min, both arms saturated at about 97 % GPU; the proxy ran with a 3 s pool idle timeout, so every probe miss was a real stall. The stock arm was marked unhealthy twice, with runs of back-to-back probes over 3 s lasting up to 29.8 s. The patched arm's worst probe was 206 ms, with 0 pool-probe misses. At 0.33 rps for 15 min neither arm was marked unhealthy; the worst probe was 245 ms patched against 6,330 ms stock.
• Stall dump on a live GPU engine (threshold lowered to 2 s): armed at startup, silent through startup, warmup and 2.5 min idle. One 64-image request produced two dumps naming the shm copy on the loop thread (2.0 s of loop CPU in the first 2.8 s), then event loop recovered after 5.3s. Each dump was about 15 KB (11 threads). At the 30 s default it logged nothing through all the runs above.

Not covered:

• The 14-155 min hang is not reproduced. No lab shape triggered it, so v3 is not a known fix for it. The stall dump is the instrument: if the hang happens on a v3 replica, the engine log will hold every thread's stack.
• The lab arm always ran --enable-dynamic-batch-tokenizer; production does not (Encode long-tier prompts off the HTTP event loop (--enable-dynamic-batch-tokenizer) #305 proposes it). Without that flag, the tokenizer manager's second tokenization pass goes through PR #30771's _tokenize_texts change instead. The PR's unit test covers that path, but the canary is its first run under GPU load.
• The 9 MP request was slower end to end on the patched arm (256 s against 81 s) in one sample. All of the extra time was before dispatch, in image processing, while the host was under memory pressure. The patches do not run before dispatch.

Recipe mechanics

• apply-patches.py now checks and applies each patch in PATCHES order. shm-off-loop.diff edits lines that sglang-pr30771.diff adds, so it cannot be git apply --checked against the unpatched base. Before-hashes are still verified before any patch; after-hashes and ast.parse are checked after all of them.
• source-manifest.json gains http_server.py, serving_base.py, tokenizer_manager.py (base to final), the new stall-dump module and the PR's three test files ("before": null = new file). tokenizer_manager.py is the only file two patches touch; PROVENANCE records its hash between them (d32617ee…). The event-loop before-hashes were checked in the real base 3eccc307… and equal fc91d24.
• test-cpu.sh gains:
  • step 5: source checks, then the PR's own tests via pytest (5 tests; on the stock files 3 fail and 2 skip);
  • step 6: the shm move with the copy stubbed, so it needs no /dev/shm (the workflow's test container keeps Docker's 64 MB default). It checks that the copy runs on sglang-mm-shm, that the loop keeps ticking, and that a cancel mid-copy leads to a discard;
  • step 7: the lifespan hook placement, then test_stall_dump.py (58 checks, shipped in the recipe).

file	sha256
sglang-pr30771.diff	a102977de7bb02776cee1053ff04979b740526fe99c59d2c63e2f3962df3c95d (= gh pr diff 30771)
shm-off-loop.diff	4bd4cc65d68deacd1b56b95e32a2242853d20bd3cfa4188e8c1a20777cc79a94
event-loop-stall-dump.diff	367dbcdec432586ae4c8ab984c08caafb2f148d62cebf257d12e89c4c32f72ab
test_stall_dump.py	59cf2daae55e46b4f597f004ca647e235bcea41de4d6485dc386c98b5e088abb

Publish workflow pins (hicache-w4afp8 case): Dockerfile f5136afd…, PROVENANCE 720fd00c…, SHA256SUMS 2f74f65b…, DEFAULT_TAG=glm53-hicache-w4afp8-v3.

Checks run (after rebasing onto #307)

• Recipe pre-PR check against the real base nearaidev/sglang@sha256:3eccc307… (pulled for the check, then removed). This reproduces the Dockerfile's RUN step and then runs the workflow's test step with the same flags (--runtime=runc --network=none --memory=16g):
  • sha256sum --check --strict SHA256SUMS: 12/12 OK;
  • apply-patches.py: Verified and applied 6 patches across 11 source files;
  • test-cpu.sh steps 1-7 all OK (step 5 5 passed, step 7 SUMMARY: 58 passed, 0 failed); 94 s in total.
• Hashes, from a git archive of the commit:
  • the workflow's three pins match the files;
  • the PATCHES and PROVENANCE hashes match the diffs;
  • the Dockerfile COPY list equals the directory, and SHA256SUMS covers every file except Dockerfile and itself.
• Every step of validate-compose.yaml, run locally in CI's own images, all pass:
  • the seven ruby validators, the GPU preflight tests, validate_public_metrics.py and validate_streaming_keepalive.py (nginx -t in the pinned images);
  • the four GLM-5.3 generators with --check, plus their unit tests (HiCache 34, W4AFP8 base 10, W4AFP8 canary 11, W4AFP8 long-context 13);
  • docker compose config on all 36 compose files with the CI dummy env;
  • the embedded OTel collector config in 33 files.
• The image itself was not built here. After merge, the publish workflow builds it, runs test-cpu.sh in it, then scans, attests and signs it.

Rollout

1. Publish. After merge, dispatch publish-glm53-admission-reserve.yaml with variant=hicache-w4afp8, source_revision=<merge sha> and an empty tag (defaults to glm53-hicache-w4afp8-v3). Record the run id and digest.
2. Wait for Put the DSA indexer split on both long-tier replicas, revert r2 to chunk 8192 #307 on both replicas. v3 goes in only after Put the DSA indexer split on both long-tier replicas, revert r2 to chunk 8192 #307 is deployed on both long-tier replicas, so its baseline is v2 with the same flags.
3. r2 first (the pdi 2 arm), in the follow-up PR below. Redeploy r2 alone as in docs/glm53-w4afp8-long-context-rollout.md:
   • compose/down, then compose/up, with services: ["model-sg-glm53-w4afp8-tp4-r2"];
   • run a dry_run first; it must plan exactly that service;
   • use a backdated tag, the full gpu-manager env and a low-traffic window (r1 carries the tier during the cold start);
   • do not recreate r1, the proxy or the registrar.
4. Watch r2.
   • The startup line event-loop stall dump armed: stall=30s repeat=60s (pid N, loop thread 0x…), and no [event-loop-stall] lines while idle.
   • TTFT p50/p95 against r2's own v2 history, and against r1 (which also differs by pdi).
   • Proxy Backend marked unhealthy / Backend recovered transitions for r2.
   • prompt_tokens against r1, and any Already borrowed or tokenizer error.
   • Any [event-loop-stall] block: keep it with the matching proxy transitions. It is the stack the hang investigation has been missing.
5. Then r1, with the same single-service pattern, once r2 reads out clean.
6. Base-tier hosts later. They run a different recipe, docker/sglang-glm53-w4afp8 (base e9d29a1c…, no HiCache). Carrying the fix there means porting the three diffs into that recipe in its own PR, after checking the before-hashes in that base.

Rollback

• r2 only: redeploy the previous tag for ["model-sg-glm53-w4afp8-tp4-r2"] (compose/down, then compose/up). That restores v2 with the same chunk, pdi and split in one step, and r1 keeps serving.
• Stall dump only: set SGLANG_EVENT_LOOP_STALL_DUMP_SECS=0 on the replica. In this repo that is a generator and validator change plus a recreate, so the full rollback above is quicker.
• Registry: nothing to undo. v1 and v2 stay published.

Risks

• PR #30771 is unmerged upstream, so this recipe carries it until it merges. Re-check it on any SGLang bump.
• Request preprocessing is serialized in one thread, as it effectively was on the loop. A multi-MB request now queues in that thread instead of on the loop. Its TTFT should not change, and /health stays fast.
• shm-off-loop.diff adds an await between prepare_for_dispatch_async and the dispatch, so a text request can now dispatch ahead of an image request whose copy is still running. The lab saw no effect. Cancellation during the copy is handled (shield plus a FIFO discard).
• A shorter GIL-held block remains for very large images, where process_mm_data_async returns just before the copy. It is the 163 / 551 / 1,258 ms above, on the same line as the stock arm. It stays under the 3 s probe timeout but grows with source image size.
• The stall dump is on by default. It logs about 15 KB per dump, repeated every 60 s during a stall, and faulthandler's timer is process-global.

Follow-up PR: pin v3 on r2 (after publish)

Dry-run today on a scratch copy of this branch with a placeholder digest. After the edits below and --write, all of the checks above pass. The regenerated prod/GLM-5.3-Flash-SGL-TP4-W4AFP8-LongContext.yaml changes only r2's image, its engine_image and config_variant telemetry, and the header. r1 is untouched.

• scripts/prepare_glm53_w4afp8_long_context.py:
  • R2_IMAGE becomes the v3 digest and R2_ENGINE_IMAGE_LABEL its first 12 hex;
  • VARIANTS[2] gains offloop- after qsplit-, so dashboards split r2 by config_variant;
  • the header names r2's image.
• scripts/validate_glm53_prod_config.rb:
  • W4AFP8_LONG_CONTEXT_R2_IMAGE becomes the v3 digest;
  • r1's entry goes back to W4AFP8_LONG_CONTEXT_IMAGE (v2); since Put the DSA indexer split on both long-tier replicas, revert r2 to chunk 8192 #307 both entries point at the r2 constant;
  • the split gate (SGLANG_DSA_INDEXER_QSPLIT=1 requires the patched image) must accept both v2 and v3, since both carry the split. Today it requires exactly the r2 image, which would reject r1;
  • the variant template gets an OFFLOOP placeholder ("1" on r2, nil on r1).
• docs/glm53-w4afp8-long-context-rollout.md needs a canary section. It still describes the pre-Put the DSA indexer split on both long-tier replicas, revert r2 to chunk 8192 #307 layout (r2 at chunk 16384, r1 on fde25985).
• If the proxy pool-idle PR (fix(glm53): expire pooled proxy connections before the engines close them #309) merges first, regenerating picks up its proxy lines. Nothing needs reconciling by hand.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-25T21:23:58.111022Z	6b1a51d	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[lloydmak99]

Recipe-only change that adds the v3 HiCache W4AFP8 image files and bumps the publish workflow's default tag/hashes; it publishes and deploys nothing on its own (deployment is a separate pin PR), so merge risk to running replicas is nil. Mechanical integrity and patch logic check out.

Optional follow-up (non-blocking):

• shm-off-loop.diff:54 — on cancel-during-copy, discard_shm_features is submitted fire-and-forget and its Future is never retrieved, so a failing /dev/shm unlink would be silently swallowed and could leak a segment. Rare edge case matching the documented intent; a warning-log on the discard result would make any leak visible.

Checks: sha256sum --check --strict SHA256SUMS → 12/12 OK; recomputed Dockerfile/PROVENANCE/SHA256SUMS hashes match the workflow's updated EXPECTED_* values; sglang-pr30771.diff byte-identical to upstream gh pr diff 30771; PATCHES dict hashes match the .diff files; confirmed discard_shm_features/wrap_shm_features exist upstream and wrap is a no-op without mm_inputs; apply-patches.py/test_stall_dump.py compile, workflow YAML parses, bash -n test-cpu.sh OK; CI Validate Compose Files = success (the code-review "fail" was a concurrency-cancelled job, not a real failure). Full image build and apply-patches.py against the real multi-GB base were skipped locally — covered by the publish workflow's in-image test-cpu.sh.