GLM-5.3 Flash: W4AFP8 + HiCache long-context file for gpu02 (both replicas = campaign-2 L2) by PierreLeGuen · Pull Request #294 · nearai/cvm-compose-files

PierreLeGuen · 2026-09-23T19:35:08Z

Summary

This adds a new generated file, prod/GLM-5.3-Flash-SGL-TP4-W4AFP8-LongContext.yaml, for the dedicated long-context tier (gpu02). Both replicas run the gpu31 campaign-2 arm L2: the W4AFP8 checkpoint with HiCache, 8192-token chunks, a fixed 406 GiB host budget per replica, and no admission reserve.

Only the two engines and their telemetry differ from prod/GLM-5.3-Flash-SGL-TP4-LongContext.yaml; the long-domain routing contract is untouched. gpu13 stays on #290's configuration.

Based on main, which now includes #293 (merged as 86d41e5): the W4AFP8 snapshot is pre-staged in the model-downloader of every dedicated GLM-5.3 Flash TP4 file (not prod/small-models.yaml).

It is independent of the base-host PR (#295). A local test merge showed the two merge cleanly in either order, and the full validator suite passes on the merged tree.

Engines

Both replicas are identical except for --dist-init-addr (127.0.0.1:29510 for r1, :29511 for r2).

Image

docker.io/nearaidev/sglang@sha256:fde25985aea3ebabf1eb581ae21d53be8540e32933eef942ee8b962a1bfbea20 (docker/sglang-glm53-hicache-w4afp8), published by workflow run 35903077821 from #291's merge commit f8106f0 (digest comment). I re-verified it before pinning:

the run concluded success;
the tag glm53-hicache-w4afp8-v1 resolves to this digest;
all ten nearai.sglang.* labels are present, among them hicache_fix_revision=230102db…, hcc_host_memory=v4-cuda-hostalloc-opt-in, pooled_transfers=v1-opt-in, admission_reserve=v10, chunked_prefill_pool_clamp=v1, w4afp8_loader_patch=modules-to-not-convert-v1 and source_revision=fc91d24…;
org.opencontainers.image.base.digest is 3eccc307… and the revision is f8106f09…;
cosign verify (v3.0.6) against the publish-glm53-admission-reserve.yaml@refs/heads/main identity passes;
Trivy reports 0 fixable critical findings.

Argv

sglang serve --model-path /root/.cache/huggingface/hub/models--graphistry--GLM-5.3-Flash-W4AFP8/snapshots/99f1fa70408c52b007d4fd69e02e5a522422e755 --served-model-name z-ai/glm-5.3-flash --tp-size 4 --ep-size 4 --mem-fraction-static 0.80 --max-running-requests 32 --max-queued-requests 8 --enable-priority-scheduling --disable-priority-preemption --chunked-prefill-size 8192 --max-prefill-tokens 32768 --prefill-decode-interval 1 --cuda-graph-max-bs-decode 32 --dsa-prefill-backend tilelang --dsa-decode-backend tilelang --kv-cache-dtype bfloat16 --speculative-algorithm EAGLE --speculative-num-steps 5 --speculative-eagle-topk 1 --speculative-num-draft-tokens 6 --speculative-adaptive --reasoning-parser glm45 --enable-strict-thinking --grammar-backend xgrammar --tool-call-parser glm47 --chat-template /root/.cache/huggingface/hub/models--zai-org--GLM-5.3-Flash/snapshots/3f1971b7b5f7a528c9c4ef6212c8785298a8c24a/chat_template.jinja --context-length 1048576 --dist-init-addr 127.0.0.1:29510 --watchdog-timeout 1800 --host 0.0.0.0 --port 8000 --enable-metrics --enable-cache-report --log-requests-level 0 --disable-fast-image-processor --limit-mm-data-per-request '{"image": 64}' --enable-hierarchical-cache --hicache-write-policy write_through --hicache-io-backend direct --hicache-mem-layout page_first_direct

The distinct --dist-init-addr ports are harmless on the bridge network, where each container has its own loopback, and they guard against host networking.

Environment

The long-context control environment, plus:

SGLANG_HICACHE_RAM_BUDGET=${GLM53_HICACHE_RAM_BUDGET:-406GiB}: a fixed per-replica budget, the qualified L2 value. A percentage resolves against MemAvailable when each engine starts, so the replica started second would get less.
SGLANG_HICACHE_CUDA_HOST_MEMORY=${GLM53_HICACHE_CUDA_HOST_MEMORY:-1}
SGLANG_HICACHE_POOLED_TRANSFERS=1
SGLANG_HICACHE_STAGING_PAGES=64

No admission reserve.

Telemetry

precision int4-weights-fp8-activations-bf16-kv;
config_variant fc91d24-long-context-w4afp8-c8192-hicache-cuda-host-pooled-v1-admission-reserve-disabled-pool-clamp-pdi1-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192;
engine_image fde25985aea3;
model_path graphistry/GLM-5.3-Flash-W4AFP8 on the engines and DCGM.

Unchanged (validator-enforced)

the domains and the :8001 discovery stub;
LONG_TIER_ONLY (default true) and the registrar;
the proxy pooling r1 and r2 with conversation affinity;
nginx, the soak relay and the perception check;
DCGM, the OTel pipeline and model-downloader.

No OpenRouter gateway or cloud-api change: the host, ports (:8000 probe, :8444 TLS), domains and model-proxy handle all stay the same.

Argv parity with campaign-2 arm L2

Compared against gpu31:/data/inference-optimizer-20260913/results/hicache-w4afp8-real-20260923/campaign2/setup/pm-c2-l2.sh. The argv was rendered with docker compose config and compared flag by flag: 74 tokens and 40 flags on each side, 35 of them identical.

flag	this file	L2	kind
`--model-path`	`/root/.cache/huggingface/hub/models--graphistry--GLM-5.3-Flash-W4AFP8/snapshots/99f1fa70…`	`/models/hub/models--graphistry--GLM-5.3-Flash-W4AFP8/snapshots/99f1fa70…`	path
`--chat-template`	`/root/.cache/huggingface/hub/models--zai-org--GLM-5.3-Flash/snapshots/3f1971b7…/chat_template.jinja`	`/configs/chat_template.jinja`	path
`--host`	`0.0.0.0`	`127.0.0.1`	net
`--port`	`8000`	`30001`	net
`--dist-init-addr`	`127.0.0.1:29510` (r1) / `127.0.0.1:29511` (r2)	`127.0.0.1:29611`	net

SGLANG_* environment: identical to L2, the 406GiB host budget included. The only delta is SGLANG_ENABLE_HEALTH_ENDPOINT_GENERATION=0, which is production-only, affects only the health endpoint, and is the same in every GLM-5.3 production file.

L2 ran the local :pr build of the same recipe: same base 3eccc307…, and the patched-file after-hashes pinned in source-manifest.json. This file pins that recipe's published digest.

Evidence (gpu31, 2026-09-23, CC off)

L2 against FP8 + HiCache (L0) on cold requests of 128K tokens and more: TTFT p50/p95 32.6/50.3 s vs 41.9/76.2 s (0.78×/0.66×), E2E p50 22.3 vs 44.9 s.
8/8 concurrent 647K–756K-token cold prefills completed, with 3.7 GiB minimum free device memory.
Chunk 16384 fell to 0.04 GiB free and was rejected.
12 engine configs ran with 0 restarts and 0 OOMs.
Tables: reports/glm53-hicache-w4afp8-gpu31-2026-09-23/campaign2/ in the lab evidence tree.

Changes

prod/GLM-5.3-Flash-SGL-TP4-W4AFP8-LongContext.yaml, generated by the new scripts/prepare_glm53_w4afp8_long_context.py. The generator uses exact-count replacements and fails on source drift. --check detects a stale file.
scripts/validate_glm53_prod_config.rb: the new file's contract.
- The exact L2 argv per replica, the image, and the environment (long-context control environment plus the 406 GiB HiCache environment, no reserve).
- Devices, a shared runtime apart from --dist-init-addr, and the proxy pooling both replicas.
- Truthful per-replica telemetry: labels, log tags, scrape labels and DCGM model_path.
- Equality with the long-context file outside the engines and their telemetry.
scripts/validate_glm53_dcgm_metrics.rb: covers the new file.
scripts/test_glm53_w4afp8_long_context.py plus a CI step: 10 tests.
- Generator sync and guards.
- Mutations, each confirmed to fail with its specific message:
  - argv: chunk, --dist-init-addr, HiCache IO backend, a re-added --revision, queue size;
  - image, host budget 80%, CUDA host memory 0, admission reserve, device IDs, runtime drift;
  - routing: LONG_TIER_ONLY, the synthetic discovery id, soak-relay routing, proxy pooling;
  - telemetry: config_variant label and log tag, engine_image label and scrape label, precision, DCGM model_path;
  - DCGM image.
docs/glm53-w4afp8-long-context-rollout.md, which references the existing gpu02 runbook, and a README row.

Rollout summary (details in the runbook)

Gates
- Merged, backdated tag; the full env map on every call; force_recreate: false; never an empty service list; dry_run before every up.
- GLM53_HICACHE_RAM_BUDGET now sets both replicas, and defaults to a fixed 406GiB. Before step 1, read gpu02's env map and confirm it is unset or 406GiB.
- 2 × 406 GiB must fit in available RAM minus the 10 GiB reserve, with room for page cache. gpu02 has about 1223 GiB available, which leaves about 400 GiB; an oversized value fails at startup.
- Both replicas' startup logs must show rank_budget_bytes=108984795136.
- Other CVMs have different RAM; gpu13 is on Make gpu13 GLM-5.3 Flash a HiCache long-context replica #290's 20%.
Orphan rule. compose-manager runs up --remove-orphans, which removes orphans with a ~10 s stop. Each engine being replaced is stopped first with a scoped compose/down of the file that defines it (5 m grace), then the new file is brought up, scoped.
Order on gpu02. r2 first while r1 serves: the W4AFP8 canary's FP8 HiCache r2 goes down, the new r2 comes up and is verified, then the proxy, collector and exporter are recreated. r1 follows the same way.
Verification. A real long-domain completion and a cache-hit follow-up, host-restore evidence in the engine logs and metrics, /backends/list on both model-proxy peers, and the telemetry labels.
Rollback. Redeploy the previous tag and file scoped to the same services, one replica at a time.

Validation

A local run of every step in validate-compose.yaml, with the CI images, passes:

OTel labels, proxy dependencies/environment, streaming keepalive (real nginx -t on this file), public metrics (32 TLS listeners), DS4F, registrar auth, GPU preflight, GLM-5.3 DCGM;
the GLM-5.3 prod contract, the HiCache tests, the W4AFP8 canary --check and tests, and this PR's generator --check and tests;
docker compose config on every file, the embedded OTel configs, and git diff --check.

Risks (accepted) and decisions

First CVM run of the HCC host-memory path. This image's cudaMallocHost path has never run in a CVM (gpu31 reports CC off). gpu02 is the first soak and a hard gate.
Chunk-8192 device headroom. It kept 3.7 GiB free under 8 concurrent ~700K cold prefills on bare metal. CVM memory-encryption overhead is unmeasured, the gpu02 soak decides, and --mem-fraction-static 0.78 is an untested fallback.
Budget (decided: keep the name, fixed 406 GiB default). GLM53_HICACHE_RAM_BUDGET keeps its single name, the same contract as gpu13 (Make gpu13 GLM-5.3 Flash a HiCache long-context replica #290), but sets both replicas. Its default is now a fixed 406GiB rather than 40%, because a percentage resolves against MemAvailable at each start: at 40%, the replica started second would get about 293 GiB against the first one's 489 GiB. The runbook guards against a leftover override:
- read gpu02's env map before step 1 (unset or 406GiB);
- the r2-first order cannot double-book;
- both replicas' mandatory startup-log checks require rank_budget_bytes=108984795136.
Single long host during the swaps (accepted). Until Make gpu13 GLM-5.3 Flash a HiCache long-context replica #290 lands, gpu02 is the only long host, so one replica carries the tier during each swap; this needs a low-traffic window.
Queue-full on the long tier. With a queue of 8, huge requests that arrive during a huge prefill are rejected as queue-full once short turns fill the queue. This was observed on both W4AFP8 and FP8.
Admission reserve (decided: stays off). The long tier keeps the reserve off, as qualified in L2. Re-enabling it is a separate follow-up once the pool clamp has production time.

chatgpt-codex-connector · 2026-09-23T19:35:17Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-23T19:43:40.155394Z	`80884e1`	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 80884e1f82

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Generate prod/GLM-5.3-Flash-SGL-TP4-W4AFP8-LongContext.yaml from the long-context file with scripts/prepare_glm53_w4afp8_long_context.py. Both replicas run the gpu31 campaign-2 arm L2: graphistry/GLM-5.3-Flash-W4AFP8@99f1fa7 on the signed hicache-w4afp8 image fde25985, 8192-token chunks with --max-prefill-tokens 32768, HiCache with CUDA-owned host memory at a 40% startup budget per replica, no admission reserve, and distinct --dist-init-addr ports. The long-domain routing contract stays byte-identical to the source; telemetry reports the W4AFP8 precision, config variant and engine image. Pin the contract in validate_glm53_prod_config.rb (exact L2 argv, image, environment, per-replica telemetry, equality with the long-context file outside the engines) and in the DCGM validator, cover it with generator and mutation tests wired into CI, and add the gpu02 rollout runbook and a README row.

A percentage budget resolves against MemAvailable when each engine starts, so with two HiCache replicas on one CVM the replica started second got less (about 489 vs 293 GiB on gpu02 at 40%), and every restart moved it again. Default each replica to the qualified L2 value, 406 GiB, which is independent of start order and fails at startup when it does not fit. GLM53_HICACHE_RAM_BUDGET keeps its name and still overrides both replicas.

State the 406 GiB budget check (rank_budget_bytes=108984795136 on both replicas, 2 x 406 GiB within available RAM minus the reserve), and add explicit start, run and cleanup steps for glm53-soak-relay and glm53-perception-check from this file, since a relay left over from the canary still routes :8009 to the removed FP8 r2.

Add explicit start, run and cleanup steps for glm53-soak-relay and glm53-perception-check, which sit behind the verification profile and only start when named. Point the gpu02 soak gate at #294, whose runbook lands with that PR.

GLM-5.3 Flash's DSA indexer is ReplicatedLinear, so every attention-TP rank computes fp8_mqa_logits and the K-pool top-k for all prefill query rows; at long context that is the only prefill cost that grows with context. With SGLANG_DSA_INDEXER_QSPLIT=1 each rank scores a quarter of the rows against the same gathered keys and the int32 top-k rows are all-gathered. It also shrinks the per-rank logits scratch, which is what lets 16384-token chunks fit on the long tier. Inert unless enabled. Evidence (gpu31/gpu32, 2026-09-24, every A/B in both island orders): stored long-tier traffic paired TTFT 0.81-0.85x with 100% completion; the chunk-16384 burst keeps 9.5 GB free vs 0.65 GB without; GSM8K, passkey retrieval and teacher-forced agreement are at the noise floor. The image built from this recipe passes test-cpu.sh (new step 4) and reproduces the cold-prefill speedup on GPU (+16.4% at 512K, chunk 16384). The publish workflow pins move to the new recipe hashes and the default tag to glm53-hicache-w4afp8-v2, so the v1 image #294 pins is untouched. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

r1 stays the #294 control at pdi 1; r2 runs pdi 2, and the replicas' config_variant differs only in pdi1/pdi2 so prod dashboards compare them directly. On stored long-tier traffic (gpu31, 2026-09-24, both island orders) pdi 2 cut TPOT p50 to 0.53-0.72x of pdi 1 at a paired TTFT cost of at most 2%, with 100% completion. The generator sets the interval per replica, the validator pins each replica's argv and variant (and rejects either taking the other's value), and the rollout doc gains an r2-only canary procedure and readout. Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…302) * Canary the DSA indexer split and chunk 16384 on the long-context r2 Pins r2 to the v2 engine image published from #300 (8ff1a487b98a), sets SGLANG_DSA_INDEXER_QSPLIT=1 and raises its prefill chunk to 16384. r1 is left completely untouched -- same #294 image, chunk 8192, pdi 1, split unset -- so it stays a live control and a targeted `compose up` recreates r2 alone. Structurally r2 now needs its own image AND environment: a YAML merge key replaces a list rather than extending it, so r2 cannot inherit the anchor's environment and add one variable. The generator derives r2's block from the anchor at generation time rather than duplicating it as a literal, so the two cannot drift. The chunk and the split are one change. At 16384 without the split a concurrent long burst left 0.04-0.65 GB free per GPU, the condition that preceded the gpu02 crash; with it, 9.3-9.5 GB. The validator now rejects --chunked-prefill-size 16384 on any replica that does not also set SGLANG_DSA_INDEXER_QSPLIT=1, and rejects the variable on any image that does not carry the patch. Both gates were verified by mutation, including the case where the expected spec itself is edited -- the first placement of the chunk gate sat in validate_w4afp8_base, where it could never fire. Lab evidence (gpu31, candidate vs #294): TTFT 0.61-0.78x, TPOT 0.53-0.85x, minimum free device memory 3.7 -> 9.3 GB, quality at the noise floor. The all-gather's cost under CC/PPCIe inside a CVM is still unmeasured; gpu31 is bare metal with CC off, so this canary is the first measurement of it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Make the generated header describe the canary, and enforce marker counts Review found the header still claimed "8192-token prefill chunks", "Both pin fde25985...", "flags differ only in --dist-init-addr and r2's canary pdi 2" and "(16384 fell to 0.04 GiB and is rejected)" -- all false once r2 moved to the v2 image at chunk 16384. That header is the rollback and incident reference for a file operators are told not to hand-edit, so a stale one is worse than none: it would have pointed an on-call at the wrong image during a rollback. It now states plainly that the replicas are not identical and lists each one's image, chunk, interval and provenance, plus why the chunk and the split are one change. Also enforce the occurrence count in resolve_engine_image_labels instead of unpacking it and ignoring it. find() silently takes the first hit, so if an "instance:N" marker ever stopped being replica-unique the nearest-placeholder binding could attach the wrong replica's image label. It now fails loudly, matching the exact-count discipline replace_exact uses elsewhere in the script. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

PierreLeGuen requested review from lloydmak99 and pranavraja99 September 23, 2026 19:35

This was referenced Sep 23, 2026

GLM-5.3 Flash: W4AFP8 base-tier file for gpu03, gpu04 and gpu23 (both replicas = campaign-2 B5) #295

Merged

GLM-5.3 Flash: pre-stage the W4AFP8 snapshot in model-downloader (no engine change) #293

Merged

PierreLeGuen force-pushed the feat/glm53-w4afp8-long-context branch from 80884e1 to 3433b87 Compare September 23, 2026 19:39

chatgpt-codex-connector Bot reviewed Sep 23, 2026

View reviewed changes

Comment thread docs/glm53-w4afp8-long-context-rollout.md Resolved

Comment thread prod/GLM-5.3-Flash-SGL-TP4-W4AFP8-LongContext.yaml Resolved

PierreLeGuen added 4 commits September 23, 2026 19:47

docs(glm53): apply the rollout rulings to the runbook

9efd77d

PierreLeGuen changed the base branch from feat/glm53-w4afp8-prestage to main September 23, 2026 19:50

PierreLeGuen force-pushed the feat/glm53-w4afp8-long-context branch from f7a0061 to ce9abda Compare September 23, 2026 19:50

lloydmak99 approved these changes Sep 23, 2026

View reviewed changes

PierreLeGuen merged commit 92f341e into main Sep 23, 2026
2 checks passed

This was referenced Sep 24, 2026

Canary --prefill-decode-interval 2 on the W4AFP8 long-context r2 #299

Merged

Add an opt-in DSA indexer query split to the HiCache W4AFP8 image #300

Merged

pranavraja99 mentioned this pull request Sep 25, 2026

Canary the DSA indexer split and chunk 16384 on the long-context r2 #302

Merged

github-actions Bot mentioned this pull request Sep 25, 2026

Put the DSA indexer split on both long-tier replicas, revert r2 to chunk 8192 #307

Merged

Navigation Menu

Repository navigation

GLM-5.3 Flash: W4AFP8 + HiCache long-context file for gpu02 (both replicas = campaign-2 L2) - #294

GLM-5.3 Flash: W4AFP8 + HiCache long-context file for gpu02 (both replicas = campaign-2 L2)#294
PierreLeGuen merged 4 commits into
mainfrom
feat/glm53-w4afp8-long-context

PierreLeGuen commented Sep 23, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot commented Sep 23, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

PierreLeGuen commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Engines

Unchanged (validator-enforced)

Argv parity with campaign-2 arm L2

Evidence (gpu31, 2026-09-23, CC off)

Changes

Rollout summary (details in the runbook)

Validation

Risks (accepted) and decisions

Uh oh!

chatgpt-codex-connector Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

PierreLeGuen commented Sep 23, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Sep 23, 2026 •

edited

Loading