[PierreLeGuen]

Summary

The model-downloader of every dedicated GLM-5.3 Flash TP4 file (canonical, HiCache, long-context and the W4AFP8 canary; not prod/small-models.yaml, whose gpu13 replica stays on FP8 under #290) now also fetches graphistry/GLM-5.3-Flash-W4AFP8 at 99f1fa70408c52b007d4fd69e02e5a522422e755 (~165 GiB). The FP8 snapshot and the corrected chat template (3f1971b7…) stay. No engine, proxy, routing or telemetry change.

This is the prep step for the W4AFP8 engine swaps, which are stacked on this branch:

• GLM-5.3 Flash: W4AFP8 + HiCache long-context file for gpu02 (both replicas = campaign-2 L2) #294: the long-context host (gpu02) moves to prod/GLM-5.3-Flash-SGL-TP4-W4AFP8-LongContext.yaml, with both replicas on campaign-2 L2 and the published W4AFP8 + HiCache digest from #291's digest comment.
• GLM-5.3 Flash: W4AFP8 base-tier file for gpu03, gpu04 and gpu23 (both replicas = campaign-2 B5) #295: the base hosts (gpu03, gpu04, gpu23) move to prod/GLM-5.3-Flash-SGL-TP4-W4AFP8.yaml, with both replicas on campaign-2 B5.

This repository does not auto-delete merged branches, so once this PR merges, #294 and #295 must be rebased onto main and retargeted explicitly.

Why the download has to come from the file already deployed

compose-manager always runs docker compose up -d --remove-orphans, and materialize_only runs only pull and build (no weight download). A W4AFP8 file names its engines model-sg-glm53-w4afp8-tp4-r*, so the first compose/up of that file on an FP8 host removes the running model-sg-glm53-fp8-tp4-r* engines as orphans. Orphan removal is a plain stop (about 10 s before SIGKILL), not the services' 5 m grace. Running the new file's downloader first would therefore take the host down for the download and the cold start.

Downloading from the file that is already deployed (same service names, so no orphans) leaves only the engine cold start for the later switch.

Changes

• prod/GLM-5.3-Flash-SGL-TP4.yaml, prod/GLM-5.3-Flash-SGL-TP4-LongContext.yaml: model-downloader adds the W4AFP8 snapshot after the chat template, with a comment explaining why.
• Regenerated prod/GLM-5.3-Flash-SGL-TP4-HiCache.yaml (prepare_glm53_hicache_canary.py) and prod/GLM-5.3-Flash-SGL-TP4-W4AFP8-Canary.yaml (prepare_glm53_w4afp8_canary.py). The canary generator now requires the source's W4AFP8 download instead of inserting its own. Its guard keys on engine snapshot paths (models--graphistry--GLM-5.3-Flash-W4AFP8/), not the checkpoint name. The committed canary file only gains the four comment lines; its YAML model is unchanged. The canary test's pinned long-context SHA-256 is updated.
• scripts/validate_glm53_prod_config.rb: new model-cache contract for the canonical, HiCache and long-context files:
  • each of the three hf download commands (FP8 84c6a6aa…, template 3f1971b7…, W4AFP8 99f1fa70…) runs exactly once, and no other download exists;
  • hf-cleanup stays behind profiles: ["maintenance"] with MODEL_NAME=${MODEL_NAME:-} and no default.
• scripts/test_glm53_hicache_canary.py: ModelCacheContractTest mutation cases, each shown to fail (below).
• README canonical row: note the pre-stage and the free-space requirement.

hf-cleanup cannot remove the new snapshot

hf-cleanup only starts when explicitly targeted with the maintenance profile. It removes exactly one directory, models--${MODEL_NAME//\//--}, and exits early if MODEL_NAME is unset. The validator now pins both properties. It can only touch the W4AFP8 snapshot if someone runs it with MODEL_NAME=graphistry/GLM-5.3-Flash-W4AFP8.

The reverse caveat matters for later cleanup: the W4AFP8 engines load the chat template from models--zai-org--GLM-5.3-Flash/snapshots/3f1971b7…. Evicting zai-org/GLM-5.3-Flash would delete it, so keep that repo cached while any GLM-5.3 Flash engine runs.

Deploy (after merge; engines untouched)

gpu03, gpu04 and gpu23 run the canonical file. For each, one at a time:

1. Confirm ≥250 GiB free on the CVM model-cache volume. The snapshot is ~165 GiB; the rest is headroom for the transfer.
2. Use a merged, backdated tag and the host's full gpu-manager env map.
3. compose/up with file: prod/GLM-5.3-Flash-SGL-TP4.yaml, services: ["model-downloader"], force_recreate: false. Run it with dry_run: true first where the host's compose-manager supports it; the plan must recreate only model-downloader and remove nothing.
4. Follow compose/logs for model-downloader until Download complete.. The engines and the proxy keep serving throughout.

gpu02 already holds the snapshot (its W4AFP8 canary r1 serves from it), so it needs no action. gpu13 uses prod/small-models.yaml and is unaffected.

Validation

A local run of every step in .github/workflows/validate-compose.yaml, with the same container images, passes:

• OTel labels, proxy dependencies/environment, streaming keepalive (real nginx -t in the pinned nginx image for every GLM-5.3 file), public metrics, DS4F migration, registrar auth, GPU preflight, GLM-5.3 DCGM;
• GLM-5.3 prod contract, HiCache canary tests (including the new mutation class);
• W4AFP8 canary --check and its tests;
• docker compose config on every file, the embedded OTel collector configs, and git diff --check.

New mutation cases, each confirmed to fail the validator with the expected message:

• remove the W4AFP8 download from the canonical, HiCache or long-context file;
• change its revision;
• duplicate it;
• change the template revision;
• drop hf-cleanup's maintenance profile;
• give hf-cleanup a default MODEL_NAME.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-23T19:20:18.762487Z	ecc6483	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ecc6483f57

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[lloydmak99]

Pure pre-staging change: adds the graphistry/GLM-5.3-Flash-W4AFP8 snapshot download to the model-downloader in the three dedicated GLM-5.3 Flash TP4 files, plus a regenerated canary and Ruby validate_model_cache contract. No engine/proxy/routing changes; all three files fetch each required download exactly once and hf-cleanup stays inert behind the maintenance profile. Looks good to merge.

Optional follow-up (non-blocking):

• The "every GLM-5.3 Flash file also fetches the W4AFP8 snapshot" note (prod/GLM-5.3-Flash-SGL-TP4.yaml:171 and the twin comments/README) overstates scope — prod/small-models.yaml:574-577 also serves GLM-5.3 Flash but downloads only FP8 + template. Harmless (the validator intentionally covers just the 3 TP4 files, nothing auto-deploys); consider narrowing the wording to "every dedicated GLM-5.3 Flash TP4 file."

Checks: W4AFP8 canary tests 11/11 pass (generator round-trip + --check); YAML parse on all 4 changed files; LongContext sha256 matches the updated pin; grep audit confirms the 3 downloads once each. Ruby validator and docker compose config skipped locally (no ruby/docker) — covered by CI.

[PierreLeGuen]

@lloydmak99 thanks. Your optional follow-up is done in 6fc9583: the pre-stage note (downloader comment, README, validator comment, test docstring) now says "every dedicated GLM-5.3 Flash TP4 file" and excludes prod/small-models.yaml, and the generated HiCache and W4AFP8 canary files are regenerated. gpu13's downloader stays on FP8 (#290; shared disk). No functional change; the CI mirror passes. Re-requesting your review for this round.

[lloydmak99]

Pure pre-staging change: the pinned graphistry/GLM-5.3-Flash-W4AFP8 @ 99f1fa70… snapshot is appended to the model-downloader command in the four GLM-5.3 Flash TP4 files (plus a regenerated canary), with a new Ruby validate_model_cache contract and Python mutation tests. No engine/proxy/routing change. Looks good to merge.

Optional follow-up (non-blocking):

• prod/GLM-5.3-Flash-SGL-TP4.yaml:344-346 (and twin files): engines gate on model-downloader via condition: service_completed_successfully, which runs under set -e, so a cold start of an FP8 host now also depends on the W4AFP8 fetch succeeding even though the running FP8 engines don't use it. The intended pre-stage deploy targets services: ["model-downloader"] only and the revision is pinned/known-good (already served on gpu02), so risk is low — just flagging the added coupling as a conscious choice.

Checks: W4AFP8 canary tests 11/11 pass (generator round-trip + --check); LongContext SHA-256 pin matches; YAML parse OK on all 4 files (11 services each); revision cross-check confirms one W4AFP8 download per file at the pinned SHA and small-models.yaml correctly excluded. Ruby validator + HiCache contract tests and docker compose config couldn't run locally (no ruby/docker) — covered by CI.