[pranavraja99]

The gpu02 canary from #302 regressed the TTFT tail in production. gpu02 has been rolled back to v0.0.443; this PR prepares the follow-up experiment.

What the canary actually measured

4 h steady-state window, r2's cold start excluded:

	r1 (control)	r2 (canary)	ratio
TTFT p95	47.3 s	76.9 s	1.62× ✗
TTFT p50	3.654 s	3.768 s	1.03
ITL p50	18.08 ms	17.53 ms	0.97 ✓
ITL p90	78.6 ms	68.8 ms	0.87 ✓

The lab predicted TTFT 0.61–0.78×. Production gave 1.62× at p95 — opposite sign. The decode win did reproduce (ITL p90 0.87×), so the change is not simply bad.

Confounds ruled out

• Traffic volume: balanced, 681 vs 688 req/h.
• Cache warmth: r2 79.7% total hit vs r1 81.6% — 1.9 points, nowhere near a 62% tail gap. (r2's HiCache host tier worked harder: 2.73% of tokens vs 0.65%.)
• Queueing: r2's average queue depth was lower, 1.23 vs 1.56.

r2 did draw ~23% more uncached prefill tokens per request. It absorbed that for +3% median TTFT but +62% at p95 — worse only at the tail, i.e. only on the largest prefills.

Why this arm

Two things bite hardest on the largest prefills, and #302 changed both at once:

1. chunk 16384 — doubles how long one prefill chunk occupies the scheduler.
2. the indexer split's all-gather — trades per-rank compute for a collective. gpu31 is bare metal with CC off, so its all-gather is cheap; under CC/PPCIe memory encryption a collective costs more and the trade can invert. Add an opt-in DSA indexer query split to the HiCache W4AFP8 image #300 flagged this as untested and said the r2 canary is where it would show. It showed.

This arm holds the split and returns the chunk to 8192:

	r1 (control)	r2 (isolation arm)
image	fde25985	8ff1a487 (v2, carries the patch)
chunk	8192	8192 (was 16384)
SGLANG_DSA_INDEXER_QSPLIT	unset	1
pdi	1	2

If p95 recovers → the chunk was at fault. If it stays bad → the all-gather under CC is.

The validator gate is one-directional by design — c16384 requires the split, not the reverse — so this combination is permitted without weakening the check guarding the 0.04–0.65 GB-free crash condition.

Tests

• The two c8192/c16384 asymmetry cases are vacuous now that both replicas run 8192; replaced with drift cases off 8192.
• The pairing-gate test now constructs the forbidden combination (raise the chunk and drop the split) and asserts the specific pairing error. Previously it passed because an unrelated equality check fired first — it would not have caught a broken gate.
• 13 tests. The 2 failures under local Ruby 2.6 are the pre-existing filter_map gap, identical on origin/main; CI runs 3.3.
• --check and validate_glm53_prod_config.rb pass.

Header

The generated file's header states plainly that r2 is an isolation arm, not a shipping config, and records the canary's actual result, so nobody reads the file as an endorsement of either setting.

Deployment

Same r2-only pair, plus the otelcol-contrib recreate from #303 (this changes r2's config_variant to c8192-qsplit). gpu02 currently sits on v0.0.443 with r1 untouched.

🤖 Generated with Claude Code

[github-actions]

🔍 OpenCodeReview found 1 issue(s) in this PR.

• ✅ 1 posted as inline comment(s)
• 📝 0 posted as summary

[github-actions]

This replacement's tail (the unchanged continuation lines right below, ending "...restores 9.3-9.5 GB, which is what makes r2's 16384 admissible") was not updated along with the new text and now contradicts both this replacement and the rest of the generated header: r2 no longer runs chunk 16384 (the engine command is --chunked-prefill-size 8192, and the new isolation-arm paragraph explicitly says not to read the file as an endorsement of either setting). The committed YAML still emits this stale sentence (header lines "16384 was rejected then because / it fell to 0.04 GiB free; the indexer split shrinks that scratch by TP and restores / 9.3-9.5 GB, which is what makes r2's 16384 admissible"), and since prod/GLM-5.3-Flash-SGL-TP4-W4AFP8-LongContext.yaml is generated and marked "Do not hand-edit this file", every regeneration will re-emit the contradiction. Rephrase the tail here (e.g. tie the split-restored headroom to the one-directional pairing gate rather than to r2 running 16384) and regenerate the YAML in the same change.

Suggestion:

Suggested change

	"# (1 on r1, 2 on r2); both now run --chunked-prefill-size 8192. r2 additionally sets\n"
	"# SGLANG_DSA_INDEXER_QSPLIT=1 and runs a different image. The 8192 chunk is the\n"
	"# concurrent 647K-756K-token cold prefills on gpu31. 16384 was rejected then because\n"
	"# it fell to 0.04 GiB free; the indexer split shrinks that scratch by TP and restores\n"
	"# 9.3-9.5 GB, which is what makes the chunk safe WITH the split -- this arm still\n"
	"# holds the chunk at 8192 while the canary is attributed. Conversation affinity stays\n"
	"# on because the prefix cache is worth ~3x in request capacity.\n",