From 9bd5fca4af9702b9333904a359671018d46172ab Mon Sep 17 00:00:00 2001 From: Pranav Raja Date: Fri, 25 Sep 2026 10:40:13 -0700 Subject: [PATCH] Isolation arm: keep the DSA indexer split, drop r2 back to chunk 8192 The 2026-09-25 gpu02 canary ran split + c16384 + pdi 2 and inverted the lab result at the tail. Over a 4 h steady-state window (cold start excluded): TTFT p95 r1 47.3 s r2 76.9 s 1.62x WORSE TTFT p50 r1 3.654 s r2 3.768 s 1.03 ITL p50 r1 18.08ms r2 17.53ms 0.97 (decode win reproduced) ITL p90 r1 78.6 ms r2 68.8 ms 0.87 (decode win reproduced) The lab predicted TTFT 0.61-0.78x. Not explained by traffic volume (681 vs 688 req/h), cache warmth (79.7% vs 81.6% hit) or queueing (r2's average queue depth was LOWER, 1.23 vs 1.56). r2 did draw ~23% more uncached prefill tokens per request, which it absorbed for +3% median TTFT but +62% at p95: worse only at the tail, i.e. only on the largest prefills. The chunk and the split both bite exactly there, and the canary changed both at once. This arm holds the split and returns the chunk to 8192 so they can be told apart: if p95 recovers the chunk was at fault, if it stays bad the all-gather under CC/PPCIe is -- the risk #300 flagged as untested, and which gpu31 cannot test with CC off. The validator gate is one-directional by design (c16384 REQUIRES the split, not the reverse), so this combination is permitted without weakening the check that guards the 0.04-0.65 GB free crash condition. Tests: the two c8192/c16384 asymmetry cases are vacuous now that both replicas run 8192, replaced with drift cases off 8192. The pairing-gate test now CONSTRUCTS the forbidden combination (raise chunk, drop split) and asserts the specific pairing error, instead of passing because an unrelated equality check fired first. 13 tests; the 2 failures under local Ruby 2.6 are the pre-existing filter_map gap, identical on origin/main. gpu02 was rolled back to v0.0.443 before this was written; r1 was never touched. Co-Authored-By: Claude Opus 5 (1M context) --- ...-5.3-Flash-SGL-TP4-W4AFP8-LongContext.yaml | 43 ++++++------ .../prepare_glm53_shared_kv.cpython-314.pyc | Bin 0 -> 11297 bytes ..._glm53_w4afp8_long_context.cpython-314.pyc | Bin 28410 -> 28928 bytes .../test_glm53_shared_kv.cpython-314.pyc | Bin 0 -> 11838 bytes ..._glm53_w4afp8_long_context.cpython-314.pyc | Bin 21724 -> 22468 bytes scripts/prepare_glm53_w4afp8_long_context.py | 61 ++++++++++++------ scripts/test_glm53_w4afp8_long_context.py | 26 +++++--- scripts/validate_glm53_prod_config.rb | 2 +- 8 files changed, 84 insertions(+), 48 deletions(-) create mode 100644 scripts/__pycache__/prepare_glm53_shared_kv.cpython-314.pyc create mode 100644 scripts/__pycache__/test_glm53_shared_kv.cpython-314.pyc diff --git a/prod/GLM-5.3-Flash-SGL-TP4-W4AFP8-LongContext.yaml b/prod/GLM-5.3-Flash-SGL-TP4-W4AFP8-LongContext.yaml index c49e0a21..8318bce4 100644 --- a/prod/GLM-5.3-Flash-SGL-TP4-W4AFP8-LongContext.yaml +++ b/prod/GLM-5.3-Flash-SGL-TP4-W4AFP8-LongContext.yaml @@ -8,23 +8,31 @@ # r1: gpu31 campaign-2 arm L2 (2026-09-23), chunk 8192, --prefill-decode-interval 1 # docker.io/nearaidev/sglang@sha256:fde25985aea3ebabf1eb581ae21d53be8540e32933eef942ee8b962a1bfbea20 # published by workflow run 35903077821 from recipe merge commit f8106f096e9c83. -# r2: chunk 16384, --prefill-decode-interval 2, SGLANG_DSA_INDEXER_QSPLIT=1 +# r2: chunk 8192, --prefill-decode-interval 2, SGLANG_DSA_INDEXER_QSPLIT=1 # docker.io/nearaidev/sglang@sha256:8ff1a487b98a52fe08b781715bebd7c8c445d4fe068f312f03f527d5a3c77e84 # published by workflow run 36073196105 from recipe merge commit 556482c78cd3, # which adds the opt-in DSA indexer query split (cvm-compose-files#300). -# On r2 the 16384 chunk and the split are ONE change: without the split a concurrent -# long burst left 0.04-0.65 GB free per GPU, the condition that preceded the gpu02 -# crash; with it, 9.3-9.5 GB. validate_glm53_prod_config.rb rejects either one alone. -# On stored long-tier traffic (gpu31, 2026-09-24) the r2 arm ran TTFT 0.61-0.78x and -# TPOT 0.53-0.85x of the r1 arm, with quality at the noise floor. -# Neither the HCC/PPCIe host-memory path nor the indexer split's all-gather has run in -# a CVM before this file, so gpu02 is their first soak. Roll out with +# +# r2 IS AN ISOLATION ARM, NOT A SHIPPING CONFIG. The 2026-09-25 canary ran the split WITH +# chunk 16384 and regressed the tail: TTFT p95 76.9 s against r1's 47.3 s (1.62x) over a +# 4 h steady-state window, though the decode win did reproduce (ITL p90 0.87x). Traffic +# volume, cache warmth and queue depth did not explain it. The chunk and the split both +# bite hardest on the largest prefills, and the canary changed both at once, so this arm +# holds the split at 8192 to tell them apart: if p95 recovers the chunk was at fault, if +# it stays bad the all-gather under CC/PPCIe is. Do not read this file as an endorsement +# of either setting until that run reports. +# +# chunk 16384 must never be set without the split: it left 0.04-0.65 GB free per GPU on a +# concurrent long burst, the condition that preceded the gpu02 crash. The reverse is +# allowed, which is what makes this arm possible. validate_glm53_prod_config.rb enforces it. +# The HCC/PPCIe host-memory path is confirmed working in a CVM as of 2026-09-25; the +# split's all-gather under CC is what this arm measures. Roll out with # docs/glm53-w4afp8-long-context-rollout.md, one replica at a time. # Do not hand-edit this file. # This is the DEDICATED LONG-CONTEXT TIER variant of prod/GLM-5.3-Flash-SGL-TP4.yaml. # Generated from the long-context file. Routing remains the dedicated long-context -# contract below; both replicas run a W4AFP8 + HiCache engine, r2 as a canary at -# --prefill-decode-interval 2 with chunk 16384 and the DSA indexer split. +# contract below; both replicas run a W4AFP8 + HiCache engine, r2 as an isolation arm +# at --prefill-decode-interval 2 with the DSA indexer split and chunk 8192. # # *** LONG-CONTEXT TIER (dedicated host, both TP4/EP4 replicas) *** # Deploy this file on exactly the host(s) chosen as the GLM-5.3 Flash long-context @@ -66,8 +74,8 @@ # - Request-level tiering trades the tail for the median: p50 TTFT -24%, # p95 TTFT 10.0 s -> 27.4 s on identical replicas. Judge it on the SLO you mean. # The replicas' engine flags differ in --dist-init-addr, --prefill-decode-interval -# (1 on r1, 2 on r2) and --chunked-prefill-size (8192 on r1, 16384 on r2); r2 also -# sets SGLANG_DSA_INDEXER_QSPLIT=1 and runs a different image. r1's 8192 is the +# (1 on r1, 2 on r2); both now run --chunked-prefill-size 8192. r2 additionally sets +# SGLANG_DSA_INDEXER_QSPLIT=1 and runs a different image. The 8192 chunk is the # W4AFP8 setting: FP8 at 8192 OOMed in the DSA indexer under concurrent 400K+ # contexts, while W4AFP8's 2.4x larger device KV pool kept 3.7 GiB free through 8 # concurrent 647K-756K-token cold prefills on gpu31. 16384 was rejected then because @@ -117,8 +125,7 @@ x-logging-conf: &logging-conf # SGLang image is the # pinned build base. The serving envelope is # intentionally conservative: BF16 KV, 0.80 static memory, 32 running requests, a -# bounded 8-request queue, prefill chunks of 8192 (r1) / 16384 (r2) with -# --max-prefill-tokens 32768, +# bounded 8-request queue, 8192-token prefill chunks with --max-prefill-tokens 32768, # decode graphs capped at batch 32, TileLang DSA, the CUTLASS W4A8 MoE path (hence no # --moe-runner-backend and no FP8 --revision), and adaptive EAGLE 5/1/6. This retains # the full 1,048,576-token model context. @@ -498,7 +505,7 @@ services: --max-queued-requests 8 --enable-priority-scheduling --disable-priority-preemption - --chunked-prefill-size 16384 + --chunked-prefill-size 8192 --max-prefill-tokens 32768 --prefill-decode-interval 2 --cuda-graph-max-bs-decode 32 @@ -576,7 +583,7 @@ services: device_ids: ["4","5","6","7"] capabilities: [gpu] labels: - com.datadoghq.ad.logs: '[{"source":"sglang","service":"sglang","tags":["model:z-ai/glm-5.3-flash","model_path:graphistry/GLM-5.3-Flash-W4AFP8","served_model:z-ai/glm-5.3-flash","precision:int4-weights-fp8-activations-bf16-kv","deployment:glm53-flash-sgl-tp4","config_variant:fc91d24-long-context-w4afp8-c16384-qsplit-hicache-cuda-host-pooled-v1-admission-reserve-disabled-pool-clamp-pdi2-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192","request_logging:disabled","engine_image:8ff1a487b98a","env:${ENV}","host:${CVM_HOST}","ip:${HOST_IP}","port:8000","instance:2"]}]' + com.datadoghq.ad.logs: '[{"source":"sglang","service":"sglang","tags":["model:z-ai/glm-5.3-flash","model_path:graphistry/GLM-5.3-Flash-W4AFP8","served_model:z-ai/glm-5.3-flash","precision:int4-weights-fp8-activations-bf16-kv","deployment:glm53-flash-sgl-tp4","config_variant:fc91d24-long-context-w4afp8-c8192-qsplit-hicache-cuda-host-pooled-v1-admission-reserve-disabled-pool-clamp-pdi2-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192","request_logging:disabled","engine_image:8ff1a487b98a","env:${ENV}","host:${CVM_HOST}","ip:${HOST_IP}","port:8000","instance:2"]}]' nearai.otel.scrape: "true" nearai.otel.job: "sglang" nearai.otel.service: "sglang" @@ -588,7 +595,7 @@ services: nearai.otel.model_path: "graphistry/GLM-5.3-Flash-W4AFP8" nearai.otel.served_model: "z-ai/glm-5.3-flash" nearai.otel.deployment: "glm53-flash-sgl-tp4" - nearai.otel.config_variant: "fc91d24-long-context-w4afp8-c16384-qsplit-hicache-cuda-host-pooled-v1-admission-reserve-disabled-pool-clamp-pdi2-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192" + nearai.otel.config_variant: "fc91d24-long-context-w4afp8-c8192-qsplit-hicache-cuda-host-pooled-v1-admission-reserve-disabled-pool-clamp-pdi2-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192" nearai.otel.thinking_budget_policy: "default8192-public-to-native" nearai.otel.request_logging: "disabled" nearai.otel.engine_image: "8ff1a487b98a" @@ -1014,7 +1021,7 @@ configs: ip: "${HOST_IP}" port: "8000" instance: "2" - config_variant: "fc91d24-long-context-w4afp8-c16384-qsplit-hicache-cuda-host-pooled-v1-admission-reserve-disabled-pool-clamp-pdi2-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192" + config_variant: "fc91d24-long-context-w4afp8-c8192-qsplit-hicache-cuda-host-pooled-v1-admission-reserve-disabled-pool-clamp-pdi2-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192" thinking_budget_policy: "default8192-public-to-native" request_logging: "disabled" engine_image: "8ff1a487b98a" diff --git a/scripts/__pycache__/prepare_glm53_shared_kv.cpython-314.pyc b/scripts/__pycache__/prepare_glm53_shared_kv.cpython-314.pyc new file mode 100644 index 0000000000000000000000000000000000000000..28163bc781b3640dec8495edea36bbef01242918 GIT binary patch literal 11297 zcmcIKTToj`c31Bw;$efqV7sx$1K3;y0t^W3F$N46+h{IWp4kD*7pa9bNV=MPuXt$U zls3CnWADz!PL)$}rCb$fYpd2t<-;FaDSym|tFj-YDj&jP7VRA;wW;jS{=s@So7(*3 zbYERbVEour(godpPoF;Befo5tN9(k!!p=Z??E1$~R}jPe2l>#Fu~2#WM8_~ZlVP~V zLyU|%8JXc2S$Bisbd!2nPfLSrpruhZ($XZGpwv&ACoHn%7z1^~q;6qTi!5C;^(y%g)XlVBE!RWc zLhChh1Jtdwen379bsMc8Jj8IdhnQHE+{o2I{Sd7;arIDdp!Fm3N4dlOA2Cb_R@%hO zn|$X=E7oej!z0a%#wl%muom43`rp>X{}XKBI8I=Zmobtf7)cRc#36Rd&j$G*d@u82 zluyV=2-D`pWUAAHVnSk(4GIYvugWMKjo}WveMU%$LF}bivFI{J*L#L9`ug|5MYBLx zUJ;Opld))!m(UW%NzySYB4H(gaAGl?IDMDL?ch9EiqwK;;u5l{ZL$sg3xqvE1+#f+Wd;h`n|@LZ?(Z>g**#j}Dkj zEySWy1cy*a2rgmKEiJ}ibapWo@9k!nJ9|ByY=P@b%j#%A458L{KzDOvm&b-@#zq5^ zSBA&O+R=&_m1Uejd|1X}`S55(5SPL+VWn-5PIV$WLRYuQM-vjd$+8jH30N*m)&mP> z7kHQ~EShXDdbK$z;xLwj5jH8n9v9IihaL z%=qQutK)%*$&ulaiLt=U#4rckpP8NJ#sU|vjgF7af}COCq(&C{)HX6bS88$VBWPx%%5QX5=YlA3(0lh#Nrcj`(O*w>CqHt`Za$AS}YMV|m`gl14 zrRicc!N<~`q$q^kxK#^He;561lq17|#Z*Fty!^PWz{Sx(_y zM(huVyZD~|zJ-B)zSk4Ro&5`a{at-sy$g6D)EDdz_Vo0IdH{Z=Kiu8r33qmfdp&)j zUcNin*N6Li6lXbwbW=Du&=vCZu;t2wUFqS&$$mEd-ky7Kf2=0$k_x6mJR1=t8Ll58 z2Ks8bi%p+?Q){eSyJX3Q#-{D1TPEG|-sd1Ez1`Yrp7QTH&|bEuvlA|hTj`UpB64HC z%WxM2MuvS}wyO*9ZlzDWiZ*-2cX1{#Gx?n{pw`p5c#HpY1H+s^1^Gl&K6mR%ISRs0 zMVwEdXo7`{4XM{nQ8%@t(qRaTLfn1={qa%{w*``dVl)Xd54U;}E~x<3O@ZS5+>u=C z0NNL*n$(mw(Qoe{>w7e(ssc2<2}D~9>YzB-ZWYkxg_949iYLJa0TrTM)I;&Y<@Y%n^$~7u1@J`4BN&M9dO(6^+G+u4n}trrYC1-5uSlD9i(vNQUg{?wV>x zJsq8^667T{f$d_JR8tsPEA}MGH(- zL$8ijCf;Pd<%StiW|Y}0z!a29s5TWC45}bfJu5Cev&%7$z>*$=f} z2f8ZAXbluJ(EzZou$>_H6C)#TpKoMRWug~JnD8IOPI9aElPWgs(lQGcVN6KL9q|xa zz+qx_gg{h z#S|o#kYX0GoDvgldc`2gqGA9w47$w>>x%QTL}qzhx#_+J)(SbbVBju`{HMHI5QDs% zn0iu@4`Rt3T#kdG9#0AqX351Txj#!`3}%KDalif0E6zZGPk@mHn-&O&)iCM={7Y`g zGR*H8qwe?)$?l)nCdI=MTKptPi=U~$uTD^um(Ya?g11pjMGpS}K!n3SjrTTz#Q2dnmER9aDqGI*Ln3&>gS4Mz?Z>qbTS^RMkAtwQ7l@|QhB|m zr?5}3FUGWum;pg@xulKc{(wB1K!32dgS@1`C~Gr{4(P1dXV&2Mh>w9qq?l)>uW=(| ziiI30V11iOv;vN3BN1>qG^5-dhp!BicUIlKbm!7N>4(#q zvAo6k{Xc!-s(&!>=-Q+4$30IBf4A}^^wpWCtfvj}VlAtKvAlO{eMeVM4}_7fFOf1^lkcbtJUlt&$s`dKDvm?JVklaENu zo}_u^Rm{aPhFO6*ieqzxb2Rx4oRNqxXY!DmPOZV{uxgTkdO5DyZvs)*b7nsr0}q+y z{&*IG$60}~g&LSlzrBOk1*PKgTBg2qYQ&{C3IVv<2VO4WPzKbwV^{(+j!pZo%=3mX# z?4o@DR{uIq9pq}~4wlic+k;z-O3gKBSY@(U7J<@Qo6BXjNrRS3Xg{ibX>!ONYS%ze z`VE>SYwcQ@w9Jv?M3dj5K|yJeD-%x4E@`iyJFejZFB|+8keFh83@%9v>=;}{Nvq%L zE6SMGgDIQk+y|ulMS2$engIZJ|BA0JEnn%KdV0x^mYiIj^ilAid>0>3t{bDb6SkNoMnKwpX?CIky+8d{s9Uh;AxP=d*L^Eg4 z^w`svPoNpraDHF7g_`e z{RD+r0Afdhf{+k!*_;bY0^lw_*wcUie?kV8%@xK#CIc5)iS|Ze@2U8A`NUqAZIz=yH#THJ*VqjATBVsK; z!SJR*W2Hhh=+waK445}y=}FW`Qyglz@5=PGtFwx;JocwJ${Z%eUh?{hnIwWm3D!K8 z6-yWb)O;+an6U&N8CGfwgZYQK$>FQB0UtMZaq=U@5W@+Pm^+GVPfJaqgBFK`?V&-i+E&NX%0@N0@O|9If>>1^Yfr|i1v*-Ey5I{P6p zkZ&7Q0)fXLKc0H>S+;TDX<+@@`gnF`HamDNd;O!F>*F67c5F=ZnU{9vK-+CI;OCu{ zc~{lFx_j1a{rRkOxb%^)ti2t{)}1Mo4c7@s2v+ZJR_LsE|idoN8)!h$1JpMPQ{`%BUxSxLflI(01fG2- zcleWB-CVA6J~Q#cV!z%0y;~1-o0bN;+vERYezv;4n*Hcz_Bx+!Sjf48S!?i{1C6&W zc~|XS;f}C*wEJf_o_&~ejc>W8H(b-3zJ<+@ko`=|xun~MysP@|(w(ITEjd>UaMQ#b zZ2H#BIBV{W{Aeg^JoKu2L+pI&8xX_fZEm|;2Udfgx-pvHLhMd(Uta6C+<|uK0HytV z=7_Fhhh$#{X`#*r+KP!de{?oh@HF1;sgzvTGCPtuY&eTq}Ez0{Y5W=yyu@_%66dW0hG@+X@Yr>qq-**=P^Qe*yJmE4JAb1=~7** zf!ymjSDAd7e5}@1D$8d%2m7lgu5zw8N8&+iK9sr=zv-Q%1p*u(k)}8OV&RI8t4f-$ zhv@4)NFeaJIEV5XaMgR~@Rt4P(pFL`=R8{7r-`EmfU=qKn|*y{fI^ik2HpI|H@D9X zXuV4_*Z9WOXi)_X0-Pi_?g0pDI^YeEzBxpNgi@(SH6aim2*Nv^Gz*bVtB5~Kfrk`I zTMEyWik*hW1G1RHBoxXAW#~+r>04#m%EH@q2yB~l8-2B;5qs4(Az|jS_W<(xNkKB) z{vcrxgpJ@>l8~Z4UfR9eDWt$bE!HlP_tf3*?w~P7flT@;dGzELB;qxo!n^-Fy^&Cl zupJ2^IEWGnN7EsJPZshV1Fw;a72aLJX%v>y<`og#0i-Z#7a0y-w_!dZh|m(A5lak* z+Kwnzc*=#o5>{Np;v&4aCuAS_6ctM#0P*ucKxyRomC@2ROaKcm((p2Dgwn)^LVzBIblVFc zcW0x=y^GHh2zN7mQz9^_C4ix1oQy{`1Jz#ZNabe!6K}%3R1h zDz+S_HXNs(jAk9Da*lz_Xx?buGS+SwYwtxLJF~{xO=IsHL7zTu$__13#7N#~{$l#O z)AuH`?_JAGZyNo1tNm`por)h;W`^??>lZ6utYlX5Rn>oG`3uV((+j)n{?V`P$F^)O z8@84$TibJ6+l%Vjhpw-y-`}e0+NkQvRdsJw4Qx~mxiib?%KicdoAMNpG%lATvRRTm51+v--m7xNE;--?G+kSnD6u zZ(5K4va;$$MfLrQKbp^6EX~U5A~W5xHf~rOAB{fg$yysXt^IkcYs=cOVQqNOx@kS} z!dh|9e1G{~B*$8-=%x!jt;vk%Ev`+=Vd!xm zEcFMjtgB_y+De8tTE4jS-Ah}>h7DuG!dn=xu;DHiysAfO0VML;SMWQ`LqAta5k`W6; zBvyymMluq06p0FjJGl`AKuCl@hY0XP6YYTvJ|uV}Ba-kY2ZG!m(!W7Q9zJSx#vPLV z{=Ckh)Bl!%?90>+DRzc+1U{i-bmyKkXYZ#rn6o>R6m|(@vUx zpEKV3V;hWj=L&^g1=xu0IdkF5Ya7gkooNb$H#Xhz=S(w!HSc^#VI08D1L6l?;tl45 zof!)J4gk+SXNCyk(9S0mb^~DE=gh#DFv`GAfWj7Z6=dd+eQDYu#czWIP7h35vlxYy zic=(;B-ss;5vx-4Lq@cl9D(auBqssx+BJy6E>aefvqs)4K~2Gn8ax|AunFEs7|};E z5)u*N$B&HoA<6#eO;jJ;hDhHqWMr@eA18wu$6YclhUG}k za%{`evSDd?tozz>Dr5Ymts?8Xkh6_s%(TkoY%@@;uFck6%2iKgTo6q#MRnOvSM$aQ zsdBgb6G*EW-~IbZid z#+hS2fIf9+31sM32M03t95b+Oc9>3W8yKDCCE32;A$Z9BoeQ4fekbd|x7|LkV~oz* e=}kjJ9&XZ!KbyES`$PXPj0YjKVLVK?`F{X|38iiT literal 0 HcmV?d00001 diff --git a/scripts/__pycache__/prepare_glm53_w4afp8_long_context.cpython-314.pyc b/scripts/__pycache__/prepare_glm53_w4afp8_long_context.cpython-314.pyc index 6da2c833dce2c0051cdec114284807408379b009..e9458f790de4bceb4d7cb48b1a85ba6cb9ab0630 100644 GIT binary patch delta 2191 zcmZuyOKclO7@pb0Nof-&A#u_)k51l6V>^!XXwo9uG-=b6yz00KYH`rUJGQr3@4CC= zx{*rF0Z|UQp!KB!s!A289Du?iBu>Bu!GV$l32`Y0qzVb4DyCGG5Fq{;_jSOBnQy=O z=YM?j&-ky~_|;#qyVqU56T$Dto1dL+Ro-;B;J%LoNAbyuK8fv#pygewcrOYCSOefP zfh$=f;BtYhSQB8kz|XQ~z!d`bv%P>TIbO!0>SZZZV}FfXvL4n7eDy=#1AHySX#=iC z>DuE0{4gZ0qzwz81e%42|N%P4EaZ1MD`u&c|7Q}7e$Kg*GvD#*X=u{ zt8MO3x99ImC{s3 zmZrkT!E-c`)>ldY@W99*k#&WbG;UJMqKYtOvZ`edkZ|}|n50IB$o^sfD6xnfmsQWK>--GG1_Oq~i(UU!hS$ z5i3MddGM5JDCsCA-BaOdh!`LNe|BX5g&wHMlw&b93XabkS~^L+P*xPG$;f7sB}DG$ z(lkwpLR0X~-RTA+^g>F5WQocOvJ~p&l|GST#b62WyP`sp3E5OAlx7sP(quCm=ie!l z##BwSydo0c`%u+rLe}FT7k5HrzA*~9SqAu5K`Ao{o)dzLF$z zN;cU6qQ->MCMW|+!mccjB~B?LgF{AQa#~|vFa*zVkRa}wMONfOMY5*(;xafiNm^H6 z93StSpC6y1M78{6!VsdEa4LoTpe9PVTb)oi9n+#osxxqG#0XV|n?;Keljmp7o+toi+U0K2#AW8a5(U-LrU{{m zWVozpWmuNF zqEY*tn^aOgW|&b}hGqZ{VSA>cc@+#HdQ3HwaEKYhT;-)g!^mWOafV9+N<6ts2YC&! zSKJ|J{{NBtaVn3KNK)BK!>w|@ub6h>-|a@vT^M)O-pAVxt_x$B7I#l#{J^{^Zr}U;EeOe!S`M86&xeQ_#J2A=xZ*Reme1MX*Pio?%zUz9Nk2LyoyV4 z@q@F#jo~eHcpb>Y`Bgz`Kpxmaqq+1Z8qFsKrvo>4MXff`18&UjgHbJl%`=2q&`-mZe9ztsgDq_Oh@} zNQ9>b!@4War@Sh7t>_fy3w+F-%8NOlEaDlSE_oC<2U~bGXG%C9E2-c+VBkE*an75S z2szJ%uSxj6eS7gr4Lnhnp~+0;S3`gUQ_+Q#gcwFK>+f}<7o%OR_5fA0uxbz;Ea(e=w!p+hKs=g=m+fLb+b?ZNCe1+x! delta 1732 zcmY+EU2GIp6vywK?OHyz%a(S#{eW^yzjnJH-Pw=Y1*L_GPy||rS}}23hVASJ-OemC zccmo7^}*;%{UEbGkeEoK4>cwdnegC)ue@p68cd90j0r(~ARUolG(pe3TS=UUbN=(Y z=bU@)&As>apYdOJusi5(*oNTi{PyPg@$?mU7mj`&J&aFz1_j!lMl;*z@g5|Jv;(w@ z>qfc@bOYBr>2A<&uA68l=p9_g=pN7>rf0C!JR?Y6^A_%^w9vi4n>Kk1a39Rm4Xl~x z#c2;{FW0TMVCv9mDCw)MCoV2H#T}bLkw}9OhbJBhoY2_mc>IdD%^==vf z-9C>iu#`?Z06!gUz>B(&w8w`ek`5*uY!?usA*l<IL1$^`O)b z+Rt@V3V~uMi$! z+HUVYSy#x!$-^X*m(`rIk6bL!yjG?>$|w~Jl*lBbshP5_D=OXAO^TYDBj?JxVJ>=z zdDGKh0hrNcBfp;~lL8HqR3siwMUv#mp$K_ZE*9ows?6q!3(5FwN!R9Pp<=d>i|FTw zuAEmg)F2AXtLQ{i6(YkFN(8DxzXsKGWsXl8rUeL~x}41xGQ^)NmE&TF#G;khP&k?j z#}WbFG^|k87lET5pt8-9wYI1_nWE|*la*=4+TAs-ALbWwq zC_~fq5|L{gp=t#~A=#p)>8!7lo6WMFJUUhx8=RaR8&^nPGiZ20SejfyHnR0s!S@gXm=yLZGfp#->k@b4nK042cDUWD7+lL=26^Yp)`$2z%iT7G!DZTyVEmKyX~##*P3(RN3%??3 zrSp_ic!;dOyTe%c6Ilbj*E#>bFDYQR)fjjS;~nPJff9C`+k!XnPV39yA5L&L#3lhh zW8I2>y%nFe-hcKJC;W|!kvjaQb@s>`PH=aR->a`2#R$jNkyusMkZ6zNdR+JD7_ftQ z9gQvn9JOEI@HoJI>nK$%uc4HEg0qvrhSpKC>RdxfdxEpc&A8`r!YMs&(h(#{&Mv{NWYV$)n8odVN9v zu$YR$MI}01KmL&LpIBf&dtKzBp}|om#89YVzLQRE)kQe+EXTT?Zgk^!&3p4De@R#1 zY!~6#fj$g&2YFS$E(XZpW>vo)3z&7=yR9q5egPjazfiww+JZ>ewVoBcl3eW+S5fmf qNHni%tKHk%`Zjzm&b|!?!u1c>)@~%Gai_3~njY}&+|bRpOUA#O$?3HK diff --git a/scripts/__pycache__/test_glm53_shared_kv.cpython-314.pyc b/scripts/__pycache__/test_glm53_shared_kv.cpython-314.pyc new file mode 100644 index 0000000000000000000000000000000000000000..120fef213e66cf353e88eb7b3d9ee48bf886d622 GIT binary patch literal 11838 zcmcgSZBQHMnY;R2gaGpmHntc10m2Aug8>ulSO$zSAj=B~ClPEGX#rueQg#j zn-=%S+0FRIZtGrrcb+M^@y*bg%!U5BCU-yLxtW`hB5ZlKzM0(Q?nnOFL$B_2?#Dgv zyIQRPVb@917xeD#^FI52?eqOUNAmM*1eBHhf13I0GJ^OLJ!r+8y4g5N62ub05Hc}N zFk~MollW?o4RAH|8PA(!(|*)u>@%OY$kzSDn5mi=qgvoSR6ND%HM4F>J$Onoq3ZoO z*|1xO%DGIAY#+-VvwAY1T6;sTb_e6^)$g@CZ5%m|F_jaHxts{?k_$?SYJ#!Avvthi zp*39n+G14i<-)NdhHAe|5CNEH6#>|Xbu8_!n$Sks5BqZf{~WgH)bKPPIbn*Y1Hs8jUXQ9*mIyB~ZfO5YSR%lxh)d)&$&fy1Q43@68PfMg9o}R-KsZcap*F>YCZkwk=uptl zDHf5J=S0C_P>hl+D#oB7!_+L3VR4p|B{Yi~WvppLg2gpOL{8x5MQ(;`3X6WOX(23L zks_R*mzwXXrWTmJ99iQOa`UtCdn@bBeK*a^OC%Et&NdI#m=&vAcSR3 z=2=$EgCP-}N!3s-5zh$|x#t;sn?n(*_U%-Y}20 zmGG<;hV+tl;D_TzjqW1`FOe0~=ry#Y`i~e#4PGNjpv{cB5ygm|seL?6!w$^{{X9Jp z3j43nb>cBPC`|KWP?j(~z~;<`1)3B5(_yiJ7LU{6d0rHQ0bZhmvZFD&m!@aK0Y2oE zrkt1{oC_`7WTf3GM_Qa>)X-sz8f-TBHs+uZ;IBGNA|egZ3YB7(WKNV7BQFFb695h$ zrfJcRTI{?q6%=?D&q_qRjrSJuz!LF=eb@C1*Dk~=TUQrWgKJW(yenbvUg~*l%fDWK zt^S7h_DI5ZV5#eIZov)z^6N3vZZRKT^usS*w5{Lx7XWe4=@=oCAbt$vdt_r75eABF z1{wucrWgZeXcLAQF$CBgSlo;uh}|i~=m4=`h!sO946$NJ4u;q;gu;+q46$QK4u<4m zNIr(xFr)xOc40^^h7@8*5r){AVmZ%C$oWhOeArUO#=ti6;xJI_CQd{v#hkbgZaT^A zyvP`uK&X6Z6cHJU6jbgH35P*Ml26HFS$~L=q(IOwv#f+VtXaa}2~)jNu8-W9$rDw8 zD53B6d&Cca+#EEGfV3PGpAbe|MgC5sPyUVPx*x%)kd;^WJvzoPUtYU_N3k0F*rzhVVcsqY|G z!u_`07m>o+k6I8u-f5HBMS_%pdaXsh>JtGRToC`^&8~u}A;p_jhyeYAc*(N;9!+tl zx+U>|uOf%u2AfoPI9iZCf5Qu$AM2w~6sm#e@-W*bL zK?L!l><-I)g7(n!&Kws~%>MA4Ad83#97eGQy=y{-4A=&uUqbtyRxFM`deutH*V!w~ ziBmA*67jg8`1<0t#dwA9Zuc+F|LytY!M6Cpw)KN26BWKhfp6J_MMnL4as6`FWBab8 zy((_6x;^yBUi-{QlpOhjvL~sEI8|}G5 zXLQc~mPO1VSd%Vdtmat|F{oZEQLp+$01=n8h-m}?g49kn?%084vxpa+S29`j8}~|T z9QY^g)$n402O`IqUF3Y@^_aeu^_ahvUM<;rWvrmDspx*Ea~ceN@MoM7xG*4woD-b? z3aD-Dk=UX~wCq2|V;Bxn}=li-`UFUn) zbA1Cn>{(Y=e~-JH9U8vKfDyoU_c5nLB#)zqw!JWP-o^B&sPN)QcW=+|sc7vsv`zgE zx-PRreQ)-hik{jsJmVVFIyv~A-hn~zrnH&qO&!jbqpiKcv+sckBc6b1VQDG0qB|@m zbJ4_m*ujey$T(V2#yHB`T+5Gglr_AzhGt%M!H_eX0zjZ3Y`{ zGvotow9SzEL;w+&v_v&wqwO2Z3gV?wip-Ewq;7`6$XN_EFsjl<8DL~F8+94GsjqAX znlXD1`g@ECUs!*zN(kE==z>}B*!{E%OA|<$AR-Gd| zi!2vDz)mb;hD^D=s=KXpZ{^9a(+egu4kntFOII2`T1>VL#M=g9t%DDkSe++Pc_}tD zyiSeie6stVZ}s5Eb-%1zb0rR)UZ-9M4!fRrEiYC!9(y~Ge0w_n_Vgnvs1xkB%sFV= zb!mUR!~t}hqKNXIN|_pnsNVLG$#e0dM{d!xs#rqlVo}V zBB7i(#fuOL;{-K)igYDCF&CKPWj8YOq5R+LLE_(j`dScf&p2!%3U(_7(5M!P=S2Us zs-*3zl2%Oy#XKd3=ORij)8iQcQ-tkud6WWGf(-DIq8 z;e-y42;C$?U-z-$Sl- zqcDvpi`+_U{6~04ffKYT!n0PG9fC9(pY7kl_ks|7n)9=3+a2_EwmBHSPG4$(CA!Dy z^iR(TSKtr>nef37M3aM2o^C&O@_4jrhsI;A&FwAGJt>pAd&tH1xx0HV_b}|GA>gXv zQ^(v83^)s$q#DdfgltpJ^_V+SoWV|pxG9N6M;j0o;l+iZ#Ith{`9?=20p-M2w)mr7 z!{59Q#)i1HrZt;ahd;jj%gaE#X+_%dQRtJF&)Pq2e{d<$dU2idq+6Ol@_lmfv${{~ zfMrg&A5jJb9OdOHgLZz<4@xEvQxHYJrJa)xNo(BS&w+39K?@Uo~ZU#Zp=DArW} zI*S3Dk`M_692^Nnp23y^$3Gd-$$r2{#X_?(q*4c^l0wdJjuIYFl_nqJXJL#*R^Y&$z&cal7a*bl2ZU2f<5qbfzYrgD)8E5H zGVJg>mC4HXcxC(A!C%+?s&1X?dQx0=Q@j;@KYFj`7mmMk-0w>^bjKUI6E!^#@*dWV z#75srjx0(@W4r5DT#xLH z5K%1Joi^xWm2GP!zpnUI#X5C1-9sAqh&uac6EXd;ANmndnYs2E0o4lF@o>>+0a&=C z)!{wR&RD)^Eg4rI(%iV{M?_gEI?drE5rMNl#=saE6JySJE6Z60gzhZIjg4mDQA3Jb zjb1XF`Hu`#O>RX?$pl^82y0qKxh12_yX!>5>>O#JL~$yiesS(njTr0t!4 z)z*-0jTvv2=x=Cu8LjA$Y)jGlg*Bp6E?hi&v^)LE*uarW$?DB%8%%Cy5=X@3ODAV! zjEqdUEyYv41ptr8Zbe0t%aKev$kGRV`x=nc@U61#3uD*Uq*4rH_gd6DSWWut_8&~i za5MA=_Yvz3{pER0nRkrUFI0uLB_tH3=iZX6^HG)K|kse5^pr_{GaGr+HZUf{@;J6Px zu}GsMI9hYge2qS#?)46Od#oCuo5IQ&;p%s@gr(ozgsjD&9j6-WeS+gV&F@90Gh<`0 zh^@h~OpqEKQz4teV;U&Hqa>14%uEA@J7_eTLMUWMhx;oe$lT&lsH1tC?1N~*rboI) zA!{7Ps@2W{y6Qn&&iA;wdl(#eQ}cizgGbf>Mrm$B97mXMLZ#$tLG)o)uTqqX7r+Eo(sVzQsaGy=OOk7Ob(}s6XC$3V&(z?@B`olfmd>nB_Kgk zj0}!Gx=3O>LN%)oNOQoUBQB~VUWsiA$QPQ1{416Zfp4LD7Lsyst0j>Q3#-b12Au_t z1e`2hy>|7E@lIc|sv%z0kf?CRDjH)2P0OY)?Cr6Rg~#^RSiAH%pT1X_tZt20x5ntU zRWWAocxp8j+Lp~5HllR*a`)rn@>`uh?!0Td8%Pu%`B68R38lOLtDx}u{Ez0J5=8kW zBh*92QFM93IA+{{)BSNHoLuTCXARKSZ8(Q6=M4h}gfeJYHlfqccdxyRX4n$1YWXOb zY#oWWjwA}a$%65C!T35mon+-WE3eOg?>Rv%kVA%N>IL;M3_Uz4sk}AtUc=Mw#Gh~dw6O%<{eEGU&d%< zRZlHMS=BA}2CE}CtvXisthr+J=}($s%xLV=<=EvnVrSlr`Nk5(ix1+>H^rvY@^n8Gzi6ca9YJyXMSw#d5`-#1fc7gs7 z#%spC1WJpDRtF%3jTak(;U)n>ZyY44%{NI?A#mS2LJ&8TWRwMDS3_wG2j0D;*oGD* znV;>s8kDh^5s?WYBHrH?E2v7v3TAk}EJ0i#5)=f8++!a{QfjyIu*hdc#GeI88*l^i z3X^&J;(7b-M3Z^-5A*7u*h?V`;B$M`9dGQ=`E`5Wb0d*|1lr*L^St^$Na)bzZ>n4- z^QWdun_taC7fa*gdt}U zsdDios!pS-3snrNyr`N$)m2n|7b>L`QokVAG3aMG8S+jh=3qOZ)5478Fh3s(u`J9A zXDT83KBK0IA-`8d6*BLnS*U*hL)d`l2FM3{iju;fTTr?ZWJp5&+>Y`Jp5p9+=S3#- z42I7D9L4sZ;t2oqT_|3L@I4s63*&c@&x^osc#52d=SLB-{|=KpCUrbM+NW7AD6njF zVrObO1HSC>6RjQG#5enMlqgn~g>hi7v7<`O2|*c=kz$QNGHob0AtK^etf^d6(Fjkf zx`T=0(^(ejIw>52Y!-M&y?oEF54_f^6zAY8Zy-;vicFZ~PJ( z!i1KM)5+A`K-yTLI#p6RExr$r=z~c=fa|*{{l%~Aa7J83v1$qHNPeBL literal 0 HcmV?d00001 diff --git a/scripts/__pycache__/test_glm53_w4afp8_long_context.cpython-314.pyc b/scripts/__pycache__/test_glm53_w4afp8_long_context.cpython-314.pyc index 2ea0216e7d9d7f154aef92d47de51d8b83ba9d3b..3a66b72fdd7f9273da248cbf384ec8eb24095c24 100644 GIT binary patch delta 2414 zcmZ{lYiv|S6vuaV+ue2h02c~&X<_IhWo^5(+hw6>ftFUR!qRR_sV}a#yVLf0V zwM|Gdfk-r>FbPz_wenKS2} zIdkrRX1+d2-hY7?q3kH&aKAwT0|S-X7TY=C&+v;SbDai+fq*= zo`sgUGqMOc*KkM5jCtJ_V}7LEYw<14mWZYeCFcE}<9WG|NwaRrK|+FJoUS2EoTZ<7 z>maBvj|VaKDntPyg;pzuSUrgclJKZ(LndBIE4C{(i(BM1y($>On$!M7lBvxtyr{sZ565hJ)PzpDj0k4;l`M z=T>|V(;xA@Ng$W5s?d5J?FK?tPUbCB$W4p`my)>$y0ZZmRcg9LHB528{T`j+QdmSK!L}+-JjA4k84= zXP230^HkoyJDVR>b*A&&GSFqe7z>O$Hd|+`mZEl{)Svb;Fk@x(g@C{ITFzJs@^2)+ zNuDy~Q@rDtCnmQ#Y=yrY;@8#X;{ITfjkMdu((-w8ZIQekBxr+|;g=@eV#$hyqmlfe zHD_63iP%984)r5qu|!l&*OL8|2dI&v#sH&rq53sJ8Xd?achMou7)WIdY1h+n&7g^l zZqQyvIorvi28+?&VL6&sHI8xmRfG9muC4)1rz%fSo=NI-Oy1_FTc`fCT8n#pN3eDi~Fh z$&>-^x=s^}v-mKqKB{J5y?QFaXeP;7WgKc5bFsmU3Y8y*HL>U}+NW_y2&fx=7k>gi zsrbHb1t@Y|Xn9KqbVt0iU#)Y~=BDr_rKPh;X$!Zs_OwQnj?VVYZC#CZVNgG6Q*JFO z)sq+Y_dfAp=)i{SU~I2t#%-CmaN6yia(gG2ZJFFUeA+#7!11NK^ytb%D<>=3CISd*|eu>%2PF2-8s2!`)Nz`|qE)KpNM_)xWBQ_#r5*}>fnSS9s<**~kZRr_)y$)4L|l&pPm05l?yK~4bxYAV>r|A4swEZ0w}?+;gds!_p^47U=H`BEu0Sk9$mFp&JdGGc za0FIA>+FIZwGr+|UWPWA(yd~;b2<4=obP;Q!FDT&k+B=5%W9^|YW}du?u;Sc>RS9a DQqZ72 delta 1801 zcmZXUZERCj7{~AFwzhO-ESu{nG;l}fwz9R3c9f0DSO>TfT^V(B1eJPePuGj7&Istl*kP*1;b1HfH5Gi0+k4UF_NH}(F9HW;CW657+JF4J3GL`C(f3c=o0}C#>WgOXK*&e0i1jE&^UE$eYng;Am-s-b4v-H}AZsAJ z>jwHy*-f;8UZ3SdF@2Kr9YghcsIfC-EbEA@;fRLUYh(zre!x6{6EK%veA->Ybv8k~ z8L$CRuNPMIBGg6StgK^RPE_7^aM@0nT*2VV$@cGp#0WiI^$Tl%(DfB!UgFjMg#_{{ zfPW|MdkDg8V;DU)vz3Jp1t;47P~C}`Vy*imLWBB!&k2L^T~;+%PJi{xFwXJ&>gz1@ z@EkWi;;-7}on@+@)8MrXy6Eaw2VLSV^kr8E+H42xf(}M*Ig=}q?*2?(R8p}@vi_lD zGadGxF|`}U4Z`H*beZp}GsesK|SpNu8 zR5hL6t+yJ|i2fKVrNPExx1=IL=uBXdn--&qKIJu0R>hRkGax2$GDQa5suaiKI$2B5 zyM9Nh$71PX<5Zs}YV3Dvx<}STNu+n@mTVMza9k5>J+*c2T91FO*xDk-3C1e#7D<{VO>h*)aE#A&r~7KXVw6Z~qLH@}WvxbR@c7&f9+2^fA?BrDR=Rn4zeHqd zO%HRGkQsZh#HThgLze6-NZQ03CHquOT!Te|*W###V>P0rWP2vDs!H8hWdg z2K$&kXxxcL=+Nw!UuI8OZV{F8Z z`WKM+5^$1!*nAz0(SZPFyc75|z)MU7T#(NI|J&r#AP2{EVyBb+kofNoztA#*cI&Gb z)gf;3s6M)QouObalD3Z2uJbkrEQ#cO(hr~sk2>6RDQ zJ02}DzgZsM7$&g4iP}5U=o;PD8Q2K-BfkL2S{T6>3A&yls~~s@@G{^PhKwl^iKU{E z2pNFX5P$$4y*3$z@B_d>z!AVHz*)dIfD3@tfCymbtBi2yg=SI!i8T-L988Q-WkuKu zH_14^Ngmkf?G;UBP?Rx8B1uVBA`#bg@*VuDAHc8w1Z@nrl*b{4EltV++$+CjQxK*B ty>x$gs*(3xMNfw-&@K9N_!HZ51B#(NZ8se5%MSOS0uRR%^rNmB{{n0Z;?)2E diff --git a/scripts/prepare_glm53_w4afp8_long_context.py b/scripts/prepare_glm53_w4afp8_long_context.py index 79eac947..ffc9a4ee 100755 --- a/scripts/prepare_glm53_w4afp8_long_context.py +++ b/scripts/prepare_glm53_w4afp8_long_context.py @@ -37,10 +37,26 @@ R2_ENGINE_IMAGE_LABEL: Final = "8ff1a487b98a" REPLICA_IMAGE: Final = {1: IMAGE, 2: R2_IMAGE} REPLICA_IMAGE_LABEL: Final = {1: ENGINE_IMAGE_LABEL, 2: R2_ENGINE_IMAGE_LABEL} -# c16384 is only safe WITH the split: without it the indexer scratch left 0.04-0.65 GB free on a -# concurrent long burst, the condition that preceded the gpu02 crash. With it, 9.3-9.5 GB. -# The validator enforces the pairing; never set one without the other. -CHUNKED_PREFILL_SIZE: Final = {1: 8192, 2: 16384} +# r2 is back to 8192 while keeping the split. This is an ISOLATION ARM, not the shipping config. +# +# The 2026-09-25 gpu02 canary ran split + c16384 + pdi 2 and produced the opposite of the lab +# result at the tail: over a 4 h steady-state window TTFT p95 was 76.9 s against r1's 47.3 s +# (1.62x), while the decode win did reproduce (ITL p90 0.87x, p50 0.97x). It was not explained by +# traffic volume (681 vs 688 req/h), cache warmth (79.7% vs 81.6% hit) or queueing (r2's average +# queue depth was LOWER, 1.23 vs 1.56). r2 did draw ~23% more uncached prefill tokens per request, +# which it absorbed for +3% median TTFT but +62% at p95 -- worse only at the tail, i.e. only on the +# largest prefills. +# +# Two candidates bite exactly there, and the canary changed both at once: +# a) the 16384 chunk, which doubles how long one prefill chunk occupies the scheduler; +# b) the indexer split's all-gather, which trades per-rank compute for a collective. gpu31 is +# bare metal with CC off, so its all-gather is cheap; under CC/PPCIe memory encryption a +# collective costs more and the trade can invert. This was flagged untested in #300. +# +# Holding the split at 8192 separates them: if p95 recovers, the chunk was at fault; if it stays +# bad, the all-gather under CC is. Note the validator gate is one-directional -- c16384 REQUIRES +# the split, but the split does not require c16384 -- so this combination is permitted by design. +CHUNKED_PREFILL_SIZE: Final = {1: 8192, 2: 8192} R2_EXTRA_ENV: Final = (" - SGLANG_DSA_INDEXER_QSPLIT=1\n",) SOURCE_SERVICE_PREFIX: Final = "model-sg-glm53-fp8-tp4-r" SERVICE_PREFIX: Final = "model-sg-glm53-w4afp8-tp4-r" @@ -54,7 +70,7 @@ VARIANTS: Final = { 1: "fc91d24-long-context-w4afp8-c8192-hicache-cuda-host-pooled-v1-admission-reserve-disabled" "-pool-clamp-pdi1-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192", - 2: "fc91d24-long-context-w4afp8-c16384-qsplit-hicache-cuda-host-pooled-v1-admission-reserve-disabled" + 2: "fc91d24-long-context-w4afp8-c8192-qsplit-hicache-cuda-host-pooled-v1-admission-reserve-disabled" "-pool-clamp-pdi2-h200-tp4-ep4-eagle-adaptive-5-1-6-strict-budget8192", } PREFILL_DECODE_INTERVAL: Final = {1: 1, 2: 2} @@ -83,17 +99,25 @@ "# r1: gpu31 campaign-2 arm L2 (2026-09-23), chunk 8192, --prefill-decode-interval 1\n" f"# {IMAGE}\n" "# published by workflow run 35903077821 from recipe merge commit f8106f096e9c83.\n" - "# r2: chunk 16384, --prefill-decode-interval 2, SGLANG_DSA_INDEXER_QSPLIT=1\n" + "# r2: chunk 8192, --prefill-decode-interval 2, SGLANG_DSA_INDEXER_QSPLIT=1\n" f"# {R2_IMAGE}\n" "# published by workflow run 36073196105 from recipe merge commit 556482c78cd3,\n" "# which adds the opt-in DSA indexer query split (cvm-compose-files#300).\n" - "# On r2 the 16384 chunk and the split are ONE change: without the split a concurrent\n" - "# long burst left 0.04-0.65 GB free per GPU, the condition that preceded the gpu02\n" - "# crash; with it, 9.3-9.5 GB. validate_glm53_prod_config.rb rejects either one alone.\n" - "# On stored long-tier traffic (gpu31, 2026-09-24) the r2 arm ran TTFT 0.61-0.78x and\n" - "# TPOT 0.53-0.85x of the r1 arm, with quality at the noise floor.\n" - "# Neither the HCC/PPCIe host-memory path nor the indexer split's all-gather has run in\n" - "# a CVM before this file, so gpu02 is their first soak. Roll out with\n" + "#\n" + "# r2 IS AN ISOLATION ARM, NOT A SHIPPING CONFIG. The 2026-09-25 canary ran the split WITH\n" + "# chunk 16384 and regressed the tail: TTFT p95 76.9 s against r1's 47.3 s (1.62x) over a\n" + "# 4 h steady-state window, though the decode win did reproduce (ITL p90 0.87x). Traffic\n" + "# volume, cache warmth and queue depth did not explain it. The chunk and the split both\n" + "# bite hardest on the largest prefills, and the canary changed both at once, so this arm\n" + "# holds the split at 8192 to tell them apart: if p95 recovers the chunk was at fault, if\n" + "# it stays bad the all-gather under CC/PPCIe is. Do not read this file as an endorsement\n" + "# of either setting until that run reports.\n" + "#\n" + "# chunk 16384 must never be set without the split: it left 0.04-0.65 GB free per GPU on a\n" + "# concurrent long burst, the condition that preceded the gpu02 crash. The reverse is\n" + "# allowed, which is what makes this arm possible. validate_glm53_prod_config.rb enforces it.\n" + "# The HCC/PPCIe host-memory path is confirmed working in a CVM as of 2026-09-25; the\n" + "# split's all-gather under CC is what this arm measures. Roll out with\n" "# docs/glm53-w4afp8-long-context-rollout.md, one replica at a time.\n" "# Do not hand-edit this file.\n" ) @@ -104,8 +128,8 @@ "# long-context contract below, while the engines intentionally form an r1 control / r2\n" "# HiCache experiment and therefore are not byte-identical to the canonical file.\n", "# Generated from the long-context file. Routing remains the dedicated long-context\n" - "# contract below; both replicas run a W4AFP8 + HiCache engine, r2 as a canary at\n" - "# --prefill-decode-interval 2 with chunk 16384 and the DSA indexer split.\n", + "# contract below; both replicas run a W4AFP8 + HiCache engine, r2 as an isolation arm\n" + "# at --prefill-decode-interval 2 with the DSA indexer split and chunk 8192.\n", ), ( "# tier; every other host keeps the canonical file. The inference-proxy and routing\n" @@ -121,8 +145,8 @@ "# halves prefill speed, and no other per-replica flag moved the tail. Conversation\n" "# affinity stays on because the prefix cache is worth ~3x in request capacity.\n", "# The replicas' engine flags differ in --dist-init-addr, --prefill-decode-interval\n" - "# (1 on r1, 2 on r2) and --chunked-prefill-size (8192 on r1, 16384 on r2); r2 also\n" - "# sets SGLANG_DSA_INDEXER_QSPLIT=1 and runs a different image. r1's 8192 is the\n" + "# (1 on r1, 2 on r2); both now run --chunked-prefill-size 8192. r2 additionally sets\n" + "# SGLANG_DSA_INDEXER_QSPLIT=1 and runs a different image. The 8192 chunk is the\n" "# W4AFP8 setting: FP8 at 8192 OOMed in the DSA indexer under concurrent 400K+\n" "# contexts, while W4AFP8's 2.4x larger device KV pool kept 3.7 GiB free through 8\n" "# concurrent 647K-756K-token cold prefills on gpu31. 16384 was rejected then because\n" @@ -178,8 +202,7 @@ "# 1,048,576-token model context while avoiding the late K-pool workspace exhaustion\n" "# observed with larger prefill chunks.\n", "# intentionally conservative: BF16 KV, 0.80 static memory, 32 running requests, a\n" - "# bounded 8-request queue, prefill chunks of 8192 (r1) / 16384 (r2) with\n" - "# --max-prefill-tokens 32768,\n" + "# bounded 8-request queue, 8192-token prefill chunks with --max-prefill-tokens 32768,\n" "# decode graphs capped at batch 32, TileLang DSA, the CUTLASS W4A8 MoE path (hence no\n" "# --moe-runner-backend and no FP8 --revision), and adaptive EAGLE 5/1/6. This retains\n" "# the full 1,048,576-token model context.\n", diff --git a/scripts/test_glm53_w4afp8_long_context.py b/scripts/test_glm53_w4afp8_long_context.py index 418ca49c..de737f9d 100644 --- a/scripts/test_glm53_w4afp8_long_context.py +++ b/scripts/test_glm53_w4afp8_long_context.py @@ -122,24 +122,30 @@ def test_rejects_engine_argv_drift(self) -> None: # r1 is the pdi 1 control and r2 the pdi 2 canary; neither may take the other's value. ("\n --prefill-decode-interval 1\n", "\n --prefill-decode-interval 2\n", "--prefill-decode-interval 1"), ("\n --prefill-decode-interval 2\n", "\n --prefill-decode-interval 1\n", "--prefill-decode-interval 2"), - # r1 is the c8192 control and r2 the c16384 canary; neither may take the other's chunk. - ("\n --chunked-prefill-size 8192\n", "\n --chunked-prefill-size 16384\n", "model-sg-glm53-w4afp8-tp4-r1 argv must be"), - ("\n --chunked-prefill-size 16384\n", "\n --chunked-prefill-size 8192\n", "model-sg-glm53-w4afp8-tp4-r2 argv must be"), + # Both replicas run 8192 in this arm, so neither may drift off it. + ("\n --chunked-prefill-size 8192\n", "\n --chunked-prefill-size 4096\n", "model-sg-glm53-w4afp8-tp4-r1 argv must be"), + ("\n --chunked-prefill-size 8192\n", "\n --chunked-prefill-size 4096\n", "model-sg-glm53-w4afp8-tp4-r2 argv must be"), ) for before, after, message in cases: with self.subTest(mutation=after.strip()[:60]): self.assert_fails(self.replace_once(before, after), message) def test_rejects_c16384_without_the_indexer_split(self) -> None: - """The 16384 chunk is only memory-safe with SGLANG_DSA_INDEXER_QSPLIT=1. + """Raising r2 to the 16384 chunk without the split must be rejected by the pairing gate. - Without the split a concurrent long burst left 0.04-0.65 GB free, the condition that - preceded the gpu02 crash; with it, 9.3-9.5 GB. Removing the variable while leaving the - chunk at 16384 must be rejected, and the message must name the pairing rather than - surfacing only as generic environment drift. + This arm runs 8192, so the forbidden pairing has to be constructed: raise the chunk AND + drop the split. Without the split that chunk left 0.04-0.65 GB free per GPU on a + concurrent long burst, the condition that preceded the gpu02 crash. The assertion names + the pairing error specifically, so the test cannot pass merely because some unrelated + equality check fired first. """ - mutated = self.replace_once(" - SGLANG_DSA_INDEXER_QSPLIT=1\n", "") - self.assert_fails(mutated, "SGLANG_DSA_INDEXER_QSPLIT") + mutated = self.replace_once("\n --chunked-prefill-size 8192\n", "\n --chunked-prefill-size 16384\n") + mutated = mutated.replace(" - SGLANG_DSA_INDEXER_QSPLIT=1\n", "", 1) + self.assert_fails(mutated, "without SGLANG_DSA_INDEXER_QSPLIT=1") + + def test_rejects_dropping_the_split_from_r2(self) -> None: + """r2 carries the split in this arm; removing it is the whole variable under test.""" + self.assert_fails(self.replace_once(" - SGLANG_DSA_INDEXER_QSPLIT=1\n", ""), "SGLANG_DSA_INDEXER_QSPLIT") def test_rejects_the_split_on_an_image_without_the_patch(self) -> None: """Only the v2 image carries the split patch; the flag is inert and misleading elsewhere.""" diff --git a/scripts/validate_glm53_prod_config.rb b/scripts/validate_glm53_prod_config.rb index 9eac883e..d60a8e2b 100644 --- a/scripts/validate_glm53_prod_config.rb +++ b/scripts/validate_glm53_prod_config.rb @@ -645,7 +645,7 @@ def validate_long_context(errors, compose, replica_services) "model-sg-glm53-w4afp8-tp4-r1" => { "devices" => %w[0 1 2 3], "dist_init" => "127.0.0.1:29510", "instance" => "1", "pdi" => "1", "image" => W4AFP8_LONG_CONTEXT_IMAGE, "chunk" => "8192", "qsplit" => nil }, "model-sg-glm53-w4afp8-tp4-r2" => { "devices" => %w[4 5 6 7], "dist_init" => "127.0.0.1:29511", "instance" => "2", "pdi" => "2", - "image" => W4AFP8_LONG_CONTEXT_R2_IMAGE, "chunk" => "16384", "qsplit" => "1" }, + "image" => W4AFP8_LONG_CONTEXT_R2_IMAGE, "chunk" => "8192", "qsplit" => "1" }, }.freeze W4AFP8_LONG_CONTEXT_ARGV = Shellwords.split(<<~'ARGV').freeze sglang serve