[Docs] Fill GLM-5.2 H200 FP8 speed cells (low-latency, balanced); fix MTP notation by Smallfu666 · Pull Request #31554 · sgl-project/sglang

Smallfu666 · 2026-07-17T06:51:32Z

Motivation

The three H200 + FP8 speed cells for GLM-5.2 have been "pending re-measurement" since the acceptance-length-pinned methodology landed. We measured the low-latency and balanced cells on 8xH200 following that methodology, and while auditing the recipes found an MTP notation mismatch in the docs.

Modifications

Fill all three pending H200 + FP8 speed cells in glm-5.2-benchmarks.jsx: low-latency (c=1/c=16), balanced (c=64/c=256), and high-throughput (c=1024), measured with the acceptance-length-pinned methodology the B200/GB300 cells use (high-throughput has no speculative decoding).
Clarify the MTP notation by quantization: FP8 balanced recipes use 1-1-2 (--speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2), while NVFP4 balanced recipes remain 2-1-3 (--speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 3). The MDX Note now states both explicitly, and the NVFP4 benchmark comment retains 2-1-3.
Annotate the mdx Note that the H200 FP8 cells were measured on v0.5.14 @ 49e384ce (the other cells remain main @ 09ca4fc).

Accuracy Tests

N/A — docs-only change (benchmark cell data + text fix); no code paths affected.

Speed Tests and Profiling

Setup


Hardware	Single node, 8× NVIDIA H200 141GB (TWCC nano4)
Parallelism	Low-latency: TP8. Balanced / High-throughput: TP8 + DP8 DP-Attention + DeepEP (high-throughput: max-running-requests 256, no speculative decoding)
SGLang	`v0.5.14` (`49e384ce`), container built 2026-07-02 from `lmsysorg/sglang:latest`
Model	`zai-org/GLM-5.2-FP8` @ HF commit `70311cfa0158cce7dd2cf5d2e04f68e3fdc3efc1` (704 GB, 141 shards)
Serve flags	exactly the cookbook recipes in `glm-5.2.jsx` (low-latency: EAGLE 5-1-6 + mfs 0.8; balanced: EAGLE 1-1-2 + chunked-prefill 32768 + max-running 256 + mfs 0.85)
Acceptance pinning	`SGLANG_SIMULATE_ACC_LEN=3.5` (low-latency), `=2` (balanced) — matching the B200/GB300 cells' methodology
Benchmark	`python3 -m sglang.bench_serving --dataset-name random --random-input-len 8192 --random-output-len 1024 --random-range-ratio 1.0 --flush-cache --warmup-requests 64`, prompts per concurrency {1: 8, 16: 64, 64: 128, 256: 512, 1024: 2048}
Conventions	`ttft_ms`/`tpot_ms` are P50; `tokens_per_sec_per_gpu` = total (input+output) throughput ÷ 8 — per the convention documented in the NVFP4 section comment

Numbers

cell	conc	TTFT p50 (ms)	TPOT p50 (ms)	total tok/s/GPU	accept_length
low-latency (ACC=3.5)	1	668	5.05	197	3.51
low-latency (ACC=3.5)	16	6148	16.11	813	3.48
balanced (ACC=2)	64	7473	23.49	2343	2.00
balanced (ACC=2)	256	80562	28.08	2391	2.00
high-throughput	1024	553480	61.71	1656	N/A

Sanity vs. the pre-re-measurement H200 numbers (before they were reset to pending): our unpinned low-latency run measured c=1 TTFT 665.5 ms / TPOT 3.03 ms vs. the historical 662 / 3.03 — independent reproduction within ~1%.

Reproducibility

Every point passed a hard completion gate (completed == num_prompts: 8/8, 64/64, 128/128, 512/512; the completed field is visible in each record below).
Low-latency and balanced points were independently repeated on a fresh node, with every metric within 1.4%.
The high-throughput c=1024 point was repeated on a different fresh node: TTFT -0.19%, TPOT -0.33%, and total throughput +0.37%. Both high-throughput runs completed 2048/2048 requests.
At max concurrency 1024 with max-running-requests 256, this point operates in a heavily queued regime. All 2048 requests completed successfully.
The records below are the bench_serving aggregate (last-line) JSON with server_info stripped for size; sanitized aggregate records are included inline below (no external attachment needed).

Raw aggregate records — run 1 (primary)

job188377-low-latency/bench_c1 (server_info omitted here — see attached zip):

{
 "tag": null,
 "backend": "sglang",
 "dataset_name": "random",
 "request_rate": Infinity,
 "max_concurrency": 1,
 "sharegpt_output_len": null,
 "random_input_len": 8192,
 "random_output_len": 1024,
 "random_range_ratio": 1.0,
 "duration": 46.87092513893731,
 "completed": 8,
 "total_input_tokens": 65536,
 "total_input_text_tokens": 65536,
 "total_input_vision_tokens": 0,
 "total_output_tokens": 8192,
 "total_output_tokens_retokenized": 2056,
 "request_throughput": 0.1706815040728548,
 "input_throughput": 1398.2228813648264,
 "output_throughput": 174.7778601706033,
 "total_throughput": 1573.0007415354296,
 "mean_e2e_latency_ms": 5856.81392022525,
 "median_e2e_latency_ms": 5840.214977972209,
 "std_e2e_latency_ms": 94.38504763094735,
 "p90_e2e_latency_ms": 5982.915663276799,
 "p95_e2e_latency_ms": 6004.9216546467505,
 "p99_e2e_latency_ms": 6022.526447742712,
 "mean_ttft_ms": 672.5768942269497,
 "median_ttft_ms": 668.5035654809326,
 "std_ttft_ms": 15.703240363759694,
 "p90_ttft_ms": 693.9584533218294,
 "p95_ttft_ms": 699.2837211349979,
 "p99_ttft_ms": 703.5439353855327,
 "mean_tpot_ms": 5.067680377319942,
 "median_tpot_ms": 5.0547804672294685,
 "std_tpot_ms": 0.09023807513109654,
 "p90_tpot_ms": 5.1879770626365875,
 "p95_tpot_ms": 5.195317298213961,
 "p99_tpot_ms": 5.201189486675861,
 "mean_itl_ms": 5.067671189499557,
 "median_itl_ms": 4.509554244577885,
 "std_itl_ms": 0.7702675321455809,
 "p90_itl_ms": 5.986686555358271,
 "p95_itl_ms": 6.030418677255511,
 "p99_itl_ms": 6.083622652416428,
 "concurrency": 0.9996498089788786,
 "accept_length": 3.5129310344827585,
 "max_output_tokens_per_s": 207.0,
 "max_concurrent_requests": 2
}

job188377-low-latency/bench_c16 (server_info omitted here — see attached zip):

{
 "tag": null,
 "backend": "sglang",
 "dataset_name": "random",
 "request_rate": Infinity,
 "max_concurrency": 16,
 "sharegpt_output_len": null,
 "random_input_len": 8192,
 "random_output_len": 1024,
 "random_range_ratio": 1.0,
 "duration": 90.69424906396307,
 "completed": 64,
 "total_input_tokens": 524288,
 "total_input_text_tokens": 524288,
 "total_input_vision_tokens": 0,
 "total_output_tokens": 65536,
 "total_output_tokens_retokenized": 16448,
 "request_throughput": 0.7056676764020983,
 "input_throughput": 5780.829605085989,
 "output_throughput": 722.6037006357486,
 "total_throughput": 6503.433305721738,
 "mean_e2e_latency_ms": 22663.672626033076,
 "median_e2e_latency_ms": 22634.03127202764,
 "std_e2e_latency_ms": 68.62604373956043,
 "p90_e2e_latency_ms": 22780.968908173963,
 "p95_e2e_latency_ms": 22781.843876792118,
 "p99_e2e_latency_ms": 22782.43900728412,
 "mean_ttft_ms": 6088.042177812895,
 "median_ttft_ms": 6147.928810445592,
 "std_ttft_ms": 2953.3753808444735,
 "p90_ttft_ms": 10248.367946152575,
 "p95_ttft_ms": 10388.81406300934,
 "p99_ttft_ms": 10474.469358159695,
 "mean_tpot_ms": 16.202962314975743,
 "median_tpot_ms": 16.1094704726192,
 "std_tpot_ms": 2.88821385325889,
 "p90_tpot_ms": 20.27970795983491,
 "p95_tpot_ms": 20.8922032804892,
 "p99_tpot_ms": 21.04937957348493,
 "mean_itl_ms": 16.202955236126876,
 "median_itl_ms": 10.483376128831878,
 "std_itl_ms": 88.85796219569306,
 "p90_itl_ms": 13.908424026643237,
 "p95_itl_ms": 13.970418728422374,
 "p99_itl_ms": 14.094835221767429,
 "concurrency": 15.993021200750603,
 "accept_length": 3.4775862068965515,
 "max_output_tokens_per_s": 1454.0,
 "max_concurrent_requests": 32
}

job188378-balanced/bench_c64 (server_info omitted here — see attached zip):

{
 "tag": null,
 "backend": "sglang",
 "dataset_name": "random",
 "request_rate": Infinity,
 "max_concurrency": 64,
 "sharegpt_output_len": null,
 "random_input_len": 8192,
 "random_output_len": 1024,
 "random_range_ratio": 1.0,
 "duration": 62.939392385073006,
 "completed": 128,
 "total_input_tokens": 1048576,
 "total_input_text_tokens": 1048576,
 "total_input_vision_tokens": 0,
 "total_output_tokens": 131072,
 "total_output_tokens_retokenized": 32895,
 "request_throughput": 2.033702505687949,
 "input_throughput": 16660.09092659568,
 "output_throughput": 2082.51136582446,
 "total_throughput": 18742.60229242014,
 "mean_e2e_latency_ms": 31446.884258515638,
 "median_e2e_latency_ms": 31445.53433859255,
 "std_e2e_latency_ms": 68.50949368152482,
 "p90_e2e_latency_ms": 31518.610538705252,
 "p95_e2e_latency_ms": 31556.508335482795,
 "p99_e2e_latency_ms": 31564.069423787296,
 "mean_ttft_ms": 7726.924181864888,
 "median_ttft_ms": 7473.413051920943,
 "std_ttft_ms": 3863.7389433935423,
 "p90_ttft_ms": 12046.830470184796,
 "p95_ttft_ms": 12188.939655397553,
 "p99_ttft_ms": 12200.1242283592,
 "mean_tpot_ms": 23.186666741594088,
 "median_tpot_ms": 23.48958427962262,
 "std_tpot_ms": 3.791163860622865,
 "p90_tpot_ms": 28.1717660805012,
 "p95_tpot_ms": 28.210898518279183,
 "p99_tpot_ms": 28.745887772389988,
 "mean_itl_ms": 23.369230913447804,
 "median_itl_ms": 18.78348900936544,
 "std_itl_ms": 125.83604515228544,
 "p90_itl_ms": 19.452188978902996,
 "p95_itl_ms": 19.80993594042957,
 "p99_itl_ms": 20.75543743558228,
 "concurrency": 63.953607312622175,
 "accept_length": 2.0,
 "max_output_tokens_per_s": 3456.0,
 "max_concurrent_requests": 128
}

job188378-balanced/bench_c256 (server_info omitted here — see attached zip):

{
 "tag": null,
 "backend": "sglang",
 "dataset_name": "random",
 "request_rate": Infinity,
 "max_concurrency": 256,
 "sharegpt_output_len": null,
 "random_input_len": 8192,
 "random_output_len": 1024,
 "random_range_ratio": 1.0,
 "duration": 246.72980459104292,
 "completed": 512,
 "total_input_tokens": 4194304,
 "total_input_text_tokens": 4194304,
 "total_input_vision_tokens": 0,
 "total_output_tokens": 524288,
 "total_output_tokens_retokenized": 131577,
 "request_throughput": 2.075144512227232,
 "input_throughput": 16999.583844165485,
 "output_throughput": 2124.9479805206856,
 "total_throughput": 19124.531824686168,
 "mean_e2e_latency_ms": 96707.55410415631,
 "median_e2e_latency_ms": 111274.56834493205,
 "std_e2e_latency_ms": 31700.323856044484,
 "p90_e2e_latency_ms": 116857.34916154761,
 "p95_e2e_latency_ms": 116880.1650033798,
 "p99_e2e_latency_ms": 174026.31403507432,
 "mean_ttft_ms": 65974.76368588832,
 "median_ttft_ms": 80561.94260250777,
 "std_ttft_ms": 29479.444804584742,
 "p90_ttft_ms": 90037.83877966926,
 "p95_ttft_ms": 92339.51969599583,
 "p99_ttft_ms": 110386.8161455936,
 "mean_tpot_ms": 30.04182836585337,
 "median_tpot_ms": 28.07623228146526,
 "std_tpot_ms": 15.543531215306043,
 "p90_tpot_ms": 33.82528970265654,
 "p95_tpot_ms": 35.7640387333123,
 "p99_tpot_ms": 116.17792877836338,
 "mean_itl_ms": 30.398047726286045,
 "median_itl_ms": 19.92835500277579,
 "std_itl_ms": 490.0420258815505,
 "p90_itl_ms": 20.98563426407054,
 "p95_itl_ms": 21.566689881728962,
 "p99_itl_ms": 24.16617702692747,
 "concurrency": 200.6821501901581,
 "accept_length": 2.0,
 "max_output_tokens_per_s": 4437.0,
 "max_concurrent_requests": 336
}

Raw aggregate records — run 2 (reproducibility check)

job189097-low-latency-repeat/bench_c1 (server_info omitted here — see attached zip):

{
 "tag": null,
 "backend": "sglang",
 "dataset_name": "random",
 "request_rate": Infinity,
 "max_concurrency": 1,
 "sharegpt_output_len": null,
 "random_input_len": 8192,
 "random_output_len": 1024,
 "random_range_ratio": 1.0,
 "duration": 47.33765984000638,
 "completed": 8,
 "total_input_tokens": 65536,
 "total_input_text_tokens": 65536,
 "total_input_vision_tokens": 0,
 "total_output_tokens": 8192,
 "total_output_tokens_retokenized": 2056,
 "request_throughput": 0.16899863717468722,
 "input_throughput": 1384.4368357350377,
 "output_throughput": 173.0546044668797,
 "total_throughput": 1557.4914402019174,
 "mean_e2e_latency_ms": 5915.235135005787,
 "median_e2e_latency_ms": 5916.122988099232,
 "std_e2e_latency_ms": 51.52819026964398,
 "p90_e2e_latency_ms": 5974.467770406045,
 "p95_e2e_latency_ms": 5985.118931147736,
 "p99_e2e_latency_ms": 5993.639859741088,
 "mean_ttft_ms": 670.0472391676158,
 "median_ttft_ms": 664.4925546133891,
 "std_ttft_ms": 16.594898690678736,
 "p90_ttft_ms": 691.5624632500112,
 "p95_ttft_ms": 698.7666545668617,
 "p99_ttft_ms": 704.5300076203421,
 "mean_tpot_ms": 5.1272608952474785,
 "median_tpot_ms": 5.124072539610997,
 "std_tpot_ms": 0.0394158073422213,
 "p90_tpot_ms": 5.174337233801829,
 "p95_tpot_ms": 5.178383211191027,
 "p99_tpot_ms": 5.181619993102385,
 "mean_itl_ms": 5.127252333856462,
 "median_itl_ms": 4.510876751737669,
 "std_itl_ms": 0.7733233419951872,
 "p90_itl_ms": 5.988019371094803,
 "p95_itl_ms": 6.011436693370342,
 "p99_itl_ms": 6.056803356235226,
 "concurrency": 0.9996666763838049,
 "accept_length": 3.4788135593220337,
 "max_output_tokens_per_s": 207.0,
 "max_concurrent_requests": 2
}

job189097-low-latency-repeat/bench_c16 (server_info omitted here — see attached zip):

{
 "tag": null,
 "backend": "sglang",
 "dataset_name": "random",
 "request_rate": Infinity,
 "max_concurrency": 16,
 "sharegpt_output_len": null,
 "random_input_len": 8192,
 "random_output_len": 1024,
 "random_range_ratio": 1.0,
 "duration": 90.63667547004297,
 "completed": 64,
 "total_input_tokens": 524288,
 "total_input_text_tokens": 524288,
 "total_input_vision_tokens": 0,
 "total_output_tokens": 65536,
 "total_output_tokens_retokenized": 16448,
 "request_throughput": 0.7061159256790386,
 "input_throughput": 5784.501663162684,
 "output_throughput": 723.0627078953355,
 "total_throughput": 6507.564371058019,
 "mean_e2e_latency_ms": 22648.411396196025,
 "median_e2e_latency_ms": 22719.608452403918,
 "std_e2e_latency_ms": 143.76929169313607,
 "p90_e2e_latency_ms": 22754.013554449193,
 "p95_e2e_latency_ms": 22754.58411106374,
 "p99_e2e_latency_ms": 22755.450878271367,
 "mean_ttft_ms": 6048.050032430183,
 "median_ttft_ms": 6107.504372950643,
 "std_ttft_ms": 2935.1149314192776,
 "p90_ttft_ms": 10182.524590892717,
 "p95_ttft_ms": 10327.294039470144,
 "p99_ttft_ms": 10409.258595537394,
 "mean_tpot_ms": 16.227137207982253,
 "median_tpot_ms": 16.117264967746777,
 "std_tpot_ms": 2.874447942581307,
 "p90_tpot_ms": 20.24662969483724,
 "p95_tpot_ms": 20.7390688460221,
 "p99_tpot_ms": 21.09710788449185,
 "mean_itl_ms": 16.227130081523164,
 "median_itl_ms": 10.537880240008235,
 "std_itl_ms": 85.07094961833693,
 "p90_itl_ms": 13.981531118042769,
 "p95_itl_ms": 14.040645367155472,
 "p99_itl_ms": 14.167000617211064,
 "concurrency": 15.992403978184642,
 "accept_length": 3.478448275862069,
 "max_output_tokens_per_s": 1456.0,
 "max_concurrent_requests": 32
}

job189098-balanced-repeat/bench_c64 (server_info omitted here — see attached zip):

{
 "tag": null,
 "backend": "sglang",
 "dataset_name": "random",
 "request_rate": Infinity,
 "max_concurrency": 64,
 "sharegpt_output_len": null,
 "random_input_len": 8192,
 "random_output_len": 1024,
 "random_range_ratio": 1.0,
 "duration": 63.34407868096605,
 "completed": 128,
 "total_input_tokens": 1048576,
 "total_input_text_tokens": 1048576,
 "total_input_vision_tokens": 0,
 "total_output_tokens": 131072,
 "total_output_tokens_retokenized": 32895,
 "request_throughput": 2.020709791118362,
 "input_throughput": 16553.65460884162,
 "output_throughput": 2069.2068261052027,
 "total_throughput": 18622.861434946826,
 "mean_e2e_latency_ms": 31641.409009343988,
 "median_e2e_latency_ms": 31641.18522289209,
 "std_e2e_latency_ms": 11.548501282324974,
 "p90_e2e_latency_ms": 31656.760893040337,
 "p95_e2e_latency_ms": 31658.407244936097,
 "p99_e2e_latency_ms": 31660.847783731297,
 "mean_ttft_ms": 7757.3084259202005,
 "median_ttft_ms": 7498.361431527883,
 "std_ttft_ms": 3886.9371135256074,
 "p90_ttft_ms": 12225.003275251947,
 "p95_ttft_ms": 12236.39358021319,
 "p99_ttft_ms": 12239.808484513778,
 "mean_tpot_ms": 23.34711689484241,
 "median_tpot_ms": 23.58934099905244,
 "std_tpot_ms": 3.8052188727822376,
 "p90_tpot_ms": 28.287102847645933,
 "p95_tpot_ms": 28.472086977801276,
 "p99_tpot_ms": 28.85318531490679,
 "mean_itl_ms": 23.530945011353964,
 "median_itl_ms": 18.956460524350405,
 "std_itl_ms": 126.24252549093102,
 "p90_itl_ms": 19.450886524282396,
 "p95_itl_ms": 19.58462258335203,
 "p99_itl_ms": 20.368880592286587,
 "concurrency": 63.93810498996215,
 "accept_length": 2.0,
 "max_output_tokens_per_s": 3424.0,
 "max_concurrent_requests": 128
}

job189098-balanced-repeat/bench_c256 (server_info omitted here — see attached zip):

{
 "tag": null,
 "backend": "sglang",
 "dataset_name": "random",
 "request_rate": Infinity,
 "max_concurrency": 256,
 "sharegpt_output_len": null,
 "random_input_len": 8192,
 "random_output_len": 1024,
 "random_range_ratio": 1.0,
 "duration": 245.8893771301955,
 "completed": 512,
 "total_input_tokens": 4194304,
 "total_input_text_tokens": 4194304,
 "total_input_vision_tokens": 0,
 "total_output_tokens": 524288,
 "total_output_tokens_retokenized": 131576,
 "request_throughput": 2.0822371668740374,
 "input_throughput": 17057.686871032114,
 "output_throughput": 2132.2108588790143,
 "total_throughput": 19189.89772991113,
 "mean_e2e_latency_ms": 96767.34893708045,
 "median_e2e_latency_ms": 110937.15275602881,
 "std_e2e_latency_ms": 31737.090643189425,
 "p90_e2e_latency_ms": 117347.78634714894,
 "p95_e2e_latency_ms": 117376.49990059435,
 "p99_e2e_latency_ms": 174104.11466171267,
 "mean_ttft_ms": 66040.63537753654,
 "median_ttft_ms": 80595.93886043876,
 "std_ttft_ms": 29491.651070825625,
 "p90_ttft_ms": 90129.17693518102,
 "p95_ttft_ms": 92648.30517515074,
 "p99_ttft_ms": 110487.71346103848,
 "mean_tpot_ms": 30.035888132496474,
 "median_tpot_ms": 27.992867913032505,
 "std_tpot_ms": 15.54591473997517,
 "p90_tpot_ms": 33.82328355663027,
 "p95_tpot_ms": 35.91894802557967,
 "p99_tpot_ms": 116.0347772182356,
 "mean_itl_ms": 30.392037093958326,
 "median_itl_ms": 19.939263467676938,
 "std_itl_ms": 489.7560616319358,
 "p90_itl_ms": 20.746644597966224,
 "p95_itl_ms": 21.102216653525822,
 "p99_itl_ms": 22.988157928921282,
 "concurrency": 201.49257049665778,
 "accept_length": 2.0,
 "max_output_tokens_per_s": 4353.0,
 "max_concurrent_requests": 336
}

Server-flag digest (from server_info of the run-1 records; full ServerArgs in the attached zip)

server arg	low-latency	balanced
`version`	`0.5.14`	`0.5.14`
`tp_size`	`8`	`8`
`dp_size`	`1`	`8`
`enable_dp_attention`	`False`	`True`
`moe_a2a_backend`	`none`	`deepep`
`ep_size`	`1`	`8`
`speculative_algorithm`	`EAGLE`	`EAGLE`
`speculative_num_steps`	`5`	`1`
`speculative_eagle_topk`	`1`	`1`
`speculative_num_draft_tokens`	`6`	`2`
`mem_fraction_static`	`0.8`	`0.85`
`chunked_prefill_size`	`8192`	`4096`
`max_running_requests`	`48`	`256`
`schedule_conservativeness`	`1.0`	`0.3`
`attention_backend`	`dsa`	`dsa`
`kv_cache_dtype`	`bfloat16`	`bfloat16`
`dsa_prefill_backend`	`flashmla_sparse`	`flashmla_sparse`
`dsa_decode_backend`	`fa3`	`fa3`
`dsa_topk_backend`	`sgl-kernel`	`sgl-kernel`
`max_total_num_tokens`	`202816`	`97216`
`weight_loader_disable_mmap`	`False`	`False`
`weight_loader_drop_cache_after_load`	`False`	`False`

Raw aggregate records — high-throughput c=1024 (both fresh-node runs)

run 1 (primary):

{
 "tag": null,
 "backend": "sglang",
 "dataset_name": "random",
 "request_rate": Infinity,
 "max_concurrency": 1024,
 "sharegpt_output_len": null,
 "random_input_len": 8192,
 "random_output_len": 1024,
 "random_range_ratio": 1.0,
 "duration": 1424.5137859559618,
 "completed": 2048,
 "total_input_tokens": 16777216,
 "total_input_text_tokens": 16777216,
 "total_input_vision_tokens": 0,
 "total_output_tokens": 2097152,
 "total_output_tokens_retokenized": 2090051,
 "request_throughput": 1.4376835241546149,
 "input_throughput": 11777.503429874605,
 "output_throughput": 1472.1879287343256,
 "total_throughput": 13249.691358608932,
 "mean_e2e_latency_ms": 543826.7195251883,
 "median_e2e_latency_ms": 634466.0517864395,
 "std_e2e_latency_ms": 235223.74403821983,
 "p90_e2e_latency_ms": 750868.5894504188,
 "p95_e2e_latency_ms": 778773.7937859027,
 "p99_e2e_latency_ms": 1178501.1734094839,
 "mean_ttft_ms": 458330.23908795667,
 "median_ttft_ms": 553480.0831214525,
 "std_ttft_ms": 210243.6957750701,
 "p90_ttft_ms": 660995.9750647191,
 "p95_ttft_ms": 682623.7465491517,
 "p99_ttft_ms": 701735.1920356136,
 "mean_tpot_ms": 83.57427217715703,
 "median_tpot_ms": 61.70527278199857,
 "std_tpot_ms": 106.69424515102686,
 "p90_tpot_ms": 81.00405583933781,
 "p95_tpot_ms": 86.87968842752592,
 "p99_tpot_ms": 667.4230744187398,
 "mean_itl_ms": 83.57518179068929,
 "median_itl_ms": 37.74788696318865,
 "std_itl_ms": 3518.4882146177906,
 "p90_itl_ms": 41.75438731908798,
 "p95_itl_ms": 44.41027110442519,
 "p99_itl_ms": 270.6557059660554,
 "concurrency": 781.8507146564161,
 "accept_length": null,
 "max_output_tokens_per_s": 3361.0,
 "max_concurrent_requests": 1122
}

run 2 (reproducibility, different node):

{
 "tag": null,
 "backend": "sglang",
 "dataset_name": "random",
 "request_rate": Infinity,
 "max_concurrency": 1024,
 "sharegpt_output_len": null,
 "random_input_len": 8192,
 "random_output_len": 1024,
 "random_range_ratio": 1.0,
 "duration": 1419.265560768079,
 "completed": 2048,
 "total_input_tokens": 16777216,
 "total_input_text_tokens": 16777216,
 "total_input_vision_tokens": 0,
 "total_output_tokens": 2097152,
 "total_output_tokens_retokenized": 2089780,
 "request_throughput": 1.4429998561309851,
 "input_throughput": 11821.05482142503,
 "output_throughput": 1477.6318526781288,
 "total_throughput": 13298.686674103159,
 "mean_e2e_latency_ms": 541644.4377911555,
 "median_e2e_latency_ms": 633149.2786440067,
 "std_e2e_latency_ms": 234600.3623954544,
 "p90_e2e_latency_ms": 747442.3129011411,
 "p95_e2e_latency_ms": 775582.2239092551,
 "p99_e2e_latency_ms": 1174030.4074160939,
 "mean_ttft_ms": 456371.10916878737,
 "median_ttft_ms": 552440.0541086216,
 "std_ttft_ms": 209673.4598428391,
 "p90_ttft_ms": 658362.8703787457,
 "p95_ttft_ms": 679339.4044729182,
 "p99_ttft_ms": 698943.4284139937,
 "mean_tpot_ms": 83.3561374607704,
 "median_tpot_ms": 61.50061101038173,
 "std_tpot_ms": 106.31406078449504,
 "p90_tpot_ms": 80.70336166784743,
 "p95_tpot_ms": 86.82641498275636,
 "p99_tpot_ms": 664.4745829015677,
 "mean_itl_ms": 83.35696498425482,
 "median_itl_ms": 37.727318704128265,
 "std_itl_ms": 3505.8912586794663,
 "p90_itl_ms": 41.74499725922942,
 "p95_itl_ms": 44.45643830113113,
 "p99_itl_ms": 269.52496128156776,
 "concurrency": 781.5928458067857,
 "accept_length": null,
 "max_output_tokens_per_s": 4046.0,
 "max_concurrent_requests": 1122
}

Notes for reviewers

Version caveat: these cells are v0.5.14, not main @ 09ca4fc — annotated in the mdx Note. Happy to re-run on a newer container if the team prefers a single version across cells.
Balanced c=256 TTFT (80.6 s) is much higher than B200's (18.7 s): on H200 141GB the KV pool saturates around c~64 for this workload (c=256 adds +2% throughput while TTFT p50 grows 10.8x) — admission queueing, consistent across both runs. We believe reporting it as measured is more useful than omitting it.

Checklist

Format your code according to the Format code with pre-commit (lint CI is green).
Add unit tests according to the Run and add unit tests — N/A, docs-only.
Update documentation according to Write documentations.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed (full protocol, raw records and reproducibility data above).
Follow the SGLang code style guidance.

CI States

Latest PR Test (Base): ✅ Run #31764309479
Latest PR Test (Extra): ❌ Run #31764309369

gemini-code-assist · 2026-07-17T06:51:35Z

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

Smallfu666 · 2026-07-17T08:46:37Z

This is a docs-only PR that fills pending H200 benchmark cells and fixes an MTP notation mismatch. Could a maintainer please run /tag-and-rerun-ci when convenient? Thank you.

zRzRzRzRzRzRzR

The H200 FP8 benchmark updates look reasonable, but the MTP notation fix seems too broad. The NVFP4 balanced recipes in glm-5.2.jsx still use 2-1-3 (num-steps=2, topk=1, draft-tokens=3), so the NVFP4 benchmark comment should remain 2-1-3. Could you scope the 1-1-2 correction to FP8 only? CC@JustinTong0323

Smallfu666 · 2026-07-23T14:53:22Z

@zRzRzRzRzRzRzR Thanks for catching this. I’ve scoped 1-1-2 to FP8 only, restored the NVFP4 benchmark comment to 2-1-3, and clarified both configurations in the MDX note. I’ve also rebased the branch onto the latest main; the updated fix is in 90f846acb. CC @JustinTong0323

zijiexia · 2026-08-14T00:33:08Z

Hi @Smallfu666 , thanks for contributing! Sorry I just saw your PR. Is there any chance you can update it or maybe just resolve the conflict? Thank you so much!

… MTP notation - Fill the pending H200+FP8 low-latency (c=1/c=16) and balanced (c=64/c=256) speed cells, measured on v0.5.14 @ 49e384c with the same acceptance-length-pinned methodology as the B200/GB300 cells (SGLANG_SIMULATE_ACC_LEN=3.5 / =2, random 8192/1024, --random-range-ratio 1.0, --flush-cache, P50, total tok/s/GPU). - High-throughput cell stays pending (requires the c=1024 point). - Fix MTP notation: the mdx Note and the NVFP4 section comment said balanced is 2-1-3; the actual recipe flags in glm-5.2.jsx are 1-1-2. - Annotate the mdx Note with the H200 cells' measurement version. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Fills the last pending H200 FP8 speed cell with the high-throughput recipe (TP8 + DP8 DP-Attention + DeepEP, mfs 0.85, max-running 256, no spec), v0.5.14 @ 49e384c, random 8192/1024, --random-range-ratio 1.0, --flush-cache. c=1024 / 2048 prompts: TTFT p50 553480 ms, TPOT p50 61.71 ms, 1656 tok/s/GPU, 2048/2048 completed. Repeated on a fresh node with all metrics within 0.4%. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Smallfu666 · 2026-08-14T02:39:09Z

@zijiexia Thanks for the review! Rebased onto latest main and resolved the conflicts — the PR should be up to date now.

… MTP notation (sgl-project#31554) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

…ph-max-bs Match the flag the benchmark numbers were measured with (and origin/main / sgl-project#31554): `--cuda-graph-max-bs` is the deprecated alias for `--cuda-graph-max-bs-decode`, so the two are functionally identical, but the shipped numbers used the old name. Keep the old name so config matches the measured recipe (esp. the low-latency cells). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… MTP notation (sgl-project#31554) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Smallfu666 requested review from JustinTong0323, sogalin, wisclmy0611 and zijiexia as code owners July 17, 2026 06:51

github-actions Bot added the documentation Improvements or additions to documentation label Jul 17, 2026

zRzRzRzRzRzRzR reviewed Jul 20, 2026

View reviewed changes

Smallfu666 force-pushed the glm52-h200-fp8-cells branch from 72610fb to 90f846a Compare July 23, 2026 14:52

zRzRzRzRzRzRzR approved these changes Jul 27, 2026

View reviewed changes

dougyster mentioned this pull request Aug 6, 2026

docs(cookbook): add Community Configs section for contributed recipes #33860

Open

Smallfu666 and others added 3 commits August 14, 2026 10:37

docs: scope GLM-5.2 MTP notation by quantization

f2ec68c

Smallfu666 force-pushed the glm52-h200-fp8-cells branch from c297ab0 to 234c89e Compare August 14, 2026 02:37

zijiexia approved these changes Aug 14, 2026

View reviewed changes

zijiexia merged commit 22dde1d into sgl-project:main Aug 14, 2026
93 of 97 checks passed

dougyster mentioned this pull request Aug 15, 2026

docs(cookbook): GLM-5.2 single-node benchmarks #31609

Open

saturn-acc pushed a commit to saturn-acc/sglang that referenced this pull request Aug 16, 2026

[Docs] Fill GLM-5.2 H200 FP8 speed cells (low-latency, balanced); fix…

25415f5

… MTP notation (sgl-project#31554) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Atituiset pushed a commit to Atituiset/sglang that referenced this pull request Sep 10, 2026

[Docs] Fill GLM-5.2 H200 FP8 speed cells (low-latency, balanced); fix…

b98f435

… MTP notation (sgl-project#31554) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Navigation Menu

[Docs] Fill GLM-5.2 H200 FP8 speed cells (low-latency, balanced); fix MTP notation - #31554

[Docs] Fill GLM-5.2 H200 FP8 speed cells (low-latency, balanced); fix MTP notation#31554
zijiexia merged 3 commits into
sgl-project:mainfrom
Smallfu666:glm52-h200-fp8-cells

Smallfu666 commented Jul 17, 2026 •

edited by github-actions Bot

Loading

Uh oh!

gemini-code-assist Bot commented Jul 17, 2026

Uh oh!

Smallfu666 commented Jul 17, 2026

Uh oh!

zRzRzRzRzRzRzR left a comment

Uh oh!

Smallfu666 commented Jul 23, 2026

Uh oh!

zijiexia commented Aug 14, 2026

Uh oh!

Smallfu666 commented Aug 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Smallfu666 commented Jul 17, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Accuracy Tests

Speed Tests and Profiling

Setup

Numbers

Reproducibility

Notes for reviewers

Checklist

CI States

Uh oh!

gemini-code-assist Bot commented Jul 17, 2026

Uh oh!

Smallfu666 commented Jul 17, 2026

Uh oh!

zRzRzRzRzRzRzR left a comment

Choose a reason for hiding this comment

Uh oh!

Smallfu666 commented Jul 23, 2026

Uh oh!

zijiexia commented Aug 14, 2026

Uh oh!

Smallfu666 commented Aug 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Smallfu666 commented Jul 17, 2026 •

edited by github-actions Bot

Loading