Skip to content

fix: add vLLM 0.25.0 measurements and correct FP8-block reuse #244

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
simone-chen merged 17 commits into from
Sep 18, 2026
Merged

fix: add vLLM 0.25.0 measurements and correct FP8-block reuse #244

Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
c89e5c1
Collect B300 vLLM 0.25.0 operator measurements
simone-chen Sep 17, 2026
15b2b86
fix(data): withdraw B200 and B300 vLLM 0.24.0 measurements
simone-chen Sep 17, 2026
e4ab5c3
Revert "fix(data): withdraw B200 and B300 vLLM 0.24.0 measurements"
simone-chen Sep 17, 2026
99b0670
fix(perfmodel): restrict reuse of eager FP8-block GEMMs
simone-chen Sep 17, 2026
3aef988
fix(data): checkpoint H200 vLLM 0.25.0 and extend safe reuse
simone-chen Sep 17, 2026
5d58d48
fix(data): add verified GB200 GB300 and H100 collection checkpoints
simone-chen Sep 17, 2026
34b930e
fix(data): complete vLLM 0.25.0 Hopper collection
simone-chen Sep 17, 2026
efaeebe
fix(data): collect vLLM 0.25.0 Grace Blackwell operator tables
simone-chen Sep 17, 2026
4af4e9f
fix(data): recover GB300 MSA measurements and retain hang evidence
simone-chen Sep 17, 2026
1f29ee4
fix(data): finalize GB200 MSA collection
simone-chen Sep 17, 2026
56c7553
Remove collection diagnostics from packaged performance data
simone-chen Sep 17, 2026
af91885
fix(data): replace invalid FP8-block rows through declared reuse
simone-chen Sep 17, 2026
0a51476
test(data): qualify corrected FP8-block prediction baselines
simone-chen Sep 17, 2026
ec43e8c
test(perfmodel): refresh goldens for corrected FP8-block data
simone-chen Sep 17, 2026
f2dc14f
test(data): verify complete FP8-block reuse coverage
simone-chen Sep 17, 2026
58510e7
test: make collection evidence and GEMM reuse auditable
simone-chen Sep 18, 2026
36bd2f9
fix: reject anomalous collection evidence before writing
simone-chen Sep 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions .github/prediction-numerical-sentinels.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"schema_version": 1,
"baseline_source_sha": "a4c77310d7609745863ac8867c207f850beac5c3",
"baseline_source_sha": "af91885dfdeddedebd256c0c4df11a6f3e75e97e",
"interpretation": "Native prediction stability sentinels; not measured-hardware accuracy. Two percent relative and 0.0001 ms absolute tolerance permit small numerical changes; baseline changes require explained review.",
"cases": [
{
Expand Down Expand Up @@ -102,7 +102,7 @@
"moe_tp_size": 1,
"moe_ep_size": 8,
"database_mode": "SILICON",
"shared_layer": false,
"shared_layer": true,
"forward_model": "op_level"
},
"method": "predict_prefill_latency",
Expand All @@ -111,7 +111,7 @@
"isl": 1024,
"prefix": 0
},
"expected_ms": 42.67633133937119,
"expected_ms": 21.898890264920386,
"rtol": 0.02,
"atol_ms": 0.0001
},
Expand All @@ -126,7 +126,7 @@
"moe_tp_size": 1,
"moe_ep_size": 8,
"database_mode": "SILICON",
"shared_layer": false,
"shared_layer": true,
"forward_model": "op_level"
},
"method": "predict_prefill_latency",
Expand All @@ -135,7 +135,7 @@
"isl": 8192,
"prefix": 128
},
"expected_ms": 116.4093545460498,
"expected_ms": 98.96825693463862,
"rtol": 0.02,
"atol_ms": 0.0001
},
Expand All @@ -150,7 +150,7 @@
"moe_tp_size": 1,
"moe_ep_size": 8,
"database_mode": "SILICON",
"shared_layer": false,
"shared_layer": true,
"forward_model": "op_level"
},
"method": "predict_decode_latency",
Expand All @@ -159,7 +159,7 @@
"isl": 1024,
"osl": 2
},
"expected_ms": 39.36455865490505,
"expected_ms": 6.956580007206089,
"rtol": 0.02,
"atol_ms": 0.0001
},
Expand All @@ -174,7 +174,7 @@
"moe_tp_size": 1,
"moe_ep_size": 8,
"database_mode": "SILICON",
"shared_layer": false,
"shared_layer": true,
"forward_model": "op_level"
},
"method": "predict_decode_latency",
Expand All @@ -183,7 +183,7 @@
"isl": 8192,
"osl": 256
},
"expected_ms": 7396.437641058326,
"expected_ms": 2192.9660386705996,
"rtol": 0.02,
"atol_ms": 0.0001
}
Expand Down
20 changes: 20 additions & 0 deletions crates/core/parity_tests/perfmodel/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,26 @@ requirement, pinned thread caps, byte-reproducible output, and
all-payloads-before-any-write. `TestGoldenComparisonGuards` proves the
comparison itself still bites.

### FP8-block correction in PR #244

The selective refresh from `0a51476b6ab90bc0e475bd41d4b1c7abbef07b95`
covers 68 engine-step records, 19 compiled-engine references, and three
per-op cases. It follows removal of eager vLLM 0.24.0 FP8-block timings
and explicit reuse of the graph-timed 0.25.0 GEMM table. Declared reuse
also fills missing GEMM shapes for the other retained precisions.

The three per-op cases change only 12 QKV/projection GEMM latency values;
other per-op latencies, energies, and source labels are unchanged. For
MiniMax-M2.5 (B200, ISL 1024, OSL 2), context QKV GEMM changes from
11.926155 to 0.948021 ms and generation QKV GEMM from 16.582272 to
0.565109 ms. Static, mixed-step, aggregated/disaggregated, chunked-prefill,
and imbalance-scale references inherit these data changes.

Only records implicated by the failed golden comparisons were refreshed,
using `pin_goldens.py --refresh`. Each refreshed record retains its source
commit in `post_freeze_pins`; test matrices and tolerances are unchanged.
These are prediction-regression baselines, not whole-model silicon validation.

## Engine-Step Benchmark

Historical Python-vs-Rust speedup numbers (dated + commit-stamped) live in
Expand Down
76 changes: 38 additions & 38 deletions crates/core/parity_tests/perfmodel/goldens/compile_engine.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,15 +16,15 @@
"post_freeze_pins": {
"chunked_prefill::ctx300_gen7_isl1000_osl64_prefix100::mixed_step": {
"engine": "rust",
"git_head": "eb4f61cb5d11f17898d7a0f1fd47def268445a6b"
"git_head": "0a51476b6ab90bc0e475bd41d4b1c7abbef07b95"
},
"chunked_prefill::ctx512_gen4_isl4096_osl128_prefix0::mixed_step": {
"engine": "rust",
"git_head": "eb4f61cb5d11f17898d7a0f1fd47def268445a6b"
"git_head": "0a51476b6ab90bc0e475bd41d4b1c7abbef07b95"
},
"chunked_prefill::ctx512_gen4_isl4096_osl128_prefix256::mixed_step": {
"engine": "rust",
"git_head": "eb4f61cb5d11f17898d7a0f1fd47def268445a6b"
"git_head": "0a51476b6ab90bc0e475bd41d4b1c7abbef07b95"
},
"gpt-oss-20b-b200-trtllm-isl1024-osl2::decode_step": {
"engine": "rust",
Expand All @@ -44,19 +44,19 @@
},
"imbalance_scale::decode_step": {
"engine": "rust",
"git_head": "eb4f61cb5d11f17898d7a0f1fd47def268445a6b"
"git_head": "0a51476b6ab90bc0e475bd41d4b1c7abbef07b95"
},
"imbalance_scale::mixed_step": {
"engine": "rust",
"git_head": "eb4f61cb5d11f17898d7a0f1fd47def268445a6b"
"git_head": "0a51476b6ab90bc0e475bd41d4b1c7abbef07b95"
},
"imbalance_scale::static_ctx": {
"engine": "rust",
"git_head": "760b6c92edc00389ff79938b62b028c01bcfefdd"
"git_head": "0a51476b6ab90bc0e475bd41d4b1c7abbef07b95"
},
"imbalance_scale::static_gen": {
"engine": "rust",
"git_head": "eb4f61cb5d11f17898d7a0f1fd47def268445a6b"
"git_head": "0a51476b6ab90bc0e475bd41d4b1c7abbef07b95"
},
"kimi-k25-b200-sglang-isl1024-osl2::decode_step": {
"engine": "rust",
Expand Down Expand Up @@ -108,51 +108,51 @@
},
"minimax-m25-b200-vllm-isl1024-osl2::decode_step": {
"engine": "rust",
"git_head": "eb4f61cb5d11f17898d7a0f1fd47def268445a6b"
"git_head": "0a51476b6ab90bc0e475bd41d4b1c7abbef07b95"
},
"minimax-m25-b200-vllm-isl1024-osl2::mixed_step": {
"engine": "rust",
"git_head": "eb4f61cb5d11f17898d7a0f1fd47def268445a6b"
"git_head": "0a51476b6ab90bc0e475bd41d4b1c7abbef07b95"
},
"minimax-m25-b200-vllm-isl1024-osl2::static_ctx": {
"engine": "rust",
"git_head": "760b6c92edc00389ff79938b62b028c01bcfefdd"
"git_head": "0a51476b6ab90bc0e475bd41d4b1c7abbef07b95"
},
"minimax-m25-b200-vllm-isl1024-osl2::static_gen": {
"engine": "rust",
"git_head": "eb4f61cb5d11f17898d7a0f1fd47def268445a6b"
"git_head": "0a51476b6ab90bc0e475bd41d4b1c7abbef07b95"
},
"minimax-m25-b200-vllm-sampled-prefix::decode_step": {
"engine": "rust",
"git_head": "eb4f61cb5d11f17898d7a0f1fd47def268445a6b"
"git_head": "0a51476b6ab90bc0e475bd41d4b1c7abbef07b95"
},
"minimax-m25-b200-vllm-sampled-prefix::mixed_step": {
"engine": "rust",
"git_head": "eb4f61cb5d11f17898d7a0f1fd47def268445a6b"
"git_head": "0a51476b6ab90bc0e475bd41d4b1c7abbef07b95"
},
"minimax-m25-b200-vllm-sampled-prefix::static_ctx": {
"engine": "rust",
"git_head": "760b6c92edc00389ff79938b62b028c01bcfefdd"
"git_head": "0a51476b6ab90bc0e475bd41d4b1c7abbef07b95"
},
"minimax-m25-b200-vllm-sampled-prefix::static_gen": {
"engine": "rust",
"git_head": "eb4f61cb5d11f17898d7a0f1fd47def268445a6b"
"git_head": "0a51476b6ab90bc0e475bd41d4b1c7abbef07b95"
},
"minimax-m27-b200-vllm-isl1024-osl2::decode_step": {
"engine": "rust",
"git_head": "eb4f61cb5d11f17898d7a0f1fd47def268445a6b"
"git_head": "0a51476b6ab90bc0e475bd41d4b1c7abbef07b95"
},
"minimax-m27-b200-vllm-isl1024-osl2::mixed_step": {
"engine": "rust",
"git_head": "eb4f61cb5d11f17898d7a0f1fd47def268445a6b"
"git_head": "0a51476b6ab90bc0e475bd41d4b1c7abbef07b95"
},
"minimax-m27-b200-vllm-isl1024-osl2::static_ctx": {
"engine": "rust",
"git_head": "760b6c92edc00389ff79938b62b028c01bcfefdd"
"git_head": "0a51476b6ab90bc0e475bd41d4b1c7abbef07b95"
},
"minimax-m27-b200-vllm-isl1024-osl2::static_gen": {
"engine": "rust",
"git_head": "eb4f61cb5d11f17898d7a0f1fd47def268445a6b"
"git_head": "0a51476b6ab90bc0e475bd41d4b1c7abbef07b95"
},
"nemotron-nas-b200-trtllm-isl1024-osl2::decode_step": {
"engine": "rust",
Expand Down Expand Up @@ -228,17 +228,17 @@
}
},
"references": {
"chunked_prefill::ctx300_gen7_isl1000_osl64_prefix100::mixed_step": 43.58332153821506,
"chunked_prefill::ctx512_gen4_isl4096_osl128_prefix0::mixed_step": 46.03136681155293,
"chunked_prefill::ctx512_gen4_isl4096_osl128_prefix256::mixed_step": 46.029415749083185,
"chunked_prefill::ctx300_gen7_isl1000_osl64_prefix100::mixed_step": 16.081171352808852,
"chunked_prefill::ctx512_gen4_isl4096_osl128_prefix0::mixed_step": 17.664446149246828,
"chunked_prefill::ctx512_gen4_isl4096_osl128_prefix256::mixed_step": 17.662495086777085,
"gpt-oss-20b-b200-trtllm-isl1024-osl2::decode_step": 2.213361158220323,
"gpt-oss-20b-b200-trtllm-isl1024-osl2::mixed_step": 11.539696083103436,
"gpt-oss-20b-b200-trtllm-isl1024-osl2::static_ctx": 11.215364152344467,
"gpt-oss-20b-b200-trtllm-isl1024-osl2::static_gen": 2.213362545722157,
"imbalance_scale::decode_step": 39.01752053715316,
"imbalance_scale::mixed_step": 53.71932103484562,
"imbalance_scale::static_ctx": 43.88750156247328,
"imbalance_scale::static_gen": 39.01750961288222,
"imbalance_scale::decode_step": 6.609541889454197,
"imbalance_scale::mixed_step": 26.06666001428854,
"imbalance_scale::static_ctx": 23.110060488022473,
"imbalance_scale::static_gen": 6.60953096518326,
"kimi-k25-b200-sglang-isl1024-osl2::decode_step": 6.491702647659921,
"kimi-k25-b200-sglang-isl1024-osl2::mixed_step": 40.250282696160745,
"kimi-k25-b200-sglang-isl1024-osl2::static_ctx": 39.65105076373316,
Expand All @@ -251,18 +251,18 @@
"minimax-m25-b200-sglang-isl1024-osl2::mixed_step": 28.09828061363558,
"minimax-m25-b200-sglang-isl1024-osl2::static_ctx": 25.330444037457216,
"minimax-m25-b200-sglang-isl1024-osl2::static_gen": 7.412970250723051,
"minimax-m25-b200-vllm-isl1024-osl2::decode_step": 39.36457150698851,
"minimax-m25-b200-vllm-isl1024-osl2::mixed_step": 52.85520178157888,
"minimax-m25-b200-vllm-isl1024-osl2::static_ctx": 42.67633133937119,
"minimax-m25-b200-vllm-isl1024-osl2::static_gen": 39.36455865490505,
"minimax-m25-b200-vllm-sampled-prefix::decode_step": 35.194031677138476,
"minimax-m25-b200-vllm-sampled-prefix::mixed_step": 60.67073561230981,
"minimax-m25-b200-vllm-sampled-prefix::static_ctx": 68.94160745246145,
"minimax-m25-b200-vllm-sampled-prefix::static_gen": 140.77605550543933,
"minimax-m27-b200-vllm-isl1024-osl2::decode_step": 39.36457150698851,
"minimax-m27-b200-vllm-isl1024-osl2::mixed_step": 52.85520178157888,
"minimax-m27-b200-vllm-isl1024-osl2::static_ctx": 42.67633133937119,
"minimax-m27-b200-vllm-isl1024-osl2::static_gen": 39.36455865490505,
"minimax-m25-b200-vllm-isl1024-osl2::decode_step": 6.956592859289543,
"minimax-m25-b200-vllm-isl1024-osl2::mixed_step": 25.202540761021798,
"minimax-m25-b200-vllm-isl1024-osl2::static_ctx": 21.898890264920386,
"minimax-m25-b200-vllm-isl1024-osl2::static_gen": 6.956580007206089,
"minimax-m25-b200-vllm-sampled-prefix::decode_step": 7.81648403057622,
"minimax-m25-b200-vllm-sampled-prefix::mixed_step": 33.61569835698758,
"minimax-m25-b200-vllm-sampled-prefix::static_ctx": 48.247495936383125,
"minimax-m25-b200-vllm-sampled-prefix::static_gen": 31.26586491919031,
"minimax-m27-b200-vllm-isl1024-osl2::decode_step": 6.956592859289543,
"minimax-m27-b200-vllm-isl1024-osl2::mixed_step": 25.202540761021798,
"minimax-m27-b200-vllm-isl1024-osl2::static_ctx": 21.898890264920386,
"minimax-m27-b200-vllm-isl1024-osl2::static_gen": 6.956580007206089,
"nemotron-nas-b200-trtllm-isl1024-osl2::decode_step": 4.86243033208606,
"nemotron-nas-b200-trtllm-isl1024-osl2::mixed_step": 24.98626430220314,
"nemotron-nas-b200-trtllm-isl1024-osl2::static_ctx": 24.91795031094951,
Expand Down
Loading