Skip to content
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
c89e5c1
Collect B300 vLLM 0.25.0 operator measurements
simone-chen Sep 17, 2026
15b2b86
fix(data): withdraw B200 and B300 vLLM 0.24.0 measurements
simone-chen Sep 17, 2026
e4ab5c3
Revert "fix(data): withdraw B200 and B300 vLLM 0.24.0 measurements"
simone-chen Sep 17, 2026
99b0670
fix(perfmodel): restrict reuse of eager FP8-block GEMMs
simone-chen Sep 17, 2026
3aef988
fix(data): checkpoint H200 vLLM 0.25.0 and extend safe reuse
simone-chen Sep 17, 2026
5d58d48
fix(data): add verified GB200 GB300 and H100 collection checkpoints
simone-chen Sep 17, 2026
34b930e
fix(data): complete vLLM 0.25.0 Hopper collection
simone-chen Sep 17, 2026
efaeebe
fix(data): collect vLLM 0.25.0 Grace Blackwell operator tables
simone-chen Sep 17, 2026
4af4e9f
fix(data): recover GB300 MSA measurements and retain hang evidence
simone-chen Sep 17, 2026
1f29ee4
fix(data): finalize GB200 MSA collection
simone-chen Sep 17, 2026
56c7553
Remove collection diagnostics from packaged performance data
simone-chen Sep 17, 2026
af91885
fix(data): replace invalid FP8-block rows through declared reuse
simone-chen Sep 17, 2026
0a51476
test(data): qualify corrected FP8-block prediction baselines
simone-chen Sep 17, 2026
ec43e8c
test(perfmodel): refresh goldens for corrected FP8-block data
simone-chen Sep 17, 2026
f2dc14f
test(data): verify complete FP8-block reuse coverage
simone-chen Sep 17, 2026
58510e7
test: make collection evidence and GEMM reuse auditable
simone-chen Sep 18, 2026
36bd2f9
fix: reject anomalous collection evidence before writing
simone-chen Sep 18, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions .github/prediction-numerical-sentinels.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"schema_version": 1,
"baseline_source_sha": "a4c77310d7609745863ac8867c207f850beac5c3",
"baseline_source_sha": "af91885dfdeddedebd256c0c4df11a6f3e75e97e",
"interpretation": "Native prediction stability sentinels; not measured-hardware accuracy. Two percent relative and 0.0001 ms absolute tolerance permit small numerical changes; baseline changes require explained review.",
"cases": [
{
Expand Down Expand Up @@ -102,7 +102,7 @@
"moe_tp_size": 1,
"moe_ep_size": 8,
"database_mode": "SILICON",
"shared_layer": false,
"shared_layer": true,
"forward_model": "op_level"
},
"method": "predict_prefill_latency",
Expand All @@ -111,7 +111,7 @@
"isl": 1024,
"prefix": 0
},
"expected_ms": 42.67633133937119,
"expected_ms": 21.898890264920386,
"rtol": 0.02,
"atol_ms": 0.0001
},
Expand All @@ -126,7 +126,7 @@
"moe_tp_size": 1,
"moe_ep_size": 8,
"database_mode": "SILICON",
"shared_layer": false,
"shared_layer": true,
"forward_model": "op_level"
},
"method": "predict_prefill_latency",
Expand All @@ -135,7 +135,7 @@
"isl": 8192,
"prefix": 128
},
"expected_ms": 116.4093545460498,
"expected_ms": 98.96825693463862,
"rtol": 0.02,
"atol_ms": 0.0001
},
Expand All @@ -150,7 +150,7 @@
"moe_tp_size": 1,
"moe_ep_size": 8,
"database_mode": "SILICON",
"shared_layer": false,
"shared_layer": true,
"forward_model": "op_level"
},
"method": "predict_decode_latency",
Expand All @@ -159,7 +159,7 @@
"isl": 1024,
"osl": 2
},
"expected_ms": 39.36455865490505,
"expected_ms": 6.956580007206089,
"rtol": 0.02,
"atol_ms": 0.0001
},
Expand All @@ -174,7 +174,7 @@
"moe_tp_size": 1,
"moe_ep_size": 8,
"database_mode": "SILICON",
"shared_layer": false,
"shared_layer": true,
"forward_model": "op_level"
},
"method": "predict_decode_latency",
Expand All @@ -183,7 +183,7 @@
"isl": 8192,
"osl": 256
},
"expected_ms": 7396.437641058326,
"expected_ms": 2192.9660386705996,
"rtol": 0.02,
"atol_ms": 0.0001
}
Expand Down
10 changes: 10 additions & 0 deletions docs/ci.md
Original file line number Diff line number Diff line change
Expand Up @@ -239,6 +239,16 @@ are 2% relative and 0.0001 ms absolute. Missing, duplicate, failed, nonfinite,
nonpositive, or out-of-tolerance results fail. Intentional modeling changes
need explained before/after evidence; do not refresh goldens merely to pass CI.

The FP8-block data correction in PR #244 changes only the MiniMax cases to
enable declared reuse: their vLLM 0.24.0 primary data no longer contains
invalid eager FP8-block measurements, so corrected GEMMs come from 0.25.0.
With the original data, enabling reuse reproduces all four old baselines
exactly. With corrected data, the prefill baselines change from
42.676331 / 116.409355 ms to 21.898890 / 98.968257 ms, and decode from
39.364559 / 7396.437641 ms to 6.956580 / 2192.966039 ms (short/long cases).
The four Qwen baselines and all tolerances remain unchanged. These are
prediction-stability values, not measured whole-model accuracy.

The broader [prediction comparison](../python/aisimulate/tools/prediction_regression_gate/report.py)
reports numerical drift, gains, and added/removed rows for review. It blocks
previously working cases becoming broken. If the comparison base predates the
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -322,6 +322,17 @@ have been invalidated by it, so backward fill preserves historical
reproducibility; forward fill silently answers "how fast is 0.5.14" with
0.5.15's kernels.

### Corrected vLLM 0.24.0 FP8-block measurements

For B200/B300/GB200/GB300 and H100/H200 SXM, the 0.24.0 GEMM tables
exclude FP8-block rows measured eagerly with host launch gaps (PR #219).
All other rows are retained unchanged. Their standard collection sidecars mark
the reduced tables partial, and existing declared reuse points to 0.25.0.
With shared-layer reuse enabled, explicit 0.24.0 requests load retained primary
rows first and fill missing keys from 0.25.0 graph-timed measurements.
Newer versions are never selected implicitly. With reuse disabled, the removed
FP8-block measurements remain unavailable.

### 6.3 Channel 2 — declared reuse (`reuse.yaml`, same backend, any direction)

When we *know* data is valid for a version we never collected — typically a
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,6 @@ runtime:
version: 0.24.0
tables:
gemm_perf:
status: complete
status: partial
rows: 107226
# Eager FP8-block timings removed; reuse.yaml declares the 0.25.0 donor.
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

schema_version: 1
reuse:
- table: gemm_perf
from_version: "0.25.0"
reason: "Replace removed eager FP8-block timings with graph-timed measurements; see PR #219."
approved_by: "simone-chen"
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

schema_version: 2
provenance: local
runtime:
framework: vllm
version: 0.25.0
image: vllm/vllm-openai
image_digest: sha256:e1c1ff1af9a15921bfa11d1d95047258c1797392cdbfa296e7639da446b23f97
tables:
context_attention_perf:
rows: 52836
status: complete
collections:
- collector_ref: cbaf51b64fa460e5ec6146bde407a4c64958212d
collector_hash: sha256:262799b47ef42d0dae5ced92845517d20da265daea5c58dee5813f84c7fe5ec0
case_plan_hash: sha256:04acb1ef5df69580c3028de4dbe34f5e7697450ce783366632247395a3ed3507
collected_at: '2026-09-16'
rows: 52836
status: complete
generation_attention_perf:
rows: 65408
status: complete
collections:
- collector_ref: cbaf51b64fa460e5ec6146bde407a4c64958212d
collector_hash: sha256:262799b47ef42d0dae5ced92845517d20da265daea5c58dee5813f84c7fe5ec0
case_plan_hash: sha256:12df4833c99374841693e9bec6da73b3e7a8300acb5ca88e92c5c43d4bb24cdc
collected_at: '2026-09-16'
rows: 65408
status: complete
Binary file not shown.
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

schema_version: 2
provenance: local
runtime:
framework: vllm
version: 0.25.0
image: vllm/vllm-openai
image_digest: sha256:e1c1ff1af9a15921bfa11d1d95047258c1797392cdbfa296e7639da446b23f97
tables:
encoder_attention_perf:
rows: 7679
status: complete
collections:
- collector_ref: cbaf51b64fa460e5ec6146bde407a4c64958212d
collector_hash: sha256:b65cba616f0be2a3d04cc4192ee7e46b096784accdf6e1bb5e193987894a8830
case_plan_hash: sha256:8a4f0e45a56a4b73354b445a9b3bccde722835cb6c6c133991182ba092eb1296
collected_at: '2026-09-16'
rows: 7679
status: complete
Binary file not shown.
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,6 @@ runtime:
version: 0.24.0
tables:
gemm_perf:
status: complete
status: partial
rows: 107226
# Eager FP8-block timings removed; reuse.yaml declares the 0.25.0 donor.
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

schema_version: 1
reuse:
- table: gemm_perf
from_version: "0.25.0"
reason: "Replace removed eager FP8-block timings with graph-timed measurements; see PR #219."
approved_by: "simone-chen"
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

schema_version: 2
provenance: local
runtime:
framework: vllm
version: 0.25.0
image: vllm/vllm-openai
image_digest: sha256:e1c1ff1af9a15921bfa11d1d95047258c1797392cdbfa296e7639da446b23f97
tables:
gemm_perf:
rows: 148962
Comment thread
simone-chen marked this conversation as resolved.
Resolved
status: complete
collections:
- collector_ref: cbaf51b64fa460e5ec6146bde407a4c64958212d
collector_hash: sha256:5d490b4e88354bf5d27eb98a392bd4b951d9a2331f093d40e51c79d32c0365d5
case_plan_hash: sha256:37d5ec63f630c7a1fc241bcb3c59e2de5e52e68b54849496160dfedf0a12fe3a
collected_at: '2026-09-16'
rows: 37518
status: complete
- collector_ref: cbaf51b64fa460e5ec6146bde407a4c64958212d
collector_hash: sha256:5d490b4e88354bf5d27eb98a392bd4b951d9a2331f093d40e51c79d32c0365d5
case_plan_hash: sha256:a26b4b0dd1684df8b9645fddcb12b080ee89d9c39f0e08b3cb44f22926a5500e
collected_at: '2026-09-16'
rows: 37518
status: complete
- collector_ref: cbaf51b64fa460e5ec6146bde407a4c64958212d
collector_hash: sha256:5d490b4e88354bf5d27eb98a392bd4b951d9a2331f093d40e51c79d32c0365d5
case_plan_hash: sha256:ac15c6379b8de03e49c419d74c5b702b2596c387a07d78f833da2ff288133956
collected_at: '2026-09-16'
rows: 37518
status: complete
- collector_ref: cbaf51b64fa460e5ec6146bde407a4c64958212d
collector_hash: sha256:5d490b4e88354bf5d27eb98a392bd4b951d9a2331f093d40e51c79d32c0365d5
case_plan_hash: sha256:e0f07c473cae107b8c2a5520c2bb1ae85045e499a268a73f0ba1ce97d035a42a
collected_at: '2026-09-16'
rows: 9348
status: complete
- collector_ref: cbaf51b64fa460e5ec6146bde407a4c64958212d
collector_hash: sha256:5d490b4e88354bf5d27eb98a392bd4b951d9a2331f093d40e51c79d32c0365d5
case_plan_hash: sha256:fee7715156b25ee24e67995e9ae17a3bede2bbeed890c479b99cc84ade4d8fc8
collected_at: '2026-09-16'
rows: 9348
status: complete
- collector_ref: cbaf51b64fa460e5ec6146bde407a4c64958212d
collector_hash: sha256:5d490b4e88354bf5d27eb98a392bd4b951d9a2331f093d40e51c79d32c0365d5
case_plan_hash: sha256:03318633ff86a012c7e9bf8310f97ba545972b973af130273420a0c381c85a66
collected_at: '2026-09-16'
rows: 8856
status: complete
- collector_ref: cbaf51b64fa460e5ec6146bde407a4c64958212d
collector_hash: sha256:5d490b4e88354bf5d27eb98a392bd4b951d9a2331f093d40e51c79d32c0365d5
case_plan_hash: sha256:c5da28e5b245195dbf0816485f8230a16d02cf3aa39e8fed9dcffb8da85e87c8
collected_at: '2026-09-16'
rows: 8856
status: complete
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

schema_version: 2
provenance: local
runtime:
framework: vllm
version: 0.25.0
image: vllm/vllm-openai
image_digest: sha256:e1c1ff1af9a15921bfa11d1d95047258c1797392cdbfa296e7639da446b23f97
tables:
gdn_perf:
rows: 10326
status: complete
collections:
- collector_ref: cbaf51b64fa460e5ec6146bde407a4c64958212d
collector_hash: sha256:ad983a508da4af6b521f77f75043cd8959537105866666aa1fbdd8da1298212f
case_plan_hash: sha256:ba375a4c3a8dbbe8b77cec5cfd0f9dafd245e18b99ace763cc738c9645962dda
collected_at: '2026-09-16'
rows: 10326
status: complete
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

schema_version: 2
provenance: local
runtime:
framework: vllm
version: 0.25.0
image: vllm/vllm-openai
image_digest: sha256:e1c1ff1af9a15921bfa11d1d95047258c1797392cdbfa296e7639da446b23f97
tables:
mhc_module_perf:
rows: 140
status: complete
collections:
- collector_ref: cbaf51b64fa460e5ec6146bde407a4c64958212d
collector_hash: sha256:fc748ba7378714bee0e03107d3dec91f0d4659792df2cd2e31a51bdde4dd5c43
case_plan_hash: sha256:4251dd697ea2e18fcaeed92b1a7cbdf452549a75d3a38284340d5da8502e03a1
collected_at: '2026-09-16'
rows: 140
status: complete
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

schema_version: 2
provenance: local
runtime:
framework: vllm
version: 0.25.0
image: vllm/vllm-openai
image_digest: sha256:e1c1ff1af9a15921bfa11d1d95047258c1797392cdbfa296e7639da446b23f97
tables:
mla_context_module_perf:
rows: 13059
status: complete
collections:
- collector_ref: cbaf51b64fa460e5ec6146bde407a4c64958212d
collector_hash: sha256:113fa413702792e5aa95bf4ee7c12f5da9b4410e97fe1be13054acc088a6b7fb
case_plan_hash: sha256:5b1dd5428859b6622f336dbc519f3c5dadc310525ee9c11d72a17557f91c8b39
collected_at: '2026-09-16'
rows: 13059
status: complete
mla_generation_module_perf:
rows: 11400
status: complete
collections:
- collector_ref: cbaf51b64fa460e5ec6146bde407a4c64958212d
collector_hash: sha256:113fa413702792e5aa95bf4ee7c12f5da9b4410e97fe1be13054acc088a6b7fb
case_plan_hash: sha256:3473224f524c629652b2a402f358af23f7f8a1cc2660955225ad5572182ae33e
collected_at: '2026-09-16'
rows: 11400
status: complete
Binary file not shown.
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

schema_version: 2
provenance: local
runtime:
framework: vllm
version: 0.25.0
image: vllm/vllm-openai
image_digest: sha256:e1c1ff1af9a15921bfa11d1d95047258c1797392cdbfa296e7639da446b23f97
tables:
mla_bmm_perf:
rows: 636
status: complete
collections:
- collector_ref: cbaf51b64fa460e5ec6146bde407a4c64958212d
collector_hash: sha256:4c2fd32c623c9841221e74bf76389706de4171beb0c22793246b0ce7ea79af87
case_plan_hash: sha256:3928a5f969ae055ba17244c0d53b504e8253870ffcbf60696731071ddb98e1e5
collected_at: '2026-09-16'
rows: 300
status: complete
- collector_ref: cbaf51b64fa460e5ec6146bde407a4c64958212d
collector_hash: sha256:4c2fd32c623c9841221e74bf76389706de4171beb0c22793246b0ce7ea79af87
case_plan_hash: sha256:faeaa20a84179b59df5283aa9a238572a431f4f16f8521470519147073fe4b53
collected_at: '2026-09-16'
rows: 336
status: complete
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

schema_version: 2
provenance: local
runtime:
framework: vllm
version: 0.25.0
image: vllm/vllm-openai
image_digest: sha256:e1c1ff1af9a15921bfa11d1d95047258c1797392cdbfa296e7639da446b23f97
tables:
moe_perf:
rows: 72924
status: complete
collections:
- collector_ref: cbaf51b64fa460e5ec6146bde407a4c64958212d
collector_hash: sha256:fe2d2d23d340ded7bff27c17b5682e950458c2946d84d1a04bf8608cdb5a689d
case_plan_hash: sha256:261fd9888e2e292a179050ceb03bf8f7ab9d4f27ccd033f5075a92c165aec537
collected_at: '2026-09-16'
rows: 72559
status: complete
- collector_ref: cbaf51b64fa460e5ec6146bde407a4c64958212d
collector_hash: sha256:fe2d2d23d340ded7bff27c17b5682e950458c2946d84d1a04bf8608cdb5a689d
case_plan_hash: sha256:999bf157086b0c2d5cad715cea1ca4587a76130bad8adf97ef81b52b94636387
collected_at: '2026-09-16'
rows: 402
status: complete
Binary file not shown.
Loading