diff --git a/README.md b/README.md index de09081d2..b04acad55 100644 --- a/README.md +++ b/README.md @@ -402,6 +402,20 @@ complete test coverage, and release gates. Use [DEVELOPMENT.md](DEVELOPMENT.md) for environment and local test details and [CONTRIBUTING.md](CONTRIBUTING.md) before sending a change. +Direct Python `ReplaySpec.workload` synthetic workloads accept +`length_sampler: numpy_random_state` for InferenceX-compatible seeded token +lengths in Gym replay. The public prediction/recommendation YAML and CLI, and +the Sweeper `Workload` schema, do not expose this option and reject it. +The supported direct path must omit `source_type`; all workload-driver inputs +with `source_type` reject `length_sampler` at the common runner entrypoint, +including AFD and AFD+PD. Materialized trace replay rejects +non-default samplers. The default `python_random` +preserves existing workloads. Both sample the full input vector before output +lengths; unknown sampler names are rejected. NumPy seeds must fit an unsigned +32-bit integer; the Python sampler retains unsigned 64-bit seed support. +Each replay initializes its own seeded sampler, so independent runs reproduce +the same request lengths. + ### FPM accuracy overview The [FPM Accuracy Overview](https://ai-dynamo.org/aisimulate/fpm-accuracy/?branch=main) diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index d311585c9..b7aa79be5 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -36,17 +36,20 @@ at the identified revision is available at: https://github.com/ai-dynamo/aiconfigurator/blob/13b5cf2697876692b0a52098266c81162add11fc/LICENSE The 18 B200 TensorRT-LLM 1.3.0rc20 performance tables under -`src/aiconfigurator_core/systems/data/b200_sxm/*/trtllm/1.3.0rc20/` include +`src/aisimulate_core/systems/data/b200_sxm/*/trtllm/1.3.0rc20/` include power measurements derived from AIConfigurator commit -`915f590680d8a79fe9c39f6f3a9ff13bc267fcce` (PR #1584). Sixteen tables are -unmodified, byte-identical copies. The context-attention and +`915f590680d8a79fe9c39f6f3a9ff13bc267fcce` (PR #1584). Fourteen tables remain +unmodified, byte-identical copies after the September 17-18 refresh. The +context-MLA table was replaced with locally collected measurements, and the +MoE table has locally refreshed MXFP4 rows with unavailable-power sentinels. +The context-attention and generation-attention tables are modified derivatives: AISimulate preserves newer local timing rows and adds the typed `0.0` / `0.0` unavailable sentinel to those local-only identities. Source paths, row counts, measured coverage, and merge details are recorded in -`src/aiconfigurator_core/systems/data/b200_sxm/README.md`. The two unmodified +`src/aisimulate_core/systems/data/b200_sxm/README.md`. The two unmodified upstream attention copies under -`src/aiconfigurator_core/systems/data/b200_sxm/power_upstream/` support focused +`src/aisimulate_core/systems/data/b200_sxm/power_upstream/` support focused import regression tests, which pin source and packaged SHA-256 digests. The corresponding energy expectations in the repository-root file diff --git a/docs/trtllm-eager-execution-plan.md b/docs/trtllm-eager-execution-plan.md new file mode 100644 index 000000000..6aa13d4ee --- /dev/null +++ b/docs/trtllm-eager-execution-plan.md @@ -0,0 +1,89 @@ +# TRT-LLM eager execution: proposed measurement contract + +Status: deferred at the user’s request. Retain collection evidence; do not +extend the data contract or implement a timing correction. The proposal below +is preserved for a future decision. +The current GPT-OSS op-based prediction still omits a material host execution +boundary. See the [trace report](../python/aisimulate/collector/trtllm/gym-gpt-eager-20260918.md) +for measured evidence and its limits. + +## Problem + +GPT-OSS-120B, B200 TP2/EP1, 1k/1k, concurrency 4 reproduces client TTFT at +48.86 ms versus a 16.53 ms prediction. The corrected MXFP4/MXFP8 MoE kernel +cost is close to the trace. Eager prefill/mixed execution has substantial CPU +submission gaps and cross-rank waiting; graph decode does not have the same +boundary. Increasing the shared MoE kernel table would charge that delay to +graph decode and erase its existing TPOT alignment. + +## Proposed coordinated change + +Keep the op-based GPU model and add explicit execution-boundary measurements. +Do not insert an E2E-derived TTFT constant or a model-wide latency multiplier. + +1. Extend the collector output contract with an execution-mode identity + (`cuda_graph` or `eager`) and CPU dispatch timing. Preserve GPU latency as + a separate measurement. Record actual capture success, runtime source, + CPU model/affinity, rank membership, warmup policy and raw timing samples. + The measured runtime version remains part of the identity: rc14 host timing + must not be published as an rc20 measurement. + Existing rows without this evidence retain their legacy behavior; they + must not be relabeled as measured eager execution. +2. Collect both modes at identical serving-selected modules, shapes, dtypes + and topology. Use phase-correct attention and fused all-reduce/norm + boundaries. Preserve the overlap within a module and between CPU and GPU; + a profiler's inflated CPU times are diagnostic evidence, not database rows. +3. Carry the resolved TRT-LLM graph policy from Gym/runner into the native + performance model. Mixed/prefill selects eager measurements where the + pinned runtime cannot capture those steps. Decode selects graph data only + for captured shapes. Graph memory reservation is a separate setting. +4. Compose host submission and GPU work with explicit dependencies. Do not + add CPU time to the sum of all GPU kernels, or separately add collective + waiting already caused by a late peer. Validate the composition against + complete, independent per-rank traces before enabling the new path. +5. Keep missing eager coverage visible. Do not silently report a legacy + graph lookup as an eager measurement. Preserve the existing path for + configurations outside the qualified collection domain. + +The exact row schema and composition must be frozen after the paired +measurement smoke, before formal collection. Approval would cover the +new execution-mode/CPU-timing contract and its producer/consumer work; it +is not an accuracy claim for an unmeasured composition algorithm. + +## Owners and affected surfaces + +- Collector: `python/aisimulate/collector/trtllm/`, the network collector for + fused communication, perf schema/metadata, and collector tests. +- Data loading and native timing: `crates/core/src/perfmodel/perf_database/`, + `operators/`, and `engine/runtime.rs`; Python lowering and diagnostics. +- Deployment identity: Gym normalization, AISimulate runner/provider config, + and native engine config. Preserve actual graph capture coverage. +- Documentation and data: measured rows, immutable provenance, paired results, + missing coverage, and unchanged controls. + +## Acceptance checks + +- The existing TP2/C4 workload retains its exact 40-request token vectors. + Repeated unprofiled serving measurements remain independent validation. +- Prefill, mixed, and graph decode are evaluated separately; complete rank + groups and timing boundaries are checked. CPU affinity is retained. +- Cover neighboring token counts and the affected GPT-OSS topologies, not + only the diagnostic 886-token step. Select collection points before looking + at validation errors. +- Replay all 31 GPT-OSS/B200 points. If shared timing or config code changes, + rerun all 263 points and retain every TTFT/TPOT regression and failure. +- Reject double counting with synthetic dependency tests; test graph/eager + selection at capture boundaries and explicit missing-data behavior. +- Report remaining queue/frontend error separately from forward error. + Do not claim the entire 32 ms client gap is a kernel or host-only delta. + +## Repository approval boundary + +`python/aisimulate/.claude/rules/collector/layer_permissions.md` requires: + +> Changing the data contract requires explicit human approval. + +Adding execution-mode/CPU-timing fields changes the producer/consumer contract. +The current collection/fix request authorizes diagnostic collection; this +proposal makes the newly discovered contract extension reviewable before its +implementation. No collector policy file is changed. diff --git a/python/aisimulate/THIRD_PARTY_NOTICES.md b/python/aisimulate/THIRD_PARTY_NOTICES.md index d311585c9..b7aa79be5 100644 --- a/python/aisimulate/THIRD_PARTY_NOTICES.md +++ b/python/aisimulate/THIRD_PARTY_NOTICES.md @@ -36,17 +36,20 @@ at the identified revision is available at: https://github.com/ai-dynamo/aiconfigurator/blob/13b5cf2697876692b0a52098266c81162add11fc/LICENSE The 18 B200 TensorRT-LLM 1.3.0rc20 performance tables under -`src/aiconfigurator_core/systems/data/b200_sxm/*/trtllm/1.3.0rc20/` include +`src/aisimulate_core/systems/data/b200_sxm/*/trtllm/1.3.0rc20/` include power measurements derived from AIConfigurator commit -`915f590680d8a79fe9c39f6f3a9ff13bc267fcce` (PR #1584). Sixteen tables are -unmodified, byte-identical copies. The context-attention and +`915f590680d8a79fe9c39f6f3a9ff13bc267fcce` (PR #1584). Fourteen tables remain +unmodified, byte-identical copies after the September 17-18 refresh. The +context-MLA table was replaced with locally collected measurements, and the +MoE table has locally refreshed MXFP4 rows with unavailable-power sentinels. +The context-attention and generation-attention tables are modified derivatives: AISimulate preserves newer local timing rows and adds the typed `0.0` / `0.0` unavailable sentinel to those local-only identities. Source paths, row counts, measured coverage, and merge details are recorded in -`src/aiconfigurator_core/systems/data/b200_sxm/README.md`. The two unmodified +`src/aisimulate_core/systems/data/b200_sxm/README.md`. The two unmodified upstream attention copies under -`src/aiconfigurator_core/systems/data/b200_sxm/power_upstream/` support focused +`src/aisimulate_core/systems/data/b200_sxm/power_upstream/` support focused import regression tests, which pin source and packaged SHA-256 digests. The corresponding energy expectations in the repository-root file diff --git a/python/aisimulate/collector/README.md b/python/aisimulate/collector/README.md index 98ed8b072..1341e89ef 100644 --- a/python/aisimulate/collector/README.md +++ b/python/aisimulate/collector/README.md @@ -127,6 +127,9 @@ capabilities.py — generation-time capability/denylist f The plan is one equation: cases = dedup(base grid ∪ model shapes), then intersected with hardware capability floors and minus the hang denylist. +TRT-LLM MLA precision and GPT-OSS TP profile collection evidence is recorded +in [the September 17 campaign](trtllm/gym-profiles-20260917.md). + ## Whole-forward FPM campaign Use the dedicated `python3 -m collector.fpm_forward` entry point for vLLM @@ -664,3 +667,12 @@ Explicit limits must be positive for one-shot module collection. Omitting the option keeps SGLang's default. The existing wide-EP and DSA paths keep their separate behavior. The ordinary context table uses the existing consumer schema; this does not add mixed-precision modeling. + +## TRT-LLM MoE autotuning failures + +TRT-LLM MoE collection requires either a loaded tactic cache or at least one +successful autotuning attempt before measuring a tuning-enabled case. Ordinary +failures may try smaller eligible token counts; if every attempt fails, the +case raises before timing or publishing rows. CUDA accelerator errors propagate +immediately. The successful collection path and historical profile values are +unchanged by this failure guard. diff --git a/python/aisimulate/collector/op_backend_facts.yaml b/python/aisimulate/collector/op_backend_facts.yaml index 3e9e812a5..0b61a9df5 100644 --- a/python/aisimulate/collector/op_backend_facts.yaml +++ b/python/aisimulate/collector/op_backend_facts.yaml @@ -175,7 +175,7 @@ ops: - {framework: "trtllm", version: "1.0.0", system: "l40s", op_name: "mla_context", kv_cache_dtype: "bfloat16", mla_dtype: "bfloat16", backends: ["trtllm_internal"], kernel_sources: ["default"]} - {framework: "trtllm", version: "1.0.0", system: "l40s", op_name: "mla_context", kv_cache_dtype: "fp8", mla_dtype: "bfloat16", backends: ["trtllm_internal"], kernel_sources: ["default"]} - {framework: "trtllm", version: "1.3.0rc20", system: "b200_sxm", op_name: "mla_context", kv_cache_dtype: "bfloat16", mla_dtype: "bfloat16", backends: ["trtllm_internal"], kernel_sources: ["default"]} - - {framework: "trtllm", version: "1.3.0rc20", system: "b200_sxm", op_name: "mla_context", kv_cache_dtype: "fp8", mla_dtype: "bfloat16", backends: ["trtllm_internal"], kernel_sources: ["default"]} + - {framework: "trtllm", version: "1.3.0rc20", system: "b200_sxm", op_name: "mla_context", kv_cache_dtype: "fp8", mla_dtype: "fp8", backends: ["trtllm_internal"], kernel_sources: ["default"]} - {framework: "trtllm", version: "1.3.0rc20", system: "b300_sxm", op_name: "mla_context", kv_cache_dtype: "bfloat16", mla_dtype: "bfloat16", backends: ["trtllm_internal"], kernel_sources: ["default"]} - {framework: "trtllm", version: "1.3.0rc20", system: "b300_sxm", op_name: "mla_context", kv_cache_dtype: "fp8", mla_dtype: "bfloat16", backends: ["trtllm_internal"], kernel_sources: ["default"]} - {framework: "trtllm", version: "1.3.0rc20", system: "gb200", op_name: "mla_context", kv_cache_dtype: "bfloat16", mla_dtype: "bfloat16", backends: ["trtllm_internal"], kernel_sources: ["default"]} @@ -185,7 +185,7 @@ ops: - {framework: "trtllm", version: "1.3.0rc20", system: "h100_sxm", op_name: "mla_context", kv_cache_dtype: "bfloat16", mla_dtype: "bfloat16", backends: ["trtllm_internal"], kernel_sources: ["default"]} - {framework: "trtllm", version: "1.3.0rc20", system: "h100_sxm", op_name: "mla_context", kv_cache_dtype: "fp8", mla_dtype: "bfloat16", backends: ["trtllm_internal"], kernel_sources: ["default"]} - {framework: "trtllm", version: "1.3.0rc20", system: "h200_sxm", op_name: "mla_context", kv_cache_dtype: "bfloat16", mla_dtype: "bfloat16", backends: ["trtllm_internal"], kernel_sources: ["default"]} - - {framework: "trtllm", version: "1.3.0rc20", system: "h200_sxm", op_name: "mla_context", kv_cache_dtype: "fp8", mla_dtype: "bfloat16", backends: ["trtllm_internal"], kernel_sources: ["default"]} + - {framework: "trtllm", version: "1.3.0rc20", system: "h200_sxm", op_name: "mla_context", kv_cache_dtype: "fp8", mla_dtype: "fp8", backends: ["trtllm_internal"], kernel_sources: ["default"]} - {framework: "trtllm", version: "1.3.0rc20", system: "rtx_pro_6000_server", op_name: "mla_context", kv_cache_dtype: "bfloat16", mla_dtype: "bfloat16", backends: ["trtllm_internal"], kernel_sources: ["default"]} - {framework: "trtllm", version: "1.3.0rc20", system: "rtx_pro_6000_server", op_name: "mla_context", kv_cache_dtype: "fp8", mla_dtype: "bfloat16", backends: ["trtllm_internal"], kernel_sources: ["default"]} - op_file: custom_allreduce_perf diff --git a/python/aisimulate/collector/trtllm/collect_mla.py b/python/aisimulate/collector/trtllm/collect_mla.py index 93ad1f4df..a7faa6105 100644 --- a/python/aisimulate/collector/trtllm/collect_mla.py +++ b/python/aisimulate/collector/trtllm/collect_mla.py @@ -651,7 +651,13 @@ def kernel_func(): log_perf( item_list=[ { - "mla_dtype": "bfloat16", + # TRT-LLM quantizes context Q/K/V internally for FP8 KV on + # SM90/100/103/120. The BF16 Python inputs do not describe + # the FMHA compute precision (attentionOp.cpp:1228-1231 at + # c25c23f71786bad54d192893d696ce8043426eca / v1.3.0rc20). + "mla_dtype": ( + dtype_str if is_context_phase and get_sm_version() in (90, 100, 103, 120) else "bfloat16" + ), "kv_cache_dtype": dtype_str, "num_heads": num_heads, "batch_size": len(context_sequence_lengths), diff --git a/python/aisimulate/collector/trtllm/collect_moe.py b/python/aisimulate/collector/trtllm/collect_moe.py index ce0dc354c..4021cad9d 100755 --- a/python/aisimulate/collector/trtllm/collect_moe.py +++ b/python/aisimulate/collector/trtllm/collect_moe.py @@ -357,6 +357,7 @@ def run_moe_torch( model_config.mapping = mapping model_config.quant_config = quant_config model_config.moe_max_num_tokens = num_tokens_lists[-1] # to avoid multi-chunk auxi stream in cuda-graph mode. + is_gpt_oss = model_name in {"openai/gpt-oss-120b", "openai/gpt-oss-20b"} swiglu_alpha = None swiglu_beta = None swiglu_limit = None @@ -417,7 +418,13 @@ def run_moe_torch( "w4afp8": 4, "nvfp4": 4, }[moe_type] - if (inter_size // moe_tp_size) % (256 // _weight_bits) != 0: + # Blackwell MXFP4 uses TRTLLMGenFusedMoE, whose weight loader pads + # before TP sharding. Its logical intermediate size need not satisfy + # the CUTLASS plugin's physical-weight alignment check below. + # TensorRT-LLM v1.3.0rc20, c25c23f71786bad54d192893d696ce8043426eca: + # _torch/modules/fused_moe/quantization.py:5634-5659,5705-5735. + native_mxfp4_padding = moe_type in _MXFP4_MOE_TYPES and 100 <= sm_version < 120 + if not native_mxfp4_padding and (inter_size // moe_tp_size) % (256 // _weight_bits) != 0: raise ValueError( f"TRT-LLM fused MoE requires the TP-sharded intermediate size to be a multiple " f"of 256/weight_bits = {256 // _weight_bits} for {moe_type} (TLLM_CHECK_WITH_INFO " @@ -425,7 +432,7 @@ def run_moe_torch( f"{inter_size // moe_tp_size}" ) - if model_name in ["openai/gpt-oss-120b", "openai/gpt-oss-20b"]: + if is_gpt_oss: swiglu_alpha = torch.tensor([1.702] * (num_experts // moe_ep_size), dtype=torch.float32).to( torch.device(device) ) @@ -499,7 +506,14 @@ def run_moe_torch( router_logits_dtype = torch.float32 else: # for low latency mode in fp4, experts > 128 is not supported. - routing_method = RenormalizeMoeRoutingMethod(topk) + # Native GPT-OSS uses BF16 routing weights for TRTLLM, FP32 otherwise: + # modeling_gpt_oss.py:159-162@1.3.0rc14 (93cb6518), :165-168@1.3.0rc20. + if is_gpt_oss: + routing_method = RenormalizeMoeRoutingMethod( + topk, output_dtype=torch.bfloat16 if model_config.moe_backend == "trtllm" else torch.float32 + ) + else: + routing_method = RenormalizeMoeRoutingMethod(topk) create_moe_kwargs = { "routing_method": routing_method, @@ -507,6 +521,9 @@ def run_moe_torch( "hidden_size": hidden_size, "intermediate_size": inter_size, "dtype": dtype, + # Native GPT-OSS expert construction requests bias=True: + # modeling_gpt_oss.py:183@1.3.0rc14 (93cb6518), :189@1.3.0rc20. + "bias": is_gpt_oss, # In both low latency and attention dp scenarios, create_moe needs not to do allreduce # inside op. "reduce_results": False, @@ -634,7 +651,11 @@ def fp32_to_mxfp4(tensor): else: continue - if moe_type != "w4a16_mxfp4": + # TRTLLMGen MXFP4 must warm its tactic cache just like serving: + # model_engine.py:827-842@1.3.0rc14 (93cb6518). A B200 GPT-OSS + # TP1/256-token graph measurement drops 26% after explicit tuning. Keep the + # separate Triton/CUTLASS MXFP4 tuning behavior unchanged. + if moe_type != "w4a16_mxfp4" or model_config.moe_backend == "trtllm": cleanup_empty_json_files(moe_tune_path) # The tuned-tactic cache MUST be SM-scoped: tactic indices are positions # in the runner's per-arch config table, and replaying another arch's @@ -648,6 +669,7 @@ def fp32_to_mxfp4(tensor): cache_path = ( f"{moe_tune_path}/sm{get_sm_version()}_" f"{moe_type}_{hidden_size}_{inter_size // moe_tp_size}_{num_experts // moe_ep_size}" + f"{'_gptoss_bias' if is_gpt_oss else ''}" ) existing_files = glob.glob(f"{cache_path}*") cache_loaded = False @@ -666,6 +688,8 @@ def fp32_to_mxfp4(tensor): if not cache_loaded: torch.cuda.synchronize() + tuning_succeeded = False + last_tuning_error = None for i in range(len(num_tokens_lists)): max_tokens_for_tuning = num_tokens_lists[-i - 1] if max_tokens_for_tuning > max_tokens: @@ -688,9 +712,16 @@ def fp32_to_mxfp4(tensor): do_finalize=not min_latency_mode, ) torch.cuda.synchronize() + tuning_succeeded = True except Exception as e: - print(f"tune failed for {max_tokens_for_tuning} tokens: {e}, fallback to samller tokens") + accelerator_error = getattr(torch, "AcceleratorError", None) + if accelerator_error is not None and isinstance(e, accelerator_error): + raise + last_tuning_error = e + print(f"tune failed for {max_tokens_for_tuning} tokens: {e}, trying smaller tokens") continue + if not tuning_succeeded: + raise RuntimeError("MoE autotuning did not succeed for any eligible token count") from last_tuning_error del hidden_states_max_tokens, logits_max_tokens if moe_type == "fp8_block": diff --git a/python/aisimulate/collector/trtllm/gym-gpt-eager-20260918.json b/python/aisimulate/collector/trtllm/gym-gpt-eager-20260918.json new file mode 100644 index 000000000..8602d1be5 --- /dev/null +++ b/python/aisimulate/collector/trtllm/gym-gpt-eager-20260918.json @@ -0,0 +1,1585 @@ +{ + "status": "diagnosis complete; user deferred execution-timing contract extension; prediction unchanged", + "baseline": { + "historical_ttft_ms": 48.31286990083754, + "reproduced_unprofiled_ttft_ms": 48.85651273, + "prediction_ttft_ms": 16.532643949999834, + "prediction_tpot_ms": 2.9374422182467463 + }, + "jobs": [ + { + "id": 1996517, + "gpus": 2, + "state": "COMPLETED", + "exit_code": "0:0", + "elapsed": "00:03:06" + }, + { + "id": 1996606, + "gpus": 2, + "state": "COMPLETED", + "exit_code": "0:0", + "elapsed": "00:04:14" + }, + { + "id": 1996733, + "gpus": 2, + "state": "COMPLETED", + "exit_code": "0:0", + "elapsed": "00:08:22" + } + ], + "phases": [ + { + "phase": "trace", + "repetitions": [ + { + "repeat": 1, + "completed": 40, + "ttft_ms": 57.456108421320096, + "tpot_ms": 3.0571908764123377 + } + ], + "completed": 40, + "mean_ttft_ms": 57.456108421320096, + "mean_tpot_ms": 3.0571908764123377, + "workers": [], + "hosts": [] + }, + { + "phase": "timing", + "repetitions": [ + { + "repeat": 1, + "completed": 40, + "ttft_ms": 55.703431530855596, + "tpot_ms": 2.8425575887027246 + }, + { + "repeat": 2, + "completed": 40, + "ttft_ms": 55.445059249177575, + "tpot_ms": 2.83089806728799 + }, + { + "repeat": 3, + "completed": 40, + "ttft_ms": 54.03229470830411, + "tpot_ms": 2.8369985792960226 + } + ], + "completed": 120, + "mean_ttft_ms": 55.06026182944576, + "mean_tpot_ms": 2.8368180784289123, + "workers": [ + { + "artifact": "timing/cpu-modules-3551295.jsonl", + "measured_prefill_steps": 114, + "means_ms": { + "forward": 31.116310271929823, + "layer": 24.388748763157896, + "attention": 8.716003614035088, + "mlp": 9.319741078947368, + "allreduce": 5.0653127719298245, + "outside_layers": 6.727561508771928 + } + }, + { + "artifact": "timing/cpu-modules-3551296.jsonl", + "measured_prefill_steps": 114, + "means_ms": { + "forward": 30.631993552631577, + "layer": 23.740045035087718, + "attention": 8.502180763157895, + "mlp": 9.137787596491227, + "allreduce": 4.854846622807018, + "outside_layers": 6.89194851754386 + } + } + ], + "hosts": [] + }, + { + "phase": "cpu-control", + "repetitions": [ + { + "repeat": 1, + "completed": 40, + "ttft_ms": 54.13114909315482, + "tpot_ms": 2.852289204539715 + }, + { + "repeat": 2, + "completed": 40, + "ttft_ms": 52.38005965366028, + "tpot_ms": 2.867921953242021 + }, + { + "repeat": 3, + "completed": 40, + "ttft_ms": 52.93121309368871, + "tpot_ms": 2.8404801422428325 + } + ], + "completed": 120, + "mean_ttft_ms": 53.1474739468346, + "mean_tpot_ms": 2.853563766674856, + "workers": [ + { + "artifact": "cpu-control/cpu-modules-3562462.jsonl", + "measured_prefill_steps": 114, + "means_ms": { + "forward": 30.040869157894736 + } + }, + { + "artifact": "cpu-control/cpu-modules-3562463.jsonl", + "measured_prefill_steps": 114, + "means_ms": { + "forward": 30.324480438596492 + } + } + ], + "hosts": [ + { + "artifact": "cpu-control/host-3562194.json", + "cpu_model": "INTEL(R) XEON(R) PLATINUM 8568Y+", + "cpu_affinity": [ + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 108, + 109, + 110, + 111, + 112, + 113, + 114, + 115 + ], + "module_timing_enabled": false + }, + { + "artifact": "cpu-control/host-3562462.json", + "cpu_model": "INTEL(R) XEON(R) PLATINUM 8568Y+", + "cpu_affinity": [ + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 108, + 109, + 110, + 111, + 112, + 113, + 114, + 115 + ], + "module_timing_enabled": false + }, + { + "artifact": "cpu-control/host-3562463.json", + "cpu_model": "INTEL(R) XEON(R) PLATINUM 8568Y+", + "cpu_affinity": [ + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 108, + 109, + 110, + 111, + 112, + 113, + 114, + 115 + ], + "module_timing_enabled": false + } + ] + }, + { + "phase": "cpu-timed", + "repetitions": [ + { + "repeat": 1, + "completed": 40, + "ttft_ms": 58.457596582593396, + "tpot_ms": 2.846298165209989 + }, + { + "repeat": 2, + "completed": 40, + "ttft_ms": 55.93435300397687, + "tpot_ms": 2.839051412946568 + }, + { + "repeat": 3, + "completed": 40, + "ttft_ms": 54.336668900214136, + "tpot_ms": 2.8414355380074627 + } + ], + "completed": 120, + "mean_ttft_ms": 56.242872828928135, + "mean_tpot_ms": 2.8422617053880064, + "workers": [ + { + "artifact": "cpu-timed/cpu-modules-3571721.jsonl", + "measured_prefill_steps": 114, + "means_ms": { + "forward": 30.842750157894738, + "layer": 24.171250377192983, + "attention": 8.663844456140351, + "mlp": 9.244890728070175, + "allreduce": 4.958306245614035, + "outside_layers": 6.671499780701755 + } + }, + { + "artifact": "cpu-timed/cpu-modules-3571722.jsonl", + "measured_prefill_steps": 114, + "means_ms": { + "forward": 31.287979280701755, + "layer": 24.43763206140351, + "attention": 8.77970399122807, + "mlp": 9.398532245614035, + "allreduce": 4.974686684210527, + "outside_layers": 6.8503472192982455 + } + } + ], + "hosts": [ + { + "artifact": "cpu-timed/host-3571224.json", + "cpu_model": "INTEL(R) XEON(R) PLATINUM 8568Y+", + "cpu_affinity": [ + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 108, + 109, + 110, + 111, + 112, + 113, + 114, + 115 + ], + "module_timing_enabled": true + }, + { + "artifact": "cpu-timed/host-3571721.json", + "cpu_model": "INTEL(R) XEON(R) PLATINUM 8568Y+", + "cpu_affinity": [ + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 108, + 109, + 110, + 111, + 112, + 113, + 114, + 115 + ], + "module_timing_enabled": true + }, + { + "artifact": "cpu-timed/host-3571722.json", + "cpu_model": "INTEL(R) XEON(R) PLATINUM 8568Y+", + "cpu_affinity": [ + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 108, + 109, + 110, + 111, + 112, + 113, + 114, + 115 + ], + "module_timing_enabled": true + } + ] + } + ], + "trace": [ + { + "artifact": "trace/trace-rank-0.json.gz", + "event_count": 324598, + "unmatched_gpu": { + "gpu_memcpy": 1007, + "gpu_memset": 201, + "kernel": 3023 + }, + "mixed_steps": [ + { + "name": "aisim/forward/step=2538/ctx=1/gen=3", + "cpu_ms": 50.314893, + "gpu_start_us": 6744262561548.644, + "gpu_end_us": 6744262612077.941, + "gpu_span_ms": 50.5292978515625, + "gpu_busy_union_ms": 33.424748046875, + "gpu_gap_ms": 17.104549804687494, + "gap_before_launch_ms": 16.364059570312495, + "gpu_event_count": 634, + "categories_sum_ms": { + "other": 0.622081, + "communication_norm": 23.547012000000006, + "norm_position_cache": 0.3757430000000001, + "gemm": 9.692959999999996, + "attention": 0.7190729999999999, + "moe": 7.491711999999998 + }, + "kernel_totals": { + "void tensorrt_llm::_v1::kernels::ar_fusion::allreduce_fusion_kernel_oneshot_lamport<(tensorrt_llm::_v1::kernels::ar_fusion::AllReduceFusionPattern)1, __nv_bfloat16, 2, false, false>(tensorrt_llm::_v1::kernels::ar_fusion::AllReduceFusionParams)": [ + 72, + 22.825669000000005 + ], + "nvjet_sm100_tst_64x16_64x16_2x4_2cta_h_bz_bias_TNT": [ + 36, + 8.485624999999999 + ], + "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_bA32_bB32_bC32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_16dp256b_rM_TN_transOut_schPd2x1x2x3_biasM_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f": [ + 36, + 4.4734989999999994 + ], + "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_bA32_bB32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_16dp256b_rM_TN_transOut_schPd2x1x2x3_biasM_bN_rgTma_clmp_dynB_sm100f": [ + 36, + 2.2680299999999995 + ], + "void tensorrt_llm::_v1::kernels::ar_fusion::allreduce_fusion_kernel_oneshot_lamport<(tensorrt_llm::_v1::kernels::ar_fusion::AllReduceFusionPattern)0, __nv_bfloat16, 2, false, true>(tensorrt_llm::_v1::kernels::ar_fusion::AllReduceFusionParams)": [ + 1, + 0.721343 + ], + "nvjet_sm100_tst_128x128_64x8_2x1_2cta_v_bz_bias_TNT": [ + 36, + 0.644259 + ], + "nvjet_sm100_tst_144x128_64x6_2x1_v_bz_bias_TNN": [ + 36, + 0.46870799999999996 + ], + "void moe::dev::finalize::finalizeKernelVecLoad >(moe::dev::finalize::KernelParams)": [ + 36, + 0.39458999999999994 + ], + "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext": [ + 18, + 0.275742 + ], + "void tensorrt_llm::_v1::kernels::quantize_with_block_size<(tensorrt_llm::_v1::BlockScaleQuantizationType)2, __nv_bfloat16, 32, true>(int, int, int, int, __nv_bfloat16 const*, float const*, unsigned int*, unsigned int*, tensorrt_llm::_v1::QuantizationSFLayout)": [ + 36, + 0.25689700000000004 + ], + "void tensorrt_llm::_v1::kernels::applyBiasRopeUpdateKVCacheV2<__nv_bfloat16, __nv_fp8_e4m3, 256, 64, false, true, false, false, tensorrt_llm::_v1::kernels::KVBlockArray, (tensorrt_llm::_v1::kernels::RotaryPositionEmbeddingType)2>(tensorrt_llm::_v1::kernels::QKVPreprocessingParams<__nv_bfloat16, tensorrt_llm::_v1::kernels::KVBlockArray>)": [ + 36, + 0.243297 + ], + "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext": [ + 18, + 0.20163299999999995 + ], + "void moe::dev::routing::routingIndicesCoopKernel > >(moe::dev::routing::routingCustom::KernelParams<__nv_bfloat16, __nv_bfloat16, 128, 8, moe::dev::routing::TopKExpertSelect >)": [ + 36, + 0.193317 + ], + "void tensorrt_llm::_v1::kernels::computeSeqAndPaddingOffsets<__nv_bfloat16, 256>(tensorrt_llm::_v1::kernels::BuildDecoderInfoParams<__nv_bfloat16>)": [ + 36, + 0.18979199999999996 + ], + "ncclDevKernel_AllGather_RING_LL(ncclDevKernelArgsStorage<4096ul>)": [ + 1, + 0.14144 + ], + "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen": [ + 18, + 0.12409799999999999 + ], + "void tensorrt_llm::_v1::kernels::applyBiasRopeUpdateKVCacheV2<__nv_bfloat16, __nv_fp8_e4m3, 256, 64, false, false, true, true, tensorrt_llm::_v1::kernels::KVBlockArray, (tensorrt_llm::_v1::kernels::RotaryPositionEmbeddingType)2>(tensorrt_llm::_v1::kernels::QKVPreprocessingParams<__nv_bfloat16, tensorrt_llm::_v1::kernels::KVBlockArray>)": [ + 36, + 0.12399899999999998 + ], + "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen": [ + 18, + 0.11760000000000001 + ], + "void moe::dev::routing::routingCustom::routingIndicesHistogramScoresKernel > >(moe::dev::routing::routingCustom::KernelParams<__nv_bfloat16, __nv_bfloat16, 128, 4, moe::dev::routing::TopKExpertSelect >)": [ + 36, + 0.115619 + ], + "nvjet_sm100_tst_64x8_64x16_2x1_v_bz_TNT": [ + 1, + 0.094368 + ], + "void moe::dev::routing::routingInitExpertCounts > >(moe::dev::routing::routingCustom::KernelParams<__nv_bfloat16, __nv_bfloat16, 128, 8, moe::dev::routing::TopKExpertSelect >)": [ + 36, + 0.046657 + ], + "Memcpy HtoD (Pinned -> Device)": [ + 7, + 0.009248 + ], + "kernel_cutlass_kernel_flashinfernormkernelsrmsnormRMSNormKernel_object_at__tensorptrbf16gmemalign128o288028801_tensorptrbf16gmemalign16o28801_tensorptrbf16gmemalign128o288028801___True_4__0": [ + 1, + 0.008447 + ], + "void at::native::unrolled_elementwise_kernel, 4, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1> >(int, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#4}::operator()() const::{lambda(long)#1}, std::array, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1>)": [ + 2, + 0.00448 + ], + "void at::native::unrolled_elementwise_kernel, 4, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1> >(int, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#7}::operator()() const::{lambda(float)#1}, std::array, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1>)": [ + 1, + 0.003904 + ], + "triton_poi_fused_bitwise_and_bitwise_not_embedding_ge_lt_masked_fill_mul_sub_unsqueeze_0": [ + 1, + 0.0037440000000000004 + ], + "void at::native::index_elementwise_kernel<128, 4, at::native::gpu_index_kernel >(at::TensorIteratorBase&, c10::ArrayRef, c10::ArrayRef)::{lambda(char*, char const*, long)#1}>(at::TensorIteratorBase&, c10::ArrayRef, c10::ArrayRef, at::native::index_kernel_impl >(at::TensorIteratorBase&, c10::ArrayRef, c10::ArrayRef)::{lambda(char*, char const*, long)#1} const&, bool)::{lambda(int)#1}>(long, at::native::gpu_index_kernel >(at::TensorIteratorBase&, c10::ArrayRef, c10::ArrayRef)::{lambda(char*, char const*, long)#1}>(at::TensorIteratorBase&, c10::ArrayRef, c10::ArrayRef, at::native::index_kernel_impl >(at::TensorIteratorBase&, c10::ArrayRef, c10::ArrayRef)::{lambda(char*, char const*, long)#1} const&, bool)::{lambda(int)#1})": [ + 1, + 0.002592 + ], + "void at::native::(anonymous namespace)::CatArrayBatchedCopy_vectorized, unsigned int, 2, 128, 1, 16, 8>(char*, at::native::(anonymous namespace)::CatArrInputTensorMetadata, unsigned int, 128, 1>, at::native::(anonymous namespace)::TensorSizeStride, int, unsigned int)": [ + 1, + 0.002048 + ], + "void at_cuda_detail::cub::detail::scan::DeviceScanKernel >::Policy1000, long const*, long*, at_cuda_detail::cub::ScanTileState, std::plus, at_cuda_detail::cub::NullType, unsigned int, long, false, at_cuda_detail::cub::NullType>(long const*, long*, at_cuda_detail::cub::ScanTileState, int, std::plus, at_cuda_detail::cub::NullType, unsigned int)": [ + 1, + 0.001824 + ], + "Memcpy DtoD (Device -> Device)": [ + 1, + 0.001696 + ], + "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)": [ + 1, + 0.001632 + ], + "void at_cuda_detail::cub::detail::scan::DeviceScanInitKernel >(at_cuda_detail::cub::ScanTileState, int)": [ + 1, + 0.001472 + ], + "void at::native::vectorized_elementwise_kernel<2, at::native::CUDAFunctorOnSelf_add, std::array >(int, at::native::CUDAFunctorOnSelf_add, std::array)": [ + 1, + 0.001312 + ] + }, + "moe_layers": {} + }, + { + "name": "aisim/forward/step=2560/ctx=1/gen=3", + "cpu_ms": 41.14012, + "gpu_start_us": 6744262684606.129, + "gpu_end_us": 6744262725705.803, + "gpu_span_ms": 41.099673828125, + "gpu_busy_union_ms": 35.3283759765625, + "gpu_gap_ms": 5.771297851562501, + "gap_before_launch_ms": 5.4986318359375, + "gpu_event_count": 634, + "categories_sum_ms": { + "other": 0.611132, + "communication_norm": 24.870439, + "norm_position_cache": 0.370051, + "gemm": 13.571389999999996, + "attention": 0.7571469999999999, + "moe": 8.040154999999997 + }, + "kernel_totals": { + "void tensorrt_llm::_v1::kernels::ar_fusion::allreduce_fusion_kernel_oneshot_lamport<(tensorrt_llm::_v1::kernels::ar_fusion::AllReduceFusionPattern)1, __nv_bfloat16, 2, false, false>(tensorrt_llm::_v1::kernels::ar_fusion::AllReduceFusionParams)": [ + 72, + 24.149192 + ], + "nvjet_sm100_tst_64x16_64x16_2x2_2cta_h_bz_bias_TNT": [ + 36, + 12.227608999999996 + ], + "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_bA32_bB32_bC32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_16dp256b_rM_TN_transOut_schPd2x1x2x3_biasM_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f": [ + 36, + 4.839612 + ], + "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_bA32_bB32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_16dp256b_rM_TN_transOut_schPd2x1x2x3_biasM_bN_rgTma_clmp_dynB_sm100f": [ + 36, + 2.4438389999999997 + ], + "nvjet_sm100_tst_144x128_64x8_1x2_2cta_h_bz_bias_TNN": [ + 36, + 0.7453120000000001 + ], + "void tensorrt_llm::_v1::kernels::ar_fusion::allreduce_fusion_kernel_oneshot_lamport<(tensorrt_llm::_v1::kernels::ar_fusion::AllReduceFusionPattern)0, __nv_bfloat16, 2, false, true>(tensorrt_llm::_v1::kernels::ar_fusion::AllReduceFusionParams)": [ + 1, + 0.721247 + ], + "nvjet_sm100_tst_176x128_64x8_1x2_2cta_h_bz_bias_TNN": [ + 36, + 0.5037170000000001 + ], + "void moe::dev::finalize::finalizeKernelVecLoad >(moe::dev::finalize::KernelParams)": [ + 36, + 0.39149 + ], + "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext": [ + 18, + 0.315805 + ], + "void tensorrt_llm::_v1::kernels::quantize_with_block_size<(tensorrt_llm::_v1::BlockScaleQuantizationType)2, __nv_bfloat16, 32, true>(int, int, int, int, __nv_bfloat16 const*, float const*, unsigned int*, unsigned int*, tensorrt_llm::_v1::QuantizationSFLayout)": [ + 36, + 0.26425299999999996 + ], + "void tensorrt_llm::_v1::kernels::applyBiasRopeUpdateKVCacheV2<__nv_bfloat16, __nv_fp8_e4m3, 256, 64, false, true, false, false, tensorrt_llm::_v1::kernels::KVBlockArray, (tensorrt_llm::_v1::kernels::RotaryPositionEmbeddingType)2>(tensorrt_llm::_v1::kernels::QKVPreprocessingParams<__nv_bfloat16, tensorrt_llm::_v1::kernels::KVBlockArray>)": [ + 36, + 0.25129700000000005 + ], + "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext": [ + 18, + 0.200863 + ], + "void moe::dev::routing::routingIndicesCoopKernel > >(moe::dev::routing::routingCustom::KernelParams<__nv_bfloat16, __nv_bfloat16, 128, 8, moe::dev::routing::TopKExpertSelect >)": [ + 36, + 0.19468800000000003 + ], + "void tensorrt_llm::_v1::kernels::computeSeqAndPaddingOffsets<__nv_bfloat16, 256>(tensorrt_llm::_v1::kernels::BuildDecoderInfoParams<__nv_bfloat16>)": [ + 36, + 0.19152 + ], + "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen": [ + 18, + 0.12659099999999998 + ], + "void moe::dev::routing::routingCustom::routingIndicesHistogramScoresKernel > >(moe::dev::routing::routingCustom::KernelParams<__nv_bfloat16, __nv_bfloat16, 128, 4, moe::dev::routing::TopKExpertSelect >)": [ + 36, + 0.124482 + ], + "ncclDevKernel_AllGather_RING_LL(ncclDevKernelArgsStorage<4096ul>)": [ + 1, + 0.11584 + ], + "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen": [ + 18, + 0.11388799999999999 + ], + "void tensorrt_llm::_v1::kernels::applyBiasRopeUpdateKVCacheV2<__nv_bfloat16, __nv_fp8_e4m3, 256, 64, false, false, true, true, tensorrt_llm::_v1::kernels::KVBlockArray, (tensorrt_llm::_v1::kernels::RotaryPositionEmbeddingType)2>(tensorrt_llm::_v1::kernels::QKVPreprocessingParams<__nv_bfloat16, tensorrt_llm::_v1::kernels::KVBlockArray>)": [ + 36, + 0.11334500000000003 + ], + "nvjet_sm100_tst_64x8_64x16_2x1_v_bz_TNT": [ + 1, + 0.09475199999999999 + ], + "void moe::dev::routing::routingInitExpertCounts > >(moe::dev::routing::routingCustom::KernelParams<__nv_bfloat16, __nv_bfloat16, 128, 8, moe::dev::routing::TopKExpertSelect >)": [ + 36, + 0.046044000000000015 + ], + "Memcpy HtoD (Pinned -> Device)": [ + 7, + 0.01408 + ], + "kernel_cutlass_kernel_flashinfernormkernelsrmsnormRMSNormKernel_object_at__tensorptrbf16gmemalign128o288028801_tensorptrbf16gmemalign16o28801_tensorptrbf16gmemalign128o288028801___True_4__0": [ + 1, + 0.005409 + ], + "void at::native::unrolled_elementwise_kernel, 4, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1> >(int, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#4}::operator()() const::{lambda(long)#1}, std::array, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1>)": [ + 2, + 0.005152 + ], + "void at::native::unrolled_elementwise_kernel, 4, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1> >(int, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#7}::operator()() const::{lambda(float)#1}, std::array, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1>)": [ + 1, + 0.004 + ], + "triton_poi_fused_bitwise_and_bitwise_not_embedding_ge_lt_masked_fill_mul_sub_unsqueeze_0": [ + 1, + 0.003487 + ], + "void at::native::index_elementwise_kernel<128, 4, at::native::gpu_index_kernel >(at::TensorIteratorBase&, c10::ArrayRef, c10::ArrayRef)::{lambda(char*, char const*, long)#1}>(at::TensorIteratorBase&, c10::ArrayRef, c10::ArrayRef, at::native::index_kernel_impl >(at::TensorIteratorBase&, c10::ArrayRef, c10::ArrayRef)::{lambda(char*, char const*, long)#1} const&, bool)::{lambda(int)#1}>(long, at::native::gpu_index_kernel >(at::TensorIteratorBase&, c10::ArrayRef, c10::ArrayRef)::{lambda(char*, char const*, long)#1}>(at::TensorIteratorBase&, c10::ArrayRef, c10::ArrayRef, at::native::index_kernel_impl >(at::TensorIteratorBase&, c10::ArrayRef, c10::ArrayRef)::{lambda(char*, char const*, long)#1} const&, bool)::{lambda(int)#1})": [ + 1, + 0.0026880000000000003 + ], + "void at::native::(anonymous namespace)::CatArrayBatchedCopy_vectorized, unsigned int, 2, 128, 1, 16, 8>(char*, at::native::(anonymous namespace)::CatArrInputTensorMetadata, unsigned int, 128, 1>, at::native::(anonymous namespace)::TensorSizeStride, int, unsigned int)": [ + 1, + 0.0022080000000000003 + ], + "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)": [ + 1, + 0.001824 + ], + "void at_cuda_detail::cub::detail::scan::DeviceScanKernel >::Policy1000, long const*, long*, at_cuda_detail::cub::ScanTileState, std::plus, at_cuda_detail::cub::NullType, unsigned int, long, false, at_cuda_detail::cub::NullType>(long const*, long*, at_cuda_detail::cub::ScanTileState, int, std::plus, at_cuda_detail::cub::NullType, unsigned int)": [ + 1, + 0.001728 + ], + "void at::native::vectorized_elementwise_kernel<2, at::native::CUDAFunctorOnSelf_add, std::array >(int, at::native::CUDAFunctorOnSelf_add, std::array)": [ + 1, + 0.001728 + ], + "Memcpy DtoD (Device -> Device)": [ + 1, + 0.001536 + ], + "void at_cuda_detail::cub::detail::scan::DeviceScanInitKernel >(at_cuda_detail::cub::ScanTileState, int)": [ + 1, + 0.001088 + ] + }, + "moe_layers": {} + } + ], + "decode_steps": 199, + "decode_means_ms": { + "cpu_ms": 1.2318665628140704, + "gpu_span_ms": 2.893328704067211, + "gpu_busy_union_ms": 2.639171914258794, + "gpu_gap_ms": 0.2541567898084171, + "gap_before_launch_ms": 0.03394866421953518 + } + }, + { + "artifact": "trace/trace-rank-1.json.gz", + "event_count": 322204, + "unmatched_gpu": { + "kernel": 3023, + "gpu_memcpy": 1007, + "gpu_memset": 201 + }, + "mixed_steps": [ + { + "name": "aisim/forward/step=2538/ctx=1/gen=3", + "cpu_ms": 50.860194, + "gpu_start_us": 6744262561558.279, + "gpu_end_us": 6744262612155.47, + "gpu_span_ms": 50.5971904296875, + "gpu_busy_union_ms": 11.4433994140625, + "gpu_gap_ms": 39.153791015625, + "gap_before_launch_ms": 35.765282226562505, + "gpu_event_count": 634, + "categories_sum_ms": { + "other": 0.40637100000000015, + "communication_norm": 1.6008019999999998, + "norm_position_cache": 0.360059, + "gemm": 1.253286, + "attention": 0.7209949999999999, + "moe": 7.483073 + }, + "kernel_totals": { + "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_bA32_bB32_bC32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_16dp256b_rM_TN_transOut_schPd2x1x2x3_biasM_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f": [ + 36, + 4.447520000000001 + ], + "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_bA32_bB32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_16dp256b_rM_TN_transOut_schPd2x1x2x3_biasM_bN_rgTma_clmp_dynB_sm100f": [ + 36, + 2.287711 + ], + "void tensorrt_llm::_v1::kernels::ar_fusion::allreduce_fusion_kernel_oneshot_lamport<(tensorrt_llm::_v1::kernels::ar_fusion::AllReduceFusionPattern)1, __nv_bfloat16, 2, false, false>(tensorrt_llm::_v1::kernels::ar_fusion::AllReduceFusionParams)": [ + 72, + 1.584514 + ], + "nvjet_sm100_tst_128x128_64x8_2x1_2cta_v_bz_bias_TNT": [ + 36, + 0.5270710000000001 + ], + "nvjet_sm100_tst_144x128_64x6_2x1_v_bz_TNN": [ + 36, + 0.420035 + ], + "void moe::dev::finalize::finalizeKernelVecLoad >(moe::dev::finalize::KernelParams)": [ + 36, + 0.3993289999999999 + ], + "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext": [ + 18, + 0.27913899999999997 + ], + "void tensorrt_llm::_v1::kernels::applyBiasRopeUpdateKVCacheV2<__nv_bfloat16, __nv_fp8_e4m3, 256, 64, false, true, false, false, tensorrt_llm::_v1::kernels::KVBlockArray, (tensorrt_llm::_v1::kernels::RotaryPositionEmbeddingType)2>(tensorrt_llm::_v1::kernels::QKVPreprocessingParams<__nv_bfloat16, tensorrt_llm::_v1::kernels::KVBlockArray>)": [ + 36, + 0.243165 + ], + "nvjet_sm100_tst_64x16_64x16_2x4_2cta_h_bz_bias_TNT": [ + 36, + 0.21277200000000002 + ], + "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext": [ + 18, + 0.20223899999999997 + ], + "void moe::dev::routing::routingIndicesCoopKernel > >(moe::dev::routing::routingCustom::KernelParams<__nv_bfloat16, __nv_bfloat16, 128, 8, moe::dev::routing::TopKExpertSelect >)": [ + 36, + 0.19766799999999998 + ], + "void tensorrt_llm::_v1::kernels::computeSeqAndPaddingOffsets<__nv_bfloat16, 256>(tensorrt_llm::_v1::kernels::BuildDecoderInfoParams<__nv_bfloat16>)": [ + 36, + 0.18159800000000004 + ], + "void tensorrt_llm::_v1::kernels::quantize_with_block_size<(tensorrt_llm::_v1::BlockScaleQuantizationType)2, __nv_bfloat16, 32, true>(int, int, int, int, __nv_bfloat16 const*, float const*, unsigned int*, unsigned int*, tensorrt_llm::_v1::QuantizationSFLayout)": [ + 36, + 0.17424099999999992 + ], + "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen": [ + 18, + 0.12265700000000002 + ], + "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen": [ + 18, + 0.11696 + ], + "void tensorrt_llm::_v1::kernels::applyBiasRopeUpdateKVCacheV2<__nv_bfloat16, __nv_fp8_e4m3, 256, 64, false, false, true, true, tensorrt_llm::_v1::kernels::KVBlockArray, (tensorrt_llm::_v1::kernels::RotaryPositionEmbeddingType)2>(tensorrt_llm::_v1::kernels::QKVPreprocessingParams<__nv_bfloat16, tensorrt_llm::_v1::kernels::KVBlockArray>)": [ + 36, + 0.11334199999999998 + ], + "void moe::dev::routing::routingCustom::routingIndicesHistogramScoresKernel > >(moe::dev::routing::routingCustom::KernelParams<__nv_bfloat16, __nv_bfloat16, 128, 4, moe::dev::routing::TopKExpertSelect >)": [ + 36, + 0.09475100000000004 + ], + "nvjet_sm100_tst_64x8_64x16_2x1_v_bz_TNT": [ + 1, + 0.093408 + ], + "void moe::dev::routing::routingInitExpertCounts > >(moe::dev::routing::routingCustom::KernelParams<__nv_bfloat16, __nv_bfloat16, 128, 8, moe::dev::routing::TopKExpertSelect >)": [ + 36, + 0.056094 + ], + "ncclDevKernel_AllGather_RING_LL(ncclDevKernelArgsStorage<4096ul>)": [ + 1, + 0.017889 + ], + "void tensorrt_llm::_v1::kernels::ar_fusion::allreduce_fusion_kernel_oneshot_lamport<(tensorrt_llm::_v1::kernels::ar_fusion::AllReduceFusionPattern)0, __nv_bfloat16, 2, false, true>(tensorrt_llm::_v1::kernels::ar_fusion::AllReduceFusionParams)": [ + 1, + 0.016288 + ], + "Memcpy HtoD (Pinned -> Device)": [ + 7, + 0.007489 + ], + "void at::native::unrolled_elementwise_kernel, 4, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1> >(int, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#4}::operator()() const::{lambda(long)#1}, std::array, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1>)": [ + 2, + 0.005024000000000001 + ], + "void at::native::unrolled_elementwise_kernel, 4, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1> >(int, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#7}::operator()() const::{lambda(float)#1}, std::array, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1>)": [ + 1, + 0.003968 + ], + "kernel_cutlass_kernel_flashinfernormkernelsrmsnormRMSNormKernel_object_at__tensorptrbf16gmemalign128o288028801_tensorptrbf16gmemalign16o28801_tensorptrbf16gmemalign128o288028801___True_4__0": [ + 1, + 0.003552 + ], + "triton_poi_fused_bitwise_and_bitwise_not_embedding_ge_lt_masked_fill_mul_sub_unsqueeze_0": [ + 1, + 0.003264 + ], + "void at::native::index_elementwise_kernel<128, 4, at::native::gpu_index_kernel >(at::TensorIteratorBase&, c10::ArrayRef, c10::ArrayRef)::{lambda(char*, char const*, long)#1}>(at::TensorIteratorBase&, c10::ArrayRef, c10::ArrayRef, at::native::index_kernel_impl >(at::TensorIteratorBase&, c10::ArrayRef, c10::ArrayRef)::{lambda(char*, char const*, long)#1} const&, bool)::{lambda(int)#1}>(long, at::native::gpu_index_kernel >(at::TensorIteratorBase&, c10::ArrayRef, c10::ArrayRef)::{lambda(char*, char const*, long)#1}>(at::TensorIteratorBase&, c10::ArrayRef, c10::ArrayRef, at::native::index_kernel_impl >(at::TensorIteratorBase&, c10::ArrayRef, c10::ArrayRef)::{lambda(char*, char const*, long)#1} const&, bool)::{lambda(int)#1})": [ + 1, + 0.002944 + ], + "void at::native::(anonymous namespace)::CatArrayBatchedCopy_vectorized, unsigned int, 2, 128, 1, 16, 8>(char*, at::native::(anonymous namespace)::CatArrInputTensorMetadata, unsigned int, 128, 1>, at::native::(anonymous namespace)::TensorSizeStride, int, unsigned int)": [ + 1, + 0.002176 + ], + "Memcpy DtoD (Device -> Device)": [ + 1, + 0.00176 + ], + "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)": [ + 1, + 0.00176 + ], + "void at_cuda_detail::cub::detail::scan::DeviceScanKernel >::Policy1000, long const*, long*, at_cuda_detail::cub::ScanTileState, std::plus, at_cuda_detail::cub::NullType, unsigned int, long, false, at_cuda_detail::cub::NullType>(long const*, long*, at_cuda_detail::cub::ScanTileState, int, std::plus, at_cuda_detail::cub::NullType, unsigned int)": [ + 1, + 0.001729 + ], + "void at::native::vectorized_elementwise_kernel<2, at::native::CUDAFunctorOnSelf_add, std::array >(int, at::native::CUDAFunctorOnSelf_add, std::array)": [ + 1, + 0.001409 + ], + "void at_cuda_detail::cub::detail::scan::DeviceScanInitKernel >(at_cuda_detail::cub::ScanTileState, int)": [ + 1, + 0.0011200000000000001 + ] + }, + "moe_layers": {} + }, + { + "name": "aisim/forward/step=2560/ctx=1/gen=3", + "cpu_ms": 41.504732000000004, + "gpu_start_us": 6744262684609.494, + "gpu_end_us": 6744262725737.914, + "gpu_span_ms": 41.128419921875, + "gpu_busy_union_ms": 12.30607421875, + "gpu_gap_ms": 28.822345703125002, + "gap_before_launch_ms": 25.671572265624988, + "gpu_event_count": 634, + "categories_sum_ms": { + "other": 0.420545, + "communication_norm": 1.7716469999999995, + "norm_position_cache": 0.3652139999999999, + "gemm": 1.4237140000000004, + "attention": 0.7612479999999998, + "moe": 8.062020999999998 + }, + "kernel_totals": { + "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_bA32_bB32_bC32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_16dp256b_rM_TN_transOut_schPd2x1x2x3_biasM_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f": [ + 36, + 4.832739 + ], + "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_bA32_bB32_t128x32x256u2_s5_et128x32_m128x32x32_c1x1x1_16dp256b_rM_TN_transOut_schPd2x1x2x3_biasM_bN_rgTma_clmp_dynB_sm100f": [ + 36, + 2.4801609999999994 + ], + "void tensorrt_llm::_v1::kernels::ar_fusion::allreduce_fusion_kernel_oneshot_lamport<(tensorrt_llm::_v1::kernels::ar_fusion::AllReduceFusionPattern)1, __nv_bfloat16, 2, false, false>(tensorrt_llm::_v1::kernels::ar_fusion::AllReduceFusionParams)": [ + 72, + 1.7547509999999995 + ], + "nvjet_sm100_tst_144x128_64x8_1x2_2cta_h_bz_bias_TNN": [ + 36, + 0.685954 + ], + "nvjet_sm100_tst_176x128_64x8_1x2_2cta_h_bz_TNN": [ + 36, + 0.428448 + ], + "void moe::dev::finalize::finalizeKernelVecLoad >(moe::dev::finalize::KernelParams)": [ + 36, + 0.40204600000000007 + ], + "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvCausalVarSeqQ128Kv128StaticContext": [ + 18, + 0.31923300000000004 + ], + "void tensorrt_llm::_v1::kernels::applyBiasRopeUpdateKVCacheV2<__nv_bfloat16, __nv_fp8_e4m3, 256, 64, false, true, false, false, tensorrt_llm::_v1::kernels::KVBlockArray, (tensorrt_llm::_v1::kernels::RotaryPositionEmbeddingType)2>(tensorrt_llm::_v1::kernels::QKVPreprocessingParams<__nv_bfloat16, tensorrt_llm::_v1::kernels::KVBlockArray>)": [ + 36, + 0.249087 + ], + "nvjet_sm100_tst_64x16_64x16_2x2_2cta_h_bz_bias_TNT": [ + 36, + 0.21414500000000003 + ], + "fmhaSm100fKernel_QkvBfloat16OBfloat16H64PackedQkvSlidingOrChunkedCausalVarSeqQ128Kv128StaticContext": [ + 18, + 0.20217600000000002 + ], + "void moe::dev::routing::routingIndicesCoopKernel > >(moe::dev::routing::routingCustom::KernelParams<__nv_bfloat16, __nv_bfloat16, 128, 8, moe::dev::routing::TopKExpertSelect >)": [ + 36, + 0.19632299999999997 + ], + "void tensorrt_llm::_v1::kernels::quantize_with_block_size<(tensorrt_llm::_v1::BlockScaleQuantizationType)2, __nv_bfloat16, 32, true>(int, int, int, int, __nv_bfloat16 const*, float const*, unsigned int*, unsigned int*, tensorrt_llm::_v1::QuantizationSFLayout)": [ + 36, + 0.1860820000000001 + ], + "void tensorrt_llm::_v1::kernels::computeSeqAndPaddingOffsets<__nv_bfloat16, 256>(tensorrt_llm::_v1::kernels::BuildDecoderInfoParams<__nv_bfloat16>)": [ + 36, + 0.179489 + ], + "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen": [ + 18, + 0.123103 + ], + "fmhaSm100fKernel_QkvE4m3OBfloat16H64PagedKvSlidingOrChunkedCausalP32MultiCtasKvCgaVarSeqQ8Kv128StaticSwapsAbForGen": [ + 18, + 0.116736 + ], + "void tensorrt_llm::_v1::kernels::applyBiasRopeUpdateKVCacheV2<__nv_bfloat16, __nv_fp8_e4m3, 256, 64, false, false, true, true, tensorrt_llm::_v1::kernels::KVBlockArray, (tensorrt_llm::_v1::kernels::RotaryPositionEmbeddingType)2>(tensorrt_llm::_v1::kernels::QKVPreprocessingParams<__nv_bfloat16, tensorrt_llm::_v1::kernels::KVBlockArray>)": [ + 36, + 0.11238300000000001 + ], + "void moe::dev::routing::routingCustom::routingIndicesHistogramScoresKernel > >(moe::dev::routing::routingCustom::KernelParams<__nv_bfloat16, __nv_bfloat16, 128, 4, moe::dev::routing::TopKExpertSelect >)": [ + 36, + 0.09516700000000003 + ], + "nvjet_sm100_tst_64x8_64x16_2x1_v_bz_TNT": [ + 1, + 0.095167 + ], + "void moe::dev::routing::routingInitExpertCounts > >(moe::dev::routing::routingCustom::KernelParams<__nv_bfloat16, __nv_bfloat16, 128, 8, moe::dev::routing::TopKExpertSelect >)": [ + 36, + 0.055585 + ], + "ncclDevKernel_AllGather_RING_LL(ncclDevKernelArgsStorage<4096ul>)": [ + 1, + 0.017728 + ], + "void tensorrt_llm::_v1::kernels::ar_fusion::allreduce_fusion_kernel_oneshot_lamport<(tensorrt_llm::_v1::kernels::ar_fusion::AllReduceFusionPattern)0, __nv_bfloat16, 2, false, true>(tensorrt_llm::_v1::kernels::ar_fusion::AllReduceFusionParams)": [ + 1, + 0.016896 + ], + "Memcpy HtoD (Pinned -> Device)": [ + 7, + 0.010781 + ], + "void at::native::unrolled_elementwise_kernel, 4, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1> >(int, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#4}::operator()() const::{lambda(long)#1}, std::array, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1>)": [ + 2, + 0.004320999999999999 + ], + "triton_poi_fused_bitwise_and_bitwise_not_embedding_ge_lt_masked_fill_mul_sub_unsqueeze_0": [ + 1, + 0.004 + ], + "void at::native::unrolled_elementwise_kernel, 4, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1> >(int, at::native::direct_copy_kernel_cuda(at::TensorIteratorBase&)::{lambda()#3}::operator()() const::{lambda()#7}::operator()() const::{lambda(float)#1}, std::array, TrivialOffsetCalculator<1, unsigned int>, TrivialOffsetCalculator<1, unsigned int>, at::native::memory::LoadWithCast<1>, at::native::memory::StoreWithCast<1>)": [ + 1, + 0.003904 + ], + "kernel_cutlass_kernel_flashinfernormkernelsrmsnormRMSNormKernel_object_at__tensorptrbf16gmemalign128o288028801_tensorptrbf16gmemalign16o28801_tensorptrbf16gmemalign128o288028801___True_4__0": [ + 1, + 0.0037440000000000004 + ], + "void at::native::index_elementwise_kernel<128, 4, at::native::gpu_index_kernel >(at::TensorIteratorBase&, c10::ArrayRef, c10::ArrayRef)::{lambda(char*, char const*, long)#1}>(at::TensorIteratorBase&, c10::ArrayRef, c10::ArrayRef, at::native::index_kernel_impl >(at::TensorIteratorBase&, c10::ArrayRef, c10::ArrayRef)::{lambda(char*, char const*, long)#1} const&, bool)::{lambda(int)#1}>(long, at::native::gpu_index_kernel >(at::TensorIteratorBase&, c10::ArrayRef, c10::ArrayRef)::{lambda(char*, char const*, long)#1}>(at::TensorIteratorBase&, c10::ArrayRef, c10::ArrayRef, at::native::index_kernel_impl >(at::TensorIteratorBase&, c10::ArrayRef, c10::ArrayRef)::{lambda(char*, char const*, long)#1} const&, bool)::{lambda(int)#1})": [ + 1, + 0.002976 + ], + "void at::native::(anonymous namespace)::CatArrayBatchedCopy_vectorized, unsigned int, 2, 128, 1, 16, 8>(char*, at::native::(anonymous namespace)::CatArrInputTensorMetadata, unsigned int, 128, 1>, at::native::(anonymous namespace)::TensorSizeStride, int, unsigned int)": [ + 1, + 0.002145 + ], + "void at_cuda_detail::cub::detail::scan::DeviceScanKernel >::Policy1000, long const*, long*, at_cuda_detail::cub::ScanTileState, std::plus, at_cuda_detail::cub::NullType, unsigned int, long, false, at_cuda_detail::cub::NullType>(long const*, long*, at_cuda_detail::cub::ScanTileState, int, std::plus, at_cuda_detail::cub::NullType, unsigned int)": [ + 1, + 0.002144 + ], + "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)": [ + 1, + 0.001951 + ], + "void at::native::vectorized_elementwise_kernel<2, at::native::CUDAFunctorOnSelf_add, std::array >(int, at::native::CUDAFunctorOnSelf_add, std::array)": [ + 1, + 0.0019199999999999998 + ], + "Memcpy DtoD (Device -> Device)": [ + 1, + 0.0015680000000000002 + ], + "void at_cuda_detail::cub::detail::scan::DeviceScanInitKernel >(at_cuda_detail::cub::ScanTileState, int)": [ + 1, + 0.001536 + ] + }, + "moe_layers": {} + } + ], + "decode_steps": 199, + "decode_means_ms": { + "cpu_ms": 1.2240419095477386, + "gpu_span_ms": 2.8902369170069093, + "gpu_busy_union_ms": 2.699371407820352, + "gpu_gap_ms": 0.19086550918655779, + "gap_before_launch_ms": 0.026161962154522614 + } + } + ], + "predicted_ops": [ + { + "context_tokens": 886, + "generation_requests": 3, + "isl": 886, + "osl": 850, + "total_ms": 12.569321151963898, + "ops_ms": { + "context_moe": 8.281386309117078, + "context_attention": 1.4687863615766688, + "context_moe_pre_dispatch": 0.4992717421101406, + "context_moe_post_dispatch": 0.4992717421101406, + "context_qkv_gemm": 0.3999140340945586, + "generation_attention": 0.3985324492823566, + "context_proj_gemm": 0.36459115037068723, + "context_add_norm_1": 0.22770327272727275, + "context_add_norm_2": 0.22770327272727275, + "context_router_gemm": 0.1844608856173904, + "context_embedding_ar": 0.01386865950305946, + "context_embedding": 0.0038312727272727274, + "context_p2p": 0.0 + }, + "raw": [ + [ + [ + "context_embedding", + 0.0038312727272727274, + 0.0, + "empirical", + null + ], + [ + "context_add_norm_1", + 0.22770327272727275, + 0.0, + "empirical", + null + ], + [ + "context_qkv_gemm", + 0.3999140340945586, + 0.0, + "silicon", + null + ], + [ + "context_proj_gemm", + 0.36459115037068723, + 0.0, + "silicon", + null + ], + [ + "context_add_norm_2", + 0.22770327272727275, + 0.0, + "empirical", + null + ], + [ + "context_router_gemm", + 0.1844608856173904, + 0.0, + "silicon", + null + ], + [ + "context_moe_pre_dispatch", + 0.4992717421101406, + 0.0, + "silicon", + null + ], + [ + "context_moe", + 8.281386309117078, + 0.0, + "silicon", + null + ], + [ + "context_moe_post_dispatch", + 0.4992717421101406, + 0.0, + "silicon", + null + ], + [ + "context_embedding_ar", + 0.01386865950305946, + 0.0, + "silicon", + null + ], + [ + "context_p2p", + 0.0, + 0.0, + "silicon", + null + ] + ], + [ + [ + "context_attention", + 1.4687863615766688, + 0.0, + "mixed", + null + ] + ], + [ + [ + "generation_attention", + 0.3985324492823566, + 0.0, + "silicon", + null + ] + ] + ] + }, + { + "context_tokens": 991, + "generation_requests": 0, + "isl": 991, + "osl": 1, + "total_ms": 12.830901337743699, + "ops_ms": { + "context_moe": 8.719455910474062, + "context_attention": 1.51736749920396, + "context_moe_pre_dispatch": 0.5440059555647894, + "context_moe_post_dispatch": 0.5440059555647894, + "context_qkv_gemm": 0.426964864461677, + "context_proj_gemm": 0.3927846909542423, + "context_add_norm_1": 0.24143750649350648, + "context_add_norm_2": 0.24143750649350648, + "context_router_gemm": 0.18440352263904916, + "context_embedding_ar": 0.015111276543466372, + "context_embedding": 0.003926649350649351, + "context_p2p": 0.0 + }, + "raw": [ + [ + [ + "context_embedding", + 0.003926649350649351, + 0.0, + "empirical", + null + ], + [ + "context_add_norm_1", + 0.24143750649350648, + 0.0, + "empirical", + null + ], + [ + "context_qkv_gemm", + 0.426964864461677, + 0.0, + "silicon", + null + ], + [ + "context_proj_gemm", + 0.3927846909542423, + 0.0, + "silicon", + null + ], + [ + "context_add_norm_2", + 0.24143750649350648, + 0.0, + "empirical", + null + ], + [ + "context_router_gemm", + 0.18440352263904916, + 0.0, + "silicon", + null + ], + [ + "context_moe_pre_dispatch", + 0.5440059555647894, + 0.0, + "silicon", + null + ], + [ + "context_moe", + 8.719455910474062, + 0.0, + "silicon", + null + ], + [ + "context_moe_post_dispatch", + 0.5440059555647894, + 0.0, + "silicon", + null + ], + [ + "context_embedding_ar", + 0.015111276543466372, + 0.0, + "silicon", + null + ], + [ + "context_p2p", + 0.0, + 0.0, + "silicon", + null + ] + ], + [ + [ + "context_attention", + 1.51736749920396, + 0.0, + "mixed", + null + ] + ], + [] + ] + }, + { + "context_tokens": 3804, + "generation_requests": 0, + "isl": 951, + "osl": 1, + "total_ms": 24.634349117429082, + "ops_ms": { + "context_moe": 13.710336443781852, + "context_attention": 2.790955593475339, + "context_moe_pre_dispatch": 2.0610443465196298, + "context_moe_post_dispatch": 2.0610443465196298, + "context_qkv_gemm": 1.3275430550126748, + "context_proj_gemm": 1.179878829847318, + "context_add_norm_1": 0.6202061298701299, + "context_add_norm_2": 0.6202061298701299, + "context_router_gemm": 0.1993260236716228, + "context_embedding_ar": 0.05725123184776749, + "context_embedding": 0.006556987012987014, + "context_p2p": 0.0 + }, + "raw": [ + [ + [ + "context_embedding", + 0.006556987012987014, + 0.0, + "empirical", + null + ], + [ + "context_add_norm_1", + 0.6202061298701299, + 0.0, + "empirical", + null + ], + [ + "context_qkv_gemm", + 1.3275430550126748, + 0.0, + "silicon", + null + ], + [ + "context_proj_gemm", + 1.179878829847318, + 0.0, + "silicon", + null + ], + [ + "context_add_norm_2", + 0.6202061298701299, + 0.0, + "empirical", + null + ], + [ + "context_router_gemm", + 0.1993260236716228, + 0.0, + "silicon", + null + ], + [ + "context_moe_pre_dispatch", + 2.0610443465196298, + 0.0, + "silicon", + null + ], + [ + "context_moe", + 13.710336443781852, + 0.0, + "silicon", + null + ], + [ + "context_moe_post_dispatch", + 2.0610443465196298, + 0.0, + "silicon", + null + ], + [ + "context_embedding_ar", + 0.05725123184776749, + 0.0, + "silicon", + null + ], + [ + "context_p2p", + 0.0, + 0.0, + "silicon", + null + ] + ], + [ + [ + "context_attention", + 2.790955593475339, + 0.0, + "mixed", + null + ] + ], + [] + ] + }, + { + "context_tokens": 8192, + "generation_requests": 3, + "isl": 8192, + "osl": 850, + "total_ms": 53.751729848343395, + "ops_ms": { + "context_moe": 26.788398417737337, + "context_attention": 10.15840175529491, + "context_moe_pre_dispatch": 4.117229697143776, + "context_moe_post_dispatch": 4.117229697143776, + "context_qkv_gemm": 2.920783864456497, + "context_proj_gemm": 2.371858125654241, + "context_add_norm_1": 1.2114514285714284, + "context_add_norm_2": 1.2114514285714284, + "generation_attention": 0.4701734224068787, + "context_router_gemm": 0.2597216626329351, + "context_embedding_ar": 0.11436749158732709, + "context_embedding": 0.010662857142857142, + "context_p2p": 0.0 + }, + "raw": [ + [ + [ + "context_embedding", + 0.010662857142857142, + 0.0, + "empirical", + null + ], + [ + "context_add_norm_1", + 1.2114514285714284, + 0.0, + "empirical", + null + ], + [ + "context_qkv_gemm", + 2.920783864456497, + 0.0, + "silicon", + null + ], + [ + "context_proj_gemm", + 2.371858125654241, + 0.0, + "silicon", + null + ], + [ + "context_add_norm_2", + 1.2114514285714284, + 0.0, + "empirical", + null + ], + [ + "context_router_gemm", + 0.2597216626329351, + 0.0, + "silicon", + null + ], + [ + "context_moe_pre_dispatch", + 4.117229697143776, + 0.0, + "silicon", + null + ], + [ + "context_moe", + 26.788398417737337, + 0.0, + "silicon", + null + ], + [ + "context_moe_post_dispatch", + 4.117229697143776, + 0.0, + "silicon", + null + ], + [ + "context_embedding_ar", + 0.11436749158732709, + 0.0, + "silicon", + null + ], + [ + "context_p2p", + 0.0, + 0.0, + "silicon", + null + ] + ], + [ + [ + "context_attention", + 10.15840175529491, + 0.0, + "mixed", + null + ] + ], + [ + [ + "generation_attention", + 0.4701734224068787, + 0.0, + "silicon", + null + ] + ] + ] + } + ], + "source": { + "trtllm": "93cb6518b6d6dbd6095748189e626db731f44545", + "image_child_digest": "sha256:fe2f17d0c9698bafeb9f437929003c6badc56d17e8382a02505233145a07f61f", + "model_revision": "b5c939de8f754692c1647ca79fbf85e8c1e70f8a", + "inferencex": "2baba8e27be8529b4453afc953f9859ec092c73d", + "aisimulate": "8bcc5d154f5c53ddc982747748eb87459e7c1d40", + "native_runtime_source": "89d2051137b772944a3e303c452108811790bef4", + "native_runtime_sha256": "7c0ccae3937ef48224f1a35cbab04735ea13914a395bc0853435c63425ef62b8", + "moe_data_sha256": "db1930fde20b419284bf1f038fadd4b398d29701b12867fd4ac9da878808e781", + "query_version": "1.3.0rc20" + }, + "artifact_sha256": { + "analyze_trace.py": "018f60bc192428e1028b465f347bd8b9751888cc1b17aad764728c7f26757e7f", + "build_report.py": "d5a9225a4cad4c16acb2e8b190846d98623e346e28c7eabbfa2c8fe2fb2c22b1", + "control-inner.sh": "4580666e70c414ec3e35b543d9f0cb24b0d5cca62b31411c8ef9ec6e12895c5e", + "control.sbatch": "c19953fdea24781eddfaa697d36ab2db180edd383514a99745bc8b25e58c47d8", + "cpu-control/client-1.log": "487034b12de45ead011f436a4bd0c635feca9af2aa12c8abe81b182a426d0ec3", + "cpu-control/client-2.log": "b39feefea86ce3d6c78ad06a9abdeee1baa74bb004a470a920f20046a1ab307d", + "cpu-control/client-3.log": "59aaa66c3259cdb4416c905793de84bef6cc0ff7766f677a6eaa8a3b0ea04a61", + "cpu-control/cpu-modules-3562462.jsonl": "b0c2c064095cd7d162ccdc501d2d2a13113b45993e0acbd149c45483a986523f", + "cpu-control/cpu-modules-3562463.jsonl": "c7a01722b87905c36f0f73bf1dcc9f25621b3a19be08c2ceec03d18c1e2d1c95", + "cpu-control/gpu-before.txt": "d4cf1b4e15eb621a4d63f801212c552cf82a092c57af2bf14841bfcaf66b27de", + "cpu-control/host-3562194.json": "5eb8833cb2fb2bfa42208b96f96d600b11e835a75fa805254dbeec1bc388d071", + "cpu-control/host-3562462.json": "1a4257774b3aaa0ffeb8aa653b30fe4402d6eb6397c92917141c4337f4c6e50a", + "cpu-control/host-3562463.json": "7b18bdc9f00a6dba56c0e63ab1956d87d441c3b263f766e862f726e8f05fc54a", + "cpu-control/metrics-1.json": "ac10e7810fe4e6d159ac27667ddde30dafeb684a94aebe437cff5f2021c7b7ae", + "cpu-control/metrics-2.json": "0a3d1b0887ed323ff6823f24b15925ea7a824a1d63d9da7e74fa377b3ecc32a6", + "cpu-control/metrics-3.json": "f90a386a09ef8b008006cd1077e2b5d7fc1b7613fc2e9e0d528143a8c1a0dff8", + "cpu-control/metrics-before-1.json": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945", + "cpu-control/metrics-before-2.json": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945", + "cpu-control/metrics-before-3.json": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945", + "cpu-control/options.yaml": "6f7f204bdb3eafcfa7583d29a44721c430b1021f1f9059e4e449569912baa82f", + "cpu-control/result-1.json": "2d1638374f0c80b255cf14e32101ba0b59054e9e1120db8ca7a251fa2bb47740", + "cpu-control/result-2.json": "8c4dd8d17be42d1d0fc4e879672a19483660cee06181a4cf1f9f450e09f90b8c", + "cpu-control/result-3.json": "cd1fc33086e6494a8c4c916e21e7867b1f340cd7522277ca0ac6ab0c66183d22", + "cpu-control/runtime.json": "870e0ebe2f083bf2fc1656672411f422199a92456e80460480532e825c24cc42", + "cpu-control/server.log": "562190621da1290dd5e49058dcc70c6a7a06a2e6af5fd56f9c239b10a6d2d0d5", + "cpu-control/telemetry.csv": "3c23085bdfb0d30dd7c6f59f2c29e5ba84a792234c8a3c811ccd0b9c8801096f", + "cpu-timed/client-1.log": "a6c8f24eb2a0adbc7b14f5aede11313ceaac185a700c2c397c0dcf47feb8ec59", + "cpu-timed/client-2.log": "0fe51048ee4c89253c2934678eb687f36062cac408da8dd849ae031890e2155a", + "cpu-timed/client-3.log": "e7ee6f2bda77af3b00686fb420643960330e96f2e561010ade605a2db8ac0299", + "cpu-timed/cpu-modules-3571721.jsonl": "9759cc092a68fa746a6761c3b1e4ea05395e8430ce121aff77d8b91163a0ca96", + "cpu-timed/cpu-modules-3571722.jsonl": "f19be2ffe525775f0092d1750da97ae8e338a2f487919be7273c50edc55d8177", + "cpu-timed/gpu-before.txt": "b6a0c40babefd6dd79eb141395114baf3e5eaa8534ae90d4f5b7a4a2ee581928", + "cpu-timed/host-3571224.json": "c25b82235754fc8c5633aae95f854368e2097a80efc5aa62db393e1cfd654a20", + "cpu-timed/host-3571721.json": "720dd475f3937f56f13cbcae3a5acfee18ca1a4cf95c700478684c31f98f4503", + "cpu-timed/host-3571722.json": "127c2705b708cb6b32af8a554a92e8247c2c9916996068b4934ce1036a3fbcd6", + "cpu-timed/metrics-1.json": "3ea3fdda5fbf34fd14a7c361785bfe9c6bfbce96e501c7c477f8c5f40fb23fab", + "cpu-timed/metrics-2.json": "0afef56065637f70d3f25b837c5e2308d647f5e484ed1334f4f1a8a5bcc80969", + "cpu-timed/metrics-3.json": "fed1263d642c0c1f4a2e7742e286a842b5693cb98b9d304365d3fbf58cf83b5f", + "cpu-timed/metrics-before-1.json": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945", + "cpu-timed/metrics-before-2.json": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945", + "cpu-timed/metrics-before-3.json": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945", + "cpu-timed/options.yaml": "6f7f204bdb3eafcfa7583d29a44721c430b1021f1f9059e4e449569912baa82f", + "cpu-timed/result-1.json": "318211a51c31c839878a654f70b9e31a2117ad9bd8d0637c2574d2314b2cb3fe", + "cpu-timed/result-2.json": "b5614bdae67c97fbf1cb87d423c2bc4f98002e316d2e504262f6eaa834a85947", + "cpu-timed/result-3.json": "8555658fd0b2a2fb3acaf470c7e72281b87a76a290244108733eb66c4c0fb7f8", + "cpu-timed/runtime.json": "870e0ebe2f083bf2fc1656672411f422199a92456e80460480532e825c24cc42", + "cpu-timed/server.log": "e35423e8e07566a8e39c067f4d73e4e65d984b50febd947cfe0a8e8f62e367db", + "cpu-timed/telemetry.csv": "741c2d0c61182547fc0301000f3c3de631dc034262f5806893c80a5ff91f0a49", + "hooks/sitecustomize.py": "92f97e4e31a53438c0b8e76662769d7f38b3aa5e27205ffaee0558009b3e31b3", + "predicted-ops.json": "c1c43c91a22dfe1c6bf656114cc75459d59d219b7f2d5424bf284e14d1971e71", + "run.sbatch": "921cf2ebc75c013b9bc048a6123a570a0dfdb6ab90cea112dfa5996a4e08ae0f", + "slurm/accounting.txt": "58d016aadf99542d1eafae0b6add32baadea1882af51c223891326e2a77f4117", + "slurm/gpt-ttft-control-0918-1996733.err": "c87b36d7d22af3c1da0e1727fd4618aa144c0ede0b0e5848ef116f047cfeb0a3", + "slurm/gpt-ttft-control-0918-1996733.out": "7e9c24472fe9d263ec84e8f1d4cc22eb8de8b29aac5eff3188f6e021134bee2d", + "slurm/gpt-ttft-timing-0918-1996606.err": "fb6f17915ec613a0fed2992a0edcbf2a1815a2b511681587118ad299712b0376", + "slurm/gpt-ttft-timing-0918-1996606.out": "b1e3607483f4aaf6c12a4a6c11060a791d848ca3b140bb49a18226b8829fafb9", + "slurm/gpt-ttft-trace-0918-1996517.err": "6354e2776ae2b92c23c0286818d93d1e304ec46899a94287ad0b4aff31d84a0c", + "slurm/gpt-ttft-trace-0918-1996517.out": "5561642d752d1d7de28c33cbc38bbbc5c9db4ab0cb68e14ba400bf9c999b4246", + "target.json": "a23a02e7abdc8a5f1392b23cbbab68fc0fff2a57cf9f77b8371b385e9b961975", + "timing/client-1.log": "3a71f1f14546bc0b60b47ae725451a1d3bcbfce170fba3627c42258cddab12c1", + "timing/client-2.log": "0b4903de6db50db6c82470f5e67d45c31061f3d6eb81b93f4fc91cce3b74a380", + "timing/client-3.log": "a4b4a219a50b646702f961030d05f8ce5065d646024182f540ae9eb214cc343c", + "timing/cpu-modules-3551295.jsonl": "627f7e5f985202cba0437761aa9034d285ba64664b35e93f0016e666f0898bf3", + "timing/cpu-modules-3551296.jsonl": "43fe3351fdbcb74fe95ab12382271187f842e762b45ddff3cc6f3664bd7fd90a", + "timing/gpu-before.txt": "5cfc9cdcff421b1a1d75bbda4233bfd04cef6087fa62606e3eb33546f5366a7c", + "timing/metrics-1.json": "56bf3b4334e21dda68283a33cf14066ebaac350dfd744879043ecf4fac6369c9", + "timing/metrics-2.json": "4e45196f6987ba6f7896aac28673477dae678e7a7eb7f61f92de84e33e1366ab", + "timing/metrics-3.json": "abc62e75a4faa416e991687782e25eea18d882cbfdb9b260196d4b2cf0a6fbbc", + "timing/metrics-before-1.json": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945", + "timing/metrics-before-2.json": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945", + "timing/metrics-before-3.json": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945", + "timing/options.yaml": "6f7f204bdb3eafcfa7583d29a44721c430b1021f1f9059e4e449569912baa82f", + "timing/result-1.json": "b53e4563de0b4279684c839ec0799906c8b63e1dc7f4a8b94e22e9b75cbe2826", + "timing/result-2.json": "b81f0253c87312a3923bcca04da19405f4ac3c8a8c33e7bfee3266e19565a7ac", + "timing/result-3.json": "fafa3102dce5a92e455b5041c173ef256f33bcbdd5202cb939d9250b812b6655", + "timing/runtime.json": "870e0ebe2f083bf2fc1656672411f422199a92456e80460480532e825c24cc42", + "timing/server.log": "f59e50454cf363dc446bc4be669938d538d6b26aceec6a515d7b3bd337301cb7", + "timing/telemetry.csv": "fb403ae13ab9e238e047d870235e2a52a84fdda3c3ac871342e656453046a14c", + "timing-hooks/sitecustomize.py": "805c9daf52c298383a95c3f30f0a2e8b22942d742e17f1f1a1a5abe5b3981c52", + "timing-hooks-job1996606.py": "8e64485294ef54165454a0b8c95bb8f2055694915158915af9e1ed846f88cff1", + "timing-inner.sh": "2f764be80aeab42345303dcee191bc39a23d9174a43fd454a3c618260cc9622b", + "timing-summary.json": "7f23d6cfe3e79edb74b972da4e72874dde679c5ee122bd0ed4cc642d85935edd", + "timing.sbatch": "e4622e4fe26e587d76f586801956bfca6d2c0ab91a72d58ac95256f9d20c6c39", + "trace/batches-3786524.jsonl": "2078e2bc7f6359c80b1b5b46329a38884189801ea80006735245a4a216c891ff", + "trace/batches-3786525.jsonl": "b3ff4feaa9955578bec342f7c94500723f5a03e78d2630216d958ea256636b05", + "trace/client-1.log": "5700d16cfd54fb6826b17cc4686a6a318703e79db898e15b63e51755b72239f9", + "trace/gpu-before.txt": "6cc5a22782da9114992a4687f39b8a11d641f4f63285c69f7bc1bfb05ffded3d", + "trace/metrics-1.json": "701a38ee02e682716d4ec1a031c729cb8135428f04e8cb3c378c932125d45efb", + "trace/metrics-before-1.json": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945", + "trace/options.yaml": "6f7f204bdb3eafcfa7583d29a44721c430b1021f1f9059e4e449569912baa82f", + "trace/result-1.json": "1ad6f9b86916386625e2852703c88ffac7707eb0415b321ac3362e596d04fd1f", + "trace/runtime.json": "870e0ebe2f083bf2fc1656672411f422199a92456e80460480532e825c24cc42", + "trace/server.log": "384afd6db70ad66bf2b96a9716491374ee39cf0826bd7339e3309aadc20e6523", + "trace/telemetry.csv": "fdfcf024ba479b9ff0f11eacead054f4bae5521c73adf92d747e6fac2bf4b5fb", + "trace/trace-rank-0.json": "8f96e98d82ebaa0e98cad795a353f4986b17b46bf42d42fb3762ceca7196cdf3", + "trace/trace-rank-0.json.gz": "ed2ecda78266fb024902301423157b3130a6dde32ad1888920048a2cee5c9041", + "trace/trace-rank-1.json": "124fc5287cdca40ce8a870e5bf09d5564e3da34c88ab0d70f80d0700ab4e86ad", + "trace/trace-rank-1.json.gz": "a1cedb16dbd763a0d50eca0743e1e617034ef3abbc6c2f61dff9700daf67ccb7", + "trace-inner.sh": "5d7143e9f44e24a2b15e920b26252dbe97b671148d16042f4897fbaf3ca253ab", + "trace-summary.json": "ae70c52323374cd663b389c69d7cc3477ad55797daa28761f82c6f1f3de4c924" + }, + "limits": [ + "predicted_ops use TRT-LLM rc20 performance data, while the traced runtime is rc14; this is a cross-version diagnostic, not version-matched timing qualification.", + "Profiler and lightweight wrapper timings are diagnostic, not performance-database rows.", + "CPU module boundaries are nested and overlap GPU work. Collective waiting is not an additional independent penalty.", + "The 886-token probe matches MoE combined tokens but uses approximate decode KV; it is not a complete exact-forward query.", + "CPU control and module-timed phases run sequentially on the same allocation, so temporal drift remains possible.", + "Historical checkpoint/image/host identities were not all pinned; reproduction is not an exact historical environment." + ], + "cpu_row_detail": "Raw per-forward rows remain in the hashed CPU JSONL artifacts; this report retains aggregate boundaries." +} diff --git a/python/aisimulate/collector/trtllm/gym-gpt-eager-20260918.md b/python/aisimulate/collector/trtllm/gym-gpt-eager-20260918.md new file mode 100644 index 000000000..0ec6e4478 --- /dev/null +++ b/python/aisimulate/collector/trtllm/gym-gpt-eager-20260918.md @@ -0,0 +1,132 @@ +# GPT-OSS TTFT: eager submission and rank waiting + +## Status + +- The requested trace collection is complete. The main missing execution + boundary is supported by two-rank kernel traces and separate CPU timings. +- **The TTFT prediction is not fixed.** The frozen point still predicts + 16.53 ms versus the original unprofiled reproduction's 48.86 ms. + No kernel rows, latency constants, or 263-point predictions are changed. +- The proposed op-based correction needs an execution-mode/CPU-timing + data contract; see the [reviewable plan](../../../../docs/trtllm-eager-execution-plan.md). + The user chose to defer this extension and retain the collection evidence. + No producer/consumer implementation is approved. + +## Matched collection + +- GPT-OSS-120B, B200 TP2/EP1, ISL1024/OSL1024, concurrency4; NumPy seed0, + length ratio0.8, eight warmup and 40 measured requests per repetition. +- TensorRT-LLM PyTorch backend rc14, source + `93cb6518b6d6dbd6095748189e626db731f44545`; image child digest + `fe2f17d0c9698bafeb9f437929003c6badc56d17e8382a02505233145a07f61f`. + Model revision `b5c939de8f754692c1647ca79fbf85e8c1e70f8a`, InferenceX + client `2baba8e27be8529b4453afc953f9859ec092c73d`. +- Job 1996517 completed 0:0 in 3m06s on two B200s. Both Torch traces parse; + all 40 measured input/output token vectors match frozen replay point0028. + Capture covers 201 forwards per rank: two mixed and 199 decode steps. +- Profiling changes timing: client TTFT 57.46 ms, TPOT 3.0572 ms. This is + diagnostic evidence, not a replacement for the unprofiled E2E baseline. + +## GPU attribution + +Associate each CPU launch inside a forward range with GPU work by its CUDA +correlation ID. Use the projected GPU envelope for that forward, and union +GPU intervals within each rank. Do not select kernels merely because they +fall inside a CPU timestamp window. CPU and GPU annotation copies are +explicitly separated. Sampler/copy work outside the forward is retained as +unmatched evidence, not assigned to a transformer operation. + +| Mixed step | Rank | GPU span (ms) | GPU busy union (ms) | Gaps (ms) | Gap before next launch starts (ms) | MoE kernel sum (ms) | +|---|---:|---:|---:|---:|---:|---:| +| 886 prefill + 3 decode | 0 | 50.529 | 33.425 | 17.105 | 16.364 | 7.492 | +| 886 prefill + 3 decode | 1 | 50.597 | 11.443 | 39.154 | 35.765 | 7.483 | +| 1014 prefill + 3 decode | 0 | 41.100 | 35.328 | 5.771 | 5.499 | 8.040 | +| 1014 prefill + 3 decode | 1 | 41.128 | 12.306 | 28.822 | 25.672 | 8.062 | + +- The 886-token native op probe is 12.569 ms, including 8.281 ms of MoE. + This is a controlled shape probe; its decode KV is an approximation, not + a reconstructed exact scheduler query. The MoE coordinate is exact: + 889 combined tokens, TP2/EP1, MXFP4 weights and MXFP8 activation. + The trace uses TRT-LLM rc14, while the predicted operations use rc20 data; + this cross-version comparison does not qualify rc14 timing accuracy. +- MoE kernel sums are close to the corrected prediction. Increasing the + shared MoE graph table is not supported by this trace. +- Fused all-reduce/norm kernel sums differ sharply between ranks: + 23.547/1.601 ms for the first mixed step and 24.870/1.772 ms for the second. + Rank 0 also has elongated router GEMMs. Kernel durations include dependency + and peer waiting; these differences are not evidence for a 20x collective + throughput correction. Summing ranks or adding waiting to host gaps would + double count parts of the dependency path. +- The pinned [GPT-OSS TP block](https://github.com/NVIDIA/TensorRT-LLM/blob/93cb6518b6d6dbd6095748189e626db731f44545/tensorrt_llm/_torch/models/modeling_gpt_oss.py#L365) + fuses all-reduce, residual add and RMSNorm. Compare that combined boundary. +- Across 199 decode steps, mean GPU spans are 2.893/2.890 ms; gap unions are + 0.254/0.191 ms. Decode does not exhibit the mixed step's large eager gaps. + +## Lightweight CPU timing + +- Separate job 1996606 completed 0:0 in 4m14s. No Torch profiler or per-op + device synchronization is enabled. Wrappers record CPU call intervals for + the forward, attention, MLP, fused all-reduce and decoder layer. +- All 120 measured requests across three repetitions match replay token + vectors. Native request timestamps exclude warmup and identify 114 + measured prefill/mixed forwards in each worker. + +| CPU boundary | Worker A mean (ms) | Worker B mean (ms) | +|---|---:|---:| +| Complete model-engine forward call | 31.116 | 30.632 | +| 36 decoder-layer calls | 24.389 | 23.740 | +| Attention calls | 8.716 | 8.502 | +| MLP calls | 9.320 | 9.138 | +| All-reduce calls, including embedding | 5.065 | 4.855 | +| Work outside decoder-layer calls | 6.728 | 6.892 | + +- These are nested boundaries. Layer totals contain their attention, MLP + and all-reduce calls; CPU time overlaps GPU work. Neither columns nor + CPU/GPU totals may be added as independent latency penalties. +- Client TTFT 55.06 ms and TPOT 2.8368 ms remain diagnostic. Worker labels + follow the ordered artifact names in the JSON; they do not assert MPI rank. +- The first trace's custom MoE CPU wrapper produced no ranges in this runtime. + MoE classification therefore uses the observed routing/quantization/expert + GEMM/finalize kernel names and verifies 36 layer occurrences. The separate + lightweight MLP wrappers do execute, with 36 calls per measured forward. + +## Same-allocation control + +- Job 1996733 completed 0:0 in 8m22s, with two successive server launches + on the same two-GPU allocation. Both retain the whole-forward CPU timer + and native request metrics; the first disables only module timing. +- Both phases complete 120/120 measured requests with identical replay token + vectors. Each worker has 114 measured prefill/mixed forwards. Match CPU + records to the post-startup native iteration sequence, assert every batch + state agrees, and select the native measured requests' `first_iter` values. + This independently agrees with the timestamp-based warmup exclusion. + +| Observation | Module timing off | Module timing on | +|---|---:|---:| +| Client mean TTFT (ms) | 53.1475 | 56.2429 | +| Client mean TPOT (ms) | 2.8536 | 2.8423 | +| Worker A mean forward CPU call (ms) | 30.0409 | 30.8428 | +| Worker B mean forward CPU call (ms) | 30.3245 | 31.2880 | + +- The module-timed phase has 5.82% higher TTFT. Sequential phases can also + have temporal drift; this is an observed difference, not a causal estimate + of instrumentation overhead alone. Even without module timing, CPU + submission remains about 30 ms. None of these values is used as a fitted + latency penalty or published kernel measurement. +- Host metadata records Intel Xeon Platinum 8568Y+ and 16 allowed logical + CPUs in every server/worker process, in both phases. The launcher did not + confine the two workers to one CPU. CPU clocks and the historical host + environment remain uncontrolled. +- Total new collection: 400/400 measured requests across the trace, + lightweight timing and paired control jobs. All three jobs completed 0:0. + +## Evidence and remaining work + +- Raw traces, scripts, logs and native metrics: + `${CAMPAIGN_ARTIFACT_ROOT}/gpt-ttft-fix-20260918/` and the same relative + directory under `${B200_ARTIFACT_ROOT}/results/trt-62-20260917/`. +- [Compact measurements and hashes](gym-gpt-eager-20260918.json). +- Freeze the new measurement contract before formal eager collection. + Qualify CPU/device composition on independent shapes and complete ranks, + then rerun GPT-OSS and all shared-code controls. Queue/frontend costs + identified in the [earlier report](gym-residual-20260918.md) remain separate. diff --git a/python/aisimulate/collector/trtllm/gym-profiles-20260917-collection.json b/python/aisimulate/collector/trtllm/gym-profiles-20260917-collection.json new file mode 100644 index 000000000..1fa301d59 --- /dev/null +++ b/python/aisimulate/collector/trtllm/gym-profiles-20260917-collection.json @@ -0,0 +1,636 @@ +{ + "collector_commit": "ce9a396642674e8008fda2b778580dc8d9e310b4", + "trtllm_version": "1.3.0rc20", + "latency_unit": "milliseconds", + "upstream_commit": "c25c23f71786bad54d192893d696ce8043426eca", + "image": "nvcr.io/nvidia/tensorrt-llm/release@sha256:9b3b4dfb811caa9420fa99a6f958155f6a1f727ffc2b5a5c2d9d2ce51fdc323d", + "squashfs_sha256": "5aaf1c87eeea80fe396ad63df7e8c3c448a17e90e93a4bc93dee0c306d63b656", + "collector_closure_hashes": { + "collector.trtllm.collect_mla": "sha256:9d396ec69d649ddfeb9c3450f0cd35881309628c5ca1ded9b4259c98cf074e2e", + "collector.trtllm.collect_moe": "sha256:d7a50a258be2fa9d139a6983be5b92709bf79f14e8e6a99a9a410eb8c5c6d4d2" + }, + "runner_sha256": { + "collect.py": "e2f1d7e061050a25e78b86d542f0fb5dac4ec99396ff8290657e502fcc84372c", + "probe.py": "a48acecd94757b12aef7d17dbb9c00cf589547099c0350a900e16f6d2b441511", + "production.sh": "11d109c260d48746d9b94054c72ddeabea6246d1f8814d07fc588541b4e4e829", + "smoke-fixed.sh": "d110cebe65f7d806dfa8f2d3ffb62c652b8dbb30db8bb0f672b75d2fda1c0ec4", + "production-pmix.sh": "3d5c9e3bdb97577b7d79d38897c2a3009a18829f3bc8722c7fe78ff1c958129a", + "smoke-fixed-pmix.sh": "541f8a1e93f531b77ab0b5a8001222507728a6fc138066d2635307e55ed4a6ab", + "h200-computelab-pmix.sh": "f0ac99b3035e0e8007dd78a22f79a9fe497c4e20faf8ce0363c970add1534af5", + "recheck_h200_outliers.py": "47cce356f5df781bf8914b1d4516bb69e9504961a68d892935c5cb761c58451d", + "recheck-h200-outliers.sh": "d7b328e4223b668585cade6a64191f89e5df3cf135cd461c4f73ca3e08bbfae5", + "collector-fixed.tar.gz": "aad1c962e75124b9fa82e705fbc77a28eb53103426a88061542007bc6cf114de", + "publish_profiles.py": "f3333760db62581ea1b7581149552ed2c5cae0af44a55cf61fd6c8d6487be413" + }, + "runs": { + "b200": { + "host": "${B200_CLUSTER}", + "job_id": 1993260, + "node": "nsc-svg-slurm-1-gpu-145", + "gpus": 4, + "state": "COMPLETED", + "exit_code": "0:0", + "elapsed": "00:04:57", + "results": "${B200_ARTIFACT_ROOT}/results/trt-62-20260917/production-b200", + "stdout": "${B200_ARTIFACT_ROOT}/logs/trt62-production-b200-1993260.out", + "stderr": "${B200_ARTIFACT_ROOT}/logs/trt62-production-b200-1993260.err", + "publication": { + "summary": { + "planned": 3083, + "passed": 3083, + "failed": 0, + "unattempted": 0, + "worker_exit_codes": [ + 0, + 0, + 0, + 0 + ], + "runner_sha256": "e2f1d7e061050a25e78b86d542f0fb5dac4ec99396ff8290657e502fcc84372c" + }, + "plan_sha256": "58c451994b841b112a69c4842df1f7f850ae9a3d2b6f404d44d4efd7618e72bc", + "tables": { + "context_mla_perf": { + "new_rows": 3080, + "total_rows": 3080, + "old_rows": 1760, + "new_duplicate_keys": 0, + "preserved_old_duplicate_keys": 0, + "sha256": "905f09a335f56c3a4c37cc7eb74113c1c026e3c57a8ff3dfc40a7c98d1fab30a", + "raw_sha256": { + "shard-0/context_mla_perf.txt": "33369b7074ffc81079246f94185d6bc4fa854da888c06b52445b5c8a59753e4c", + "shard-1/context_mla_perf.txt": "04c443fbcf510b75d7532b7ecfe88248be7cdfe64f24ad7730a0471cf4a3f8cd", + "shard-2/context_mla_perf.txt": "ed6bf7fcaf7e3ac49d167bfc127e6ede12e26269964d9bc92ba58fef80938435", + "shard-3/context_mla_perf.txt": "4d5283828931b2dda41b504032e375c5112e67cf4d268023eed8146f189e3b6b" + } + }, + "moe_perf": { + "new_rows": 81, + "total_rows": 218376, + "old_rows": 218295, + "new_duplicate_keys": 0, + "preserved_old_duplicate_keys": 36045, + "sha256": "0bc472446e0993221f28c454645fc4054eb73162270868c2ea1332b5572d7803", + "raw_sha256": { + "shard-0/moe_perf.txt": "81b1a57c0cea0bce8126c6c53615f295c4e170c68d2a7d6ecf6ea588a58163b1", + "shard-1/moe_perf.txt": "3d365f88674d52dd0a7f0647b1ce6404429218b2de693166ea9c5f85848a4aa3", + "shard-2/moe_perf.txt": "dcec978c34319bf7fc7a410497c51d5b8613ff2651345db1992201bd36b0862d" + } + } + }, + "mla_provenance": { + "collector_ref": "ce9a396642674e8008fda2b778580dc8d9e310b4", + "collector_hash": "sha256:9d396ec69d649ddfeb9c3450f0cd35881309628c5ca1ded9b4259c98cf074e2e", + "case_plan_hash": "sha256:984b7c01ed49ccc195f4b9668b59865da2b367a9c270800f9850275e7ae16150", + "collected_at": "2026-09-17", + "rows": 3080, + "classified_failures": 0, + "status": "complete" + } + }, + "hardware": [ + { + "name": "NVIDIA B200", + "uuid": "GPU-d420a4c4-9b52-15b2-1537-eb22183b113f", + "driver_version": "580.82.07", + "power.limit [W]": "1000.00 W", + "clocks.current.sm [MHz]": "120 MHz", + "clocks.current.memory [MHz]": "3996 MHz" + }, + { + "name": "NVIDIA B200", + "uuid": "GPU-ef87e4a9-479b-2297-0dfe-6e44a2565b0d", + "driver_version": "580.82.07", + "power.limit [W]": "1000.00 W", + "clocks.current.sm [MHz]": "120 MHz", + "clocks.current.memory [MHz]": "3996 MHz" + }, + { + "name": "NVIDIA B200", + "uuid": "GPU-00a767de-2002-93bc-9751-23f79b3fd34e", + "driver_version": "580.82.07", + "power.limit [W]": "1000.00 W", + "clocks.current.sm [MHz]": "120 MHz", + "clocks.current.memory [MHz]": "3996 MHz" + }, + { + "name": "NVIDIA B200", + "uuid": "GPU-5bc7ea44-0aec-ccb3-f990-737a8c6afd12", + "driver_version": "580.82.07", + "power.limit [W]": "1000.00 W", + "clocks.current.sm [MHz]": "120 MHz", + "clocks.current.memory [MHz]": "3996 MHz" + } + ], + "mla_latency_new_over_old_by_kv_dtype": { + "bfloat16": { + "count": 880, + "0": 0.866952780764719, + "0.05": 0.9241575099229368, + "0.5": 0.9955788789882782, + "0.95": 1.015815337558837, + "1": 1.0994007955142837 + }, + "fp8": { + "count": 880, + "0": 0.8394674979123624, + "0.05": 0.9302425753491017, + "0.5": 0.9952240950000397, + "0.95": 1.0075401769330723, + "1": 1.095697827682327 + } + }, + "mla_smoke": [ + { + "case": "mla-bf16", + "runtime": "1.3.0rc20", + "torch": "2.11.0a0+eb65b36914.nv26.02", + "device": "NVIDIA B200", + "capability": [ + 10, + 0 + ], + "source_hashes": { + "_torch/attention_backend/trtllm.py": "bf935b2562fb3e0bc198753a146e6eafa6c5a7a076f8a4b16162903e37b0f54d", + "_torch/modules/attention.py": "70fb8c8750d759da734fa839cbd51361354fdb36398d5d7533ab639b747fa091", + "_torch/modules/fused_moe/quantization.py": "97daa1cbcd3d4f0157df3b74d518ad11d55833c67a5f701998a1f34657b1a878" + }, + "input_dtypes": [ + "torch.bfloat16", + "torch.bfloat16", + "torch.bfloat16" + ], + "quant_config": "quant_algo=None kv_cache_quant_algo=None group_size=128 smoothquant_val=0.5 clamp_val=None use_meta_recipe=False has_zero_point=False pre_quant_scale=False exclude_modules=None mamba_ssm_cache_dtype=None mamba_ssm_stochastic_rounding=False mamba_ssm_philox_rounds=10", + "cuda_kernels": [ + "fmhaSm100fKernel_QkvBfloat16OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext", + "void tensorrt_llm::_v1::kernels::applyMLARopeAndAssignQKVKernelOptContext<__nv_bfloat16, 256, 512, 64, tensorrt_llm::_v1::kernels::KVBlockArray>(__nv_bfloat16*, __nv_bfloat16*, __nv_bfloat16*, __nv_bfloat16 const*, tensorrt_llm::_v1::kernels::KVBlockArray, int, int, float2 const*, unsigned long, int, int, int*, int const*, unsigned int, tensorrt_llm::_v1::kernels::KvCacheDataType, float const*, int const*, bool)", + "void tensorrt_llm::_v1::kernels::computeSeqAndPaddingOffsets<__nv_bfloat16, 256>(tensorrt_llm::_v1::kernels::BuildDecoderInfoParams<__nv_bfloat16>)" + ], + "status": "passed", + "exit_code": 0 + }, + { + "case": "mla-fp8", + "runtime": "1.3.0rc20", + "torch": "2.11.0a0+eb65b36914.nv26.02", + "device": "NVIDIA B200", + "capability": [ + 10, + 0 + ], + "source_hashes": { + "_torch/attention_backend/trtllm.py": "bf935b2562fb3e0bc198753a146e6eafa6c5a7a076f8a4b16162903e37b0f54d", + "_torch/modules/attention.py": "70fb8c8750d759da734fa839cbd51361354fdb36398d5d7533ab639b747fa091", + "_torch/modules/fused_moe/quantization.py": "97daa1cbcd3d4f0157df3b74d518ad11d55833c67a5f701998a1f34657b1a878" + }, + "input_dtypes": [ + "torch.bfloat16", + "torch.bfloat16", + "torch.bfloat16" + ], + "quant_config": "quant_algo=None kv_cache_quant_algo= group_size=128 smoothquant_val=0.5 clamp_val=None use_meta_recipe=False has_zero_point=False pre_quant_scale=False exclude_modules=None mamba_ssm_cache_dtype=None mamba_ssm_stochastic_rounding=False mamba_ssm_philox_rounds=10", + "cuda_kernels": [ + "fmhaSm100fKernel_QkvE4m3OBfloat16HQk192HV128SeparateQkvCausalVarSeqQ128Kv128StaticContext", + "void tensorrt_llm::_v1::kernels::applyMLARopeAndAssignQKVKernelOptContext<__nv_bfloat16, 256, 512, 64, tensorrt_llm::_v1::kernels::KVBlockArray>(__nv_bfloat16*, __nv_bfloat16*, __nv_bfloat16*, __nv_bfloat16 const*, tensorrt_llm::_v1::kernels::KVBlockArray, int, int, float2 const*, unsigned long, int, int, int*, int const*, unsigned int, tensorrt_llm::_v1::kernels::KvCacheDataType, float const*, int const*, bool)", + "void tensorrt_llm::_v1::kernels::computeSeqAndPaddingOffsets<__nv_bfloat16, 256>(tensorrt_llm::_v1::kernels::BuildDecoderInfoParams<__nv_bfloat16>)", + "void tensorrt_llm::_v1::kernels::quantizeCopyInputToFp8Kernel<__nv_bfloat16, 384, 128, 64, 128, false>(__nv_bfloat16 const*, __nv_fp8_e4m3*, __nv_bfloat16 const*, __nv_fp8_e4m3*, __nv_bfloat16 const*, __nv_fp8_e4m3*, int, int, float const*, float*, float*, float const*, float const*, float const*, float)" + ], + "status": "passed", + "exit_code": 0 + } + ], + "telemetry": { + "hard_clock_lock": false, + "active_samples": 91, + "active_ranges": { + "power.draw [W]": { + "min": 213.38, + "max": 693.35 + }, + "power.limit [W]": { + "min": 1000.0, + "max": 1000.0 + }, + "clocks.current.sm [MHz]": { + "min": 1402.0, + "max": 1965.0 + }, + "clocks.current.memory [MHz]": { + "min": 3996.0, + "max": 3996.0 + } + } + }, + "artifacts_sha256": { + "gpu-before.csv": "8af46526085e43bae001bbf261f57e401f56fdbea50fb9f5369f83026de4196e", + "gpu-samples.csv": "1383314c98298fba043a0e2cbf82bcf35004cb8f8fb641b587d98006b5d5eca9", + "plan.json": "58c451994b841b112a69c4842df1f7f850ae9a3d2b6f404d44d4efd7618e72bc", + "shard-0/context_mla_perf.txt": "33369b7074ffc81079246f94185d6bc4fa854da888c06b52445b5c8a59753e4c", + "shard-0/moe_perf.txt": "81b1a57c0cea0bce8126c6c53615f295c4e170c68d2a7d6ecf6ea588a58163b1", + "shard-0/outcomes.jsonl": "73b3d9f7d0ecd512c2ae39621e5ebf646330b5e4c1e0feeadb979adeaffac266", + "shard-1/context_mla_perf.txt": "04c443fbcf510b75d7532b7ecfe88248be7cdfe64f24ad7730a0471cf4a3f8cd", + "shard-1/moe_perf.txt": "3d365f88674d52dd0a7f0647b1ce6404429218b2de693166ea9c5f85848a4aa3", + "shard-1/outcomes.jsonl": "ce9c04763cffc7c870601dc83cb65394219dbec0d0451517c5999c97738ed63e", + "shard-2/context_mla_perf.txt": "ed6bf7fcaf7e3ac49d167bfc127e6ede12e26269964d9bc92ba58fef80938435", + "shard-2/moe_perf.txt": "dcec978c34319bf7fc7a410497c51d5b8613ff2651345db1992201bd36b0862d", + "shard-2/outcomes.jsonl": "7f39a45d4165e7f7bfdc1f2048ceb81f7ed7a6027a0e18f4070abdd86f9e5de4", + "shard-3/context_mla_perf.txt": "4d5283828931b2dda41b504032e375c5112e67cf4d268023eed8146f189e3b6b", + "shard-3/outcomes.jsonl": "e697120664591a6337288a67adc47a6984c4c06ca7ea5d7630a975dac77b4877", + "source.sha256": "673442c40c7b1f50224ae040022f55c831239fc04f90afefe5dc5d06aaaa1488", + "summary.json": "cacaa1f86e5ad364a9226cee8fa886643b4e5c1c5cea92833753f31c01602d7e", + "trtllm-rc20-9b3b4dfb.sqsh.sha256": "736540c04926e748f12c34406a2d0e4b6348ec775ff0f78e20f5d7053c06b0c9" + } + }, + "h200": { + "host": "${H200_CLUSTER}", + "job_id": 4375636, + "node": "viking-prod-214", + "gpus": 2, + "state": "COMPLETED", + "exit_code": "0:0", + "elapsed": "00:10:46", + "smoke_elapsed": "00:01:46", + "production_elapsed": "00:08:52", + "mpi": "pmix", + "results": "${H200_ARTIFACT_ROOT}/results/trt-62-20260917/production-h200", + "stdout": "${H200_ARTIFACT_ROOT}/logs/trt62-h200-pmix-4375636.out", + "stderr": "${H200_ARTIFACT_ROOT}/logs/trt62-h200-pmix-4375636.err", + "publication": { + "summary": { + "planned": 3080, + "passed": 3080, + "failed": 0, + "unattempted": 0, + "worker_exit_codes": [ + 0, + 0 + ], + "runner_sha256": "e2f1d7e061050a25e78b86d542f0fb5dac4ec99396ff8290657e502fcc84372c" + }, + "plan_sha256": "8f9117995de2d54841e7704b8f4310f6333326d13d9e68a1c2f0d1144dc49d87", + "tables": { + "context_mla_perf": { + "new_rows": 3080, + "total_rows": 3080, + "old_rows": 1760, + "new_duplicate_keys": 0, + "preserved_old_duplicate_keys": 0, + "sha256": "72a342dadc601b8a183ca614f9cc18bf0e14b3a501fcade518c47c90732b7c97", + "raw_sha256": { + "shard-0/context_mla_perf.txt": "4a16319ffecda0e445723293e86dcd32061a2d5e9c3e075759105ab70c337bd1", + "shard-1/context_mla_perf.txt": "c7348aa367f629a81a5c304660a9b447002d93e138cba02b145a12e7ea1d4b3a" + } + } + }, + "mla_provenance": { + "collector_ref": "ce9a396642674e8008fda2b778580dc8d9e310b4", + "collector_hash": "sha256:9d396ec69d649ddfeb9c3450f0cd35881309628c5ca1ded9b4259c98cf074e2e", + "case_plan_hash": "sha256:c0fd09dd4f806c0c23a84de0e452f6093b889ec3bc4adcdf1fdf4f30d2b66606", + "collected_at": "2026-09-17", + "rows": 3080, + "classified_failures": 0, + "status": "complete" + } + }, + "hardware": [ + { + "name": "NVIDIA H200", + "uuid": "GPU-8806907d-d1c2-65f6-775b-79462bf69ac5", + "driver_version": "595.58.03", + "power.limit [W]": "700.00 W", + "clocks.current.sm [MHz]": "345 MHz", + "clocks.current.memory [MHz]": "3201 MHz" + }, + { + "name": "NVIDIA H200", + "uuid": "GPU-bedb0afa-7fb5-484a-5abf-f4afc0fff9ba", + "driver_version": "595.58.03", + "power.limit [W]": "700.00 W", + "clocks.current.sm [MHz]": "345 MHz", + "clocks.current.memory [MHz]": "3201 MHz" + } + ], + "mla_latency_new_over_old_by_kv_dtype": { + "bfloat16": { + "count": 880, + "0": 0.02169219598693033, + "0.05": 0.9514747575137581, + "0.5": 0.9933915693409964, + "0.95": 1.0270679156503228, + "1": 1.3147809244539361 + }, + "fp8": { + "count": 880, + "0": 0.025165438421725764, + "0.05": 0.9595226271714019, + "0.5": 0.9949348961309541, + "0.95": 1.01720251468336, + "1": 1.1663598497881915 + } + }, + "mla_smoke": [ + { + "case": "mla-bf16", + "runtime": "1.3.0rc20", + "torch": "2.11.0a0+eb65b36914.nv26.02", + "device": "NVIDIA H200", + "capability": [ + 9, + 0 + ], + "source_hashes": { + "_torch/attention_backend/trtllm.py": "bf935b2562fb3e0bc198753a146e6eafa6c5a7a076f8a4b16162903e37b0f54d", + "_torch/modules/attention.py": "70fb8c8750d759da734fa839cbd51361354fdb36398d5d7533ab639b747fa091", + "_torch/modules/fused_moe/quantization.py": "97daa1cbcd3d4f0157df3b74d518ad11d55833c67a5f701998a1f34657b1a878" + }, + "input_dtypes": [ + "torch.bfloat16", + "torch.bfloat16", + "torch.bfloat16" + ], + "quant_config": "quant_algo=None kv_cache_quant_algo=None group_size=128 smoothquant_val=0.5 clamp_val=None use_meta_recipe=False has_zero_point=False pre_quant_scale=False exclude_modules=None mamba_ssm_cache_dtype=None mamba_ssm_stochastic_rounding=False mamba_ssm_philox_rounds=10", + "cuda_kernels": [ + "Memset (Device)", + "fmha_v2_flash_attention_bf16_64_128_S_q_k_v_192x128_causal_tma_ws_sm90_kernel", + "void tensorrt_llm::_v1::kernels::applyMLARopeAndAssignQKVKernelOptContext<__nv_bfloat16, 256, 512, 64, tensorrt_llm::_v1::kernels::KVBlockArray>(__nv_bfloat16*, __nv_bfloat16*, __nv_bfloat16*, __nv_bfloat16 const*, tensorrt_llm::_v1::kernels::KVBlockArray, int, int, float2 const*, unsigned long, int, int, int*, int const*, unsigned int, tensorrt_llm::_v1::kernels::KvCacheDataType, float const*, int const*, bool)", + "void tensorrt_llm::_v1::kernels::computeSeqAndPaddingOffsets<__nv_bfloat16, 256>(tensorrt_llm::_v1::kernels::BuildDecoderInfoParams<__nv_bfloat16>)" + ], + "status": "passed", + "exit_code": 0 + }, + { + "case": "mla-fp8", + "runtime": "1.3.0rc20", + "torch": "2.11.0a0+eb65b36914.nv26.02", + "device": "NVIDIA H200", + "capability": [ + 9, + 0 + ], + "source_hashes": { + "_torch/attention_backend/trtllm.py": "bf935b2562fb3e0bc198753a146e6eafa6c5a7a076f8a4b16162903e37b0f54d", + "_torch/modules/attention.py": "70fb8c8750d759da734fa839cbd51361354fdb36398d5d7533ab639b747fa091", + "_torch/modules/fused_moe/quantization.py": "97daa1cbcd3d4f0157df3b74d518ad11d55833c67a5f701998a1f34657b1a878" + }, + "input_dtypes": [ + "torch.bfloat16", + "torch.bfloat16", + "torch.bfloat16" + ], + "quant_config": "quant_algo=None kv_cache_quant_algo= group_size=128 smoothquant_val=0.5 clamp_val=None use_meta_recipe=False has_zero_point=False pre_quant_scale=False exclude_modules=None mamba_ssm_cache_dtype=None mamba_ssm_stochastic_rounding=False mamba_ssm_philox_rounds=10", + "cuda_kernels": [ + "Memset (Device)", + "fmha_v2_flash_attention_e4m3_64_128_S_q_k_v_192x128_causal_output_bf16_tma_ws_sm90_kernel", + "void tensorrt_llm::_v1::kernels::applyMLARopeAndAssignQKVKernelOptContext<__nv_bfloat16, 256, 512, 64, tensorrt_llm::_v1::kernels::KVBlockArray>(__nv_bfloat16*, __nv_bfloat16*, __nv_bfloat16*, __nv_bfloat16 const*, tensorrt_llm::_v1::kernels::KVBlockArray, int, int, float2 const*, unsigned long, int, int, int*, int const*, unsigned int, tensorrt_llm::_v1::kernels::KvCacheDataType, float const*, int const*, bool)", + "void tensorrt_llm::_v1::kernels::computeSeqAndPaddingOffsets<__nv_bfloat16, 256>(tensorrt_llm::_v1::kernels::BuildDecoderInfoParams<__nv_bfloat16>)", + "void tensorrt_llm::_v1::kernels::quantizeCopyInputToFp8Kernel<__nv_bfloat16, 384, 128, 64, 128, false>(__nv_bfloat16 const*, __nv_fp8_e4m3*, __nv_bfloat16 const*, __nv_fp8_e4m3*, __nv_bfloat16 const*, __nv_fp8_e4m3*, int, int, float const*, float*, float*, float const*, float const*, float const*, float)" + ], + "status": "passed", + "exit_code": 0 + } + ], + "telemetry": { + "hard_clock_lock": false, + "active_samples": 96, + "active_ranges": { + "power.draw [W]": { + "min": 119.22, + "max": 691.92 + }, + "power.limit [W]": { + "min": 700.0, + "max": 700.0 + }, + "clocks.current.sm [MHz]": { + "min": 1665.0, + "max": 1980.0 + }, + "clocks.current.memory [MHz]": { + "min": 3201.0, + "max": 3201.0 + } + } + }, + "artifacts_sha256": { + "gpu-before.csv": "471ff331e51cb47cd30981b66e87626879aefdfeb2c995bc18cbbce060ac85da", + "gpu-samples.csv": "7cb669c2c13e60f6768df2abd5a8b1f9b51b48e1e57b1b28342f5ff3c51bd56d", + "plan.json": "8f9117995de2d54841e7704b8f4310f6333326d13d9e68a1c2f0d1144dc49d87", + "shard-0/context_mla_perf.txt": "4a16319ffecda0e445723293e86dcd32061a2d5e9c3e075759105ab70c337bd1", + "shard-0/outcomes.jsonl": "e5e7c6e3595e7d86c745f9370c4e38d8365440b002fa75064152783fed16b02c", + "shard-1/context_mla_perf.txt": "c7348aa367f629a81a5c304660a9b447002d93e138cba02b145a12e7ea1d4b3a", + "shard-1/outcomes.jsonl": "76fe27475994ea81a2675d953e7e01bc8219d1191603e614857572a549dcec61", + "source.sha256": "673442c40c7b1f50224ae040022f55c831239fc04f90afefe5dc5d06aaaa1488", + "summary.json": "c979e8cf119160cbe9d2182c360ede84e3c700de7d1a1d55ce1e8368ac6bdd58", + "trtllm-rc20-9b3b4dfb.sqsh.sha256": "b7639a94178fa077c2bdaba122583254e9a781fba8411cee3005baebddf9b194" + } + } + }, + "h200_outlier_recheck": { + "job_id": 4375895, + "node": "viking-prod-216", + "state": "COMPLETED", + "exit_code": "0:0", + "elapsed": "00:04:39", + "gpus": 1, + "summary": { + "planned": 18, + "passed": 18, + "failed": 0, + "unattempted": 0, + "shapes": [ + [ + "fp8", + 4, + 8, + 1 + ], + [ + "fp8", + 4, + 2, + 32 + ], + [ + "bfloat16", + 8, + 1, + 128 + ], + [ + "fp8", + 128, + 16, + 128 + ], + [ + "bfloat16", + 16, + 32, + 1024 + ], + [ + "bfloat16", + 1, + 32, + 2048 + ] + ], + "runtime": "1.3.0rc20" + }, + "results": "${H200_ARTIFACT_ROOT}/results/trt-62-20260917/recheck-h200-outliers", + "stdout": "${H200_ARTIFACT_ROOT}/logs/trt62-h200-recheck-4375895.out", + "stderr": "${H200_ARTIFACT_ROOT}/logs/trt62-h200-recheck-4375895.err", + "points": [ + { + "key": { + "kv_cache_dtype": "bfloat16", + "num_heads": 1, + "batch_size": 32, + "isl": 2048 + }, + "old_latency_ms": 0.9880053202311198, + "production_latency_ms": 0.1635413368542989, + "repeat_latencies_ms": [ + 0.1635466714700063, + 0.1631253361701965, + 0.1627466678619384 + ], + "repeat_over_production": [ + 1.0000326193720195, + 0.9974562964195834, + 0.9951408676995928 + ] + }, + { + "key": { + "kv_cache_dtype": "bfloat16", + "num_heads": 8, + "batch_size": 1, + "isl": 128 + }, + "old_latency_ms": 0.7806186676025391, + "production_latency_ms": 0.0169333331286907, + "repeat_latencies_ms": [ + 0.0170719996094703, + 0.017205332716306, + 0.0172266662120819 + ], + "repeat_over_production": [ + 1.0081889654993355, + 1.0160629679666813, + 1.0173228200946567 + ] + }, + { + "key": { + "kv_cache_dtype": "bfloat16", + "num_heads": 16, + "batch_size": 32, + "isl": 1024 + }, + "old_latency_ms": 1.3052000204722087, + "production_latency_ms": 0.4743359883626302, + "repeat_latencies_ms": [ + 0.47489066918691, + 0.4750826756159464, + 0.4744373162587483 + ], + "repeat_over_production": [ + 1.0011693838078668, + 1.0015741737326187, + 1.0002136205107859 + ] + }, + { + "key": { + "kv_cache_dtype": "fp8", + "num_heads": 4, + "batch_size": 2, + "isl": 32 + }, + "old_latency_ms": 0.7970720132191976, + "production_latency_ms": 0.0200586666663487, + "repeat_latencies_ms": [ + 0.0204426671067873, + 0.0202400001386801, + 0.0198773331940174 + ], + "repeat_over_production": [ + 1.019143866679973, + 1.0090401558262898, + 0.990959844173715 + ] + }, + { + "key": { + "kv_cache_dtype": "fp8", + "num_heads": 4, + "batch_size": 8, + "isl": 1 + }, + "old_latency_ms": 0.37910934289296466, + "production_latency_ms": 0.0181813339392344, + "repeat_latencies_ms": [ + 0.0179039997359116, + 0.0177920001248518, + 0.0178240003685156 + ], + "repeat_over_production": [ + 0.9847462125579066, + 0.9785860698844304, + 0.9803461301622268 + ] + }, + { + "key": { + "kv_cache_dtype": "fp8", + "num_heads": 128, + "batch_size": 16, + "isl": 128 + }, + "old_latency_ms": 1.0207466284434001, + "production_latency_ms": 0.2195840080579122, + "repeat_latencies_ms": [ + 0.2199359933535258, + 0.2198293407758077, + 0.2198719978332519 + ], + "repeat_over_production": [ + 1.001602964162676, + 1.001117261316365, + 1.001311524358658 + ] + } + ], + "raw_sha256": { + "case-0-repeat-0.txt": "699a889dfe109fda753b9253c1ce01c00afe934f4af2029da181a966583e70c1", + "case-0-repeat-1.txt": "013afffdd72d7fa709bb16a82ffaac483b6a1103007ed81a3e1c3b8399fc4274", + "case-0-repeat-2.txt": "987d7f74b79898ed01e6e14c16984592b9074251209238d7c1a9d0c7a80965d5", + "case-1-repeat-0.txt": "d57e00ff465108a35197c9139e8856164aeb7a37823c091b13ce340a562a2c73", + "case-1-repeat-1.txt": "7d8b507785fe5b71bddc21b7698ef1d9b9082135bd2d6bcb38afd7d78ce4bf7e", + "case-1-repeat-2.txt": "0f941611bdc8bf936d395c100b62ea42a850e487618db6eeda2220651a31e008", + "case-2-repeat-0.txt": "c62f0fda944fdbdc5b6490e1dae1d44d36623c416966429b0248b431ddf7f0de", + "case-2-repeat-1.txt": "722794a85c5978f590267e1ffa71351323bbf7032319687fec9466c97f0f1290", + "case-2-repeat-2.txt": "dedffe6175da60651a654982db2dd6c2d7b4183716ac6dfc6a300b7dbf2b7273", + "case-3-repeat-0.txt": "eb7516eaf9b376722c89376bffe67be62b680374de6e614c957aaafc50970776", + "case-3-repeat-1.txt": "bd0de4af91945471b4226e39cae68d1bdfc78acdfb0de609e926a9004379a68c", + "case-3-repeat-2.txt": "ab045e2526f8e987d08ef13032d280db0dc4144f38a40c1e75726e2d9f4414c5", + "case-4-repeat-0.txt": "97b1a741a14baa08e9b1859b217af369a1947d468ac16b3f3ec223327a465385", + "case-4-repeat-1.txt": "dd3b15de76ff99abf8b9bf3b3b6121292c48e493654b81c50e98169990e6b56b", + "case-4-repeat-2.txt": "2330b1baf1777b2272eae4a11910c840cb4d381c5087bd8d5937f01e87d5d8ce", + "case-5-repeat-0.txt": "2196548ba6d05aea88c81c0c8745e708bee355b389d7ffcc2f143222dca69d40", + "case-5-repeat-1.txt": "58b32fd25ff3a236b34b33eb54fe153348cf4974767064f02369138433d3791e", + "case-5-repeat-2.txt": "1936e6ebb82aa06185ab03809cb04231722929540fcb859ba0471bb8c13ae43e", + "gpu-before.csv": "20422a3f3131c15b6b8dacd6cd8cfd18fb5f9fefe5841dc216bb988c01d055c1", + "summary.json": "463531b661def9f547c33268f5b39509f6c25a8dccb915ae734397df56bd81b3" + }, + "interpretation": "All six shapes reproduced the new measurements on a different H200 node. The cause of the larger old values is not established. Diagnostic repeats are separate from published production rows." + }, + "limitations": [ + "Native TRT-LLM operator measurements; no timing borrowed from another backend or GPU.", + "Replay verifies profile availability and request completion, not prediction accuracy against matched serving measurements.", + "Original silicon recipe image versions remain unchanged; database query and collector runtime are both 1.3.0rc20.", + "GPU clocks were observed, not locked. Production timings have no profiler wrapper.", + "Existing MoE rows and power values are preserved exactly; new MoE power fields are null." + ] +} diff --git a/python/aisimulate/collector/trtllm/gym-profiles-20260917-replay.json b/python/aisimulate/collector/trtllm/gym-profiles-20260917-replay.json new file mode 100644 index 000000000..3f68355af --- /dev/null +++ b/python/aisimulate/collector/trtllm/gym-profiles-20260917-replay.json @@ -0,0 +1,1719 @@ +{ + "runtime_commit": "89d2051137b772944a3e303c452108811790bef4", + "collector_commit": "ce9a396642674e8008fda2b778580dc8d9e310b4", + "gym_commit": "734e9cf31fdbca74a1be496a309026ab7a58f119", + "baseline_failures_reproduced": 62, + "recovered_points": 62, + "regression_points_passed": 201, + "total_passed": 263, + "by_model": { + "DeepSeek-R1-0528": 41, + "gpt-oss-120b": 64, + "Qwen-3.5-397B-A17B": 116, + "MiniMax-M2.5": 42 + }, + "completed_requests": 293180, + "recovered_requests": 35520, + "truncated_requests": 0, + "context_violations": 0, + "recovered_native_specs_byte_identical": true, + "source_fields_unchanged": true, + "package_files_checked": 1773, + "package_files_added": 0, + "package_files_removed": 0, + "package_changes": [ + { + "path": "aiconfigurator_core/systems/data/b200_sxm/mla/trtllm/1.3.0rc20/context_mla_perf.parquet", + "before": "42e8fda14862686a3d344ad456347816b725f9b3cc84333bac619c3c98695d64", + "after": "905f09a335f56c3a4c37cc7eb74113c1c026e3c57a8ff3dfc40a7c98d1fab30a" + }, + { + "path": "aiconfigurator_core/systems/data/b200_sxm/moe/trtllm/1.3.0rc20/moe_perf.parquet", + "before": "cf4c6a1e9ba26115188ad22d17099e9c9e91ef72d5d18dce4c420b722e251628", + "after": "0bc472446e0993221f28c454645fc4054eb73162270868c2ea1332b5572d7803" + }, + { + "path": "aiconfigurator_core/systems/data/h200_sxm/mla/trtllm/1.3.0rc20/context_mla_perf.parquet", + "before": "c5aae589679e89c6f3c89b3a8fa68f2d5883f6db5260c97292dabebeb902fea9", + "after": "72a342dadc601b8a183ca614f9cc18bf0e14b3a501fcade518c47c90732b7c97" + } + ], + "predictions_sha256": "c4d9e9f777216218459fb68001b2a989dacd09b9178031a8b55b6cde439be121", + "input_sha256": "df7a49def9d411a14aec283f3044f1da0d401d6b0eebe8d96717943d74840486", + "points": [ + { + "selected_index": 0, + "source_index": 10, + "config_id": 75, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "conc": 64, + "source_row_sha256": "ddbc7cc6fea3ffa2d147bcf4165ab2e1176ff74ee6caac28a69cb7291096d7de", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 159.83202601562516, + "dynamo_tpot_ms": 30.772882970770873, + "dynamo_request_latency_ms": 28397.693276909336, + "dynamo_interactivity": 32.49614281995723, + "dynamo_tput_per_gpu_output": 251.99148614024324, + "dynamo_tput_per_gpu_total": 504.4649022406912 + }, + "completed_requests": 640, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "2e6e0174bdde4c732db69f92adacd5ff9d3a17c0b01df14f4c3b5d498a9ba7fa", + "audit_path": "b200-output-0-reports/0001.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 1, + "source_index": 11, + "config_id": 75, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "conc": 4, + "source_row_sha256": "b163173e0e86f588b41d89bf5d22d777fd5842de65a25612d307c8a3433d3712", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 66.71030040000043, + "dynamo_tpot_ms": 12.775904542260609, + "dynamo_request_latency_ms": 11875.633185725059, + "dynamo_interactivity": 78.27234437234273, + "dynamo_tput_per_gpu_output": 38.215301686279126, + "dynamo_tput_per_gpu_total": 76.41821290431271 + }, + "completed_requests": 40, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "1ff905505e9b59e95088def93a4b30aaa9fccbd5e29c553108441e9f40fa4ac4", + "audit_path": "b200-output-1-reports/0001.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 2, + "source_index": 12, + "config_id": 75, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "conc": 8, + "source_row_sha256": "901a45477f39061f25df9cf5bbbe513f87de56f9b52c4171c74644ed9b7b2932", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 77.12835784999945, + "dynamo_tpot_ms": 15.673729699299201, + "dynamo_request_latency_ms": 14553.531238662525, + "dynamo_interactivity": 63.80102369921, + "dynamo_tput_per_gpu_output": 62.348274047581235, + "dynamo_tput_per_gpu_total": 124.73448052715248 + }, + "completed_requests": 80, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "32a771dfc65b8dd51db97696fee90359a3f35da207bac68999ce077257381dca", + "audit_path": "b200-output-2-reports/0001.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 3, + "source_index": 13, + "config_id": 75, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "conc": 8, + "source_row_sha256": "2cae225d84121da77d855cb726571c26768268f84d9b1f5454209121e142bb57", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 285.6528664874992, + "dynamo_tpot_ms": 17.113342453175882, + "dynamo_request_latency_ms": 16112.379906099968, + "dynamo_interactivity": 58.43393847438732, + "dynamo_tput_per_gpu_output": 56.69158912470797, + "dynamo_tput_per_gpu_total": 508.742439502696 + }, + "completed_requests": 80, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "658b6bd76a1c4e72b94f0a315f93d9b9d13ce4c6d85a90d064636b25b6ef6103", + "audit_path": "b200-output-3-reports/0001.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 4, + "source_index": 14, + "config_id": 75, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "conc": 64, + "source_row_sha256": "9c9f34dd0534dd1c9974017739c2465221afb3e6a67adc482bd577e683009d60", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 883.8029458484368, + "dynamo_tpot_ms": 42.485899719985895, + "dynamo_request_latency_ms": 39888.99012242362, + "dynamo_interactivity": 23.53722073889817, + "dynamo_tput_per_gpu_output": 180.65654366201483, + "dynamo_tput_per_gpu_total": 1629.2605046596602 + }, + "completed_requests": 640, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "fbb80237fc248c1ac9b303eb026c35220a2255cbc00c24fc85af8ddafa4e84aa", + "audit_path": "b200-output-0-reports/0002.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 5, + "source_index": 15, + "config_id": 75, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "conc": 4, + "source_row_sha256": "72d3a998e0fb10569675863253d23ccefeaf486c75190f663a82dbfb1350ebe6", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 231.35404255000066, + "dynamo_tpot_ms": 13.538788294853068, + "dynamo_request_latency_ms": 12748.525050225007, + "dynamo_interactivity": 73.86185367712426, + "dynamo_tput_per_gpu_output": 35.631125573502786, + "dynamo_tput_per_gpu_total": 320.7600356875201 + }, + "completed_requests": 40, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "ddaa845bd5cc2aeace637c21969db9fc5ff4471dc486de01b62e2f61bb9139ea", + "audit_path": "b200-output-1-reports/0002.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 6, + "source_index": 16, + "config_id": 75, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "conc": 128, + "source_row_sha256": "106d47c92c42aa58209dbc8215983e761470512baf343dc8b8a111c917e10740", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 248.20115803984396, + "dynamo_tpot_ms": 36.70182049045927, + "dynamo_request_latency_ms": 34055.944344947646, + "dynamo_interactivity": 27.24660484511804, + "dynamo_tput_per_gpu_output": 422.51369697805524, + "dynamo_tput_per_gpu_total": 844.0014005219156 + }, + "completed_requests": 1280, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "d0c50db45dec09af52bc904eb978e328b1a9e8f4370e5b1402dd8c8b112a779e", + "audit_path": "b200-output-2-reports/0002.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 7, + "source_index": 17, + "config_id": 75, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "conc": 256, + "source_row_sha256": "212decc83757393b05732df747abe929ee62662b02f93aedf36311e81da4b645", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 3575.3454541824203, + "dynamo_tpot_ms": 93.35945034490524, + "dynamo_request_latency_ms": 89752.36788412071, + "dynamo_interactivity": 10.711288426673685, + "dynamo_tput_per_gpu_output": 323.8711625844791, + "dynamo_tput_per_gpu_total": 2908.602548825979 + }, + "completed_requests": 2560, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "f5980b8642eeedcb878ba11f01698b2561309eb12a2695da282205c83db350b5", + "audit_path": "b200-output-3-reports/0002.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 8, + "source_index": 18, + "config_id": 75, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "conc": 128, + "source_row_sha256": "09b957755d9ef7d1ffac53ac1c5b3e9031d6239a122b58d6d848b3ef672c8520", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 1612.691693432814, + "dynamo_tpot_ms": 59.931205628683315, + "dynamo_request_latency_ms": 56887.40344390002, + "dynamo_interactivity": 16.685798149893984, + "dynamo_tput_per_gpu_output": 255.39529944334626, + "dynamo_tput_per_gpu_total": 2294.083754262735 + }, + "completed_requests": 1280, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "9cf9bd924ba1f1184bfec3842d4ac854f07820c6ace77ab4fb0b6e49be89b05b", + "audit_path": "b200-output-0-reports/0003.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 9, + "source_index": 29, + "config_id": 82, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "conc": 16, + "source_row_sha256": "796848e3e52b602155aca0dc4320ef6728153b126ae7391e5ec88a3c9a2d8c2e", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 96.66793040624994, + "dynamo_tpot_ms": 11.108461524979543, + "dynamo_request_latency_ms": 10300.078325368751, + "dynamo_interactivity": 90.02146676668997, + "dynamo_tput_per_gpu_output": 349.2553822478099, + "dynamo_tput_per_gpu_total": 700.7268338610882 + }, + "completed_requests": 160, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "46cf3c1d7d38ea8e4745401f43b018bf78d02d51bd0fce328e17ca7fc0d5919b", + "audit_path": "b200-output-1-reports/0003.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 10, + "source_index": 30, + "config_id": 82, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "conc": 4, + "source_row_sha256": "cd0c5c692095198a47c95043e7147d5b4ef37fd0129e574074080f4883c86cf0", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 74.32791005000028, + "dynamo_tpot_ms": 7.554796660340204, + "dynamo_request_latency_ms": 7057.780256975003, + "dynamo_interactivity": 132.366236307804, + "dynamo_tput_per_gpu_output": 128.82183130238244, + "dynamo_tput_per_gpu_total": 257.6018949687748 + }, + "completed_requests": 40, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "2e6f0d98b3cb8e17dd9a877cde71358f786819658908b4fdffda1b19f397e349", + "audit_path": "b200-output-2-reports/0003.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 11, + "source_index": 31, + "config_id": 82, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "conc": 8, + "source_row_sha256": "b2d2f342c8b80a19dba222b8ba13c8ee1990b25bb1831f2207b579e59b7e4f9e", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 84.59517222500001, + "dynamo_tpot_ms": 8.844276931642518, + "dynamo_request_latency_ms": 8254.912046887459, + "dynamo_interactivity": 113.0674681185367, + "dynamo_tput_per_gpu_output": 219.90858507395578, + "dynamo_tput_per_gpu_total": 439.9509616212885 + }, + "completed_requests": 80, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "e0b8cae81dd05f5e39c17d28fc44547b18e99b4123e92ea8ed4db6dad6a3eb61", + "audit_path": "b200-output-3-reports/0003.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 12, + "source_index": 32, + "config_id": 82, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "conc": 32, + "source_row_sha256": "a9aaaa7efa50d8872ba693d993c25ebdd520592b85ab258df344a0e4dcd878f2", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 618.9049051281246, + "dynamo_tpot_ms": 22.026895896605613, + "dynamo_request_latency_ms": 20851.595600871886, + "dynamo_interactivity": 45.39904327391414, + "dynamo_tput_per_gpu_output": 346.9493909129449, + "dynamo_tput_per_gpu_total": 3134.3910866689284 + }, + "completed_requests": 320, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "a2fcb7969b630c15841db497684f009456ddfcf5d0865e834cec1d2cba85ee75", + "audit_path": "b200-output-0-reports/0004.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 13, + "source_index": 33, + "config_id": 82, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "conc": 4, + "source_row_sha256": "8aea058a7107c5e66c33eaeb3c51ce8132643259f6c02bee045ce3afbaad4df7", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 271.4033068250004, + "dynamo_tpot_ms": 8.343063029363035, + "dynamo_request_latency_ms": 7987.282868800015, + "dynamo_interactivity": 119.86005577094946, + "dynamo_tput_per_gpu_output": 113.91609493227953, + "dynamo_tput_per_gpu_total": 1025.5003199515486 + }, + "completed_requests": 40, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "11f25dfe54c13d05d0d17d1ddd3547c91b69d5a464a670b74e4f2ef0df7fe9d5", + "audit_path": "b200-output-1-reports/0004.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 14, + "source_index": 34, + "config_id": 82, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "conc": 8, + "source_row_sha256": "84882cc8f076ee2ce1bd5beef4a0825464176f5b2c218f59a73da63d4c364d89", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 332.8740406999995, + "dynamo_tpot_ms": 10.621622902810511, + "dynamo_request_latency_ms": 10166.05959117497, + "dynamo_interactivity": 94.14757134104217, + "dynamo_tput_per_gpu_output": 179.8830494964013, + "dynamo_tput_per_gpu_total": 1614.2454787195 + }, + "completed_requests": 80, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "0149db803f065df9bc2524a75e9f46e982a6135f7352ca9cbb8c350e2456b740", + "audit_path": "b200-output-2-reports/0004.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 15, + "source_index": 35, + "config_id": 82, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "conc": 16, + "source_row_sha256": "85fcabaca2e69871cab89d751ad706c1eee1d56867b14598c2a60b809943c47d", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 420.1878244750004, + "dynamo_tpot_ms": 14.60733386103891, + "dynamo_request_latency_ms": 13861.983748993798, + "dynamo_interactivity": 68.45876253073314, + "dynamo_tput_per_gpu_output": 261.6711125493793, + "dynamo_tput_per_gpu_total": 2363.793618722229 + }, + "completed_requests": 160, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "d6bc957beb863348d322b5182bd12e502aede246ea7c6058f26fd1d580eeee94", + "audit_path": "b200-output-3-reports/0004.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 16, + "source_index": 36, + "config_id": 91, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "conc": 4, + "source_row_sha256": "839c5121595385e2f39702e56628d9b8fcc5465855710dc3805b3513a8e85660", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 53.661098550000624, + "dynamo_tpot_ms": 6.308918493282411, + "dynamo_request_latency_ms": 5885.501780575005, + "dynamo_interactivity": 158.50577259236692, + "dynamo_tput_per_gpu_output": 77.08396391345401, + "dynamo_tput_per_gpu_total": 154.14293505267977 + }, + "completed_requests": 40, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "c93e2fd2700e7f6db7fc6eab246371c5dc5138445951f5de0712f0205b40d20c", + "audit_path": "b200-output-0-reports/0005.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 17, + "source_index": 37, + "config_id": 91, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "conc": 4, + "source_row_sha256": "8ef8eebf5f0914dd354f473e46570bb8f23e3e1e48f808144d68d67ae77aa521", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 212.81671027499996, + "dynamo_tpot_ms": 7.269940879807199, + "dynamo_request_latency_ms": 6935.889711975021, + "dynamo_interactivity": 137.55270043221043, + "dynamo_tput_per_gpu_output": 65.52892867280435, + "dynamo_tput_per_gpu_total": 589.9073116928022 + }, + "completed_requests": 40, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "8b724f50b56d3518e2e073b46d2479be13e900a7cf67cb68db2bb12833d190ab", + "audit_path": "b200-output-1-reports/0005.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 28, + "source_index": 48, + "config_id": 155, + "display_name": "gpt-oss-120b", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "conc": 4, + "source_row_sha256": "e249f96f8f5bd6e3d315dfdf0e8d4e4c4581541bac9a605f7afdb2ef5eb0031a", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 50.957391000000186, + "dynamo_tpot_ms": 5.628635476576109, + "dynamo_request_latency_ms": 5253.7933536749915, + "dynamo_interactivity": 177.66295297706836, + "dynamo_tput_per_gpu_output": 346.35018697204913, + "dynamo_tput_per_gpu_total": 692.5880775391466 + }, + "completed_requests": 40, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "9d17c43da79079a0963b419644d4410d08388768f98c1df568ac3436587cd441", + "audit_path": "b200-output-2-reports/0005.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: MoE data missing for MoeKey { quant: \"w4a16_mxfp4\", distribution: \"power_law_1.2\", topk: 4, num_experts: 128, hidden_size: 2880, inter_size: 2880, moe_tp_size: 2, moe_ep_size: 1 } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 29, + "source_index": 49, + "config_id": 155, + "display_name": "gpt-oss-120b", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "conc": 16, + "source_row_sha256": "9aa0f378d9982e39fa4fef847b7ffe383020d3d4ef27d53ea30ccfab49a0d4de", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 71.41464054375031, + "dynamo_tpot_ms": 10.721460101966251, + "dynamo_request_latency_ms": 9917.591405174968, + "dynamo_interactivity": 93.27087826560172, + "dynamo_tput_per_gpu_output": 728.3472584114817, + "dynamo_tput_per_gpu_total": 1461.3159718636857 + }, + "completed_requests": 160, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "b54af5a2888f9d2f46b39c20b186b2d07e044f80cad3ceb1462b746514816fdb", + "audit_path": "b200-output-3-reports/0005.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: MoE data missing for MoeKey { quant: \"w4a16_mxfp4\", distribution: \"power_law_1.2\", topk: 4, num_experts: 128, hidden_size: 2880, inter_size: 2880, moe_tp_size: 2, moe_ep_size: 1 } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 30, + "source_index": 50, + "config_id": 155, + "display_name": "gpt-oss-120b", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "conc": 8, + "source_row_sha256": "36f83ca124b1107495bff4b95293fc78586e804b733fb56b7ce4ceb652ccc8e7", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 58.798741987500605, + "dynamo_tpot_ms": 8.119567818976149, + "dynamo_request_latency_ms": 7558.359840999983, + "dynamo_interactivity": 123.15926442081208, + "dynamo_tput_per_gpu_output": 482.08693076623894, + "dynamo_tput_per_gpu_total": 964.4671612267182 + }, + "completed_requests": 80, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "71264d9d760262886e2670af8ad36e61b2d415f32c4e2f066665c991c64feda8", + "audit_path": "b200-output-0-reports/0006.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: MoE data missing for MoeKey { quant: \"w4a16_mxfp4\", distribution: \"power_law_1.2\", topk: 4, num_experts: 128, hidden_size: 2880, inter_size: 2880, moe_tp_size: 2, moe_ep_size: 1 } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 31, + "source_index": 51, + "config_id": 155, + "display_name": "gpt-oss-120b", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "conc": 32, + "source_row_sha256": "d28d95b39191fa0cfbe514cd4e1b70c4bec6a8ac1b72986e995b1dcc861ef210", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 87.55258437187527, + "dynamo_tpot_ms": 14.33472636697976, + "dynamo_request_latency_ms": 13239.455182362508, + "dynamo_interactivity": 69.76066193377181, + "dynamo_tput_per_gpu_output": 1084.1199826936283, + "dynamo_tput_per_gpu_total": 2172.6417676204082 + }, + "completed_requests": 320, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "c19c756d43ec4384adf8e271c4441c38f4e1e7d97dbeec046fb03ddb781836f0", + "audit_path": "b200-output-1-reports/0006.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: MoE data missing for MoeKey { quant: \"w4a16_mxfp4\", distribution: \"power_law_1.2\", topk: 4, num_experts: 128, hidden_size: 2880, inter_size: 2880, moe_tp_size: 2, moe_ep_size: 1 } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 32, + "source_index": 52, + "config_id": 155, + "display_name": "gpt-oss-120b", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "conc": 64, + "source_row_sha256": "368364054ae2578d1e64f01f38e6482c42a7be0d43a580652c3d3317568cf4bd", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 120.37670136875049, + "dynamo_tpot_ms": 22.024346516216802, + "dynamo_request_latency_ms": 20331.463069645353, + "dynamo_interactivity": 45.40429834155068, + "dynamo_tput_per_gpu_output": 1416.3785255038597, + "dynamo_tput_per_gpu_total": 2835.4658538204103 + }, + "completed_requests": 640, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "24f3f4a905f305291b5f7ae5c42b22f5a43cc5860808b2ae48c39907c63c89da", + "audit_path": "b200-output-2-reports/0006.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: MoE data missing for MoeKey { quant: \"w4a16_mxfp4\", distribution: \"power_law_1.2\", topk: 4, num_experts: 128, hidden_size: 2880, inter_size: 2880, moe_tp_size: 2, moe_ep_size: 1 } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 33, + "source_index": 53, + "config_id": 155, + "display_name": "gpt-oss-120b", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "conc": 32, + "source_row_sha256": "e35d6a136630ea5878c4607b2a51b39fcf40b578624cf890820083722906f4d6", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 352.84388155625055, + "dynamo_tpot_ms": 17.593443184272264, + "dynamo_request_latency_ms": 16506.00383315321, + "dynamo_interactivity": 56.83935711310646, + "dynamo_tput_per_gpu_output": 876.3650250161089, + "dynamo_tput_per_gpu_total": 7917.208662193954 + }, + "completed_requests": 320, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "1b6951cb0647a2f6217792cbdee47c6c7ac1f9b27cf9970eb0fd318487bc71a0", + "audit_path": "b200-output-3-reports/0006.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: MoE data missing for MoeKey { quant: \"w4a16_mxfp4\", distribution: \"power_law_1.2\", topk: 4, num_experts: 128, hidden_size: 2880, inter_size: 2880, moe_tp_size: 2, moe_ep_size: 1 } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 34, + "source_index": 54, + "config_id": 155, + "display_name": "gpt-oss-120b", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "conc": 64, + "source_row_sha256": "d936e8e4f4790bd86e9661411cfca3ee3bfba293757317555b62fab31c513296", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 567.1430806296866, + "dynamo_tpot_ms": 28.693716917846963, + "dynamo_request_latency_ms": 26912.478150651346, + "dynamo_interactivity": 34.85083521466048, + "dynamo_tput_per_gpu_output": 1075.2728596693607, + "dynamo_tput_per_gpu_total": 9697.404624707742 + }, + "completed_requests": 640, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "65d896d7028d2ca45f6117cd3364fcb3086e5b9680f45c164c9d5bac3089beb9", + "audit_path": "b200-output-0-reports/0007.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: MoE data missing for MoeKey { quant: \"w4a16_mxfp4\", distribution: \"power_law_1.2\", topk: 4, num_experts: 128, hidden_size: 2880, inter_size: 2880, moe_tp_size: 2, moe_ep_size: 1 } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 35, + "source_index": 55, + "config_id": 155, + "display_name": "gpt-oss-120b", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "conc": 8, + "source_row_sha256": "988e17a14ae405d994126dc929652809ca294fcf4b8efe32a68bb2ef84318d24", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 193.99316447499976, + "dynamo_tpot_ms": 8.831456725739006, + "dynamo_request_latency_ms": 8363.928976737498, + "dynamo_interactivity": 113.23160278706129, + "dynamo_tput_per_gpu_output": 438.22987411641026, + "dynamo_tput_per_gpu_total": 3932.6139673120406 + }, + "completed_requests": 80, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "791b6ccd4880a910909d3ecf6915a1a1910b0b3c9a9ee2e89f126d6dc9b94280", + "audit_path": "b200-output-1-reports/0007.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: MoE data missing for MoeKey { quant: \"w4a16_mxfp4\", distribution: \"power_law_1.2\", topk: 4, num_experts: 128, hidden_size: 2880, inter_size: 2880, moe_tp_size: 2, moe_ep_size: 1 } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 36, + "source_index": 56, + "config_id": 155, + "display_name": "gpt-oss-120b", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "conc": 4, + "source_row_sha256": "1c7f6341b7f9f4e599db5a3cac445f4eff03bf53a478769252cff26512788290", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 162.37680167499914, + "dynamo_tpot_ms": 5.976221446639023, + "dynamo_request_latency_ms": 5688.233266725003, + "dynamo_interactivity": 167.32981013653563, + "dynamo_tput_per_gpu_output": 320.24345144503883, + "dynamo_tput_per_gpu_total": 2882.909233453863 + }, + "completed_requests": 40, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "f056cf8fd0ef758066aa85c46b14bf9a03844638599cedc18b6e8af192036bec", + "audit_path": "b200-output-2-reports/0007.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: MoE data missing for MoeKey { quant: \"w4a16_mxfp4\", distribution: \"power_law_1.2\", topk: 4, num_experts: 128, hidden_size: 2880, inter_size: 2880, moe_tp_size: 2, moe_ep_size: 1 } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 37, + "source_index": 57, + "config_id": 155, + "display_name": "gpt-oss-120b", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "conc": 16, + "source_row_sha256": "8686322effb3dd7656251057742e273e1eff3cfdabda9774496141cdb65b1437", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 250.4786091624995, + "dynamo_tpot_ms": 12.215786142232005, + "dynamo_request_latency_ms": 11486.054168987492, + "dynamo_interactivity": 81.86128902034667, + "dynamo_tput_per_gpu_output": 632.0744583325367, + "dynamo_tput_per_gpu_total": 5709.814723556132 + }, + "completed_requests": 160, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "7bba886bd5ecb5fb160c8f683bfc98467258e3ea42819213c96309fdf404a11f", + "audit_path": "b200-output-3-reports/0007.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: MoE data missing for MoeKey { quant: \"w4a16_mxfp4\", distribution: \"power_law_1.2\", topk: 4, num_experts: 128, hidden_size: 2880, inter_size: 2880, moe_tp_size: 2, moe_ep_size: 1 } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 38, + "source_index": 58, + "config_id": 155, + "display_name": "gpt-oss-120b", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "conc": 128, + "source_row_sha256": "283581360c323bbbd2fcb8c37c1d16d8a08be457380ad5e71940f4e581dee18e", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 177.0887033820302, + "dynamo_tpot_ms": 31.084675913962123, + "dynamo_request_latency_ms": 28810.915529633483, + "dynamo_interactivity": 32.17019224417378, + "dynamo_tput_per_gpu_output": 2010.6345845528972, + "dynamo_tput_per_gpu_total": 4016.386728851026 + }, + "completed_requests": 1280, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "cdc3eaffe42a1042faa2b430b60b899e7461c55e01b641d05648148eecb3a996", + "audit_path": "b200-output-0-reports/0008.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: MoE data missing for MoeKey { quant: \"w4a16_mxfp4\", distribution: \"power_law_1.2\", topk: 4, num_experts: 128, hidden_size: 2880, inter_size: 2880, moe_tp_size: 2, moe_ep_size: 1 } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 39, + "source_index": 59, + "config_id": 155, + "display_name": "gpt-oss-120b", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "conc": 128, + "source_row_sha256": "108c4f4d083a3423ded2ba164dd1a32cfd2bd517eec06c4a53c3b59a8defd2e8", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 1011.4303657695318, + "dynamo_tpot_ms": 44.44776375377569, + "dynamo_request_latency_ms": 42006.14301618998, + "dynamo_interactivity": 22.498319725141478, + "dynamo_tput_per_gpu_output": 1387.4681074968653, + "dynamo_tput_per_gpu_total": 12462.90770387609 + }, + "completed_requests": 1280, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "abcde72dfb00f3972958165146d9fde3da3f425ea2d0e7154e37689a6e19f1e0", + "audit_path": "b200-output-1-reports/0008.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: MoE data missing for MoeKey { quant: \"w4a16_mxfp4\", distribution: \"power_law_1.2\", topk: 4, num_experts: 128, hidden_size: 2880, inter_size: 2880, moe_tp_size: 2, moe_ep_size: 1 } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 40, + "source_index": 60, + "config_id": 155, + "display_name": "gpt-oss-120b", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "conc": 256, + "source_row_sha256": "d71663f1442a995135a7d1ef902ce5391a09ff68241b3f5961eb52dadcc93215", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 288.98425323984264, + "dynamo_tpot_ms": 45.09754938613715, + "dynamo_request_latency_ms": 41887.36384945455, + "dynamo_interactivity": 22.174153886672098, + "dynamo_tput_per_gpu_output": 2770.549332469917, + "dynamo_tput_per_gpu_total": 5533.969543282892 + }, + "completed_requests": 2560, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "c765f0cd6fcb30b2d6c08a2f7438b30ff83a9072cdb5a71d294592276b4f570d", + "audit_path": "b200-output-2-reports/0008.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: MoE data missing for MoeKey { quant: \"w4a16_mxfp4\", distribution: \"power_law_1.2\", topk: 4, num_experts: 128, hidden_size: 2880, inter_size: 2880, moe_tp_size: 2, moe_ep_size: 1 } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 41, + "source_index": 61, + "config_id": 155, + "display_name": "gpt-oss-120b", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "conc": 256, + "source_row_sha256": "ed92915801eba2026e6f03096e114058c5f2d17c35ad8edc5c7d13c31a11bdad", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 1903.9726791242151, + "dynamo_tpot_ms": 72.5948197616222, + "dynamo_request_latency_ms": 68941.4886245334, + "dynamo_interactivity": 13.775087579026644, + "dynamo_tput_per_gpu_output": 1694.0922210776544, + "dynamo_tput_per_gpu_total": 15214.20095834388 + }, + "completed_requests": 2560, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "d357d764d8403f0f6f333fd6c17becf85f2ab2162a9d22caf1a55430f03cd02f", + "audit_path": "b200-output-3-reports/0008.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: MoE data missing for MoeKey { quant: \"w4a16_mxfp4\", distribution: \"power_law_1.2\", topk: 4, num_experts: 128, hidden_size: 2880, inter_size: 2880, moe_tp_size: 2, moe_ep_size: 1 } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 50, + "source_index": 70, + "config_id": 139, + "display_name": "gpt-oss-120b", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "conc": 4, + "source_row_sha256": "7703305318de4d79f478412e6c0ae61785379e02939df28bd59de670774a98c6", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 22.737700074999918, + "dynamo_tpot_ms": 3.412202753397756, + "dynamo_request_latency_ms": 3176.8283204499994, + "dynamo_interactivity": 293.06582060642023, + "dynamo_tput_per_gpu_output": 142.77493323167639, + "dynamo_tput_per_gpu_total": 285.5035748419759 + }, + "completed_requests": 40, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "436dab1b082b3d368a9d98ca27f0b3f79d42020e3c6bbe813332aa95b124919e", + "audit_path": "b200-output-0-reports/0009.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: MoE data missing for MoeKey { quant: \"w4a16_mxfp4\", distribution: \"power_law_1.2\", topk: 4, num_experts: 128, hidden_size: 2880, inter_size: 2880, moe_tp_size: 8, moe_ep_size: 1 } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 51, + "source_index": 71, + "config_id": 139, + "display_name": "gpt-oss-120b", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "conc": 4, + "source_row_sha256": "3f76e1372edf4b41f69b0a50b1accf9ed673f71607c1d91e6b918a0fed61c338", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 68.95015359999982, + "dynamo_tpot_ms": 3.527819314867833, + "dynamo_request_latency_ms": 3330.6307212500024, + "dynamo_interactivity": 283.4612293734959, + "dynamo_tput_per_gpu_output": 136.30801173050614, + "dynamo_tput_per_gpu_total": 1227.0777867226905 + }, + "completed_requests": 40, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "c39445a8f611fc1217480e410473df29c2dcdcc0aea206ec944c16838cfe108b", + "audit_path": "b200-output-1-reports/0009.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: MoE data missing for MoeKey { quant: \"w4a16_mxfp4\", distribution: \"power_law_1.2\", topk: 4, num_experts: 128, hidden_size: 2880, inter_size: 2880, moe_tp_size: 8, moe_ep_size: 1 } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 52, + "source_index": 72, + "config_id": 133, + "display_name": "gpt-oss-120b", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "conc": 4, + "source_row_sha256": "17575b05a7f7208d17103f3c548c6c146caf231d75a87f2038eb6f4fc60cd5e8", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 96.8023692000004, + "dynamo_tpot_ms": 4.459698827044163, + "dynamo_request_latency_ms": 4219.972681899993, + "dynamo_interactivity": 224.23038836969815, + "dynamo_tput_per_gpu_output": 215.82619326745953, + "dynamo_tput_per_gpu_total": 1942.9197461629974 + }, + "completed_requests": 40, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "7c44d55afcb285331abf386e19b944d88c6059f84f81b1737ec2429f017d524b", + "audit_path": "b200-output-2-reports/0009.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: MoE data missing for MoeKey { quant: \"w4a16_mxfp4\", distribution: \"power_law_1.2\", topk: 4, num_experts: 128, hidden_size: 2880, inter_size: 2880, moe_tp_size: 4, moe_ep_size: 1 } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 53, + "source_index": 73, + "config_id": 133, + "display_name": "gpt-oss-120b", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "conc": 8, + "source_row_sha256": "5d5b06f10221b69c503ed36b3c8a037adb929e5ecd1cee8447bd98cf61d51722", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 115.01224456249966, + "dynamo_tpot_ms": 5.774134161940712, + "dynamo_request_latency_ms": 5455.979925150004, + "dynamo_interactivity": 173.18613872731623, + "dynamo_tput_per_gpu_output": 335.3783592694484, + "dynamo_tput_per_gpu_total": 3009.6387715614196 + }, + "completed_requests": 80, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "559fa24b66fab3f4a01800b1a8e9a58700648429a521327559735491930428e2", + "audit_path": "b200-output-3-reports/0009.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: MoE data missing for MoeKey { quant: \"w4a16_mxfp4\", distribution: \"power_law_1.2\", topk: 4, num_experts: 128, hidden_size: 2880, inter_size: 2880, moe_tp_size: 4, moe_ep_size: 1 } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 54, + "source_index": 74, + "config_id": 133, + "display_name": "gpt-oss-120b", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "conc": 32, + "source_row_sha256": "6056b73c4e634e3ecc45ed471513f469928d3784303eb09536c81d3b5dd6b903", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 207.61326144687501, + "dynamo_tpot_ms": 10.278810806609261, + "dynamo_request_latency_ms": 9645.263760312415, + "dynamo_interactivity": 97.28751883992274, + "dynamo_tput_per_gpu_output": 748.0612247438173, + "dynamo_tput_per_gpu_total": 6758.093533324546 + }, + "completed_requests": 320, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "906872e10cdba8a8cbefb15593c7a4716ce064da42f0c0c6d2b20169d4a20ea3", + "audit_path": "b200-output-0-reports/0010.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: MoE data missing for MoeKey { quant: \"w4a16_mxfp4\", distribution: \"power_law_1.2\", topk: 4, num_experts: 128, hidden_size: 2880, inter_size: 2880, moe_tp_size: 4, moe_ep_size: 1 } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 55, + "source_index": 75, + "config_id": 133, + "display_name": "gpt-oss-120b", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "conc": 4, + "source_row_sha256": "e19696ee3268071fd645ea5f3b120da7c8276cdad263219244e881a5d90aca6f", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 31.156744500000297, + "dynamo_tpot_ms": 4.238479012539097, + "dynamo_request_latency_ms": 3948.8256388999953, + "dynamo_interactivity": 235.93369155340974, + "dynamo_tput_per_gpu_output": 230.43709150642184, + "dynamo_tput_per_gpu_total": 460.799468871154 + }, + "completed_requests": 40, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "e22507ef646202acc0131f3256b8f77d066f8412a912dc4504047ffad6e4516f", + "audit_path": "b200-output-1-reports/0010.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: MoE data missing for MoeKey { quant: \"w4a16_mxfp4\", distribution: \"power_law_1.2\", topk: 4, num_experts: 128, hidden_size: 2880, inter_size: 2880, moe_tp_size: 4, moe_ep_size: 1 } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 56, + "source_index": 76, + "config_id": 133, + "display_name": "gpt-oss-120b", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "conc": 16, + "source_row_sha256": "ffbbd03a7f98d2adaa6e259e26e8d34b35057b88c41137086dc32fc913566934", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 147.9352771125, + "dynamo_tpot_ms": 7.533136417768352, + "dynamo_request_latency_ms": 7076.399452525016, + "dynamo_interactivity": 132.7468327324204, + "dynamo_tput_per_gpu_output": 512.0576330334668, + "dynamo_tput_per_gpu_total": 4625.648408760085 + }, + "completed_requests": 160, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "0aea753400ece3dccdfd849e0d658650bcc317ecb1de1b5f78609cebb370d0ed", + "audit_path": "b200-output-2-reports/0010.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: MoE data missing for MoeKey { quant: \"w4a16_mxfp4\", distribution: \"power_law_1.2\", topk: 4, num_experts: 128, hidden_size: 2880, inter_size: 2880, moe_tp_size: 4, moe_ep_size: 1 } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 80, + "source_index": 100, + "config_id": 183, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "conc": 64, + "source_row_sha256": "b1b6e3aa693564278b2758d04cae3e87fb0022a397906de4e5fc0e9c08a546f3", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 2285.250759109376, + "dynamo_tpot_ms": 122.62922839486582, + "dynamo_request_latency_ms": 115132.35003878936, + "dynamo_interactivity": 8.15466274304526, + "dynamo_tput_per_gpu_output": 63.30104659657805, + "dynamo_tput_per_gpu_total": 570.8838054400934 + }, + "completed_requests": 640, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "74bbe3fd6de1f969c38c22cea57cbc206d238f4cc7a293ebd832f12554da45b8", + "audit_path": "h200-output-reports/0001.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/h200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 81, + "source_index": 101, + "config_id": 191, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "conc": 128, + "source_row_sha256": "5b2b794d953cb7a0107619d2752a3644ddb75873bb1f3810a960f55cdb28a768", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 1254.5940279749964, + "dynamo_tpot_ms": 94.04044973903737, + "dynamo_request_latency_ms": 88186.65404354926, + "dynamo_interactivity": 10.633722007657386, + "dynamo_tput_per_gpu_output": 166.12587583438452, + "dynamo_tput_per_gpu_total": 1492.2227376344872 + }, + "completed_requests": 1280, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "14eabc394f1649258551e8a50a43eea6246d69de1fc7c704f68e04593690d0a6", + "audit_path": "b200-output-3-reports/0010.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 82, + "source_index": 102, + "config_id": 191, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "conc": 256, + "source_row_sha256": "3afa3764e03dc9c53b72f5f164807bbe4d42552eab72142b1db2b0315f00b4a7", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 1867.4909992453133, + "dynamo_tpot_ms": 157.22783261618162, + "dynamo_request_latency_ms": 147328.62172637047, + "dynamo_interactivity": 6.360197067914563, + "dynamo_tput_per_gpu_output": 199.25430217880975, + "dynamo_tput_per_gpu_total": 1789.4509858705205 + }, + "completed_requests": 2560, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "c402c8f1781b1fc5e04995820e4b47321b86152142a52bb530189fabaccbd045", + "audit_path": "b200-output-0-reports/0011.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 83, + "source_index": 103, + "config_id": 191, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "conc": 256, + "source_row_sha256": "1dbe4d7ce158fb0d5b920cd22150e79e92a1c29408ab8162fa3bb838949c9b1e", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 271.7718375367169, + "dynamo_tpot_ms": 44.503371063312294, + "dynamo_request_latency_ms": 41360.0378325022, + "dynamo_interactivity": 22.47020789902319, + "dynamo_tput_per_gpu_output": 700.0693046827643, + "dynamo_tput_per_gpu_total": 1398.3372051519734 + }, + "completed_requests": 2560, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "fdef0811cc8110cd54ce5bf6ab64c59f36c3e2e16d19713e3a64d35330b2abd1", + "audit_path": "b200-output-1-reports/0011.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 84, + "source_index": 104, + "config_id": 191, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "conc": 128, + "source_row_sha256": "163afb9fe500fc8c2a8dd1de9d825d2c45be59c935591ae5872de7d041db0124", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 200.01187842968827, + "dynamo_tpot_ms": 31.05881124551152, + "dynamo_request_latency_ms": 28845.1914382611, + "dynamo_interactivity": 32.19698243101676, + "dynamo_tput_per_gpu_output": 500.49733080937847, + "dynamo_tput_per_gpu_total": 999.779299894592 + }, + "completed_requests": 1280, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "1b01476f8b830dfc79e80ddec344b2e5aa3e6415aeae6234a9bde3f335d7feaa", + "audit_path": "b200-output-2-reports/0011.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 85, + "source_index": 105, + "config_id": 186, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "conc": 32, + "source_row_sha256": "a582de33008a85a22e3c270a8b57b9d6e7bd2bc0f56321259f228f8eee708498", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 523.595259462501, + "dynamo_tpot_ms": 21.98830108507814, + "dynamo_request_latency_ms": 20715.195352103212, + "dynamo_interactivity": 45.478729626757165, + "dynamo_tput_per_gpu_output": 348.8122389657923, + "dynamo_tput_per_gpu_total": 3151.220325991987 + }, + "completed_requests": 320, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "9d6a5e0f6ea73e6b95ff7bcbfce508ca0f183912836cf25b37486edd50ffb062", + "audit_path": "b200-output-3-reports/0011.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 86, + "source_index": 106, + "config_id": 181, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "conc": 4, + "source_row_sha256": "b974a1baf5483e027fe0b43935c03315463220a05797cafed93e2c75ccbfcae9", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 336.1681141250013, + "dynamo_tpot_ms": 11.630069742041902, + "dynamo_request_latency_ms": 11090.645464900019, + "dynamo_interactivity": 85.98400716249093, + "dynamo_tput_per_gpu_output": 41.091968339057324, + "dynamo_tput_per_gpu_total": 369.91986693533806 + }, + "completed_requests": 40, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "cefe865d6a0237f47d53f79f42c19149a4344c38d88fdbbe2df886b31f241123", + "audit_path": "h200-output-reports/0002.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/h200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 87, + "source_index": 107, + "config_id": 181, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "conc": 32, + "source_row_sha256": "f87f3df4d0c66d676ecdf1b062653057acdf4bf024d6e1265dd1b5e04b2e715d", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 145.42489761249956, + "dynamo_tpot_ms": 23.496811877106445, + "dynamo_request_latency_ms": 21702.262365746774, + "dynamo_interactivity": 42.55896524303903, + "dynamo_tput_per_gpu_output": 164.88507413421178, + "dynamo_tput_per_gpu_total": 330.43962351020815 + }, + "completed_requests": 320, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "7f343cf93439fb5d55aa5a18df21e37e324107df66d9d07da3c1632a128e8967", + "audit_path": "h200-output-reports/0003.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/h200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 88, + "source_index": 108, + "config_id": 181, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "conc": 8, + "source_row_sha256": "1772159067836e24014357d8d4a453babeee06d73afdf6f5ce2e116850be156e", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 414.4809144374999, + "dynamo_tpot_ms": 16.734132112828654, + "dynamo_request_latency_ms": 15899.036504050013, + "dynamo_interactivity": 59.75810357283985, + "dynamo_tput_per_gpu_output": 57.627521981538365, + "dynamo_tput_per_gpu_total": 517.1413708458837 + }, + "completed_requests": 80, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "b50260f204dcb2b87cfee8a9b377a516739a26cc7268bc76a6956090a1f50c88", + "audit_path": "h200-output-reports/0004.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/h200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 89, + "source_index": 109, + "config_id": 181, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "conc": 16, + "source_row_sha256": "33db1b19c4cf1079c344874967a823f8352f49fe229acd1d010032b7726d6823", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 524.0097046500005, + "dynamo_tpot_ms": 22.229874807483224, + "dynamo_request_latency_ms": 20973.601796724997, + "dynamo_interactivity": 44.98450884947723, + "dynamo_tput_per_gpu_output": 86.50959745071471, + "dynamo_tput_per_gpu_total": 781.4803568492463 + }, + "completed_requests": 160, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "c89b364dd3961b7338994b7a79ceb996ba1e5a1edfb458f395b139b785ce3138", + "audit_path": "h200-output-reports/0005.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/h200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 90, + "source_index": 110, + "config_id": 181, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "conc": 32, + "source_row_sha256": "24c7090c63661ff0e32ac40f60ded34348eb51d61e011ce24c0288730c81631b", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 774.0153813343748, + "dynamo_tpot_ms": 32.85620264412994, + "dynamo_request_latency_ms": 30944.734562596735, + "dynamo_interactivity": 30.435653530358877, + "dynamo_tput_per_gpu_output": 116.89089025030079, + "dynamo_tput_per_gpu_total": 1056.0092454673588 + }, + "completed_requests": 320, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "3e11aa83cf6f5bab19a1f0aeb951ed6483cf4ed9975d3080d617fd9a18da8f08", + "audit_path": "h200-output-reports/0006.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/h200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 91, + "source_index": 111, + "config_id": 181, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "conc": 64, + "source_row_sha256": "2b1049c5eafbedf75a35ef8d6adeedb74e79add61377d82cb65f9d273d4a4ae1", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 206.9225214890625, + "dynamo_tpot_ms": 30.42413382587153, + "dynamo_request_latency_ms": 28128.260946198418, + "dynamo_interactivity": 32.86864321999655, + "dynamo_tput_per_gpu_output": 254.91223002158725, + "dynamo_tput_per_gpu_total": 510.3119758825058 + }, + "completed_requests": 640, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "6e9f7efb4cc4f2a4eaec4b5723692d195c572ef220d5dae0d94b986bbec72bce", + "audit_path": "h200-output-reports/0007.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/h200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 92, + "source_index": 112, + "config_id": 181, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "conc": 4, + "source_row_sha256": "89ef3e2af32129448883842277c7deb1fae7d1006dcfdc34861887edda95ffe6", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 79.24857170000087, + "dynamo_tpot_ms": 10.445483570473504, + "dynamo_request_latency_ms": 9734.042903925043, + "dynamo_interactivity": 95.73515608475262, + "dynamo_tput_per_gpu_output": 46.76519281596531, + "dynamo_tput_per_gpu_total": 93.51522305016326 + }, + "completed_requests": 40, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "b0208e0c6cf88df36edcbaf8f190e4dfd51462154bbc05e5a27b1fd003e390cc", + "audit_path": "h200-output-reports/0008.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/h200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 93, + "source_index": 113, + "config_id": 181, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "conc": 16, + "source_row_sha256": "ed97e731376f837bc3f4ad978b3ed009188b94d14eb4bd63cb9f0bbe5bd77a95", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 110.75285956874973, + "dynamo_tpot_ms": 17.578949364323005, + "dynamo_request_latency_ms": 16253.743427643762, + "dynamo_interactivity": 56.8862210860866, + "dynamo_tput_per_gpu_output": 110.86276671112925, + "dynamo_tput_per_gpu_total": 222.42897163271186 + }, + "completed_requests": 160, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "aa6a6adc3552814c1cfe025ac3380829bd879949e03bd1bdc39760b2c72d9213", + "audit_path": "h200-output-reports/0009.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/h200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 94, + "source_index": 114, + "config_id": 181, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "conc": 8, + "source_row_sha256": "a19f21e2eae77dccd5f2df461968d379e10a4015ee0db1f1e3a402875b873b67", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 95.35389578749951, + "dynamo_tpot_ms": 14.422718780066884, + "dynamo_request_latency_ms": 13416.370537149993, + "dynamo_interactivity": 69.33505500932762, + "dynamo_tput_per_gpu_output": 67.84281368976573, + "dynamo_tput_per_gpu_total": 135.72690266670838 + }, + "completed_requests": 80, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "303f65f3fc412abd6d471d0e4d6661c58ae3bc4d2a0fbb93de0937be098c7f92", + "audit_path": "h200-output-reports/0010.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/h200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 95, + "source_index": 115, + "config_id": 192, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "conc": 256, + "source_row_sha256": "c068e8e2e90e70166c80283e657a0892825df1226c88033707463a7ae142db88", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 2348.8296846347675, + "dynamo_tpot_ms": 140.0112185163047, + "dynamo_request_latency_ms": 131812.36493323566, + "dynamo_interactivity": 7.14228481543818, + "dynamo_tput_per_gpu_output": 444.5193767228089, + "dynamo_tput_per_gpu_total": 3992.1127334121547 + }, + "completed_requests": 2560, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "557e297f67ea87adf79b4f210aa9e21095f12aeea2f1ef76024df0188fe3d02d", + "audit_path": "b200-output-0-reports/0012.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 96, + "source_index": 116, + "config_id": 192, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "conc": 256, + "source_row_sha256": "df3b84d7f35e1d7472218a1f881bc32e5545b07fdaa7cc80e27045d8c78055ad", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 336.1148590371094, + "dynamo_tpot_ms": 54.91752841189877, + "dynamo_request_latency_ms": 51020.76521302083, + "dynamo_interactivity": 18.209122459038667, + "dynamo_tput_per_gpu_output": 1135.744492527404, + "dynamo_tput_per_gpu_total": 2268.5665102359926 + }, + "completed_requests": 2560, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "4ef18de97ad9c81084876d604b1caef25a9b8668037cf5442b13fee0fa79da54", + "audit_path": "b200-output-1-reports/0012.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 97, + "source_index": 117, + "config_id": 178, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "conc": 64, + "source_row_sha256": "bb0bb2eeb73527959b15ea6c325c71856703ac7b8b4f5c349e31c7935f03d782", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 113.17266760312495, + "dynamo_tpot_ms": 15.062078526833583, + "dynamo_request_latency_ms": 13938.789472257698, + "dynamo_interactivity": 66.39189924673859, + "dynamo_tput_per_gpu_output": 512.575279426061, + "dynamo_tput_per_gpu_total": 1026.1308514318412 + }, + "completed_requests": 640, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "e79469298e1ea967ca3b2edd7842a7e1e46f06b04d0f5d9a60a9a40230fa7ba6", + "audit_path": "b200-output-2-reports/0012.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 100, + "source_index": 145, + "config_id": 473, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "conc": 32, + "source_row_sha256": "148cd89656bf3bd88c67e38c5abfc3e2c07acb5b09d2bbf9166b55f6f78ee898", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 30731.560811728163, + "dynamo_tpot_ms": 24.313442471826082, + "dynamo_request_latency_ms": 53036.394408787564, + "dynamo_interactivity": 41.12951101674637, + "dynamo_tput_per_gpu_output": 133.092709891281, + "dynamo_tput_per_gpu_total": 1202.3788325038965 + }, + "completed_requests": 320, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "49b1ea942a1b70641967292141cc328279ec00478a76a4a7e467c02dfa8911bb", + "audit_path": "b200-output-3-reports/0012.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 101, + "source_index": 146, + "config_id": 473, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "conc": 16, + "source_row_sha256": "e574c7b3dc61ffca7463d22355e28cfbd38daa067ea6d3a92bbd7c5bffd3e385", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 4970.566047712493, + "dynamo_tpot_ms": 24.15142487617139, + "dynamo_request_latency_ms": 27166.517675368686, + "dynamo_interactivity": 41.40542452990563, + "dynamo_tput_per_gpu_output": 130.97765535888115, + "dynamo_tput_per_gpu_total": 1183.180454717171 + }, + "completed_requests": 160, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "e8f501530d21ecbd08301cd98c0bb1b327ac707940d3266f315125e4b4cd8bed", + "audit_path": "b200-output-0-reports/0013.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 102, + "source_index": 147, + "config_id": 473, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "conc": 16, + "source_row_sha256": "cd179628bb8035edab43e738e2c9fc9d4b76de6d50253dd4f4d0d830143ee697", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 105.25585718749896, + "dynamo_tpot_ms": 22.492551213127662, + "dynamo_request_latency_ms": 20758.67436385006, + "dynamo_interactivity": 44.45916297019945, + "dynamo_tput_per_gpu_output": 173.30267193565567, + "dynamo_tput_per_gpu_total": 347.7049711404995 + }, + "completed_requests": 160, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "c6b83626fd9898bd05bda42b4d61302c9131932214b22f48c51ef1d4ba4c4179", + "audit_path": "b200-output-1-reports/0013.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 103, + "source_index": 148, + "config_id": 473, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "conc": 8, + "source_row_sha256": "512115fcec50ad8101ab8570fad0b7ed444ed6121b1562835c8fafd6ff742bdd", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 329.79144853750086, + "dynamo_tpot_ms": 20.10812375391757, + "dynamo_request_latency_ms": 18925.761042225076, + "dynamo_interactivity": 49.73114410066104, + "dynamo_tput_per_gpu_output": 96.60821715265288, + "dynamo_tput_per_gpu_total": 866.9487101892913 + }, + "completed_requests": 80, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "9582ef2d1031233916093d127d556692b195779be997dc45d55ddb95e75d1012", + "audit_path": "b200-output-2-reports/0013.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + }, + { + "selected_index": 104, + "source_index": 149, + "config_id": 473, + "display_name": "DeepSeek-R1-0528", + "aic_backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "conc": 8, + "source_row_sha256": "fe0d68a4e3f8c0007cd2a9cd2e786d4234fd21526e17345835db9a1989fa57c0", + "status": "success", + "metrics": { + "dynamo_ttft_ms": 91.19394122499946, + "dynamo_tpot_ms": 18.12601714198523, + "dynamo_request_latency_ms": 16832.41006168745, + "dynamo_interactivity": 55.169317791480154, + "dynamo_tput_per_gpu_output": 107.905702026951, + "dynamo_tput_per_gpu_total": 215.8770534365994 + }, + "completed_requests": 80, + "truncated_requests": 0, + "context_violations": 0, + "spec_sha256": "83eb28ff9d9d0c98417acfc5ca21ded346a02bd2947fd7d30afbef4ba267e9d3", + "audit_path": "b200-output-3-reports/0013.audit.json", + "baseline_error": "AISimulate replay failed: engine error: generalized engine worker 0 is now poisoned because a rank may have been partially mutated: executing attention-DP rank 0: external prefill prediction failed: AIC prefill prediction failed: PerfDataNotAvailableError: perf database error: context MLA data missing for ContextKey { fmha_quant: \"fp8\", kv_quant: \"fp8\" } at aiconfigurator_core/systems/data/b200_sxm/trtllm/1.3.0rc20" + } + ] +} diff --git a/python/aisimulate/collector/trtllm/gym-profiles-20260917.md b/python/aisimulate/collector/trtllm/gym-profiles-20260917.md new file mode 100644 index 000000000..76e8ee6e4 --- /dev/null +++ b/python/aisimulate/collector/trtllm/gym-profiles-20260917.md @@ -0,0 +1,270 @@ +# TRT-LLM profiles for the 62 remaining Gym failures + +## Scope + +The `max_model_len` replay at AISimulate `89d2051137b772944a3e303c452108811790bef4` +left 62 missing-profile failures: 31 DeepSeek-R1 points on B200, 10 on H200, +and 21 GPT-OSS-120B points on B200. All use the original TRT-LLM database +query version `1.3.0rc20`. + +This campaign measures TRT-LLM directly. It does not borrow SGLang timings. +The existing row schemas and consumer keys are unchanged. + +The refreshed context MLA tables and their `op_backend_facts.yaml` entries +cover B200 and H200 only. The B300, GB200, GB300, H100 and RTX Pro 6000 +Server rc20 tables retain their historical labels and have not been remeasured +or certified by this campaign. The facts registry joins persisted table keys; +it is not a prediction of the current collector's output on every system. +The source-backed collector correction applies to future runs on the listed +SMs. Refreshing each remaining system requires matching collection evidence +and an accompanying table/facts update; rewriting old labels alone would not +establish that evidence. `tools/perf_database/backend_facts.py --check` checks +all persisted slices, including the two refreshed systems. + +## Results + +| Production run | Planned / passed cases | Failed / unattempted | Measured rows | +| --- | ---: | ---: | ---: | +| B200, job `1993260`, 4 GPUs | 3,083 / 3,083 | 0 / 0 | 3,080 MLA + 81 MoE | +| H200, job `4375636`, 2 GPUs | 3,080 / 3,080 | 0 / 0 | 3,080 MLA | + +Both jobs completed with exit `0:0`. Each MoE case produces 27 token-count +rows. All new physical keys are unique and all latencies are finite and positive. +See [collection evidence](gym-profiles-20260917-collection.json) for plans, +source/image/raw/parquet hashes, runtime kernel names, and observed telemetry. + +Across overlapping MLA shapes, new/old median latency ratios are 0.993–0.996. +Six H200 shapes have much larger old timings. Job `4375895` repeated each +three times on another H200 node (`viking-prod-216`): all 18 passed, within +2.2% of the new production measurements. The cause of the old values is not +established. These diagnostic repeats remain separate from the published rows. + +- Reproduced all 62 original missing-profile failures before replacing data. +- Recovered all 62 points, and reran the other 201 points: **263/263 passed**. +- Completed **293,180 requests**, including 35,520 in the recovered points, + with zero truncated requests and zero context-limit violations. +- All 62 recovered native request specs are byte-identical to the failing + baseline. Original workload fields are unchanged across all 263 points. +- Checked 1,773 installed package files: only the three published parquet + files differ; no source or native binary changed, and no files were added + or removed (excluding Python bytecode caches). + +The replay uses the frozen `89d2051137b772944a3e303c452108811790bef4` build from +[PR #261](https://github.com/ai-dynamo/aisimulate/pull/261), with the measured +data overlaid. [Per-point replay evidence](gym-profiles-20260917-replay.json) +includes metrics, native spec hashes, original errors, and package hashes. +This verifies prediction availability and request completion; it does not +establish matched-silicon latency accuracy. Original silicon recipe image +versions are retained; the collector and database query both use `1.3.0rc20`. + +Validation: 44 focused collector tests, Ruff check/format for changed Python +files, all seven collector-data checks, and the backend-facts registry check +passed. The latter matches 1,922 fact slices; its existing unrelated vLLM/FPM +warnings remain. + +## Collector corrections + +- Context MLA with FP8 KV internally quantizes Q/K/V to FP8 on + SM90/100/103/120. The Python inputs remain BF16; logging their dtype as + compute precision mislabels the measurements. Record FP8/FP8 for that path + and BF16/BF16 for the control. The SM list mirrors the upstream + `mFP8ContextMLA` condition, whose false branch leaves context compute BF16; + it is not a list of the only architectures audited by this campaign. + Generation tables are outside this refresh. +- Blackwell MXFP4 MoE pads weights before TP sharding through its native + weight loader. Do not reject the logical intermediate dimension using + the CUTLASS plugin's physical-weight alignment rule. Keep the original + GPT-OSS shape (`hidden=2880`, `intermediate=2880`, 128 experts, top-k 4) + and record TP=2/4/8, EP=1 under their original keys. + +The measured runtime sources match TensorRT-LLM commit +[`c25c23f71786bad54d192893d696ce8043426eca`](https://github.com/NVIDIA/TensorRT-LLM/tree/c25c23f71786bad54d192893d696ce8043426eca): + +- [MLA precision dispatch](https://github.com/NVIDIA/TensorRT-LLM/blob/c25c23f71786bad54d192893d696ce8043426eca/cpp/tensorrt_llm/thop/attentionOp.cpp#L1228-L1231). +- [MXFP4 weight creation and padding](https://github.com/NVIDIA/TensorRT-LLM/blob/c25c23f71786bad54d192893d696ce8043426eca/tensorrt_llm/_torch/modules/fused_moe/quantization.py#L5634-L5735). + +## Reproduction + +- OCI index: `sha256:1532b38814b3faf2affdb5ef01ca91468685d314ffb7e8926a0567595355ed88`. +- Pinned amd64 image: `nvcr.io/nvidia/tensorrt-llm/release@sha256:9b3b4dfb811caa9420fa99a6f958155f6a1f727ffc2b5a5c2d9d2ce51fdc323d`. +- Source snapshot: `c0317fccb25385093f9e8948675f765e4a6a0e96` plus the two + collector corrections above; the source archive hash identifies the exact bytes. +- MLA uses the full existing `get_context_mla_test_cases()` grid. +- MoE selects the three matching `get_moe_test_cases()` tuples at runtime, + preserving all 27 canonical token counts, through 16,384 tokens, and the + `power_law_1.2` distribution. +- Both MLA precision slices are recollected; the fresh grids preserve every + previously stored KV-storage/shape combination. Other GPUs are outside + this data refresh. +- The 81 new MoE rows are finalized separately and appended using the old + Arrow schema. All 218,295 existing rows retain their exact values, including + power fields. New rows have null power fields; node telemetry is separate. +- Production invokes `run_mla` / `run_moe_torch` without profiler wrappers. + MoE is the last case in each worker; `TRTLLM_MOE_RESTART_WORKER=0` lets the + worker write its outcome before exiting normally. +- Smoke runs are separate. The original smoke wrapper labeled MoE exit 10 + as a crash; the collector explicitly uses that code to request worker + recycling after finishing. The smoke CSVs retain all three expected token + counts. No failed measurement is treated as a successful production case. + +Artifact location aliases below are resolved through the campaign owner's +access-controlled retention manifest. Measurement hashes and artifact filenames +are unchanged. Artifacts are retained under `results/trt-62-20260917` on: + +- B200: `${B200_CLUSTER}`, storage root + `${B200_ARTIFACT_ROOT}`. +- H200: `${H200_CLUSTER}`, storage root `${H200_ARTIFACT_ROOT}`. +- Initial H200 preparation and failed launch: `${H200_PREPARATION_CLUSTER}`, + storage root `${H200_PREPARATION_ARTIFACT_ROOT}`. + +Local evidence, runners, source archives, native replay reports, and package +hashes are under `${CAMPAIGN_ARTIFACT_ROOT}/`. + +## Infrastructure retries + +- Aria image job `8899` failed registry authentication; a separate public + Enroot configuration succeeded in job `8900`. Smoke job `8905` then failed + before the runner started on `neb-cdg-slurm-1-gpu-11` (signal 53), and its + dependent production job `8908` did not run. +- ComputeLab import job `4374730` hit unsupported overlay whiteout operations + on NFS. It and its pending dependent GPU job `4374761` were cancelled. + The completed B200 squashfs was transferred in eight parallel streams + through the user's desktop, preserving SSH host verification. CPU job + `4375480` verified its full SHA-256 before GPU use (exit `0:0`). +- ComputeLab job `4375492` failed during MPI initialization before smoke + measurements. B200 defaults to `pmix`; ComputeLab has no MPI default. + Retry `4375636` explicitly selects `srun --mpi=pmix` with the same image + and collector source. +- These are preparation/launch failures, separate from measured case results. + Their logs remain in each cluster's `logs/trt62-*` paths. + +## GPT-OSS TPOT follow-up + +The coverage replay exposed a large GPT-OSS/B200 TPOT gap. An independent +same-input graph benchmark confirmed that explicit MXFP4 autotuning lowers +TP1/tokens256 MoE latency from 1.7934 to 1.3227 ms (-26.25%). This is an +operator measurement, not an E2E accuracy result. + +The collector now matches native GPT-OSS expert bias and routing-weight dtype, +and warms TRTLLMGen MXFP4 tactics instead of skipping them. GPT-OSS cache names +are separate from the earlier unbiased configuration. Native source evidence: + +- [GPT-OSS bias and routing dtype, rc14](https://github.com/NVIDIA/TensorRT-LLM/blob/93cb6518b6d6dbd6095748189e626db731f44545/tensorrt_llm/_torch/models/modeling_gpt_oss.py#L159-L190). +- [Serving autotuner warmup, rc14](https://github.com/NVIDIA/TensorRT-LLM/blob/93cb6518b6d6dbd6095748189e626db731f44545/tensorrt_llm/_torch/pyexecutor/model_engine.py#L827-L842). +- [Autotuner enabled by default](https://github.com/NVIDIA/TensorRT-LLM/blob/93cb6518b6d6dbd6095748189e626db731f44545/tensorrt_llm/llmapi/llm_args.py#L3715-L3719). + +The original coverage and collection results above remain historical evidence. +The follow-up refresh is scoped to GPT-OSS-120B/B200 MXFP4; other hardware +corpora are not implicitly recollected by this source change. Source-level +parity with the recipe does not replace an original serving kernel trace. + +### Refreshed B200 profiles and replay + +- Source `3830b5a4`; B200 job `1994597` completed `0:0` in 1m46s on two GPUs. + All six canonical TP/EP cases passed: `(1,1)`, `(2,1)`, `(4,1)`, `(8,1)`, + `(1,2)`, `(1,4)`, each with 27 token counts. All 162 timings used CUDA graphs. +- Replaced exactly those 162 MXFP4/power_law_1.2 rows. The other **218,214** + MoE rows retain their exact values. Shape identities and Arrow schema are + unchanged. This supersedes the earlier 81-row append for these GPT-OSS keys. +- Reran all **263 native specs**, with byte-identical spec serialization and + the frozen runtime. Only one of 1,773 installed package files changed: + B200/TRT-LLM rc20 `moe_perf.parquet`. All **293,180 requests** completed with + identical sampled lengths, zero truncations, and zero context violations. +- All 232 points outside GPT-OSS/B200 retain exactly the same TTFT and TPOT. + Among the 31 affected points, nine have worse TPOT APE; the largest increase + is config283/ISL1024/concurrency64, **191.71% to 200.14%**. No rows were + selected or discarded based on improved error. + +| Backend / subset | Points | TTFT MAPE before / after | TPOT MAPE before / after | +| --- | ---: | ---: | ---: | +| TRT-LLM, all | 166 | 66.25% / 66.00% | 68.16% / 66.72% | +| TRT-LLM, GPT-OSS/B200 | 31 | 86.99% / 85.67% | 176.52% / 168.78% | +| SGLang, all | 97 | 69.61% / 69.61% | 12.55% / 12.55% | + +MAPE is the unweighted mean of per-point absolute percentage errors against +positive silicon measurements. This is a data-only E2E ablation; it does not +claim exact runtime-version parity or that the remaining large gap is fixed. +Silicon uses rc14/rc18, while these profiles use rc20. + +The controlled four-arm job `1994556` keeps routing hashes identical. At +TP1/tokens256, baseline/tuned/bias+tuned/full-contract medians are +1.82016/1.34733/1.34962/1.34993 ms. Kernel traces show a `128x8` to `128x16` +Bfloat16/MXFP4 GEMM tactic change. Bias and routing dtype fix correctness but +have negligible timing impact in this experiment. + +[Follow-up evidence](gym-tpot-20260917.json) contains each paired point, +source/data hashes, coverage, and validation provenance. Raw reports, scripts, +and diagnostics remain under `trt-62-20260917/tpot-fix` in the local cache and +B200 storage. Validation: 52 focused tests, Ruff, and seven data checks passed. + +An independent EP4 check (`1994626`, one B200, exit `0:0`) also shows that +alignment need not improve every shape. With identical routing hashes, +baseline/full-contract TP1/EP4 latencies at tokens160/192/256 are +0.29861/0.33427, 0.30739/0.34978, and 0.35100/0.42240 ms. The slowdown is +reproducible in direction, but does not isolate the full production change +(up to +51.3%): production uses balanced maximum-shape warmup, while this +controlled probe tunes the measured routing inputs. Neither result is used +to discard higher timings or select a better MAPE. + +Original GPT-OSS config131 server artifacts from InferenceX run +`26016885799` are expired. Source-level recipe alignment and new same-version +microbenchmarks cannot establish its historical executed kernel/tactic. + +The pinned original [GPT-OSS/B200 launch script](https://github.com/SemiAnalysisAI/InferenceX/blob/2baba8e27be8529b4453afc953f9859ec092c73d/benchmarks/single_node/gptoss_fp4_b200_trt.sh) +sets `TRTLLM_ENABLE_PDL=1`. The native environment helper defaults PDL to true +in both [rc14](https://github.com/NVIDIA/TensorRT-LLM/blob/93cb6518b6d6dbd6095748189e626db731f44545/cpp/tensorrt_llm/common/envUtils.cpp#L249-L275) +and [rc20](https://github.com/NVIDIA/TensorRT-LLM/blob/c25c23f71786bad54d192893d696ce8043426eca/cpp/tensorrt_llm/common/envUtils.cpp#L234-L260). +No PDL-default mismatch was identified. This source check does not establish +full serving-environment parity or a historical executed-kernel match. + +### rc14 runtime cross-check + +CPU image import `1994625` and one-B200 probe `1994650` both completed `0:0`. +The latter produced 18/18 graph timings (two arms, three shapes, three repeats) +in 1m33s. Its pinned amd64 image is +`sha256:fe2f17d0c9698bafeb9f437929003c6badc56d17e8382a02505233145a07f61f`. +The runner verifies runtime version `1.3.0rc14` and SM100. Routing hashes match +both arms and the rc20 probe. Each timed closure contains five forwards; +the following medians divide its elapsed time by five, like the collector. + +| Tokens | rc14 baseline / aligned (ms) | rc20 baseline / aligned (ms) | +| ---: | ---: | ---: | +| 64 | 0.95476 / 0.93937 | 0.89840 / 0.89107 | +| 256 | 1.92882 / 1.40442 | 1.82016 / 1.34993 | +| 16,384 | 12.52036 / 12.56028 | 11.35655 / 11.26267 | + +At 256 tokens, rc14 also switches the native BF16/MXFP4 GEMM tactic from +`128x8` to `128x16`. The aligned rc14 timing is 4.04% higher than rc20; +the other two shapes are 5.42% and 11.52% higher. These observations do not +support the runtime version alone explaining the original roughly 5x E2E +overprediction. Different B200 nodes and image dependencies prevent treating +this as an isolated version-only effect. The tag's resolved image is not +proof of the historical May serving image digest. No rc14 timings replace +rc20 production data. + +The remaining attribution requires a matched full-model serving trace: +actual expert routing and decode shapes, MoE tactics, and per-step timing. +The expired original server artifacts prevent checking those historical facts. +A new matched serving run can test the mechanism but cannot recover them. + +## Full serving reproduction: activation mapping correction + +The [GPT-OSS/B200 reproduction](gym-tpot-serving-20260917.md) supersedes the +assumption that W4A16 was the effective serving precision. The real rc14 server +selects **W4A8 MXFP4/MXFP8** by its Blackwell runtime default. Source, live model +metadata, and executed GPU kernels agree. The earlier W4A16 collector ablation +remains valid for that operator, but did not establish full serving parity. + +At config131/ISL1024/OSL1024/C256, uninstrumented TPOT is **16.54925 ms** +(historical **16.79212 ms**). Replaying the exact client lengths changes TPOT +**65.16758 → 17.17212 ms** when only `aic_moe_dtype` changes from W4A16 to W4A8; +APE falls **293.78% → 3.76%**. All 2,560 requests complete in both arms. +This is a one-point verification, not a replacement 31/263-point MAPE. +The automatic Gym mapping and missing W4A8 TP2/4/8 profiles still need correction; +the earlier W4A16 timings must not be relabeled as W4A8. + +## Full correction follow-up + +The mapping, W4A8 collection, and complete 263-point rerun are recorded in +[the 2026-09-18 report](gym-w4a8-20260918.md). Earlier metrics above remain +historical ablations. diff --git a/python/aisimulate/collector/trtllm/gym-residual-20260918.md b/python/aisimulate/collector/trtllm/gym-residual-20260918.md new file mode 100644 index 000000000..5ebf466c1 --- /dev/null +++ b/python/aisimulate/collector/trtllm/gym-residual-20260918.md @@ -0,0 +1,122 @@ +# TRT residual diagnosis after activation correction + +## Status + +- Review cleanup is complete: final MoE provenance, NumPy seed bounds, and + artifact-location aliases. The three changes do not alter the seed-0 + prediction results in the [263-point report](gym-w4a8-20260918.md). +- Qwen scheduling and kernel-selection contracts remain incomplete. The + evidence below supports a scheduling contribution; it does not quantify the + contribution or establish a corrected Qwen MAPE. + +## Qwen: source controls and mixed-step amplification + +- Frozen point: config1553, B200, attention DP8 / EP8, ISL8192, OSL1024, + concurrency512, 5,120 completed requests and zero readmissions. +- Historical TPOT **35.0937 ms**, replay **116.8493 ms**. Replay ITL median + **21.6066 ms**, p75 **315.7660 ms**, p90 **349.6741 ms**. + Historical TTFT **2,915.4292 ms**, replay **1,399.7996 ms**. +- Controlled native per-op probes, using the same model, topology and rc20 + data: 64 decode requests cost **21.2968 ms**; 8,192 prefill tokens plus + 63 decode requests cost **348.0735 ms**; 16,384 prefill tokens plus + 62 decode requests cost **686.1578 ms**. These are model probes, not + observed silicon batches. Their agreement with the replay tail supports + investigating prefill interruptions before treating the full gap as a + pure-decode kernel error. +- The immutable [InferenceX recipe](https://github.com/SemiAnalysisAI/InferenceX/blob/f9426a550344a64a9d9b11fc58f10cb5c61d015e/benchmarks/single_node/fixed_seq_len/qwen3.5_fp4_b200_trt.sh) + explicitly disables chunked prefill and selects the controls below. + The retained resolved source arguments verify the attention-DP point. + Non-DP values below come only from `residual-20260918/upstream/qwen-recipe.sh`; + this summary retains no resolved non-DP arguments and no non-DP readmission + counts. + The frozen engine spec does not express these controls. + +| Recipe scope | Effective source control | Replay limitation | +|---|---|---| +| Seven attention-DP points | Balance enabled, batching wait 10 iterations, timeout 500 iterations | Missing TRT cross-rank prefill waiting | +| Twelve non-DP points | Batch wait timeout 50 iterations, token threshold 0.45 | Missing TRT token-threshold batching wait | +| Seven attention-DP points | CUTEDSL MoE, low-precision combine | Kernel selection absent from standard replay identity | +| All 19 points | MAX_UTILIZATION capacity policy | Engine supports GUARANTEED_NO_EVICT only | +| All 19 points | Serving rc18 | Frozen performance data is rc20 | + +- Pinned [TRT rc18 scheduling implementation](https://github.com/NVIDIA/TensorRT-LLM/blob/15d06c0923b63ac1781784d5f59e1747bb47d5f1/tensorrt_llm/_torch/pyexecutor/py_executor.py#L3682) + confirms that balanced DP prefill can wait for context requests on all + ranks, then apply the configured batching delay while generation continues. + Non-DP batching uses a token threshold and iteration timeout. A fixed vLLM + prefill cadence is not an equivalent implementation. +- Next discriminating check: capture matched rc18 per-rank batch composition + and CUTEDSL timings, then implement the supported scheduling contract and + rerun all 19 points. The capacity-policy difference is a confirmed config + mismatch; its numerical effect is not established by the attention-DP + point's zero readmissions. The JSON records that readmission total from all 5,120 + `per_request[].readmission_count` values in the hashed replay artifact. + +## GPT-OSS: matched TTFT reproduction and timing boundaries + +- Slurm job **1996181**, two B200 GPUs, completed `0:0` in 9m27s. + Config155, TP2/EP1, ISL1024/OSL1024, concurrency4; each repetition has + eight warmup and 40 measured requests, NumPy seed0 and length ratio0.8. + All **240/240 measured requests** across six repetitions completed. + Every input and output length matches the frozen replay request vector. +- The previous reproduction's rc14 image and checkpoint were reused. Pin + the image child digest + `fe2f17d0c9698bafeb9f437929003c6badc56d17e8382a02505233145a07f61f`, + checkpoint `b5c939de8f754692c1647ca79fbf85e8c1e70f8a`, and InferenceX client + `2baba8e27be8529b4453afc953f9859ec092c73d`. Runtime confirms rc14, + Torch 2.11.0a0+eb65b36914.nv26.02, CUDA 13.1 and SM100. + The historical recipe did not pin its image digest or checkpoint revision; + the historical host environment is not recreated exactly. +- Baseline keeps native request metrics disabled. A separate server enables + `return_perf_metrics` and retains all 48 request records per repetition. + Sort by server arrival, exclude the first eight warmups, and verify that + every warmup finished before any measured request arrived. + +| Observation | Mean TTFT (ms) | Mean TPOT (ms) | +|---|---:|---:| +| Historical observation | 48.3129 | 2.8212 | +| Frozen corrected replay | 16.5326 | 2.9374 | +| Uninstrumented repeat 1 | 50.3386 | 2.8087 | +| Uninstrumented repeat 2 | 48.4294 | 2.8042 | +| Uninstrumented repeat 3 | 47.8015 | 2.8132 | +| Uninstrumented repeats, equal-weight mean | 48.8565 | 2.8087 | +| Native-metrics repeats, equal-weight mean | 50.5588 | 2.8374 | + +- The historical TTFT residual reproduces; TPOT remains close to silicon. + Instrumented runs average 3.48% higher TTFT than the uninstrumented runs, + so retain them as diagnostic evidence rather than replace the baseline. +- Native timing means over 120 measured diagnostic requests: + +| Boundary | Mean time (ms) | +|---|---:| +| Server arrival to executor arrival | 2.6915 | +| Executor arrival to first scheduled iteration | 5.6543 | +| First scheduled iteration to first token | 34.9355 | +| First token to server response timestamp | 2.0356 | +| Server arrival to server response | 45.3170 | +| Client mean minus server mean | 5.2418 | + +- The client/server difference is a difference of means, not an ID-paired + transport measurement. Native timestamps and response boundaries follow + [rc14 openai_server.py](https://github.com/NVIDIA/TensorRT-LLM/blob/93cb6518b6d6dbd6095748189e626db731f44545/tensorrt_llm/serve/openai_server.py#L830). +- Each diagnostic repetition has 38 distinct first-prefill iterations for + 40 requests. Match `first_iter` to rank0 iteration logs and use the next + iteration's `prev_device_step_time`, as defined by the pinned + [ping-pong CUDA-event logger](https://github.com/NVIDIA/TensorRT-LLM/blob/93cb6518b6d6dbd6095748189e626db731f44545/tensorrt_llm/_torch/pyexecutor/py_executor.py#L926). + Mean prefill-step device intervals are **33.44 / 31.40 / 31.75 ms**. + They can include host launch gaps and synchronization; they are not sums + of GPU kernel durations. Frontend-only overhead cannot explain the full gap. +- [Matched two-rank traces and CPU timing](gym-gpt-eager-20260918.md) now + support eager submission gaps and peer waiting as a missing boundary. + The user deferred the execution-mode/CPU-timing extension; the prediction + correction remains open. + Adding a constant TTFT offset is not justified by this single point. +- [Timing summary, per-request diagnostic durations and artifact hashes](gym-residual-gpt-20260918.json). + Remote results: `${B200_ARTIFACT_ROOT}/results/trt-62-20260917/residual-20260918/gpt-ttft/`; + logs: `${B200_ARTIFACT_ROOT}/logs/gpt-ttft-tp2-0918-1996181.{out,err}`. + +## Evidence retention + +- Raw artifacts: `${CAMPAIGN_ARTIFACT_ROOT}/residual-20260918/`. +- [Machine-readable Qwen probes and hashes](gym-residual-qwen-20260918.json). +- Runtime, data and original-cohort identities remain those in the linked + full replay report. No performance rows or prediction values were changed. diff --git a/python/aisimulate/collector/trtllm/gym-residual-gpt-20260918.json b/python/aisimulate/collector/trtllm/gym-residual-gpt-20260918.json new file mode 100644 index 000000000..cec234c44 --- /dev/null +++ b/python/aisimulate/collector/trtllm/gym-residual-gpt-20260918.json @@ -0,0 +1,1383 @@ +{ + "job": { + "id": "1996181", + "state": "COMPLETED", + "exit_code": "0:0", + "elapsed": "00:09:27", + "gpus": 2 + }, + "point": { + "config_id": 155, + "hardware": "b200", + "tp": 2, + "ep": 1, + "isl": 1024, + "osl": 1024, + "concurrency": 4, + "measured_requests_per_repeat": 40, + "warmups_per_repeat": 8, + "random_range_ratio": 0.8, + "seed": 0 + }, + "historical": { + "ttft_ms": 48.31286990083754, + "tpot_ms": 2.821235826337538 + }, + "prediction": { + "ttft_ms": 16.532643949999834, + "tpot_ms": 2.9374422182467463 + }, + "phases": { + "baseline": { + "repetitions": [ + { + "repeat": 1, + "completed": 40, + "ttft_ms": 50.33858924289234, + "tpot_ms": 2.80866272547643 + }, + { + "repeat": 2, + "completed": 40, + "ttft_ms": 48.42941244132817, + "tpot_ms": 2.804163560849715 + }, + { + "repeat": 3, + "completed": 40, + "ttft_ms": 47.801536513725296, + "tpot_ms": 2.813150691485603 + } + ], + "completed": 120, + "mean_ttft_ms": 48.8565127326486, + "mean_tpot_ms": 2.808658992603916 + }, + "diagnostic": { + "repetitions": [ + { + "repeat": 1, + "completed": 40, + "ttft_ms": 53.981779725290835, + "tpot_ms": 2.850134083805169, + "timing_means_ms": { + "server_to_executor_ms": 3.405750001547858, + "executor_queue_ms": 6.289525015745312, + "scheduled_to_first_token_ms": 36.39722499065101, + "first_token_to_server_response_ms": 2.133199980016798, + "server_ttft_ms": 48.22569998796098 + }, + "client_minus_server_ttft_ms": 5.756079737329856, + "distinct_prefill_steps": 38, + "mean_prefill_step_host_ms": 34.90876524071945, + "mean_prefill_step_device_ms": 33.44207151312577 + }, + { + "repeat": 2, + "completed": 40, + "ttft_ms": 48.56108608073555, + "tpot_ms": 2.831353669734967, + "timing_means_ms": { + "server_to_executor_ms": 2.3050750023685396, + "executor_queue_ms": 5.358524998882785, + "scheduled_to_first_token_ms": 34.14272498921491, + "first_token_to_server_response_ms": 1.966475002700463, + "server_ttft_ms": 43.7727999931667 + }, + "client_minus_server_ttft_ms": 4.788286087568849, + "distinct_prefill_steps": 38, + "mean_prefill_step_host_ms": 32.911175175717005, + "mean_prefill_step_device_ms": 31.40388287995991 + }, + { + "repeat": 3, + "completed": 40, + "ttft_ms": 49.13342778454535, + "tpot_ms": 2.830779588983459, + "timing_means_ms": { + "server_to_executor_ms": 2.363600011449307, + "executor_queue_ms": 5.3149750048760325, + "scheduled_to_first_token_ms": 34.266649989876896, + "first_token_to_server_response_ms": 2.0072000101208687, + "server_ttft_ms": 43.952425016323105 + }, + "client_minus_server_ttft_ms": 5.181002768222243, + "distinct_prefill_steps": 38, + "mean_prefill_step_host_ms": 33.24831159491288, + "mean_prefill_step_device_ms": 31.747168691534746 + } + ], + "completed": 120, + "mean_ttft_ms": 50.55876453019058, + "mean_tpot_ms": 2.837422447507865 + } + }, + "diagnostic_timing_means_ms": { + "server_to_executor_ms": 2.6914750051219016, + "executor_queue_ms": 5.654341673168044, + "scheduled_to_first_token_ms": 34.935533323247604, + "first_token_to_server_response_ms": 2.0356249976127097, + "server_ttft_ms": 45.31697499915026 + }, + "diagnostic_requests": [ + { + "server_to_executor_ms": 3.8499999791383743, + "executor_queue_ms": 43.35100017488003, + "scheduled_to_first_token_ms": 42.224999982863665, + "first_token_to_server_response_ms": 2.42499983869493, + "server_ttft_ms": 91.850999975577, + "first_iter": 1708, + "host_step_ms": 42.49930381774902, + "device_step_ms": 42.49910354614258 + }, + { + "server_to_executor_ms": 2.5930001866072416, + "executor_queue_ms": 1.892999978736043, + "scheduled_to_first_token_ms": 81.70400001108646, + "first_token_to_server_response_ms": 2.7019998524338007, + "server_ttft_ms": 88.89200002886355, + "first_iter": 1707, + "host_step_ms": 53.89690399169922, + "device_step_ms": 53.888065338134766 + }, + { + "server_to_executor_ms": 2.8490000404417515, + "executor_queue_ms": 39.914000080898404, + "scheduled_to_first_token_ms": 42.24199987947941, + "first_token_to_server_response_ms": 1.8220001365989447, + "server_ttft_ms": 86.82700013741851, + "first_iter": 1708, + "host_step_ms": 42.49930381774902, + "device_step_ms": 42.49910354614258 + }, + { + "server_to_executor_ms": 2.1929999347776175, + "executor_queue_ms": 39.71300018019974, + "scheduled_to_first_token_ms": 42.24599990993738, + "first_token_to_server_response_ms": 1.918999943882227, + "server_ttft_ms": 86.07099996879697, + "first_iter": 1708, + "host_step_ms": 42.49930381774902, + "device_step_ms": 42.49910354614258 + }, + { + "server_to_executor_ms": 3.8670001085847616, + "executor_queue_ms": 1.8889999482780695, + "scheduled_to_first_token_ms": 43.246000073850155, + "first_token_to_server_response_ms": 3.0499999411404133, + "server_ttft_ms": 52.0520000718534, + "first_iter": 2541, + "host_step_ms": 42.1137809753418, + "device_step_ms": 40.8474235534668 + }, + { + "server_to_executor_ms": 3.5940001253038645, + "executor_queue_ms": 3.6720000207424164, + "scheduled_to_first_token_ms": 32.67199988476932, + "first_token_to_server_response_ms": 2.0099999383091927, + "server_ttft_ms": 41.947999969124794, + "first_iter": 2564, + "host_step_ms": 33.304691314697266, + "device_step_ms": 32.32998275756836 + }, + { + "server_to_executor_ms": 3.357000183314085, + "executor_queue_ms": 2.7069998905062675, + "scheduled_to_first_token_ms": 32.87100000306964, + "first_token_to_server_response_ms": 1.896000001579523, + "server_ttft_ms": 40.831000078469515, + "first_iter": 2718, + "host_step_ms": 31.908035278320312, + "device_step_ms": 29.9484806060791 + }, + { + "server_to_executor_ms": 2.8649999294430017, + "executor_queue_ms": 4.781000083312392, + "scheduled_to_first_token_ms": 36.189999897032976, + "first_token_to_server_response_ms": 1.7140000127255917, + "server_ttft_ms": 45.54999992251396, + "first_iter": 2726, + "host_step_ms": 36.427974700927734, + "device_step_ms": 34.6888313293457 + }, + { + "server_to_executor_ms": 3.5749999806284904, + "executor_queue_ms": 5.46300015412271, + "scheduled_to_first_token_ms": 35.81999987363815, + "first_token_to_server_response_ms": 2.025000052526593, + "server_ttft_ms": 46.88300006091595, + "first_iter": 3422, + "host_step_ms": 36.283016204833984, + "device_step_ms": 34.58319854736328 + }, + { + "server_to_executor_ms": 2.859000116586685, + "executor_queue_ms": 5.158999934792519, + "scheduled_to_first_token_ms": 32.20600006170571, + "first_token_to_server_response_ms": 2.0520000252872705, + "server_ttft_ms": 42.27600013837218, + "first_iter": 3493, + "host_step_ms": 32.48286247253418, + "device_step_ms": 30.57593536376953 + }, + { + "server_to_executor_ms": 3.3939999993890524, + "executor_queue_ms": 4.206999903544784, + "scheduled_to_first_token_ms": 38.01200003363192, + "first_token_to_server_response_ms": 2.6730000972747803, + "server_ttft_ms": 48.28600003384054, + "first_iter": 3575, + "host_step_ms": 36.206722259521484, + "device_step_ms": 34.690399169921875 + }, + { + "server_to_executor_ms": 3.047000151127577, + "executor_queue_ms": 4.098000004887581, + "scheduled_to_first_token_ms": 36.758999805897474, + "first_token_to_server_response_ms": 2.701000077649951, + "server_ttft_ms": 46.60500003956258, + "first_iter": 3753, + "host_step_ms": 36.14091873168945, + "device_step_ms": 34.431968688964844 + }, + { + "server_to_executor_ms": 2.426999853923917, + "executor_queue_ms": 2.171000000089407, + "scheduled_to_first_token_ms": 36.19100013747811, + "first_token_to_server_response_ms": 2.2499999031424522, + "server_ttft_ms": 43.03899989463389, + "first_iter": 4398, + "host_step_ms": 34.39807891845703, + "device_step_ms": 32.7191047668457 + }, + { + "server_to_executor_ms": 2.9039999935775995, + "executor_queue_ms": 2.7020000852644444, + "scheduled_to_first_token_ms": 33.79299980588257, + "first_token_to_server_response_ms": 1.7500000540167093, + "server_ttft_ms": 41.148999938741326, + "first_iter": 4481, + "host_step_ms": 32.32741355895996, + "device_step_ms": 30.396896362304688 + }, + { + "server_to_executor_ms": 2.3139999248087406, + "executor_queue_ms": 2.971000038087368, + "scheduled_to_first_token_ms": 32.46799996122718, + "first_token_to_server_response_ms": 1.7250000964850187, + "server_ttft_ms": 39.478000020608306, + "first_iter": 4603, + "host_step_ms": 31.354427337646484, + "device_step_ms": 29.539424896240234 + }, + { + "server_to_executor_ms": 3.496999852359295, + "executor_queue_ms": 4.484999924898148, + "scheduled_to_first_token_ms": 32.706000143662095, + "first_token_to_server_response_ms": 3.100999863818288, + "server_ttft_ms": 43.788999784737825, + "first_iter": 4692, + "host_step_ms": 32.49382972717285, + "device_step_ms": 30.566144943237305 + }, + { + "server_to_executor_ms": 4.2749999556690454, + "executor_queue_ms": 3.8219999987632036, + "scheduled_to_first_token_ms": 35.869000013917685, + "first_token_to_server_response_ms": 1.9000000320374966, + "server_ttft_ms": 45.86600000038743, + "first_iter": 5335, + "host_step_ms": 34.679412841796875, + "device_step_ms": 32.85747146606445 + }, + { + "server_to_executor_ms": 3.456000005826354, + "executor_queue_ms": 3.261999925598502, + "scheduled_to_first_token_ms": 34.813000122085214, + "first_token_to_server_response_ms": 2.5599999353289604, + "server_ttft_ms": 44.09099998883903, + "first_iter": 5406, + "host_step_ms": 34.01041030883789, + "device_step_ms": 32.160545349121094 + }, + { + "server_to_executor_ms": 5.637000082060695, + "executor_queue_ms": 4.275999963283539, + "scheduled_to_first_token_ms": 32.89300017058849, + "first_token_to_server_response_ms": 1.5999998431652784, + "server_ttft_ms": 44.406000059098005, + "first_iter": 5463, + "host_step_ms": 32.96828269958496, + "device_step_ms": 31.15228843688965 + }, + { + "server_to_executor_ms": 3.114999970421195, + "executor_queue_ms": 3.7429998628795147, + "scheduled_to_first_token_ms": 33.614000072702765, + "first_token_to_server_response_ms": 2.16599996201694, + "server_ttft_ms": 42.637999868020415, + "first_iter": 5645, + "host_step_ms": 33.6453914642334, + "device_step_ms": 31.820831298828125 + }, + { + "server_to_executor_ms": 2.982999896630645, + "executor_queue_ms": 1.8809998873621225, + "scheduled_to_first_token_ms": 37.20400016754866, + "first_token_to_server_response_ms": 1.9789999350905418, + "server_ttft_ms": 44.046999886631966, + "first_iter": 6288, + "host_step_ms": 35.805702209472656, + "device_step_ms": 34.44499206542969 + }, + { + "server_to_executor_ms": 3.9470000192523003, + "executor_queue_ms": 4.932000068947673, + "scheduled_to_first_token_ms": 33.220999874174595, + "first_token_to_server_response_ms": 2.4160000029951334, + "server_ttft_ms": 44.5159999653697, + "first_iter": 6341, + "host_step_ms": 34.07645225524902, + "device_step_ms": 32.899009704589844 + }, + { + "server_to_executor_ms": 2.3819999769330025, + "executor_queue_ms": 2.339000115171075, + "scheduled_to_first_token_ms": 31.910999910905957, + "first_token_to_server_response_ms": 2.5290001649409533, + "server_ttft_ms": 39.16100016795099, + "first_iter": 6395, + "host_step_ms": 30.590295791625977, + "device_step_ms": 29.664384841918945 + }, + { + "server_to_executor_ms": 3.123000031337142, + "executor_queue_ms": 3.4340000711381435, + "scheduled_to_first_token_ms": 32.63299982063472, + "first_token_to_server_response_ms": 2.426000079140067, + "server_ttft_ms": 41.616000002250075, + "first_iter": 6603, + "host_step_ms": 32.98616409301758, + "device_step_ms": 31.990047454833984 + }, + { + "server_to_executor_ms": 2.937000012025237, + "executor_queue_ms": 1.3770000077784061, + "scheduled_to_first_token_ms": 36.27600008621812, + "first_token_to_server_response_ms": 2.363999839872122, + "server_ttft_ms": 42.953999945893884, + "first_iter": 7151, + "host_step_ms": 34.29269790649414, + "device_step_ms": 32.95462417602539 + }, + { + "server_to_executor_ms": 3.0139998998492956, + "executor_queue_ms": 2.6740001048892736, + "scheduled_to_first_token_ms": 34.02299992740154, + "first_token_to_server_response_ms": 2.171000000089407, + "server_ttft_ms": 41.88199993222952, + "first_iter": 7183, + "host_step_ms": 32.1047306060791, + "device_step_ms": 30.167007446289062 + }, + { + "server_to_executor_ms": 3.3370000310242176, + "executor_queue_ms": 5.374000174924731, + "scheduled_to_first_token_ms": 33.088999800384045, + "first_token_to_server_response_ms": 1.9420001190155745, + "server_ttft_ms": 43.74200012534857, + "first_iter": 7299, + "host_step_ms": 34.311532974243164, + "device_step_ms": 33.19772720336914 + }, + { + "server_to_executor_ms": 3.880999982357025, + "executor_queue_ms": 3.6379999946802855, + "scheduled_to_first_token_ms": 36.564999958500266, + "first_token_to_server_response_ms": 2.1359999664127827, + "server_ttft_ms": 46.21999990195036, + "first_iter": 7560, + "host_step_ms": 37.01210021972656, + "device_step_ms": 35.95199966430664 + }, + { + "server_to_executor_ms": 3.8580000400543213, + "executor_queue_ms": 3.172999946400523, + "scheduled_to_first_token_ms": 37.09100000560284, + "first_token_to_server_response_ms": 1.8549999222159386, + "server_ttft_ms": 45.97699991427362, + "first_iter": 8051, + "host_step_ms": 35.6745719909668, + "device_step_ms": 33.684574127197266 + }, + { + "server_to_executor_ms": 3.022999968379736, + "executor_queue_ms": 3.949000034481287, + "scheduled_to_first_token_ms": 32.62199996970594, + "first_token_to_server_response_ms": 2.0660001318901777, + "server_ttft_ms": 41.66000010445714, + "first_iter": 8081, + "host_step_ms": 32.76515007019043, + "device_step_ms": 31.15555191040039 + }, + { + "server_to_executor_ms": 3.639000002294779, + "executor_queue_ms": 4.701999947428703, + "scheduled_to_first_token_ms": 33.866000128909945, + "first_token_to_server_response_ms": 2.076999982818961, + "server_ttft_ms": 44.28400006145239, + "first_iter": 8310, + "host_step_ms": 33.92434120178223, + "device_step_ms": 31.922815322875977 + }, + { + "server_to_executor_ms": 3.6769998259842396, + "executor_queue_ms": 4.535000072792172, + "scheduled_to_first_token_ms": 34.98500003479421, + "first_token_to_server_response_ms": 1.7629999201744795, + "server_ttft_ms": 44.9599998537451, + "first_iter": 8416, + "host_step_ms": 36.15140914916992, + "device_step_ms": 34.97536087036133 + }, + { + "server_to_executor_ms": 4.2940001003444195, + "executor_queue_ms": 1.40299997292459, + "scheduled_to_first_token_ms": 36.44599998369813, + "first_token_to_server_response_ms": 2.3739999160170555, + "server_ttft_ms": 44.516999972984195, + "first_iter": 8907, + "host_step_ms": 35.317420959472656, + "device_step_ms": 34.667518615722656 + }, + { + "server_to_executor_ms": 3.4729999024420977, + "executor_queue_ms": 3.3530001528561115, + "scheduled_to_first_token_ms": 33.709999872371554, + "first_token_to_server_response_ms": 1.9920000340789557, + "server_ttft_ms": 42.52799996174872, + "first_iter": 8996, + "host_step_ms": 33.516883850097656, + "device_step_ms": 32.04617691040039 + }, + { + "server_to_executor_ms": 4.763999953866005, + "executor_queue_ms": 3.9510000497102737, + "scheduled_to_first_token_ms": 32.80899999663234, + "first_token_to_server_response_ms": 1.4269999228417873, + "server_ttft_ms": 42.950999923050404, + "first_iter": 9201, + "host_step_ms": 32.88769721984863, + "device_step_ms": 31.11030387878418 + }, + { + "server_to_executor_ms": 3.6810000892728567, + "executor_queue_ms": 3.224999876692891, + "scheduled_to_first_token_ms": 34.09800003282726, + "first_token_to_server_response_ms": 1.9159999210387468, + "server_ttft_ms": 42.91999991983175, + "first_iter": 9410, + "host_step_ms": 34.377098083496094, + "device_step_ms": 32.96540832519531 + }, + { + "server_to_executor_ms": 4.0160000789910555, + "executor_queue_ms": 2.867999952286482, + "scheduled_to_first_token_ms": 34.91000016219914, + "first_token_to_server_response_ms": 2.450999803841114, + "server_ttft_ms": 44.24499999731779, + "first_iter": 9789, + "host_step_ms": 33.856868743896484, + "device_step_ms": 32.1190071105957 + }, + { + "server_to_executor_ms": 3.3759998623281717, + "executor_queue_ms": 4.591000033542514, + "scheduled_to_first_token_ms": 32.07499999552965, + "first_token_to_server_response_ms": 2.022000029683113, + "server_ttft_ms": 42.06399992108345, + "first_iter": 9856, + "host_step_ms": 31.754493713378906, + "device_step_ms": 29.912288665771484 + }, + { + "server_to_executor_ms": 3.003000048920512, + "executor_queue_ms": 3.352999920025468, + "scheduled_to_first_token_ms": 36.70400008559227, + "first_token_to_server_response_ms": 1.755999866873026, + "server_ttft_ms": 44.815999921411276, + "first_iter": 10128, + "host_step_ms": 35.57872772216797, + "device_step_ms": 34.023582458496094 + }, + { + "server_to_executor_ms": 4.153999965637922, + "executor_queue_ms": 2.5500000920146704, + "scheduled_to_first_token_ms": 33.1109999679029, + "first_token_to_server_response_ms": 1.6250000335276127, + "server_ttft_ms": 41.440000059083104, + "first_iter": 10354, + "host_step_ms": 32.407283782958984, + "device_step_ms": 31.250240325927734 + }, + { + "server_to_executor_ms": 2.809000201523304, + "executor_queue_ms": 3.0559999868273735, + "scheduled_to_first_token_ms": 69.69799986109138, + "first_token_to_server_response_ms": 2.0439999643713236, + "server_ttft_ms": 77.60700001381338, + "first_iter": 12971, + "host_step_ms": 48.34747314453125, + "device_step_ms": 48.33843231201172 + }, + { + "server_to_executor_ms": 2.394000068306923, + "executor_queue_ms": 34.57599994726479, + "scheduled_to_first_token_ms": 37.136999890208244, + "first_token_to_server_response_ms": 2.064000116661191, + "server_ttft_ms": 76.17100002244115, + "first_iter": 12972, + "host_step_ms": 37.454843521118164, + "device_step_ms": 37.33561706542969 + }, + { + "server_to_executor_ms": 2.972999820485711, + "executor_queue_ms": 33.37900014594197, + "scheduled_to_first_token_ms": 37.15700004249811, + "first_token_to_server_response_ms": 1.5579999890178442, + "server_ttft_ms": 75.06699999794364, + "first_iter": 12972, + "host_step_ms": 37.454843521118164, + "device_step_ms": 37.33561706542969 + }, + { + "server_to_executor_ms": 2.9980000108480453, + "executor_queue_ms": 32.88100007921457, + "scheduled_to_first_token_ms": 37.16399986296892, + "first_token_to_server_response_ms": 2.1379999816417694, + "server_ttft_ms": 75.18099993467331, + "first_iter": 12972, + "host_step_ms": 37.454843521118164, + "device_step_ms": 37.33561706542969 + }, + { + "server_to_executor_ms": 2.3819999769330025, + "executor_queue_ms": 1.884999917820096, + "scheduled_to_first_token_ms": 35.49899999052286, + "first_token_to_server_response_ms": 1.5030000358819962, + "server_ttft_ms": 41.268999921157956, + "first_iter": 13805, + "host_step_ms": 34.261226654052734, + "device_step_ms": 32.502174377441406 + }, + { + "server_to_executor_ms": 2.3120001424103975, + "executor_queue_ms": 3.683999879285693, + "scheduled_to_first_token_ms": 31.69200010597706, + "first_token_to_server_response_ms": 1.8179998733103275, + "server_ttft_ms": 39.50600000098348, + "first_iter": 13828, + "host_step_ms": 32.45425224304199, + "device_step_ms": 31.285472869873047 + }, + { + "server_to_executor_ms": 2.1559998858720064, + "executor_queue_ms": 2.9329999815672636, + "scheduled_to_first_token_ms": 31.88600018620491, + "first_token_to_server_response_ms": 1.6339998692274094, + "server_ttft_ms": 38.60899992287159, + "first_iter": 13980, + "host_step_ms": 30.800819396972656, + "device_step_ms": 28.890560150146484 + }, + { + "server_to_executor_ms": 2.065999899059534, + "executor_queue_ms": 3.4020000603049994, + "scheduled_to_first_token_ms": 31.610999954864383, + "first_token_to_server_response_ms": 1.6060001216828823, + "server_ttft_ms": 38.6850000359118, + "first_iter": 13989, + "host_step_ms": 30.90691566467285, + "device_step_ms": 29.02604866027832 + }, + { + "server_to_executor_ms": 2.3620000574737787, + "executor_queue_ms": 2.8089999686926603, + "scheduled_to_first_token_ms": 33.327999990433455, + "first_token_to_server_response_ms": 2.53800000064075, + "server_ttft_ms": 41.03700001724064, + "first_iter": 14684, + "host_step_ms": 32.63044357299805, + "device_step_ms": 31.0118408203125 + }, + { + "server_to_executor_ms": 2.199999988079071, + "executor_queue_ms": 3.2090002205222845, + "scheduled_to_first_token_ms": 37.826999789103866, + "first_token_to_server_response_ms": 1.9630000460892916, + "server_ttft_ms": 45.19900004379451, + "first_iter": 14756, + "host_step_ms": 38.01584243774414, + "device_step_ms": 36.69692611694336 + }, + { + "server_to_executor_ms": 2.2049997933208942, + "executor_queue_ms": 2.7940000873059034, + "scheduled_to_first_token_ms": 30.892000067979097, + "first_token_to_server_response_ms": 2.259999979287386, + "server_ttft_ms": 38.15099992789328, + "first_iter": 14835, + "host_step_ms": 29.6628475189209, + "device_step_ms": 27.826879501342773 + }, + { + "server_to_executor_ms": 2.2980000358074903, + "executor_queue_ms": 2.429999876767397, + "scheduled_to_first_token_ms": 33.40200008824468, + "first_token_to_server_response_ms": 1.4470000751316547, + "server_ttft_ms": 39.57700007595122, + "first_iter": 15015, + "host_step_ms": 32.71961212158203, + "device_step_ms": 31.457439422607422 + }, + { + "server_to_executor_ms": 2.0349998958408833, + "executor_queue_ms": 1.7399999778717756, + "scheduled_to_first_token_ms": 35.392000107094646, + "first_token_to_server_response_ms": 1.6840000171214342, + "server_ttft_ms": 40.85099999792874, + "first_iter": 15659, + "host_step_ms": 34.075021743774414, + "device_step_ms": 33.03750228881836 + }, + { + "server_to_executor_ms": 1.9689998589456081, + "executor_queue_ms": 2.15000007301569, + "scheduled_to_first_token_ms": 32.58599992841482, + "first_token_to_server_response_ms": 1.8080000299960375, + "server_ttft_ms": 38.51299989037216, + "first_iter": 15743, + "host_step_ms": 31.525135040283203, + "device_step_ms": 29.571168899536133 + }, + { + "server_to_executor_ms": 2.114000031724572, + "executor_queue_ms": 2.2700000554323196, + "scheduled_to_first_token_ms": 32.79499989002943, + "first_token_to_server_response_ms": 2.552000107243657, + "server_ttft_ms": 39.73100008442998, + "first_iter": 15862, + "host_step_ms": 31.09884262084961, + "device_step_ms": 29.715744018554688 + }, + { + "server_to_executor_ms": 2.3700001183897257, + "executor_queue_ms": 2.5549998972564936, + "scheduled_to_first_token_ms": 31.552999978885055, + "first_token_to_server_response_ms": 1.4929999597370625, + "server_ttft_ms": 37.970999954268336, + "first_iter": 15953, + "host_step_ms": 30.862092971801758, + "device_step_ms": 29.27292823791504 + }, + { + "server_to_executor_ms": 2.3799999617040157, + "executor_queue_ms": 3.3760000951588154, + "scheduled_to_first_token_ms": 34.38599989749491, + "first_token_to_server_response_ms": 1.4400000218302011, + "server_ttft_ms": 41.581999976187944, + "first_iter": 16596, + "host_step_ms": 33.4477424621582, + "device_step_ms": 31.683135986328125 + }, + { + "server_to_executor_ms": 1.9759999122470617, + "executor_queue_ms": 2.9249999206513166, + "scheduled_to_first_token_ms": 32.38800005055964, + "first_token_to_server_response_ms": 2.2200001403689384, + "server_ttft_ms": 39.50900002382696, + "first_iter": 16666, + "host_step_ms": 31.525135040283203, + "device_step_ms": 29.5982723236084 + }, + { + "server_to_executor_ms": 2.2960000205785036, + "executor_queue_ms": 3.8930000737309456, + "scheduled_to_first_token_ms": 30.627999920397997, + "first_token_to_server_response_ms": 3.5370001569390297, + "server_ttft_ms": 40.354000171646476, + "first_iter": 16720, + "host_step_ms": 30.536413192749023, + "device_step_ms": 28.99510383605957 + }, + { + "server_to_executor_ms": 1.9060000777244568, + "executor_queue_ms": 2.6779999025166035, + "scheduled_to_first_token_ms": 30.914000002667308, + "first_token_to_server_response_ms": 1.7840000800788403, + "server_ttft_ms": 37.28200006298721, + "first_iter": 16904, + "host_step_ms": 30.76791763305664, + "device_step_ms": 29.48201560974121 + }, + { + "server_to_executor_ms": 2.1310001611709595, + "executor_queue_ms": 2.2950000129640102, + "scheduled_to_first_token_ms": 33.78699999302626, + "first_token_to_server_response_ms": 1.9679998513311148, + "server_ttft_ms": 40.18100001849234, + "first_iter": 17548, + "host_step_ms": 32.69767761230469, + "device_step_ms": 31.089887619018555 + }, + { + "server_to_executor_ms": 2.5659999810159206, + "executor_queue_ms": 4.368999972939491, + "scheduled_to_first_token_ms": 31.386999879032373, + "first_token_to_server_response_ms": 2.27900012396276, + "server_ttft_ms": 40.600999956950545, + "first_iter": 17597, + "host_step_ms": 31.917810440063477, + "device_step_ms": 30.330047607421875 + }, + { + "server_to_executor_ms": 2.139999996870756, + "executor_queue_ms": 2.0710001699626446, + "scheduled_to_first_token_ms": 33.93599996343255, + "first_token_to_server_response_ms": 1.7430000007152557, + "server_ttft_ms": 39.89000013098121, + "first_iter": 17654, + "host_step_ms": 32.921552658081055, + "device_step_ms": 31.50694465637207 + }, + { + "server_to_executor_ms": 2.1730000153183937, + "executor_queue_ms": 2.5889999233186245, + "scheduled_to_first_token_ms": 31.236999901011586, + "first_token_to_server_response_ms": 1.726000104099512, + "server_ttft_ms": 37.72499994374812, + "first_iter": 17861, + "host_step_ms": 30.469417572021484, + "device_step_ms": 28.863040924072266 + }, + { + "server_to_executor_ms": 2.148999832570553, + "executor_queue_ms": 3.716000122949481, + "scheduled_to_first_token_ms": 36.490999860689044, + "first_token_to_server_response_ms": 1.8629999831318855, + "server_ttft_ms": 44.21899979934096, + "first_iter": 18411, + "host_step_ms": 35.10022163391113, + "device_step_ms": 33.145118713378906 + }, + { + "server_to_executor_ms": 1.846000086516142, + "executor_queue_ms": 2.4389999452978373, + "scheduled_to_first_token_ms": 30.79200000502169, + "first_token_to_server_response_ms": 1.9990000873804092, + "server_ttft_ms": 37.07600012421608, + "first_iter": 18438, + "host_step_ms": 29.736995697021484, + "device_step_ms": 27.74947166442871 + }, + { + "server_to_executor_ms": 2.4239998310804367, + "executor_queue_ms": 2.9299999587237835, + "scheduled_to_first_token_ms": 32.34500018879771, + "first_token_to_server_response_ms": 2.5439998134970665, + "server_ttft_ms": 40.242999792099, + "first_iter": 18557, + "host_step_ms": 31.51679039001465, + "device_step_ms": 29.738399505615234 + }, + { + "server_to_executor_ms": 2.2690000478178263, + "executor_queue_ms": 3.6780000664293766, + "scheduled_to_first_token_ms": 32.491999911144376, + "first_token_to_server_response_ms": 1.8070000223815441, + "server_ttft_ms": 40.24600004777312, + "first_iter": 18817, + "host_step_ms": 32.04011917114258, + "device_step_ms": 30.373184204101562 + }, + { + "server_to_executor_ms": 2.4010001216083765, + "executor_queue_ms": 2.1460000425577164, + "scheduled_to_first_token_ms": 33.847999991849065, + "first_token_to_server_response_ms": 1.7659999430179596, + "server_ttft_ms": 40.16100009903312, + "first_iter": 19304, + "host_step_ms": 32.74106979370117, + "device_step_ms": 31.56153678894043 + }, + { + "server_to_executor_ms": 2.1379999816417694, + "executor_queue_ms": 4.449000116437674, + "scheduled_to_first_token_ms": 31.316999811679125, + "first_token_to_server_response_ms": 1.7430000007152557, + "server_ttft_ms": 39.64699991047382, + "first_iter": 19340, + "host_step_ms": 31.39352798461914, + "device_step_ms": 29.463136672973633 + }, + { + "server_to_executor_ms": 2.308000111952424, + "executor_queue_ms": 3.0349998269230127, + "scheduled_to_first_token_ms": 31.95099998265505, + "first_token_to_server_response_ms": 1.5840001869946718, + "server_ttft_ms": 38.87800010852516, + "first_iter": 19567, + "host_step_ms": 31.749248504638672, + "device_step_ms": 30.256895065307617 + }, + { + "server_to_executor_ms": 2.4980001617223024, + "executor_queue_ms": 5.581999896094203, + "scheduled_to_first_token_ms": 35.17500008456409, + "first_token_to_server_response_ms": 1.7639999277889729, + "server_ttft_ms": 45.01900007016957, + "first_iter": 19672, + "host_step_ms": 37.244319915771484, + "device_step_ms": 35.40492630004883 + }, + { + "server_to_executor_ms": 2.2350000217556953, + "executor_queue_ms": 3.069999860599637, + "scheduled_to_first_token_ms": 33.866000128909945, + "first_token_to_server_response_ms": 1.868999795988202, + "server_ttft_ms": 41.03999980725348, + "first_iter": 20165, + "host_step_ms": 33.19048881530762, + "device_step_ms": 31.70515251159668 + }, + { + "server_to_executor_ms": 2.305000089108944, + "executor_queue_ms": 3.035999834537506, + "scheduled_to_first_token_ms": 32.13700000196695, + "first_token_to_server_response_ms": 1.984999980777502, + "server_ttft_ms": 39.462999906390905, + "first_iter": 20248, + "host_step_ms": 31.255006790161133, + "device_step_ms": 29.93939208984375 + }, + { + "server_to_executor_ms": 2.693000016734004, + "executor_queue_ms": 4.055999917909503, + "scheduled_to_first_token_ms": 31.10500006005168, + "first_token_to_server_response_ms": 1.9179999362677336, + "server_ttft_ms": 39.77199993096292, + "first_iter": 20456, + "host_step_ms": 31.672000885009766, + "device_step_ms": 30.09075164794922 + }, + { + "server_to_executor_ms": 2.1649999544024467, + "executor_queue_ms": 2.5110000278800726, + "scheduled_to_first_token_ms": 33.34700013510883, + "first_token_to_server_response_ms": 1.9429998937994242, + "server_ttft_ms": 39.96600001119077, + "first_iter": 20665, + "host_step_ms": 32.19342231750488, + "device_step_ms": 30.33772850036621 + }, + { + "server_to_executor_ms": 2.222999930381775, + "executor_queue_ms": 3.0720001086592674, + "scheduled_to_first_token_ms": 35.36399989388883, + "first_token_to_server_response_ms": 1.947999931871891, + "server_ttft_ms": 42.606999864801764, + "first_iter": 21046, + "host_step_ms": 34.22713279724121, + "device_step_ms": 32.45283126831055 + }, + { + "server_to_executor_ms": 2.512000035494566, + "executor_queue_ms": 3.438000101596117, + "scheduled_to_first_token_ms": 31.959000043570995, + "first_token_to_server_response_ms": 1.7579998821020126, + "server_ttft_ms": 39.66700006276369, + "first_iter": 21107, + "host_step_ms": 31.58116340637207, + "device_step_ms": 29.87548828125 + }, + { + "server_to_executor_ms": 2.4429999757558107, + "executor_queue_ms": 2.478000009432435, + "scheduled_to_first_token_ms": 32.14400005526841, + "first_token_to_server_response_ms": 3.0960000585764647, + "server_ttft_ms": 40.16100009903312, + "first_iter": 21382, + "host_step_ms": 30.515670776367188, + "device_step_ms": 29.442975997924805 + }, + { + "server_to_executor_ms": 2.382999984547496, + "executor_queue_ms": 4.755999892950058, + "scheduled_to_first_token_ms": 33.094000071287155, + "first_token_to_server_response_ms": 2.265000017359853, + "server_ttft_ms": 42.49799996614456, + "first_iter": 21608, + "host_step_ms": 35.36844253540039, + "device_step_ms": 34.29337692260742 + }, + { + "server_to_executor_ms": 2.4620001204311848, + "executor_queue_ms": 1.6909998375922441, + "scheduled_to_first_token_ms": 66.0260000731796, + "first_token_to_server_response_ms": 1.8990000244230032, + "server_ttft_ms": 72.07800005562603, + "first_iter": 24225, + "host_step_ms": 48.5687255859375, + "device_step_ms": 48.56630325317383 + }, + { + "server_to_executor_ms": 3.1159999780356884, + "executor_queue_ms": 33.88900007121265, + "scheduled_to_first_token_ms": 34.792999969795346, + "first_token_to_server_response_ms": 2.1039999555796385, + "server_ttft_ms": 73.90199997462332, + "first_iter": 24226, + "host_step_ms": 34.3167781829834, + "device_step_ms": 34.32380676269531 + }, + { + "server_to_executor_ms": 3.3810001332312822, + "executor_queue_ms": 33.14700000919402, + "scheduled_to_first_token_ms": 34.8089998587966, + "first_token_to_server_response_ms": 1.8619999755173922, + "server_ttft_ms": 73.19899997673929, + "first_iter": 24226, + "host_step_ms": 34.3167781829834, + "device_step_ms": 34.32380676269531 + }, + { + "server_to_executor_ms": 2.059000078588724, + "executor_queue_ms": 32.1690000128001, + "scheduled_to_first_token_ms": 34.81399989686906, + "first_token_to_server_response_ms": 2.1810000762343407, + "server_ttft_ms": 71.22300006449223, + "first_iter": 24226, + "host_step_ms": 34.3167781829834, + "device_step_ms": 34.32380676269531 + }, + { + "server_to_executor_ms": 2.2559999488294125, + "executor_queue_ms": 3.782000159844756, + "scheduled_to_first_token_ms": 33.68699993006885, + "first_token_to_server_response_ms": 1.8990000244230032, + "server_ttft_ms": 41.62400006316602, + "first_iter": 25060, + "host_step_ms": 32.767534255981445, + "device_step_ms": 30.840415954589844 + }, + { + "server_to_executor_ms": 2.6000000070780516, + "executor_queue_ms": 3.93400015309453, + "scheduled_to_first_token_ms": 30.981000047177076, + "first_token_to_server_response_ms": 1.4959999825805426, + "server_ttft_ms": 39.0110001899302, + "first_iter": 25081, + "host_step_ms": 31.070232391357422, + "device_step_ms": 29.734079360961914 + }, + { + "server_to_executor_ms": 1.938000088557601, + "executor_queue_ms": 2.3499999660998583, + "scheduled_to_first_token_ms": 31.02299990132451, + "first_token_to_server_response_ms": 1.8420000560581684, + "server_ttft_ms": 37.15300001204014, + "first_iter": 25235, + "host_step_ms": 29.77466583251953, + "device_step_ms": 28.49817657470703 + }, + { + "server_to_executor_ms": 1.9340000580996275, + "executor_queue_ms": 3.2629999332129955, + "scheduled_to_first_token_ms": 30.663999961689115, + "first_token_to_server_response_ms": 2.1120000164955854, + "server_ttft_ms": 37.97299996949732, + "first_iter": 25243, + "host_step_ms": 29.9224853515625, + "device_step_ms": 27.9399356842041 + }, + { + "server_to_executor_ms": 2.1589999087154865, + "executor_queue_ms": 3.5349999088793993, + "scheduled_to_first_token_ms": 33.103000139817595, + "first_token_to_server_response_ms": 2.1349999587982893, + "server_ttft_ms": 40.93199991621077, + "first_iter": 25938, + "host_step_ms": 32.407522201538086, + "device_step_ms": 30.438880920410156 + }, + { + "server_to_executor_ms": 2.2410000674426556, + "executor_queue_ms": 3.2019999343901873, + "scheduled_to_first_token_ms": 31.324999872595072, + "first_token_to_server_response_ms": 1.508000073954463, + "server_ttft_ms": 38.27599994838238, + "first_iter": 26011, + "host_step_ms": 31.26978874206543, + "device_step_ms": 29.965951919555664 + }, + { + "server_to_executor_ms": 2.076999982818961, + "executor_queue_ms": 2.1200000774115324, + "scheduled_to_first_token_ms": 30.747000128030777, + "first_token_to_server_response_ms": 1.7969999462366104, + "server_ttft_ms": 36.74100013449788, + "first_iter": 26090, + "host_step_ms": 29.584646224975586, + "device_step_ms": 28.224288940429688 + }, + { + "server_to_executor_ms": 2.0759999752044678, + "executor_queue_ms": 2.9100000392645597, + "scheduled_to_first_token_ms": 30.97600000910461, + "first_token_to_server_response_ms": 2.093000104650855, + "server_ttft_ms": 38.05500012822449, + "first_iter": 26269, + "host_step_ms": 29.672861099243164, + "device_step_ms": 27.87708854675293 + }, + { + "server_to_executor_ms": 1.9529999699443579, + "executor_queue_ms": 1.984999980777502, + "scheduled_to_first_token_ms": 34.86800007522106, + "first_token_to_server_response_ms": 1.9149999134242535, + "server_ttft_ms": 40.720999939367175, + "first_iter": 26913, + "host_step_ms": 33.71000289916992, + "device_step_ms": 32.675968170166016 + }, + { + "server_to_executor_ms": 2.300000051036477, + "executor_queue_ms": 1.5199999324977398, + "scheduled_to_first_token_ms": 35.78800009563565, + "first_token_to_server_response_ms": 2.398999873548746, + "server_ttft_ms": 42.006999952718616, + "first_iter": 26998, + "host_step_ms": 34.57760810852051, + "device_step_ms": 33.333438873291016 + }, + { + "server_to_executor_ms": 1.752000069245696, + "executor_queue_ms": 1.9950000569224358, + "scheduled_to_first_token_ms": 34.40499980933964, + "first_token_to_server_response_ms": 1.7920001409947872, + "server_ttft_ms": 39.94400007650256, + "first_iter": 27117, + "host_step_ms": 33.05697441101074, + "device_step_ms": 31.39263916015625 + }, + { + "server_to_executor_ms": 2.4320001248270273, + "executor_queue_ms": 3.4260000102221966, + "scheduled_to_first_token_ms": 30.931999906897545, + "first_token_to_server_response_ms": 1.8470000941306353, + "server_ttft_ms": 38.637000136077404, + "first_iter": 27207, + "host_step_ms": 31.131744384765625, + "device_step_ms": 29.772735595703125 + }, + { + "server_to_executor_ms": 2.269999822601676, + "executor_queue_ms": 2.202000003308058, + "scheduled_to_first_token_ms": 37.59600012563169, + "first_token_to_server_response_ms": 1.7170000355690718, + "server_ttft_ms": 43.784999987110496, + "first_iter": 27850, + "host_step_ms": 36.97037696838379, + "device_step_ms": 35.78976058959961 + }, + { + "server_to_executor_ms": 2.3759999312460423, + "executor_queue_ms": 6.568999961018562, + "scheduled_to_first_token_ms": 33.995999954640865, + "first_token_to_server_response_ms": 1.9920000340789557, + "server_ttft_ms": 44.932999880984426, + "first_iter": 27920, + "host_step_ms": 36.81612014770508, + "device_step_ms": 34.86249542236328 + }, + { + "server_to_executor_ms": 2.4510000366717577, + "executor_queue_ms": 4.251000005751848, + "scheduled_to_first_token_ms": 31.137999845668674, + "first_token_to_server_response_ms": 1.7709999810904264, + "server_ttft_ms": 39.610999869182706, + "first_iter": 27976, + "host_step_ms": 31.067609786987305, + "device_step_ms": 29.180639266967773 + }, + { + "server_to_executor_ms": 2.2499999031424522, + "executor_queue_ms": 3.0770001467317343, + "scheduled_to_first_token_ms": 32.300999853760004, + "first_token_to_server_response_ms": 1.9710001070052385, + "server_ttft_ms": 39.59900001063943, + "first_iter": 28159, + "host_step_ms": 31.931161880493164, + "device_step_ms": 30.817312240600586 + }, + { + "server_to_executor_ms": 2.28599994443357, + "executor_queue_ms": 2.9780000913888216, + "scheduled_to_first_token_ms": 37.58599981665611, + "first_token_to_server_response_ms": 2.517000073567033, + "server_ttft_ms": 45.36699992604554, + "first_iter": 28803, + "host_step_ms": 36.25655174255371, + "device_step_ms": 34.35126495361328 + }, + { + "server_to_executor_ms": 2.6499999221414328, + "executor_queue_ms": 3.535000141710043, + "scheduled_to_first_token_ms": 32.30299986898899, + "first_token_to_server_response_ms": 3.3190001267939806, + "server_ttft_ms": 41.80700005963445, + "first_iter": 28854, + "host_step_ms": 32.384395599365234, + "device_step_ms": 30.408447265625 + }, + { + "server_to_executor_ms": 1.8080000299960375, + "executor_queue_ms": 2.4029999040067196, + "scheduled_to_first_token_ms": 32.36600011587143, + "first_token_to_server_response_ms": 1.8480001017451286, + "server_ttft_ms": 38.425000151619315, + "first_iter": 28908, + "host_step_ms": 31.159162521362305, + "device_step_ms": 29.228607177734375 + }, + { + "server_to_executor_ms": 2.3380001075565815, + "executor_queue_ms": 1.9950000569224358, + "scheduled_to_first_token_ms": 32.359999837353826, + "first_token_to_server_response_ms": 2.022000029683113, + "server_ttft_ms": 38.715000031515956, + "first_iter": 29116, + "host_step_ms": 31.70609474182129, + "device_step_ms": 30.434080123901367 + }, + { + "server_to_executor_ms": 2.219000132754445, + "executor_queue_ms": 1.8970000091940165, + "scheduled_to_first_token_ms": 35.31800000928342, + "first_token_to_server_response_ms": 1.7629999201744795, + "server_ttft_ms": 41.197000071406364, + "first_iter": 29665, + "host_step_ms": 34.3470573425293, + "device_step_ms": 33.1104621887207 + }, + { + "server_to_executor_ms": 1.8619999755173922, + "executor_queue_ms": 3.172999946400523, + "scheduled_to_first_token_ms": 32.45800011791289, + "first_token_to_server_response_ms": 2.0779999904334545, + "server_ttft_ms": 39.57100003026426, + "first_iter": 29695, + "host_step_ms": 31.241655349731445, + "device_step_ms": 29.303743362426758 + }, + { + "server_to_executor_ms": 2.2769998759031296, + "executor_queue_ms": 3.7320000119507313, + "scheduled_to_first_token_ms": 32.407999970018864, + "first_token_to_server_response_ms": 2.4560000747442245, + "server_ttft_ms": 40.87299993261695, + "first_iter": 29808, + "host_step_ms": 32.7610969543457, + "device_step_ms": 31.70796775817871 + }, + { + "server_to_executor_ms": 2.858000108972192, + "executor_queue_ms": 2.930999966338277, + "scheduled_to_first_token_ms": 35.468999994918704, + "first_token_to_server_response_ms": 1.8799998797476292, + "server_ttft_ms": 43.1379999499768, + "first_iter": 30073, + "host_step_ms": 34.53183174133301, + "device_step_ms": 32.608192443847656 + }, + { + "server_to_executor_ms": 2.495999913662672, + "executor_queue_ms": 3.2939999364316463, + "scheduled_to_first_token_ms": 36.86400013975799, + "first_token_to_server_response_ms": 1.8070000223815441, + "server_ttft_ms": 44.46100001223385, + "first_iter": 30562, + "host_step_ms": 35.8736515045166, + "device_step_ms": 33.87744140625 + }, + { + "server_to_executor_ms": 2.409999957308173, + "executor_queue_ms": 3.3480001147836447, + "scheduled_to_first_token_ms": 32.620999962091446, + "first_token_to_server_response_ms": 1.8050000071525574, + "server_ttft_ms": 40.18400004133582, + "first_iter": 30594, + "host_step_ms": 33.09273719787598, + "device_step_ms": 31.946943283081055 + }, + { + "server_to_executor_ms": 2.283999929204583, + "executor_queue_ms": 3.5570000763982534, + "scheduled_to_first_token_ms": 32.68099995329976, + "first_token_to_server_response_ms": 1.436999998986721, + "server_ttft_ms": 39.95899995788932, + "first_iter": 30818, + "host_step_ms": 32.89055824279785, + "device_step_ms": 31.504255294799805 + }, + { + "server_to_executor_ms": 2.6020000223070383, + "executor_queue_ms": 4.808999830856919, + "scheduled_to_first_token_ms": 33.59400015324354, + "first_token_to_server_response_ms": 2.2670000325888395, + "server_ttft_ms": 43.27200003899634, + "first_iter": 30929, + "host_step_ms": 33.965110778808594, + "device_step_ms": 32.02467346191406 + }, + { + "server_to_executor_ms": 2.4060001596808434, + "executor_queue_ms": 2.3219999857246876, + "scheduled_to_first_token_ms": 35.37299996241927, + "first_token_to_server_response_ms": 1.660000067204237, + "server_ttft_ms": 41.76100017502904, + "first_iter": 31419, + "host_step_ms": 34.41882133483887, + "device_step_ms": 33.1987190246582 + }, + { + "server_to_executor_ms": 2.5740000419318676, + "executor_queue_ms": 2.423000056296587, + "scheduled_to_first_token_ms": 31.666999915614724, + "first_token_to_server_response_ms": 1.8740000668913126, + "server_ttft_ms": 38.53800008073449, + "first_iter": 31506, + "host_step_ms": 30.857086181640625, + "device_step_ms": 29.199871063232422 + }, + { + "server_to_executor_ms": 2.5220001116394997, + "executor_queue_ms": 5.4889998864382505, + "scheduled_to_first_token_ms": 33.71600015088916, + "first_token_to_server_response_ms": 2.4839998222887516, + "server_ttft_ms": 44.21099997125566, + "first_iter": 31708, + "host_step_ms": 33.82515907287598, + "device_step_ms": 32.07923126220703 + }, + { + "server_to_executor_ms": 2.2360000293701887, + "executor_queue_ms": 2.15000007301569, + "scheduled_to_first_token_ms": 35.10700003243983, + "first_token_to_server_response_ms": 2.319999970495701, + "server_ttft_ms": 41.81300010532141, + "first_iter": 31922, + "host_step_ms": 34.10601615905762, + "device_step_ms": 32.70646286010742 + }, + { + "server_to_executor_ms": 2.4210000410676003, + "executor_queue_ms": 2.7649998664855957, + "scheduled_to_first_token_ms": 35.10500001721084, + "first_token_to_server_response_ms": 2.3360000923275948, + "server_ttft_ms": 42.62700001709163, + "first_iter": 32301, + "host_step_ms": 34.01374816894531, + "device_step_ms": 32.13056182861328 + }, + { + "server_to_executor_ms": 2.472999971359968, + "executor_queue_ms": 3.0620000325143337, + "scheduled_to_first_token_ms": 33.2979999948293, + "first_token_to_server_response_ms": 2.3049998562783003, + "server_ttft_ms": 41.1379998549819, + "first_iter": 32366, + "host_step_ms": 32.75299072265625, + "device_step_ms": 30.858400344848633 + }, + { + "server_to_executor_ms": 2.2709998302161694, + "executor_queue_ms": 1.8339999951422215, + "scheduled_to_first_token_ms": 32.64800016768277, + "first_token_to_server_response_ms": 1.9389998633414507, + "server_ttft_ms": 38.69199985638261, + "first_iter": 32634, + "host_step_ms": 31.461000442504883, + "device_step_ms": 30.097856521606445 + }, + { + "server_to_executor_ms": 3.4680000972002745, + "executor_queue_ms": 3.8850000128149986, + "scheduled_to_first_token_ms": 33.75199995934963, + "first_token_to_server_response_ms": 1.8390000332146883, + "server_ttft_ms": 42.944000102579594, + "first_iter": 32866, + "host_step_ms": 33.17427635192871, + "device_step_ms": 31.381311416625977 + } + ], + "identity": { + "trtllm_source": "93cb6518b6d6dbd6095748189e626db731f44545", + "image_amd64_digest": "sha256:fe2f17d0c9698bafeb9f437929003c6badc56d17e8382a02505233145a07f61f", + "client_commit": "2baba8e27be8529b4453afc953f9859ec092c73d", + "checkpoint_revision": "b5c939de8f754692c1647ca79fbf85e8c1e70f8a", + "runtime": { + "trtllm": "1.3.0rc14", + "torch": "2.11.0a0+eb65b36914.nv26.02", + "cuda": "13.1", + "gpu": "NVIDIA B200", + "architecture": "x86_64", + "model_revision": "b5c939de8f754692c1647ca79fbf85e8c1e70f8a" + } + }, + "historical_identity_limits": [ + "Historical model revision was not pinned by the original recipe. This run pins the revision used to resolve the replay config.", + "Original image digest is unavailable; current resolved rc14 tag digest is pinned.", + "CPU allocation, GPU node and serving dependencies are recorded but do not recreate the historical host." + ], + "interpretation": "The uninstrumented TTFT residual reproduces. Diagnostic timing localizes most server TTFT to scheduled-to-first-token execution. Device event intervals include possible host launch gaps and synchronization; they do not establish a pure kernel timing defect. Metrics exclude 8 warmups by server arrival order and verify the warmup completion boundary. Client-minus-server is a difference of aggregate means, not an ID-paired transport measurement.", + "hashes": { + "gpt-ttft/baseline/client-1.log": "31dd499fbda1c64e6307325c5cc8e9a11317a41790eb40684d64643acecc34ee", + "gpt-ttft/baseline/client-2.log": "c54dd1116ebc432560c543aa8df001d43abdca4fe069d2a35c9fd8a4fe852357", + "gpt-ttft/baseline/client-3.log": "17ef1834a77a4ef567cdd9e479eb4622da8f5fa2edca40f26c7f28e51758d159", + "gpt-ttft/baseline/gpu-before.txt": "b232d54d42e62b8a6c7fb1a0d0cf38f5f356e791fee534bcb3d2d8f946f96bb4", + "gpt-ttft/baseline/metrics-1.json": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945", + "gpt-ttft/baseline/metrics-2.json": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945", + "gpt-ttft/baseline/metrics-3.json": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945", + "gpt-ttft/baseline/metrics-before-1.json": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945", + "gpt-ttft/baseline/metrics-before-2.json": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945", + "gpt-ttft/baseline/metrics-before-3.json": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945", + "gpt-ttft/baseline/options.yaml": "57893feeae1c24208e86379d06e1f63fbdef0a3a02de1e5c4d55da99fd41009f", + "gpt-ttft/baseline/result-1.json": "17f5a3c2f431c0525a2e43a6a41dddc5db442e2cc2d16bcd4ffd182f6e042c0b", + "gpt-ttft/baseline/result-2.json": "d64ebb8abc5d242880ef399401c91ec30ba0c663b18bb64786e1dace6bdbce4f", + "gpt-ttft/baseline/result-3.json": "4309a6be258d095991222cd9adfc388cfe8e78d06f3f46f11fd7210497dbe153", + "gpt-ttft/baseline/runtime.json": "870e0ebe2f083bf2fc1656672411f422199a92456e80460480532e825c24cc42", + "gpt-ttft/baseline/server.log": "cbc1f1bae82b15aa2aa80681ea03510aa2a24a39877cc3f45e77ceec420576e2", + "gpt-ttft/baseline/telemetry.csv": "b6a6b3a940773a93104c96d4f9c94b8de927dae6a351255e7cf1fc44b201fc08", + "gpt-ttft/diagnostic/client-1.log": "8c9a5a9d0b88df0cd988af1868b5d1092db316600fdd85a24cea3259c812acda", + "gpt-ttft/diagnostic/client-2.log": "e9159164e6da6038adb12d3cb9d6b71a1f4252cd22f71aff15d98d4e1d72b3ef", + "gpt-ttft/diagnostic/client-3.log": "e5aede669de1448b36e43c95b486d34d7e9bbbe80db47b829a8899d948198d83", + "gpt-ttft/diagnostic/gpu-before.txt": "5e7789c8c265ccdae4a9f855a10477b6e0363d13a9eccf6067bb4dde59051f2c", + "gpt-ttft/diagnostic/metrics-1.json": "3faff3d6b76a6e24f154ceb9c24ce58dd49f2a51843af01e6a72ead207221f33", + "gpt-ttft/diagnostic/metrics-2.json": "0ce07c5d7065aa81cfd4af005987cc30254533a014e42a1b2ba8c59efea8cde2", + "gpt-ttft/diagnostic/metrics-3.json": "07b0b12eabe29e18720888770b70daca50dc5b541d43329db3d1f25f27b51f11", + "gpt-ttft/diagnostic/metrics-before-1.json": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945", + "gpt-ttft/diagnostic/metrics-before-2.json": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945", + "gpt-ttft/diagnostic/metrics-before-3.json": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945", + "gpt-ttft/diagnostic/options.yaml": "6f7f204bdb3eafcfa7583d29a44721c430b1021f1f9059e4e449569912baa82f", + "gpt-ttft/diagnostic/result-1.json": "f60d7186311d494ebf01cebb0d50c130b0adbb41cfee4dba7792baf0c7247d2b", + "gpt-ttft/diagnostic/result-2.json": "7b83117d0d88449180caf792cdca1a7546b87690e8e9fd8e69faf27ed1a26501", + "gpt-ttft/diagnostic/result-3.json": "694da6ddc137bca594523ebc64d307d73fcd21a0d7753087e9830eedf2bcc576", + "gpt-ttft/diagnostic/runtime.json": "870e0ebe2f083bf2fc1656672411f422199a92456e80460480532e825c24cc42", + "gpt-ttft/diagnostic/server.log": "8c31d723ce317a8f44a04b14a04a9e917730c88685752def2480d39dce0434f9", + "gpt-ttft/diagnostic/telemetry.csv": "40a413204d77dc165120354b86989ec034696eb471d83dd4f124929e7882438f", + "gpt-ttft/phase.sh": "fd9f6c7256f2cf46edaedb60a67bb612eb7d0d0f7b25e2742f31d5c89e5c7c16", + "gpt-ttft/run.sbatch": "c08a943eb19c7007bc60637b0aae613a3a0adb310e863ed631df30f23615faa8", + "gpt-ttft/target.json": "a23a02e7abdc8a5f1392b23cbbab68fc0fff2a57cf9f77b8371b385e9b961975" + } +} diff --git a/python/aisimulate/collector/trtllm/gym-residual-qwen-20260918.json b/python/aisimulate/collector/trtllm/gym-residual-qwen-20260918.json new file mode 100644 index 000000000..170c4328d --- /dev/null +++ b/python/aisimulate/collector/trtllm/gym-residual-qwen-20260918.json @@ -0,0 +1,1385 @@ +{ + "point": { + "config_id": 1553, + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "conc": 512, + "silicon_ttft_ms": 2915.429193386558, + "silicon_tpot_ms": 35.09371775002287, + "silicon_head_sha": "f9426a550344a64a9d9b11fc58f10cb5c61d015e", + "silicon_image": "nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc18" + }, + "replay": { + "completed_requests": 5120, + "mean_ttft_ms": 1399.7995774718843, + "mean_tpot_ms": 116.84926175958326, + "median_itl_ms": 21.606636747685545, + "p75_itl_ms": 315.76599538560885, + "p90_itl_ms": 349.67405611266713, + "readmissions": 0, + "readmissions_source": "Sum of per_request[].readmission_count in w4a8-fix/client_lengths/0249.report.json.gz" + }, + "source_args": { + "attention_dp_config": { + "batching_wait_iters": 10, + "enable_balance": true, + "timeout_iters": 500 + }, + "backend": "pytorch", + "block_size": 32, + "cuda_graph_config": { + "enable_padding": true, + "max_batch_size": 128 + }, + "disable_overlap_scheduler": false, + "enable_attention_dp": true, + "enable_chunked_prefill": false, + "enable_expert_parallel": true, + "enable_iter_perf_stats": true, + "enable_layerwise_nvtx_marker": false, + "enable_prefix_caching": false, + "ep_size": 8, + "free_gpu_memory_fraction": 0.9, + "kv_cache_config": { + "dtype": "fp8", + "enable_block_reuse": false, + "free_gpu_memory_fraction": 0.9 + }, + "kv_cache_dtype": "fp8", + "max_batch_size": 128, + "max_model_len": 9472, + "max_num_batched_tokens": 32768, + "max_num_seqs": 128, + "max_num_tokens": 32768, + "max_seq_len": 9472, + "moe_config": { + "backend": "CUTEDSL", + "use_low_precision_moe_combine": true + }, + "moe_expert_parallel_size": 8, + "num_postprocess_workers": 4, + "port": 8000, + "print_iter_log": true, + "recipe_environment": {}, + "scheduler_config": { + "capacity_scheduler_policy": "MAX_UTILIZATION", + "context_chunking_policy": "FIRST_COME_FIRST_SERVED" + }, + "stream_interval": 20, + "tensor_parallel_size": 8, + "tp_size": 8, + "trust_remote_code": true + }, + "controlled_shape_probes": [ + { + "prefill_tokens": 0, + "decode_requests": 64, + "total_ms": 21.296758499874564, + "ops_ms": { + "generation_gdn_moe": 5.877504229545593, + "generation_gdn_recurrence": 3.2862239778041844, + "generation_full_moe": 1.9591680765151978, + "generation_attention": 1.7407140162666999, + "generation_gdn_in_proj_gemm": 1.4266746755920425, + "generation_gdn_moe_post_dispatch": 1.22085, + "generation_gdn_moe_pre_dispatch": 0.9267749999999999, + "generation_gdn_out_proj_gemm": 0.7170000113546848, + "generation_qkv_gemm": 0.4091212255861758, + "generation_full_moe_post_dispatch": 0.40695, + "generation_logits_gemm": 0.3743678187554012, + "generation_gdn_shared_gate_up_gemm": 0.3271680101752281, + "generation_full_moe_pre_dispatch": 0.308925, + "generation_gdn_router_gemm": 0.2823360040783882, + "generation_proj_gemm": 0.23900000378489497, + "generation_gdn_in_proj_ba_gemm": 0.2103840038180351, + "generation_gdn_conv1d": 0.19843200221657753, + "generation_gdn_shared_down_gemm": 0.1534079946577549, + "generation_gdn_norm": 0.1503201038961039, + "generation_gdn_ffn_norm": 0.1503201038961039, + "generation_gdn_shared_merge": 0.14649007792207794, + "generation_gdn_shared_expert_gate_mul": 0.14266005194805195, + "generation_gdn_shared_act_gate": 0.13787251948051948, + "generation_full_shared_gate_up_gemm": 0.1090560033917427, + "generation_full_router_gemm": 0.09411200135946274, + "generation_full_shared_down_gemm": 0.05113599821925163, + "generation_full_attn_norm": 0.050106701298701296, + "generation_full_ffn_norm": 0.050106701298701296, + "generation_full_shared_merge": 0.048830025974025984, + "generation_full_shared_expert_gate_mul": 0.04755335064935065, + "generation_full_shared_act_gate": 0.045957506493506495, + "generation_gdn_shared_expert_gate_gemm": 0.003112644155844156, + "generation_embedding": 0.0030851116883116882, + "generation_full_shared_expert_gate_gemm": 0.001037548051948052, + "generation_embedding_ar": 0.0, + "generation_gdn_ar": 0.0, + "generation_full_ar": 0.0, + "generation_p2p": 0.0 + }, + "raw": [ + [ + [ + "generation_embedding", + 0.0030851116883116882, + 0.0, + "empirical", + null + ], + [ + "generation_embedding_ar", + 0.0, + 0.0, + "empirical", + null + ], + [ + "generation_gdn_norm", + 0.1503201038961039, + 0.0, + "empirical", + null + ], + [ + "generation_gdn_in_proj_gemm", + 1.4266746755920425, + 0.0, + "silicon", + null + ], + [ + "generation_gdn_in_proj_ba_gemm", + 0.2103840038180351, + 0.0, + "silicon", + null + ], + [ + "generation_gdn_conv1d", + 0.19843200221657753, + 0.0, + "silicon", + null + ], + [ + "generation_gdn_recurrence", + 3.2862239778041844, + 0.0, + "silicon", + null + ], + [ + "generation_gdn_out_proj_gemm", + 0.7170000113546848, + 0.0, + "silicon", + null + ], + [ + "generation_gdn_ar", + 0.0, + 0.0, + "empirical", + null + ], + [ + "generation_gdn_ffn_norm", + 0.1503201038961039, + 0.0, + "empirical", + null + ], + [ + "generation_gdn_router_gemm", + 0.2823360040783882, + 0.0, + "silicon", + null + ], + [ + "generation_gdn_moe_pre_dispatch", + 0.9267749999999999, + 0.0, + "silicon", + null + ], + [ + "generation_gdn_moe", + 5.877504229545593, + 3339.940757641196, + "silicon", + null + ], + [ + "generation_gdn_shared_expert_gate_gemm", + 0.003112644155844156, + 0.0, + "sol", + null + ], + [ + "generation_gdn_shared_gate_up_gemm", + 0.3271680101752281, + 0.0, + "silicon", + null + ], + [ + "generation_gdn_shared_act_gate", + 0.13787251948051948, + 0.0, + "empirical", + null + ], + [ + "generation_gdn_shared_down_gemm", + 0.1534079946577549, + 0.0, + "silicon", + null + ], + [ + "generation_gdn_shared_expert_gate_mul", + 0.14266005194805195, + 0.0, + "empirical", + null + ], + [ + "generation_gdn_shared_merge", + 0.14649007792207794, + 0.0, + "empirical", + null + ], + [ + "generation_gdn_moe_post_dispatch", + 1.22085, + 0.0, + "silicon", + null + ], + [ + "generation_full_attn_norm", + 0.050106701298701296, + 0.0, + "empirical", + null + ], + [ + "generation_qkv_gemm", + 0.4091212255861758, + 0.0, + "silicon", + null + ], + [ + "generation_attention", + 1.7407140162666999, + 0.0, + "mixed", + null + ], + [ + "generation_proj_gemm", + 0.23900000378489497, + 0.0, + "silicon", + null + ], + [ + "generation_full_ar", + 0.0, + 0.0, + "empirical", + null + ], + [ + "generation_full_ffn_norm", + 0.050106701298701296, + 0.0, + "empirical", + null + ], + [ + "generation_full_router_gemm", + 0.09411200135946274, + 0.0, + "silicon", + null + ], + [ + "generation_full_moe_pre_dispatch", + 0.308925, + 0.0, + "silicon", + null + ], + [ + "generation_full_moe", + 1.9591680765151978, + 1113.3135858803987, + "silicon", + null + ], + [ + "generation_full_shared_expert_gate_gemm", + 0.001037548051948052, + 0.0, + "sol", + null + ], + [ + "generation_full_shared_gate_up_gemm", + 0.1090560033917427, + 0.0, + "silicon", + null + ], + [ + "generation_full_shared_act_gate", + 0.045957506493506495, + 0.0, + "empirical", + null + ], + [ + "generation_full_shared_down_gemm", + 0.05113599821925163, + 0.0, + "silicon", + null + ], + [ + "generation_full_shared_expert_gate_mul", + 0.04755335064935065, + 0.0, + "empirical", + null + ], + [ + "generation_full_shared_merge", + 0.048830025974025984, + 0.0, + "empirical", + null + ], + [ + "generation_full_moe_post_dispatch", + 0.40695, + 0.0, + "silicon", + null + ], + [ + "generation_logits_gemm", + 0.3743678187554012, + 0.0, + "silicon", + null + ], + [ + "generation_p2p", + 0.0, + 0.0, + "silicon", + null + ] + ] + ] + }, + { + "prefill_tokens": 8192, + "decode_requests": 63, + "total_ms": 348.07352138803776, + "ops_ms": { + "context_gdn_moe": 106.67924380720706, + "context_gdn_in_proj_gemm": 39.263254786581655, + "context_gdn_scan": 38.95699240488466, + "context_full_moe": 35.559747935735686, + "context_gdn_moe_post_dispatch": 34.70198648071289, + "context_gdn_out_proj_gemm": 15.344664687654586, + "context_attention": 11.618762380313546, + "context_full_moe_post_dispatch": 11.567328826904298, + "context_qkv_gemm": 10.812369121078838, + "context_gdn_moe_pre_dispatch": 10.553711881256104, + "context_proj_gemm": 5.114888229218195, + "context_gdn_shared_gate_up_gemm": 4.193096553819487, + "context_full_moe_pre_dispatch": 3.5179039604187015, + "context_gdn_norm": 2.111054025974026, + "context_gdn_ffn_norm": 2.111054025974026, + "context_gdn_shared_down_gemm": 2.0929180279126736, + "generation_attention": 1.8725684262581672, + "context_gdn_shared_merge": 1.6170405194805195, + "context_full_shared_gate_up_gemm": 1.3976988512731623, + "context_gdn_conv1d": 1.352346191970355, + "context_gdn_router_gemm": 1.1344584530852444, + "context_gdn_shared_expert_gate_mul": 1.1230270129870128, + "context_full_attn_norm": 0.7036846753246753, + "context_full_ffn_norm": 0.7036846753246753, + "context_full_shared_down_gemm": 0.6976393426375579, + "context_full_shared_merge": 0.5390135064935065, + "context_gdn_shared_act_gate": 0.5055101298701299, + "context_gdn_in_proj_ba_gemm": 0.438045983111806, + "context_gdn_shared_expert_gate_gemm": 0.3953551675324675, + "context_full_router_gemm": 0.37815281769508147, + "context_full_shared_expert_gate_mul": 0.3743423376623376, + "context_logits_gemm": 0.3277096512950256, + "context_full_shared_act_gate": 0.16850337662337664, + "context_full_shared_expert_gate_gemm": 0.13178505584415584, + "context_embedding": 0.013978077922077921, + "context_embedding_ar": 0.0, + "context_gdn_ar": 0.0, + "context_full_ar": 0.0, + "context_p2p": 0.0 + }, + "raw": [ + [ + [ + "context_embedding", + 0.013978077922077921, + 0.0, + "empirical", + null + ], + [ + "context_embedding_ar", + 0.0, + 0.0, + "empirical", + null + ], + [ + "context_gdn_norm", + 2.111054025974026, + 0.0, + "empirical", + null + ], + [ + "context_gdn_in_proj_gemm", + 39.263254786581655, + 0.0, + "silicon", + null + ], + [ + "context_gdn_in_proj_ba_gemm", + 0.438045983111806, + 0.0, + "silicon", + null + ], + [ + "context_gdn_conv1d", + 1.352346191970355, + 0.0, + "silicon", + null + ], + [ + "context_gdn_scan", + 38.95699240488466, + 0.0, + "silicon", + null + ], + [ + "context_gdn_out_proj_gemm", + 15.344664687654586, + 0.0, + "silicon", + null + ], + [ + "context_gdn_ar", + 0.0, + 0.0, + "empirical", + null + ], + [ + "context_gdn_ffn_norm", + 2.111054025974026, + 0.0, + "empirical", + null + ], + [ + "context_gdn_router_gemm", + 1.1344584530852444, + 0.0, + "silicon", + null + ], + [ + "context_gdn_moe_pre_dispatch", + 10.553711881256104, + 0.0, + "silicon", + null + ], + [ + "context_gdn_moe", + 106.67924380720706, + 57773.49575421454, + "silicon", + null + ], + [ + "context_gdn_moe_post_dispatch", + 34.70198648071289, + 0.0, + "silicon", + null + ], + [ + "context_gdn_shared_expert_gate_gemm", + 0.3953551675324675, + 0.0, + "sol", + null + ], + [ + "context_gdn_shared_gate_up_gemm", + 4.193096553819487, + 0.0, + "silicon", + null + ], + [ + "context_gdn_shared_act_gate", + 0.5055101298701299, + 0.0, + "empirical", + null + ], + [ + "context_gdn_shared_down_gemm", + 2.0929180279126736, + 0.0, + "silicon", + null + ], + [ + "context_gdn_shared_expert_gate_mul", + 1.1230270129870128, + 0.0, + "empirical", + null + ], + [ + "context_gdn_shared_merge", + 1.6170405194805195, + 0.0, + "empirical", + null + ], + [ + "context_full_attn_norm", + 0.7036846753246753, + 0.0, + "empirical", + null + ], + [ + "context_qkv_gemm", + 10.812369121078838, + 0.0, + "silicon", + null + ], + [ + "context_proj_gemm", + 5.114888229218195, + 0.0, + "silicon", + null + ], + [ + "context_full_ar", + 0.0, + 0.0, + "empirical", + null + ], + [ + "context_full_ffn_norm", + 0.7036846753246753, + 0.0, + "empirical", + null + ], + [ + "context_full_router_gemm", + 0.37815281769508147, + 0.0, + "silicon", + null + ], + [ + "context_full_moe_pre_dispatch", + 3.5179039604187015, + 0.0, + "silicon", + null + ], + [ + "context_full_moe", + 35.559747935735686, + 19257.831918071515, + "silicon", + null + ], + [ + "context_full_moe_post_dispatch", + 11.567328826904298, + 0.0, + "silicon", + null + ], + [ + "context_full_shared_expert_gate_gemm", + 0.13178505584415584, + 0.0, + "sol", + null + ], + [ + "context_full_shared_gate_up_gemm", + 1.3976988512731623, + 0.0, + "silicon", + null + ], + [ + "context_full_shared_act_gate", + 0.16850337662337664, + 0.0, + "empirical", + null + ], + [ + "context_full_shared_down_gemm", + 0.6976393426375579, + 0.0, + "silicon", + null + ], + [ + "context_full_shared_expert_gate_mul", + 0.3743423376623376, + 0.0, + "empirical", + null + ], + [ + "context_full_shared_merge", + 0.5390135064935065, + 0.0, + "empirical", + null + ], + [ + "context_logits_gemm", + 0.3277096512950256, + 0.0, + "silicon", + null + ], + [ + "context_p2p", + 0.0, + 0.0, + "silicon", + null + ] + ], + [ + [ + "context_attention", + 11.618762380313546, + 0.0, + "mixed", + null + ] + ], + [ + [ + "generation_attention", + 1.8725684262581672, + 0.0, + "mixed", + null + ] + ] + ] + }, + { + "prefill_tokens": 16384, + "decode_requests": 62, + "total_ms": 686.1578198016927, + "ops_ms": { + "context_gdn_moe": 212.53141655400694, + "context_gdn_in_proj_gemm": 78.34115899381607, + "context_gdn_scan": 76.60430236626415, + "context_full_moe": 70.84380551800231, + "context_gdn_moe_post_dispatch": 69.13493272705078, + "context_gdn_out_proj_gemm": 30.456656743772328, + "context_full_moe_post_dispatch": 23.044977575683593, + "context_attention": 22.08116329355207, + "context_qkv_gemm": 21.447794611656352, + "context_gdn_moe_pre_dispatch": 19.823836837005615, + "context_proj_gemm": 10.152218914590776, + "context_gdn_shared_gate_up_gemm": 7.921644625224871, + "context_full_moe_pre_dispatch": 6.607945612335205, + "context_gdn_shared_down_gemm": 4.161134881069302, + "context_gdn_norm": 4.0717879480519485, + "context_gdn_ffn_norm": 4.0717879480519485, + "context_gdn_shared_merge": 3.087590961038961, + "context_full_shared_gate_up_gemm": 2.64054820840829, + "context_gdn_conv1d": 2.481584580724302, + "context_gdn_router_gemm": 2.385850335514988, + "context_gdn_shared_expert_gate_mul": 2.103393974025974, + "generation_attention": 1.8508555535017286, + "context_full_shared_down_gemm": 1.3870449603564337, + "context_full_attn_norm": 1.3572626493506494, + "context_full_ffn_norm": 1.3572626493506494, + "context_gdn_in_proj_ba_gemm": 1.117341085417138, + "context_full_shared_merge": 1.029196987012987, + "context_gdn_shared_act_gate": 0.8731477402597403, + "context_full_router_gemm": 0.7952834451716626, + "context_gdn_shared_expert_gate_gemm": 0.7875976909090908, + "context_full_shared_expert_gate_mul": 0.7011313246753247, + "context_logits_gemm": 0.3277096512950256, + "context_full_shared_act_gate": 0.2910492467532468, + "context_full_shared_expert_gate_gemm": 0.2625325636363636, + "context_embedding": 0.024871044155844155, + "context_embedding_ar": 0.0, + "context_gdn_ar": 0.0, + "context_full_ar": 0.0, + "context_p2p": 0.0 + }, + "raw": [ + [ + [ + "context_embedding", + 0.024871044155844155, + 0.0, + "empirical", + null + ], + [ + "context_embedding_ar", + 0.0, + 0.0, + "empirical", + null + ], + [ + "context_gdn_norm", + 4.0717879480519485, + 0.0, + "empirical", + null + ], + [ + "context_gdn_in_proj_gemm", + 78.34115899381607, + 0.0, + "silicon", + null + ], + [ + "context_gdn_in_proj_ba_gemm", + 1.117341085417138, + 0.0, + "silicon", + null + ], + [ + "context_gdn_conv1d", + 2.481584580724302, + 0.0, + "silicon", + null + ], + [ + "context_gdn_scan", + 76.60430236626415, + 0.0, + "silicon", + null + ], + [ + "context_gdn_out_proj_gemm", + 30.456656743772328, + 0.0, + "silicon", + null + ], + [ + "context_gdn_ar", + 0.0, + 0.0, + "empirical", + null + ], + [ + "context_gdn_ffn_norm", + 4.0717879480519485, + 0.0, + "empirical", + null + ], + [ + "context_gdn_router_gemm", + 2.385850335514988, + 0.0, + "silicon", + null + ], + [ + "context_gdn_moe_pre_dispatch", + 19.823836837005615, + 0.0, + "silicon", + null + ], + [ + "context_gdn_moe", + 212.53141655400694, + 115099.08069943215, + "silicon", + null + ], + [ + "context_gdn_moe_post_dispatch", + 69.13493272705078, + 0.0, + "silicon", + null + ], + [ + "context_gdn_shared_expert_gate_gemm", + 0.7875976909090908, + 0.0, + "sol", + null + ], + [ + "context_gdn_shared_gate_up_gemm", + 7.921644625224871, + 0.0, + "silicon", + null + ], + [ + "context_gdn_shared_act_gate", + 0.8731477402597403, + 0.0, + "empirical", + null + ], + [ + "context_gdn_shared_down_gemm", + 4.161134881069302, + 0.0, + "silicon", + null + ], + [ + "context_gdn_shared_expert_gate_mul", + 2.103393974025974, + 0.0, + "empirical", + null + ], + [ + "context_gdn_shared_merge", + 3.087590961038961, + 0.0, + "empirical", + null + ], + [ + "context_full_attn_norm", + 1.3572626493506494, + 0.0, + "empirical", + null + ], + [ + "context_qkv_gemm", + 21.447794611656352, + 0.0, + "silicon", + null + ], + [ + "context_proj_gemm", + 10.152218914590776, + 0.0, + "silicon", + null + ], + [ + "context_full_ar", + 0.0, + 0.0, + "empirical", + null + ], + [ + "context_full_ffn_norm", + 1.3572626493506494, + 0.0, + "empirical", + null + ], + [ + "context_full_router_gemm", + 0.7952834451716626, + 0.0, + "silicon", + null + ], + [ + "context_full_moe_pre_dispatch", + 6.607945612335205, + 0.0, + "silicon", + null + ], + [ + "context_full_moe", + 70.84380551800231, + 38366.36023314405, + "silicon", + null + ], + [ + "context_full_moe_post_dispatch", + 23.044977575683593, + 0.0, + "silicon", + null + ], + [ + "context_full_shared_expert_gate_gemm", + 0.2625325636363636, + 0.0, + "sol", + null + ], + [ + "context_full_shared_gate_up_gemm", + 2.64054820840829, + 0.0, + "silicon", + null + ], + [ + "context_full_shared_act_gate", + 0.2910492467532468, + 0.0, + "empirical", + null + ], + [ + "context_full_shared_down_gemm", + 1.3870449603564337, + 0.0, + "silicon", + null + ], + [ + "context_full_shared_expert_gate_mul", + 0.7011313246753247, + 0.0, + "empirical", + null + ], + [ + "context_full_shared_merge", + 1.029196987012987, + 0.0, + "empirical", + null + ], + [ + "context_logits_gemm", + 0.3277096512950256, + 0.0, + "silicon", + null + ], + [ + "context_p2p", + 0.0, + 0.0, + "silicon", + null + ] + ], + [ + [ + "context_attention", + 22.08116329355207, + 0.0, + "mixed", + null + ] + ], + [ + [ + "generation_attention", + 1.8508555535017286, + 0.0, + "mixed", + null + ] + ] + ] + }, + { + "prefill_tokens": 32768, + "decode_requests": 60, + "total_ms": 1363.860381400988, + "ops_ms": { + "context_gdn_moe": 424.2357620476067, + "context_gdn_in_proj_gemm": 155.20523459183642, + "context_gdn_scan": 152.4656435650096, + "context_full_moe": 141.41192068253557, + "context_gdn_moe_post_dispatch": 138.00082521972655, + "context_gdn_out_proj_gemm": 61.02545473346254, + "context_full_moe_post_dispatch": 46.000275073242186, + "context_attention": 44.09956014795271, + "context_qkv_gemm": 42.866419816310945, + "context_gdn_moe_pre_dispatch": 38.56117231521606, + "context_proj_gemm": 20.341818244487513, + "context_gdn_shared_gate_up_gemm": 15.82644628401613, + "context_full_moe_pre_dispatch": 12.853724105072022, + "context_gdn_shared_down_gemm": 8.30679812640301, + "context_gdn_norm": 7.993255792207792, + "context_gdn_ffn_norm": 7.993255792207792, + "context_gdn_shared_merge": 6.028691844155844, + "context_full_shared_gate_up_gemm": 5.275482094672043, + "context_gdn_conv1d": 4.849421634855441, + "context_gdn_router_gemm": 4.761855174670927, + "context_gdn_shared_expert_gate_mul": 4.064127896103896, + "context_full_shared_down_gemm": 2.768932708801003, + "context_full_attn_norm": 2.664418597402597, + "context_full_ffn_norm": 2.664418597402597, + "context_gdn_in_proj_ba_gemm": 2.221437309810864, + "context_full_shared_merge": 2.009563948051948, + "generation_attention": 1.8074298079888513, + "context_gdn_shared_act_gate": 1.6084229610389609, + "context_full_router_gemm": 1.5872850582236424, + "context_gdn_shared_expert_gate_gemm": 1.5720827376623379, + "context_full_shared_expert_gate_mul": 1.3547092987012987, + "context_full_shared_act_gate": 0.536140987012987, + "context_full_shared_expert_gate_gemm": 0.5240275792207792, + "context_logits_gemm": 0.3277096512950256, + "context_embedding": 0.04665697662337662, + "context_embedding_ar": 0.0, + "context_gdn_ar": 0.0, + "context_full_ar": 0.0, + "context_p2p": 0.0 + }, + "raw": [ + [ + [ + "context_embedding", + 0.04665697662337662, + 0.0, + "empirical", + null + ], + [ + "context_embedding_ar", + 0.0, + 0.0, + "empirical", + null + ], + [ + "context_gdn_norm", + 7.993255792207792, + 0.0, + "empirical", + null + ], + [ + "context_gdn_in_proj_gemm", + 155.20523459183642, + 0.0, + "silicon", + null + ], + [ + "context_gdn_in_proj_ba_gemm", + 2.221437309810864, + 0.0, + "silicon", + null + ], + [ + "context_gdn_conv1d", + 4.849421634855441, + 0.0, + "silicon", + null + ], + [ + "context_gdn_scan", + 152.4656435650096, + 0.0, + "silicon", + null + ], + [ + "context_gdn_out_proj_gemm", + 61.02545473346254, + 0.0, + "silicon", + null + ], + [ + "context_gdn_ar", + 0.0, + 0.0, + "empirical", + null + ], + [ + "context_gdn_ffn_norm", + 7.993255792207792, + 0.0, + "empirical", + null + ], + [ + "context_gdn_router_gemm", + 4.761855174670927, + 0.0, + "silicon", + null + ], + [ + "context_gdn_moe_pre_dispatch", + 38.56117231521606, + 0.0, + "silicon", + null + ], + [ + "context_gdn_moe", + 424.2357620476067, + 229750.25058986738, + "silicon", + null + ], + [ + "context_gdn_moe_post_dispatch", + 138.00082521972655, + 0.0, + "silicon", + null + ], + [ + "context_gdn_shared_expert_gate_gemm", + 1.5720827376623379, + 0.0, + "sol", + null + ], + [ + "context_gdn_shared_gate_up_gemm", + 15.82644628401613, + 0.0, + "silicon", + null + ], + [ + "context_gdn_shared_act_gate", + 1.6084229610389609, + 0.0, + "empirical", + null + ], + [ + "context_gdn_shared_down_gemm", + 8.30679812640301, + 0.0, + "silicon", + null + ], + [ + "context_gdn_shared_expert_gate_mul", + 4.064127896103896, + 0.0, + "empirical", + null + ], + [ + "context_gdn_shared_merge", + 6.028691844155844, + 0.0, + "empirical", + null + ], + [ + "context_full_attn_norm", + 2.664418597402597, + 0.0, + "empirical", + null + ], + [ + "context_qkv_gemm", + 42.866419816310945, + 0.0, + "silicon", + null + ], + [ + "context_proj_gemm", + 20.341818244487513, + 0.0, + "silicon", + null + ], + [ + "context_full_ar", + 0.0, + 0.0, + "empirical", + null + ], + [ + "context_full_ffn_norm", + 2.664418597402597, + 0.0, + "empirical", + null + ], + [ + "context_full_router_gemm", + 1.5872850582236424, + 0.0, + "silicon", + null + ], + [ + "context_full_moe_pre_dispatch", + 12.853724105072022, + 0.0, + "silicon", + null + ], + [ + "context_full_moe", + 141.41192068253557, + 76583.41686328914, + "silicon", + null + ], + [ + "context_full_moe_post_dispatch", + 46.000275073242186, + 0.0, + "silicon", + null + ], + [ + "context_full_shared_expert_gate_gemm", + 0.5240275792207792, + 0.0, + "sol", + null + ], + [ + "context_full_shared_gate_up_gemm", + 5.275482094672043, + 0.0, + "silicon", + null + ], + [ + "context_full_shared_act_gate", + 0.536140987012987, + 0.0, + "empirical", + null + ], + [ + "context_full_shared_down_gemm", + 2.768932708801003, + 0.0, + "silicon", + null + ], + [ + "context_full_shared_expert_gate_mul", + 1.3547092987012987, + 0.0, + "empirical", + null + ], + [ + "context_full_shared_merge", + 2.009563948051948, + 0.0, + "empirical", + null + ], + [ + "context_logits_gemm", + 0.3277096512950256, + 0.0, + "silicon", + null + ], + [ + "context_p2p", + 0.0, + 0.0, + "silicon", + null + ] + ], + [ + [ + "context_attention", + 44.09956014795271, + 0.0, + "mixed", + null + ] + ], + [ + [ + "generation_attention", + 1.8074298079888513, + 0.0, + "mixed", + null + ] + ] + ] + } + ], + "hashes": { + "residual-20260918/upstream/qwen-recipe.sh": "b2e0acd41478295bef2d8d32cce98830eae9bd74760966a722b24f94e62c350b", + "residual-20260918/upstream/rc18-py_executor.py": "1e0ff56ed5bab5ae6aecba507218b77cf2ce92e24ba615d939be662ba3bf14f7", + "w4a8-fix/client_lengths/0249.spec.json": "3e1bb3f11f9b2d9aff8886a6e947b0b22578c924c02f535e51f9143e8fadd0a7", + "w4a8-fix/client_lengths/0249.report.json.gz": "9eb473a01ff764089caf4308184c7aff488979bbaeaa369e3abd06d4783d28fe" + }, + "interpretation": "Scheduling controls are confirmed absent from replay; attribution of the residual TPOT error remains supported, not quantified or fixed. Controlled shape probes are not observed silicon step timings.", + "non_dp_controls_evidence": { + "source": "residual-20260918/upstream/qwen-recipe.sh", + "batch_wait_timeout_iters": 50, + "batch_wait_max_tokens_ratio": 0.45, + "scope": "Recipe only; no resolved non-DP source_args or non-DP readmission count retained in this summary." + } +} diff --git a/python/aisimulate/collector/trtllm/gym-tpot-20260917.json b/python/aisimulate/collector/trtllm/gym-tpot-20260917.json new file mode 100644 index 000000000..cedd626b6 --- /dev/null +++ b/python/aisimulate/collector/trtllm/gym-tpot-20260917.json @@ -0,0 +1,6010 @@ +{ + "collector_ref": "3830b5a456c2d6c71c23db6ac923b7ccf0ab1fad", + "summary": { + "planned": 6, + "passed": 6, + "failed": 0, + "unattempted": 0, + "worker_exit_codes": [ + 0, + 0 + ], + "runner_sha256": "84ba6f19e9a88b127b43eeafc5e4af3ffe63672db11e2e89a4ef7f4e19cabece" + }, + "cuda_graph_measurements": 162, + "replaced_rows": 162, + "unchanged_rows": 218214, + "old_parquet_sha256": "0bc472446e0993221f28c454645fc4054eb73162270868c2ea1332b5572d7803", + "new_parquet_sha256": "1ff05df3a5715ca5d80e2435ba5ec8221e25da7f414ae7834a87af84c46b9b87", + "collector_archive_sha256": "d2faeb65a3b63b714ebf6b67c342a3638c174f1342d1bec071a946d01a3cba6f", + "min_ratio": 0.7350538558813405, + "median_ratio": 0.9967503388437886, + "max_ratio": 1.5131040897951098, + "runtime_commit": "89d2051137b772944a3e303c452108811790bef4", + "collection_job": 1994597, + "contract_probe_job": 1994556, + "source_rc14": "93cb6518b6d6dbd6095748189e626db731f44545", + "source_rc20": "c25c23f71786bad54d192893d696ce8043426eca", + "image_digest": "sha256:9b3b4dfb811caa9420fa99a6f958155f6a1f727ffc2b5a5c2d9d2ce51fdc323d", + "mape": { + "trtllm": { + "points": 166, + "before_ttft_mape_pct": 66.24674189164078, + "before_tpot_mape_pct": 68.1603347011892, + "after_ttft_mape_pct": 65.99932710369265, + "after_tpot_mape_pct": 66.71573133021964 + }, + "DeepSeek-R1-0528/b200": { + "points": 31, + "before_ttft_mape_pct": 64.00001913374423, + "before_tpot_mape_pct": 49.31907317174891, + "after_ttft_mape_pct": 64.00001913374423, + "after_tpot_mape_pct": 49.31907317174891 + }, + "gpt-oss-120b/h200": { + "points": 33, + "before_ttft_mape_pct": 79.53530478234222, + "before_tpot_mape_pct": 21.34985420167682, + "after_ttft_mape_pct": 79.53530478234222, + "after_tpot_mape_pct": 21.34985420167682 + }, + "gpt-oss-120b/b200": { + "points": 31, + "before_ttft_mape_pct": 86.99302235262971, + "before_tpot_mape_pct": 176.5171818822163, + "after_ttft_mape_pct": 85.6681560687784, + "after_tpot_mape_pct": 168.78156383121805 + }, + "DeepSeek-R1-0528/h200": { + "points": 10, + "before_ttft_mape_pct": 39.50005031788265, + "before_tpot_mape_pct": 24.21499332147967, + "after_ttft_mape_pct": 39.50005031788265, + "after_tpot_mape_pct": 24.21499332147967 + }, + "sglang": { + "points": 97, + "before_ttft_mape_pct": 69.60881280689533, + "before_tpot_mape_pct": 12.550584383154693, + "after_ttft_mape_pct": 69.60881280689533, + "after_tpot_mape_pct": 12.550584383154693 + }, + "MiniMax-M2.5/b300": { + "points": 21, + "before_ttft_mape_pct": 48.53914152597352, + "before_tpot_mape_pct": 44.033185511753715, + "after_ttft_mape_pct": 48.53914152597352, + "after_tpot_mape_pct": 44.033185511753715 + }, + "MiniMax-M2.5/b200": { + "points": 21, + "before_ttft_mape_pct": 54.47878535865214, + "before_tpot_mape_pct": 39.13142389133739, + "after_ttft_mape_pct": 54.47878535865214, + "after_tpot_mape_pct": 39.13142389133739 + }, + "Qwen-3.5-397B-A17B/b200": { + "points": 19, + "before_ttft_mape_pct": 59.638570650606255, + "before_tpot_mape_pct": 85.29156496786527, + "after_ttft_mape_pct": 59.638570650606255, + "after_tpot_mape_pct": 85.29156496786527 + } + }, + "paired": [ + { + "index": 0, + "config_id": 75, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "2e6e0174bdde4c732db69f92adacd5ff9d3a17c0b01df14f4c3b5d498a9ba7fa", + "completed": 640, + "silicon_ttft_ms": 569.0219993219216, + "before_ttft_ms": 159.83202601562516, + "after_ttft_ms": 159.83202601562516, + "before_ttft_ape_pct": 71.91109900740395, + "after_ttft_ape_pct": 71.91109900740395, + "silicon_tpot_ms": 20.063968440222716, + "before_tpot_ms": 30.772882970770873, + "after_tpot_ms": 30.772882970770873, + "before_tpot_ape_pct": 53.37386052242656, + "after_tpot_ape_pct": 53.37386052242656 + }, + { + "index": 1, + "config_id": 75, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "1ff905505e9b59e95088def93a4b30aaa9fccbd5e29c553108441e9f40fa4ac4", + "completed": 40, + "silicon_ttft_ms": 189.9082841584459, + "before_ttft_ms": 66.71030040000043, + "after_ttft_ms": 66.71030040000043, + "before_ttft_ape_pct": 64.87235894125496, + "after_ttft_ape_pct": 64.87235894125496, + "silicon_tpot_ms": 8.689942316597076, + "before_tpot_ms": 12.775904542260609, + "after_tpot_ms": 12.775904542260609, + "before_tpot_ape_pct": 47.0194401389717, + "after_tpot_ape_pct": 47.0194401389717 + }, + { + "index": 2, + "config_id": 75, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "32a771dfc65b8dd51db97696fee90359a3f35da207bac68999ce077257381dca", + "completed": 80, + "silicon_ttft_ms": 180.97360140527599, + "before_ttft_ms": 77.12835784999945, + "after_ttft_ms": 77.12835784999945, + "before_ttft_ape_pct": 57.381431738612186, + "after_ttft_ape_pct": 57.381431738612186, + "silicon_tpot_ms": 10.862899920330408, + "before_tpot_ms": 15.673729699299201, + "after_tpot_ms": 15.673729699299201, + "before_tpot_ape_pct": 44.28679095132884, + "after_tpot_ape_pct": 44.28679095132884 + }, + { + "index": 3, + "config_id": 75, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "658b6bd76a1c4e72b94f0a315f93d9b9d13ce4c6d85a90d064636b25b6ef6103", + "completed": 80, + "silicon_ttft_ms": 337.99734528001864, + "before_ttft_ms": 285.6528664874992, + "after_ttft_ms": 285.6528664874992, + "before_ttft_ape_pct": 15.486653822429863, + "after_ttft_ape_pct": 15.486653822429863, + "silicon_tpot_ms": 11.564257697095588, + "before_tpot_ms": 17.113342453175882, + "after_tpot_ms": 17.113342453175882, + "before_tpot_ape_pct": 47.98478987089651, + "after_tpot_ape_pct": 47.98478987089651 + }, + { + "index": 4, + "config_id": 75, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "fbb80237fc248c1ac9b303eb026c35220a2255cbc00c24fc85af8ddafa4e84aa", + "completed": 640, + "silicon_ttft_ms": 878.5957236948889, + "before_ttft_ms": 883.8029458484368, + "after_ttft_ms": 883.8029458484368, + "before_ttft_ape_pct": 0.5926755631872727, + "after_ttft_ape_pct": 0.5926755631872727, + "silicon_tpot_ms": 30.99640643828281, + "before_tpot_ms": 42.485899719985895, + "after_tpot_ms": 42.485899719985895, + "before_tpot_ape_pct": 37.06717843108653, + "after_tpot_ape_pct": 37.06717843108653 + }, + { + "index": 5, + "config_id": 75, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "ddaa845bd5cc2aeace637c21969db9fc5ff4471dc486de01b62e2f61bb9139ea", + "completed": 40, + "silicon_ttft_ms": 322.6864557269437, + "before_ttft_ms": 231.35404255000066, + "after_ttft_ms": 231.35404255000066, + "before_ttft_ape_pct": 28.303764089258287, + "after_ttft_ape_pct": 28.303764089258287, + "silicon_tpot_ms": 9.291597712567906, + "before_tpot_ms": 13.538788294853068, + "after_tpot_ms": 13.538788294853068, + "before_tpot_ape_pct": 45.710013645342926, + "after_tpot_ape_pct": 45.710013645342926 + }, + { + "index": 6, + "config_id": 75, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 128, + "spec_sha256": "d0c50db45dec09af52bc904eb978e328b1a9e8f4370e5b1402dd8c8b112a779e", + "completed": 1280, + "silicon_ttft_ms": 591.6610352298449, + "before_ttft_ms": 248.20115803984396, + "after_ttft_ms": 248.20115803984396, + "before_ttft_ape_pct": 58.05010922454539, + "after_ttft_ape_pct": 58.05010922454539, + "silicon_tpot_ms": 24.537629970691082, + "before_tpot_ms": 36.70182049045927, + "after_tpot_ms": 36.70182049045927, + "before_tpot_ape_pct": 49.573616255105634, + "after_tpot_ape_pct": 49.573616255105634 + }, + { + "index": 7, + "config_id": 75, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 256, + "spec_sha256": "f5980b8642eeedcb878ba11f01698b2561309eb12a2695da282205c83db350b5", + "completed": 2560, + "silicon_ttft_ms": 3615.081174194711, + "before_ttft_ms": 3575.3454541824203, + "after_ttft_ms": 3575.3454541824203, + "before_ttft_ape_pct": 1.0991653602672402, + "after_ttft_ape_pct": 1.0991653602672402, + "silicon_tpot_ms": 77.20712247274294, + "before_tpot_ms": 93.35945034490524, + "after_tpot_ms": 93.35945034490524, + "before_tpot_ape_pct": 20.920774346776994, + "after_tpot_ape_pct": 20.920774346776994 + }, + { + "index": 8, + "config_id": 75, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 128, + "spec_sha256": "9cf9bd924ba1f1184bfec3842d4ac854f07820c6ace77ab4fb0b6e49be89b05b", + "completed": 1280, + "silicon_ttft_ms": 1674.3441834467376, + "before_ttft_ms": 1612.691693432814, + "after_ttft_ms": 1612.691693432814, + "before_ttft_ape_pct": 3.6821873676538996, + "after_ttft_ape_pct": 3.6821873676538996, + "silicon_tpot_ms": 47.08023753882907, + "before_tpot_ms": 59.931205628683315, + "after_tpot_ms": 59.931205628683315, + "before_tpot_ape_pct": 27.295886260674674, + "after_tpot_ape_pct": 27.295886260674674 + }, + { + "index": 9, + "config_id": 82, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "46cf3c1d7d38ea8e4745401f43b018bf78d02d51bd0fce328e17ca7fc0d5919b", + "completed": 160, + "silicon_ttft_ms": 173.85605943854898, + "before_ttft_ms": 96.66793040624994, + "after_ttft_ms": 96.66793040624994, + "before_ttft_ape_pct": 44.39772147233205, + "after_ttft_ape_pct": 44.39772147233205, + "silicon_tpot_ms": 11.64908875804333, + "before_tpot_ms": 11.108461524979543, + "after_tpot_ms": 11.108461524979543, + "before_tpot_ape_pct": 4.640940113796466, + "after_tpot_ape_pct": 4.640940113796466 + }, + { + "index": 10, + "config_id": 82, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "2e6f0d98b3cb8e17dd9a877cde71358f786819658908b4fdffda1b19f397e349", + "completed": 40, + "silicon_ttft_ms": 161.359184270259, + "before_ttft_ms": 74.32791005000028, + "after_ttft_ms": 74.32791005000028, + "before_ttft_ape_pct": 53.936362292518055, + "after_ttft_ape_pct": 53.936362292518055, + "silicon_tpot_ms": 7.576374324475022, + "before_tpot_ms": 7.554796660340204, + "after_tpot_ms": 7.554796660340204, + "before_tpot_ape_pct": 0.28480198061376916, + "after_tpot_ape_pct": 0.28480198061376916 + }, + { + "index": 11, + "config_id": 82, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "e0b8cae81dd05f5e39c17d28fc44547b18e99b4123e92ea8ed4db6dad6a3eb61", + "completed": 80, + "silicon_ttft_ms": 154.2601879540598, + "before_ttft_ms": 84.59517222500001, + "after_ttft_ms": 84.59517222500001, + "before_ttft_ape_pct": 45.16072270688968, + "after_ttft_ape_pct": 45.16072270688968, + "silicon_tpot_ms": 9.302715206467827, + "before_tpot_ms": 8.844276931642518, + "after_tpot_ms": 8.844276931642518, + "before_tpot_ape_pct": 4.9280050463822995, + "after_tpot_ape_pct": 4.9280050463822995 + }, + { + "index": 12, + "config_id": 82, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "a2fcb7969b630c15841db497684f009456ddfcf5d0865e834cec1d2cba85ee75", + "completed": 320, + "silicon_ttft_ms": 559.6523069958494, + "before_ttft_ms": 618.9049051281246, + "after_ttft_ms": 618.9049051281246, + "before_ttft_ape_pct": 10.58739460046838, + "after_ttft_ape_pct": 10.58739460046838, + "silicon_tpot_ms": 19.846736777679933, + "before_tpot_ms": 22.026895896605613, + "after_tpot_ms": 22.026895896605613, + "before_tpot_ape_pct": 10.98497522966866, + "after_tpot_ape_pct": 10.98497522966866 + }, + { + "index": 13, + "config_id": 82, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "11f25dfe54c13d05d0d17d1ddd3547c91b69d5a464a670b74e4f2ef0df7fe9d5", + "completed": 40, + "silicon_ttft_ms": 251.3039122102782, + "before_ttft_ms": 271.4033068250004, + "after_ttft_ms": 271.4033068250004, + "before_ttft_ape_pct": 7.9980428628998235, + "after_ttft_ape_pct": 7.9980428628998235, + "silicon_tpot_ms": 7.9779975635324645, + "before_tpot_ms": 8.343063029363035, + "after_tpot_ms": 8.343063029363035, + "before_tpot_ape_pct": 4.575903451002428, + "after_tpot_ape_pct": 4.575903451002428 + }, + { + "index": 14, + "config_id": 82, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "0149db803f065df9bc2524a75e9f46e982a6135f7352ca9cbb8c350e2456b740", + "completed": 80, + "silicon_ttft_ms": 302.61298562109005, + "before_ttft_ms": 332.8740406999995, + "after_ttft_ms": 332.8740406999995, + "before_ttft_ape_pct": 9.99991953973851, + "after_ttft_ape_pct": 9.99991953973851, + "silicon_tpot_ms": 9.78609197222243, + "before_tpot_ms": 10.621622902810511, + "after_tpot_ms": 10.621622902810511, + "before_tpot_ape_pct": 8.537942755491311, + "after_tpot_ape_pct": 8.537942755491311 + }, + { + "index": 15, + "config_id": 82, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "d6bc957beb863348d322b5182bd12e502aede246ea7c6058f26fd1d580eeee94", + "completed": 160, + "silicon_ttft_ms": 388.53895208030735, + "before_ttft_ms": 420.1878244750004, + "after_ttft_ms": 420.1878244750004, + "before_ttft_ape_pct": 8.145611199402092, + "after_ttft_ape_pct": 8.145611199402092, + "silicon_tpot_ms": 13.670078590016566, + "before_tpot_ms": 14.60733386103891, + "after_tpot_ms": 14.60733386103891, + "before_tpot_ape_pct": 6.85625371390941, + "after_tpot_ape_pct": 6.85625371390941 + }, + { + "index": 16, + "config_id": 91, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "c93e2fd2700e7f6db7fc6eab246371c5dc5138445951f5de0712f0205b40d20c", + "completed": 40, + "silicon_ttft_ms": 153.4111074755856, + "before_ttft_ms": 53.661098550000624, + "after_ttft_ms": 53.661098550000624, + "before_ttft_ape_pct": 65.02137333273575, + "after_ttft_ape_pct": 65.02137333273575, + "silicon_tpot_ms": 6.435443899636045, + "before_tpot_ms": 6.308918493282411, + "after_tpot_ms": 6.308918493282411, + "before_tpot_ape_pct": 1.9660711572792922, + "after_tpot_ape_pct": 1.9660711572792922 + }, + { + "index": 17, + "config_id": 91, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "8b724f50b56d3518e2e073b46d2479be13e900a7cf67cb68db2bb12833d190ab", + "completed": 40, + "silicon_ttft_ms": 220.66775356652215, + "before_ttft_ms": 212.81671027499996, + "after_ttft_ms": 212.81671027499996, + "before_ttft_ape_pct": 3.5578570790840147, + "after_ttft_ape_pct": 3.5578570790840147, + "silicon_tpot_ms": 6.8127523183471475, + "before_tpot_ms": 7.269940879807199, + "after_tpot_ms": 7.269940879807199, + "before_tpot_ape_pct": 6.710776204630475, + "after_tpot_ape_pct": 6.710776204630475 + }, + { + "index": 18, + "config_id": 125, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "20a71bdc30925e893e5e4bc77b4b402cda4974ce6e51c7a64027e2ebbe13c803", + "completed": 80, + "silicon_ttft_ms": 328.4683411824517, + "before_ttft_ms": 76.26697876249938, + "after_ttft_ms": 76.26697876249938, + "before_ttft_ape_pct": 76.78102599235402, + "after_ttft_ape_pct": 76.78102599235402, + "silicon_tpot_ms": 9.930043911716696, + "before_tpot_ms": 9.617460231421022, + "after_tpot_ms": 9.617460231421022, + "before_tpot_ape_pct": 3.1478579860744538, + "after_tpot_ape_pct": 3.1478579860744538 + }, + { + "index": 19, + "config_id": 125, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "a486ba3060ce3c0a4bb0da73271a1aa15ba2d23b88d1e9796a3e14d899e0b19d", + "completed": 640, + "silicon_ttft_ms": 854.1142078014673, + "before_ttft_ms": 171.5880333234373, + "after_ttft_ms": 171.5880333234373, + "before_ttft_ape_pct": 79.91041107194394, + "after_ttft_ape_pct": 79.91041107194394, + "silicon_tpot_ms": 23.91051600109078, + "before_tpot_ms": 23.483916183696394, + "after_tpot_ms": 23.483916183696394, + "before_tpot_ape_pct": 1.7841514477350717, + "after_tpot_ape_pct": 1.7841514477350717 + }, + { + "index": 20, + "config_id": 125, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "12a2aa569a4cd3964a201fe90868a9e5b8b28ba7aadd366f90a2c308aaa30e90", + "completed": 40, + "silicon_ttft_ms": 417.6355976611376, + "before_ttft_ms": 64.6294955750003, + "after_ttft_ms": 64.6294955750003, + "before_ttft_ape_pct": 84.52490737452904, + "after_ttft_ape_pct": 84.52490737452904, + "silicon_tpot_ms": 7.47405043224896, + "before_tpot_ms": 6.673103527928511, + "after_tpot_ms": 6.673103527928511, + "before_tpot_ape_pct": 10.716370080465765, + "after_tpot_ape_pct": 10.716370080465765 + }, + { + "index": 21, + "config_id": 125, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "684eec940e4b07128bc063f76c598be4430e08a54a9e60f02026ec95bf111e86", + "completed": 160, + "silicon_ttft_ms": 448.35579777136445, + "before_ttft_ms": 95.52575174374928, + "after_ttft_ms": 95.52575174374928, + "before_ttft_ape_pct": 78.69420843477931, + "after_ttft_ape_pct": 78.69420843477931, + "silicon_tpot_ms": 13.187293928832572, + "before_tpot_ms": 12.959885068564537, + "after_tpot_ms": 12.959885068564537, + "before_tpot_ape_pct": 1.724454323193858, + "after_tpot_ape_pct": 1.724454323193858 + }, + { + "index": 22, + "config_id": 125, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "665fe937486e2e41cdec8795a4d7abc5373368b97dadf31a94c1c9b600339be7", + "completed": 160, + "silicon_ttft_ms": 1267.726926296018, + "before_ttft_ms": 441.85094366249933, + "after_ttft_ms": 441.85094366249933, + "before_ttft_ape_pct": 65.14620503064666, + "after_ttft_ape_pct": 65.14620503064666, + "silicon_tpot_ms": 19.92078562886774, + "before_tpot_ms": 16.71277308524882, + "after_tpot_ms": 16.71277308524882, + "before_tpot_ape_pct": 16.103845517870063, + "after_tpot_ape_pct": 16.103845517870063 + }, + { + "index": 23, + "config_id": 125, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "40c183db6901371e0ae3d7d68d2433b27d700faafc872525566ffc3435c843e8", + "completed": 40, + "silicon_ttft_ms": 1089.3548449734226, + "before_ttft_ms": 290.8274480999996, + "after_ttft_ms": 290.8274480999996, + "before_ttft_ape_pct": 73.3027810504579, + "after_ttft_ape_pct": 73.3027810504579, + "silicon_tpot_ms": 9.76237989263331, + "before_tpot_ms": 7.856292897879969, + "after_tpot_ms": 7.856292897879969, + "before_tpot_ape_pct": 19.52481890396085, + "after_tpot_ape_pct": 19.52481890396085 + }, + { + "index": 24, + "config_id": 125, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "9640989fa4eb4ab551686d284954d0c2ca1c3726f5a8811fa725c54cae9fa9e3", + "completed": 320, + "silicon_ttft_ms": 655.3529687953414, + "before_ttft_ms": 120.63162777812484, + "after_ttft_ms": 120.63162777812484, + "before_ttft_ape_pct": 81.5928768889431, + "after_ttft_ape_pct": 81.5928768889431, + "silicon_tpot_ms": 17.487370547082197, + "before_tpot_ms": 15.966571668405035, + "after_tpot_ms": 15.966571668405035, + "before_tpot_ape_pct": 8.696555463170597, + "after_tpot_ape_pct": 8.696555463170597 + }, + { + "index": 25, + "config_id": 125, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "300d449404886148de7299f6a34efe02fa49258de4e50e301d842dbb22f7b62e", + "completed": 80, + "silicon_ttft_ms": 1215.1382524287328, + "before_ttft_ms": 344.6975668375004, + "after_ttft_ms": 344.6975668375004, + "before_ttft_ape_pct": 71.6330577077511, + "after_ttft_ape_pct": 71.6330577077511, + "silicon_tpot_ms": 14.080757332967282, + "before_tpot_ms": 11.811036097497983, + "after_tpot_ms": 11.811036097497983, + "before_tpot_ape_pct": 16.119312205993353, + "after_tpot_ape_pct": 16.119312205993353 + }, + { + "index": 26, + "config_id": 125, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "436adf79fbf54469415c844f4bc9287f9033134bd167bfb6806e5ac1ca0e3b61", + "completed": 320, + "silicon_ttft_ms": 1696.2571826588828, + "before_ttft_ms": 617.5010440874997, + "after_ttft_ms": 617.5010440874997, + "before_ttft_ape_pct": 63.59626061423263, + "after_ttft_ape_pct": 63.59626061423263, + "silicon_tpot_ms": 28.445715111731737, + "before_tpot_ms": 23.67497148437073, + "after_tpot_ms": 23.67497148437073, + "before_tpot_ape_pct": 16.771396354853565, + "after_tpot_ape_pct": 16.771396354853565 + }, + { + "index": 27, + "config_id": 125, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "a18a095616467243a57906d32943dbaa06517c24e2939c3817934aece39c61fd", + "completed": 640, + "silicon_ttft_ms": 2346.8190781743033, + "before_ttft_ms": 984.437036396875, + "after_ttft_ms": 984.437036396875, + "before_ttft_ape_pct": 58.05228253203425, + "after_ttft_ape_pct": 58.05228253203425, + "silicon_tpot_ms": 43.790624972024055, + "before_tpot_ms": 39.645000979091115, + "after_tpot_ms": 39.645000979091115, + "before_tpot_ape_pct": 9.466921277285723, + "after_tpot_ape_pct": 9.466921277285723 + }, + { + "index": 28, + "config_id": 155, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "9d17c43da79079a0963b419644d4410d08388768f98c1df568ac3436587cd441", + "completed": 40, + "silicon_ttft_ms": 48.31286990083754, + "before_ttft_ms": 50.957391000000186, + "after_ttft_ms": 51.1974757250001, + "before_ttft_ape_pct": 5.473740443468866, + "after_ttft_ape_pct": 5.9706778547480965, + "silicon_tpot_ms": 2.821235826337538, + "before_tpot_ms": 5.628635476576109, + "after_tpot_ms": 5.623694327879703, + "before_tpot_ape_pct": 99.50957038153989, + "after_tpot_ape_pct": 99.33442909592746 + }, + { + "index": 29, + "config_id": 155, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "b54af5a2888f9d2f46b39c20b186b2d07e044f80cad3ceb1462b746514816fdb", + "completed": 160, + "silicon_ttft_ms": 61.46908490045462, + "before_ttft_ms": 71.41464054375031, + "after_ttft_ms": 71.40839443750049, + "before_ttft_ape_pct": 16.179768511930682, + "after_ttft_ape_pct": 16.169607133637932, + "silicon_tpot_ms": 4.2379004007864705, + "before_tpot_ms": 10.721460101966251, + "after_tpot_ms": 10.529318180839569, + "before_tpot_ape_pct": 152.98990273524504, + "after_tpot_ape_pct": 148.45600851982118 + }, + { + "index": 30, + "config_id": 155, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "71264d9d760262886e2670af8ad36e61b2d415f32c4e2f066665c991c64feda8", + "completed": 80, + "silicon_ttft_ms": 50.03589815023588, + "before_ttft_ms": 58.798741987500605, + "after_ttft_ms": 58.96638807499994, + "before_ttft_ape_pct": 17.51311390664707, + "after_ttft_ape_pct": 17.848165526977677, + "silicon_tpot_ms": 3.429161593620214, + "before_tpot_ms": 8.119567818976149, + "after_tpot_ms": 8.076405889252559, + "before_tpot_ape_pct": 136.77997076842937, + "after_tpot_ape_pct": 135.52129780872133 + }, + { + "index": 31, + "config_id": 155, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "c19c756d43ec4384adf8e271c4441c38f4e1e7d97dbeec046fb03ddb781836f0", + "completed": 320, + "silicon_ttft_ms": 75.31092280405574, + "before_ttft_ms": 87.55258437187527, + "after_ttft_ms": 87.41478278125028, + "before_ttft_ape_pct": 16.254828797769406, + "after_ttft_ape_pct": 16.071851899473355, + "silicon_tpot_ms": 5.459498672386496, + "before_tpot_ms": 14.33472636697976, + "after_tpot_ms": 14.057645762552752, + "before_tpot_ape_pct": 162.56488419867426, + "after_tpot_ape_pct": 157.4896818576892 + }, + { + "index": 32, + "config_id": 155, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "24f3f4a905f305291b5f7ae5c42b22f5a43cc5860808b2ae48c39907c63c89da", + "completed": 640, + "silicon_ttft_ms": 84.26808239491947, + "before_ttft_ms": 120.37670136875049, + "after_ttft_ms": 120.15610879218727, + "before_ttft_ape_pct": 42.849698186567494, + "after_ttft_ape_pct": 42.58792341930815, + "silicon_tpot_ms": 7.28058002120034, + "before_tpot_ms": 22.024346516216802, + "after_tpot_ms": 21.75387975979179, + "before_tpot_ape_pct": 202.5081305621812, + "after_tpot_ape_pct": 198.79322384269673 + }, + { + "index": 33, + "config_id": 155, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "1b6951cb0647a2f6217792cbdee47c6c7ac1f9b27cf9970eb0fd318487bc71a0", + "completed": 320, + "silicon_ttft_ms": 170.92277605297568, + "before_ttft_ms": 352.84388155625055, + "after_ttft_ms": 351.0740027500002, + "before_ttft_ape_pct": 106.43467752178934, + "after_ttft_ape_pct": 105.39919304913967, + "silicon_tpot_ms": 6.152468073023378, + "before_tpot_ms": 17.593443184272264, + "after_tpot_ms": 17.286436006159096, + "before_tpot_ape_pct": 185.95748853072368, + "after_tpot_ape_pct": 180.96750443865997 + }, + { + "index": 34, + "config_id": 155, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "65d896d7028d2ca45f6117cd3364fcb3086e5b9680f45c164c9d5bac3089beb9", + "completed": 640, + "silicon_ttft_ms": 265.4917608353571, + "before_ttft_ms": 567.1430806296866, + "after_ttft_ms": 564.5412011468759, + "before_ttft_ape_pct": 113.61984222982966, + "after_ttft_ape_pct": 112.6398195448981, + "silicon_tpot_ms": 9.084122183924551, + "before_tpot_ms": 28.693716917846963, + "after_tpot_ms": 28.366115328424275, + "before_tpot_ape_pct": 215.86669946628362, + "after_tpot_ape_pct": 212.2603896568183 + }, + { + "index": 35, + "config_id": 155, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "791b6ccd4880a910909d3ecf6915a1a1910b0b3c9a9ee2e89f126d6dc9b94280", + "completed": 80, + "silicon_ttft_ms": 105.4353361716494, + "before_ttft_ms": 193.99316447499976, + "after_ttft_ms": 193.0984894, + "before_ttft_ape_pct": 83.99255080780286, + "after_ttft_ape_pct": 83.1439974598596, + "silicon_tpot_ms": 3.520632999406209, + "before_tpot_ms": 8.831456725739006, + "after_tpot_ms": 8.781293671104624, + "before_tpot_ape_pct": 150.84854704334484, + "after_tpot_ape_pct": 149.4237164903493 + }, + { + "index": 36, + "config_id": 155, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "f056cf8fd0ef758066aa85c46b14bf9a03844638599cedc18b6e8af192036bec", + "completed": 40, + "silicon_ttft_ms": 121.98465588735417, + "before_ttft_ms": 162.37680167499914, + "after_ttft_ms": 161.6277330499996, + "before_ttft_ape_pct": 33.11248082295268, + "after_ttft_ape_pct": 32.49841291453372, + "silicon_tpot_ms": 2.94067039260922, + "before_tpot_ms": 5.976221446639023, + "after_tpot_ms": 5.968363369348593, + "before_tpot_ape_pct": 103.22649766050102, + "after_tpot_ape_pct": 102.95927705290828 + }, + { + "index": 37, + "config_id": 155, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "7bba886bd5ecb5fb160c8f683bfc98467258e3ea42819213c96309fdf404a11f", + "completed": 160, + "silicon_ttft_ms": 126.18746852676848, + "before_ttft_ms": 250.4786091624995, + "after_ttft_ms": 249.23025734375005, + "before_ttft_ape_pct": 98.49721377789969, + "after_ttft_ape_pct": 97.50793026716451, + "silicon_tpot_ms": 4.628018480740254, + "before_tpot_ms": 12.215786142232005, + "after_tpot_ms": 12.008940854198586, + "before_tpot_ape_pct": 163.95283841386225, + "after_tpot_ape_pct": 159.4834247999319 + }, + { + "index": 38, + "config_id": 155, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 128, + "spec_sha256": "cdc3eaffe42a1042faa2b430b60b899e7461c55e01b641d05648148eecb3a996", + "completed": 1280, + "silicon_ttft_ms": 111.10496407600294, + "before_ttft_ms": 177.0887033820302, + "after_ttft_ms": 175.94377667577962, + "before_ttft_ape_pct": 59.38865095252641, + "after_ttft_ape_pct": 58.35815990671915, + "silicon_tpot_ms": 10.238327636666467, + "before_tpot_ms": 31.084675913962123, + "after_tpot_ms": 30.041033305448003, + "before_tpot_ape_pct": 203.61087295779382, + "after_tpot_ape_pct": 193.41738584202184 + }, + { + "index": 39, + "config_id": 155, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 128, + "spec_sha256": "abcde72dfb00f3972958165146d9fde3da3f425ea2d0e7154e37689a6e19f1e0", + "completed": 1280, + "silicon_ttft_ms": 432.8474949681379, + "before_ttft_ms": 1011.4303657695318, + "after_ttft_ms": 1006.3848852109343, + "before_ttft_ape_pct": 133.66898908447737, + "after_ttft_ape_pct": 132.5033405322156, + "silicon_tpot_ms": 14.210807042334933, + "before_tpot_ms": 44.44776375377569, + "after_tpot_ms": 43.29585658210997, + "before_tpot_ape_pct": 212.77438094376245, + "after_tpot_ape_pct": 204.6685276432841 + }, + { + "index": 40, + "config_id": 155, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 256, + "spec_sha256": "c765f0cd6fcb30b2d6c08a2f7438b30ff83a9072cdb5a71d294592276b4f570d", + "completed": 2560, + "silicon_ttft_ms": 167.59310493898738, + "before_ttft_ms": 288.98425323984264, + "after_ttft_ms": 289.1686911652332, + "before_ttft_ape_pct": 72.43206595226455, + "after_ttft_ape_pct": 72.54211697462473, + "silicon_tpot_ms": 14.513360429082425, + "before_tpot_ms": 45.09754938613715, + "after_tpot_ms": 45.95950562940375, + "before_tpot_ape_pct": 210.73127141367593, + "after_tpot_ape_pct": 216.67032493252455 + }, + { + "index": 41, + "config_id": 155, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 256, + "spec_sha256": "d357d764d8403f0f6f333fd6c17becf85f2ab2162a9d22caf1a55430f03cd02f", + "completed": 2560, + "silicon_ttft_ms": 787.6301363097809, + "before_ttft_ms": 1903.9726791242151, + "after_ttft_ms": 1896.7471262972686, + "before_ttft_ape_pct": 141.73436126311043, + "after_ttft_ape_pct": 140.81698234452313, + "silicon_tpot_ms": 23.56258238616526, + "before_tpot_ms": 72.5948197616222, + "after_tpot_ms": 73.25568125475864, + "before_tpot_ape_pct": 208.09364853084253, + "after_tpot_ape_pct": 210.89835593644702 + }, + { + "index": 42, + "config_id": 131, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "785cef99c372ead7bf8fcb98f2d7f5e11bb6f69c4e5b3afb14df42a8c980874d", + "completed": 40, + "silicon_ttft_ms": 107.72685264964821, + "before_ttft_ms": 283.20722317499946, + "after_ttft_ms": 280.9375477250001, + "before_ttft_ape_pct": 162.89380614882774, + "after_ttft_ape_pct": 160.78692620740696, + "silicon_tpot_ms": 3.4467271748276604, + "before_tpot_ms": 8.647490762418695, + "after_tpot_ms": 8.600466113807567, + "before_tpot_ape_pct": 150.8899116116168, + "after_tpot_ape_pct": 149.52558405605743 + }, + { + "index": 43, + "config_id": 131, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "aee695a4f15f28dbd8c4ba417036546c6275d497c31cb802e47841fd4b43997a", + "completed": 320, + "silicon_ttft_ms": 218.55169775299146, + "before_ttft_ms": 618.5116404031256, + "after_ttft_ms": 614.1135474750005, + "before_ttft_ape_pct": 183.00472920698672, + "after_ttft_ape_pct": 180.99234816701153, + "silicon_tpot_ms": 8.784771588033268, + "before_tpot_ms": 29.963245427477666, + "after_tpot_ms": 30.610015326143003, + "before_tpot_ape_pct": 241.08166760185316, + "after_tpot_ape_pct": 248.4440661819867 + }, + { + "index": 44, + "config_id": 131, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "b02d3028c672db50df512dd58d46debdf95fffbf1ecafc53f1b3c6d1dc10a378", + "completed": 640, + "silicon_ttft_ms": 354.15133345741197, + "before_ttft_ms": 996.9082094656256, + "after_ttft_ms": 988.4572353999998, + "before_ttft_ape_pct": 181.4921518813108, + "after_ttft_ape_pct": 179.10589118786012, + "silicon_tpot_ms": 13.489782813179733, + "before_tpot_ms": 51.240700626854746, + "after_tpot_ms": 50.55306759365148, + "before_tpot_ape_pct": 279.8482254050211, + "after_tpot_ape_pct": 274.7507894957384 + }, + { + "index": 45, + "config_id": 131, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "24647f6297762b218d84b910ec575d255e2aadc1b03981a9eef364f98079e17c", + "completed": 80, + "silicon_ttft_ms": 124.24866264627781, + "before_ttft_ms": 338.9711816499994, + "after_ttft_ms": 336.1652482125, + "before_ttft_ape_pct": 172.8167647284968, + "after_ttft_ape_pct": 170.55844389208858, + "silicon_tpot_ms": 4.481681645668317, + "before_tpot_ms": 13.963113597982783, + "after_tpot_ms": 13.81703086948122, + "before_tpot_ape_pct": 211.55969347975804, + "after_tpot_ape_pct": 208.30014181921658 + }, + { + "index": 46, + "config_id": 131, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "c468f3881732b5ab8b4c83887da93c9fc7ea8703992b8213840fe8e365b35f10", + "completed": 160, + "silicon_ttft_ms": 158.8904605001062, + "before_ttft_ms": 438.79311995000063, + "after_ttft_ms": 434.8542025187493, + "before_ttft_ape_pct": 176.1607704886143, + "after_ttft_ape_pct": 173.68175606644343, + "silicon_tpot_ms": 6.246452861067533, + "before_tpot_ms": 20.588772218549312, + "after_tpot_ms": 20.134649088970512, + "before_tpot_ape_pct": 229.60742162761863, + "after_tpot_ape_pct": 222.33732546777682 + }, + { + "index": 47, + "config_id": 131, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 128, + "spec_sha256": "f439e94c70b79f60c6e0a033ee77769923e8ac3f3d53069b983942d174430f93", + "completed": 1280, + "silicon_ttft_ms": 584.5364274708118, + "before_ttft_ms": 1778.7267073164064, + "after_ttft_ms": 1758.6902468640578, + "before_ttft_ape_pct": 204.2969819712776, + "after_ttft_ape_pct": 200.8692297370768, + "silicon_tpot_ms": 21.757863187438684, + "before_tpot_ms": 78.07290039293541, + "after_tpot_ms": 72.27176041905065, + "before_tpot_ape_pct": 258.82613894736085, + "after_tpot_ape_pct": 232.1638701211009 + }, + { + "index": 48, + "config_id": 131, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 256, + "spec_sha256": "b120aeee2162a07425eaf2a9ff0bc218161cb1892938684179fe445fdd62bcca", + "completed": 2560, + "silicon_ttft_ms": 183.61345084202867, + "before_ttft_ms": 514.3101918210953, + "after_ttft_ms": 487.3287541230487, + "before_ttft_ape_pct": 180.10485586024993, + "after_ttft_ape_pct": 165.41016025145169, + "silicon_tpot_ms": 16.792121521213446, + "before_tpot_ms": 83.6627005691706, + "after_tpot_ms": 65.15635446416822, + "before_tpot_ape_pct": 398.22591185681756, + "after_tpot_ape_pct": 288.01740674551667 + }, + { + "index": 49, + "config_id": 131, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 256, + "spec_sha256": "575801d040796237d6839ee51fe5d9c7aeab0d138d50a95db7d9133c467c2395", + "completed": 2560, + "silicon_ttft_ms": 1092.9731557473133, + "before_ttft_ms": 3358.538542046094, + "after_ttft_ms": 3295.056858285152, + "before_ttft_ape_pct": 207.2846322332331, + "after_ttft_ape_pct": 201.47646728177676, + "silicon_tpot_ms": 33.567037846932344, + "before_tpot_ms": 131.60538299719968, + "after_tpot_ms": 114.07934358727346, + "before_tpot_ape_pct": 292.0673119782804, + "after_tpot_ape_pct": 239.8552595182272 + }, + { + "index": 50, + "config_id": 139, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "436dab1b082b3d368a9d98ca27f0b3f79d42020e3c6bbe813332aa95b124919e", + "completed": 40, + "silicon_ttft_ms": 58.963354578008875, + "before_ttft_ms": 22.737700074999918, + "after_ttft_ms": 22.64627395000003, + "before_ttft_ape_pct": 61.43757383254407, + "after_ttft_ape_pct": 61.59262967299652, + "silicon_tpot_ms": 2.3126213294103786, + "before_tpot_ms": 3.412202753397756, + "after_tpot_ms": 3.436141902838563, + "before_tpot_ape_pct": 47.546972347077876, + "after_tpot_ape_pct": 48.582124498291094 + }, + { + "index": 51, + "config_id": 139, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "c39445a8f611fc1217480e410473df29c2dcdcc0aea206ec944c16838cfe108b", + "completed": 40, + "silicon_ttft_ms": 82.97494126018137, + "before_ttft_ms": 68.95015359999982, + "after_ttft_ms": 69.3485427999996, + "before_ttft_ape_pct": 16.902437587999675, + "after_ttft_ape_pct": 16.42230564219863, + "silicon_tpot_ms": 2.381786584786486, + "before_tpot_ms": 3.527819314867833, + "after_tpot_ms": 3.553234754494375, + "before_tpot_ape_pct": 48.116516290819675, + "after_tpot_ape_pct": 49.183590888891636 + }, + { + "index": 52, + "config_id": 133, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "7c44d55afcb285331abf386e19b944d88c6059f84f81b1737ec2429f017d524b", + "completed": 40, + "silicon_ttft_ms": 79.17967028915882, + "before_ttft_ms": 96.8023692000004, + "after_ttft_ms": 97.27858409999988, + "before_ttft_ape_pct": 22.25659547013099, + "after_ttft_ape_pct": 22.85803129104358, + "silicon_tpot_ms": 2.5448711826222143, + "before_tpot_ms": 4.459698827044163, + "after_tpot_ms": 4.479264208720761, + "before_tpot_ape_pct": 75.24261571656159, + "after_tpot_ape_pct": 76.01143190695272 + }, + { + "index": 53, + "config_id": 133, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "559fa24b66fab3f4a01800b1a8e9a58700648429a521327559735491930428e2", + "completed": 80, + "silicon_ttft_ms": 96.38779677916318, + "before_ttft_ms": 115.01224456249966, + "after_ttft_ms": 115.4633842874999, + "before_ttft_ape_pct": 19.32241259337786, + "after_ttft_ape_pct": 19.790459109716284, + "silicon_tpot_ms": 2.9437545627989232, + "before_tpot_ms": 5.774134161940712, + "after_tpot_ms": 5.68068103723264, + "before_tpot_ape_pct": 96.14862716172445, + "after_tpot_ape_pct": 92.974003642187 + }, + { + "index": 54, + "config_id": 133, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "906872e10cdba8a8cbefb15593c7a4716ce064da42f0c0c6d2b20169d4a20ea3", + "completed": 320, + "silicon_ttft_ms": 150.2321477913938, + "before_ttft_ms": 207.61326144687501, + "after_ttft_ms": 208.60419264062512, + "before_ttft_ape_pct": 38.19496326123107, + "after_ttft_ape_pct": 38.8545632258978, + "silicon_tpot_ms": 4.874465690738827, + "before_tpot_ms": 10.278810806609261, + "after_tpot_ms": 10.379735318389491, + "before_tpot_ape_pct": 110.87051297003372, + "after_tpot_ape_pct": 112.94098629333517 + }, + { + "index": 55, + "config_id": 133, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "e22507ef646202acc0131f3256b8f77d066f8412a912dc4504047ffad6e4516f", + "completed": 40, + "silicon_ttft_ms": 51.1586994107347, + "before_ttft_ms": 31.156744500000297, + "after_ttft_ms": 31.287166774999882, + "before_ttft_ape_pct": 39.097856554456044, + "after_ttft_ape_pct": 38.84291990340385, + "silicon_tpot_ms": 2.5058461734992914, + "before_tpot_ms": 4.238479012539097, + "after_tpot_ms": 4.257226167941055, + "before_tpot_ape_pct": 69.14362331428623, + "after_tpot_ape_pct": 69.89176003553513 + }, + { + "index": 56, + "config_id": 133, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "0aea753400ece3dccdfd849e0d658650bcc317ecb1de1b5f78609cebb370d0ed", + "completed": 160, + "silicon_ttft_ms": 110.5373818998487, + "before_ttft_ms": 147.9352771125, + "after_ttft_ms": 148.50363218749982, + "before_ttft_ape_pct": 33.83280350038984, + "after_ttft_ape_pct": 34.346978040469665, + "silicon_tpot_ms": 3.6373014517101683, + "before_tpot_ms": 7.533136417768352, + "after_tpot_ms": 7.433308076047574, + "before_tpot_ape_pct": 107.10783853855332, + "after_tpot_ape_pct": 104.36326696410103 + }, + { + "index": 57, + "config_id": 179, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "bbd0901cdf08e9ebc876587c585b0cc060c62505b92a6e2154f5e939c0815855", + "completed": 160, + "silicon_ttft_ms": 1227.4979356850963, + "before_ttft_ms": 207.52815129374994, + "after_ttft_ms": 207.52815129374994, + "before_ttft_ape_pct": 83.0934011976221, + "after_ttft_ape_pct": 83.0934011976221, + "silicon_tpot_ms": 14.321917700868159, + "before_tpot_ms": 8.873528600498792, + "after_tpot_ms": 8.873528600498792, + "before_tpot_ape_pct": 38.0423153809849, + "after_tpot_ape_pct": 38.0423153809849 + }, + { + "index": 58, + "config_id": 179, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "e8f44d0ac15f46092784472bdb7c695229a350a826f4e522cbfd7f093f1dc99e", + "completed": 80, + "silicon_ttft_ms": 1399.06541993696, + "before_ttft_ms": 161.2009397125001, + "after_ttft_ms": 161.2009397125001, + "before_ttft_ape_pct": 88.47795553979431, + "after_ttft_ape_pct": 88.47795553979431, + "silicon_tpot_ms": 12.463472013874146, + "before_tpot_ms": 7.105510686066762, + "after_tpot_ms": 7.105510686066762, + "before_tpot_ape_pct": 42.98931567257489, + "after_tpot_ape_pct": 42.98931567257489 + }, + { + "index": 59, + "config_id": 179, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "27c6351ef3088decc47be05a52d9936ba2ba189d78458afe58f33d68dad254e1", + "completed": 40, + "silicon_ttft_ms": 3374.0832482781743, + "before_ttft_ms": 135.44331384999992, + "after_ttft_ms": 135.44331384999992, + "before_ttft_ape_pct": 95.98577439015122, + "after_ttft_ape_pct": 95.98577439015122, + "silicon_tpot_ms": 17.198503006038003, + "before_tpot_ms": 5.632142917406516, + "after_tpot_ms": 5.632142917406516, + "before_tpot_ape_pct": 67.25213284302012, + "after_tpot_ape_pct": 67.25213284302012 + }, + { + "index": 60, + "config_id": 179, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "aa4ccd5d4ed9f6eaa5ae03eb4d683d25b88e352c1df55d609dcbf38b6ebdbee6", + "completed": 320, + "silicon_ttft_ms": 1770.5642221233575, + "before_ttft_ms": 291.1010300531253, + "after_ttft_ms": 291.1010300531253, + "before_ttft_ape_pct": 83.5588550578515, + "after_ttft_ape_pct": 83.5588550578515, + "silicon_tpot_ms": 19.219130845535762, + "before_tpot_ms": 11.651643536907372, + "after_tpot_ms": 11.651643536907372, + "before_tpot_ape_pct": 39.37476345547735, + "after_tpot_ape_pct": 39.37476345547735 + }, + { + "index": 61, + "config_id": 179, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "8e4acaee2aab55e3712147555b1cde985798888461a282fc0cf4de7b7280e677", + "completed": 320, + "silicon_ttft_ms": 432.4044505803613, + "before_ttft_ms": 66.25021815624999, + "after_ttft_ms": 66.25021815624999, + "before_ttft_ape_pct": 84.67864563666475, + "after_ttft_ape_pct": 84.67864563666475, + "silicon_tpot_ms": 12.176334064092252, + "before_tpot_ms": 8.305791000146103, + "after_tpot_ms": 8.305791000146103, + "before_tpot_ape_pct": 31.787425045772178, + "after_tpot_ape_pct": 31.787425045772178 + }, + { + "index": 62, + "config_id": 179, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "2d189e72a4bd961560d5b8412ac2af14ac012663ee2ae9d31466f4278d872c90", + "completed": 640, + "silicon_ttft_ms": 2375.119664151862, + "before_ttft_ms": 464.45958585937444, + "after_ttft_ms": 464.45958585937444, + "before_ttft_ape_pct": 80.44479219849204, + "after_ttft_ape_pct": 80.44479219849204, + "silicon_tpot_ms": 26.724138022896334, + "before_tpot_ms": 17.27073927070276, + "after_tpot_ms": 17.27073927070276, + "before_tpot_ape_pct": 35.37400811241965, + "after_tpot_ape_pct": 35.37400811241965 + }, + { + "index": 63, + "config_id": 179, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "bbebf17ff0105ce242c4f84619e049b96d903ec62189a6aabc12f56917c98092", + "completed": 40, + "silicon_ttft_ms": 356.07514197472483, + "before_ttft_ms": 39.839020874999505, + "after_ttft_ms": 39.839020874999505, + "before_ttft_ape_pct": 88.81162536254008, + "after_ttft_ape_pct": 88.81162536254008, + "silicon_tpot_ms": 6.2471491204929785, + "before_tpot_ms": 5.211467671034348, + "after_tpot_ms": 5.211467671034348, + "before_tpot_ape_pct": 16.57846530445718, + "after_tpot_ape_pct": 16.57846530445718 + }, + { + "index": 64, + "config_id": 179, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "f9eb8ae453a1fc165dd895b3c4303f97de9e59fc7e6497cec9e8efc3537a52d6", + "completed": 80, + "silicon_ttft_ms": 133.98434043629095, + "before_ttft_ms": 45.2051811249997, + "after_ttft_ms": 45.2051811249997, + "before_ttft_ape_pct": 66.26084736634235, + "after_ttft_ape_pct": 66.26084736634235, + "silicon_tpot_ms": 6.761216498959709, + "before_tpot_ms": 6.230211239222688, + "after_tpot_ms": 6.230211239222688, + "before_tpot_ape_pct": 7.853694077370887, + "after_tpot_ape_pct": 7.853694077370887 + }, + { + "index": 65, + "config_id": 179, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "5f4364a68388b947bcc34ae8f105b7fa57667ccb5e94d0298ad613377194b051", + "completed": 160, + "silicon_ttft_ms": 330.1494436105713, + "before_ttft_ms": 54.442010712499815, + "after_ttft_ms": 54.442010712499815, + "before_ttft_ape_pct": 83.5098886985504, + "after_ttft_ape_pct": 83.5098886985504, + "silicon_tpot_ms": 9.337547488204864, + "before_tpot_ms": 7.06672010545645, + "after_tpot_ms": 7.06672010545645, + "before_tpot_ape_pct": 24.319312813315385, + "after_tpot_ape_pct": 24.319312813315385 + }, + { + "index": 66, + "config_id": 180, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "70796771b87191da8aaab0c27618e39a6b6daab19f485e38882ea93fc0549138", + "completed": 160, + "silicon_ttft_ms": 1269.9310178228188, + "before_ttft_ms": 296.50016702499977, + "after_ttft_ms": 296.50016702499977, + "before_ttft_ape_pct": 76.6522619840153, + "after_ttft_ape_pct": 76.6522619840153, + "silicon_tpot_ms": 16.438979066881668, + "before_tpot_ms": 11.71842974684327, + "after_tpot_ms": 11.71842974684327, + "before_tpot_ape_pct": 28.715586903742217, + "after_tpot_ape_pct": 28.715586903742217 + }, + { + "index": 67, + "config_id": 180, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "9017202b5dc75112880410fa846b00d3de8cb82164445774fcdcaa41e377894a", + "completed": 40, + "silicon_ttft_ms": 1026.2379905674607, + "before_ttft_ms": 194.64018802499987, + "after_ttft_ms": 194.64018802499987, + "before_ttft_ape_pct": 81.03362087410414, + "after_ttft_ape_pct": 81.03362087410414, + "silicon_tpot_ms": 8.68482414844018, + "before_tpot_ms": 6.379729435883675, + "after_tpot_ms": 6.379729435883675, + "before_tpot_ape_pct": 26.541639452429287, + "after_tpot_ape_pct": 26.541639452429287 + }, + { + "index": 68, + "config_id": 180, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "b78e0082970d899eb4810713cb37e650a90a9fc6b546b1cba48d62525c68956b", + "completed": 320, + "silicon_ttft_ms": 1612.6139393571066, + "before_ttft_ms": 415.22058341249976, + "after_ttft_ms": 415.22058341249976, + "before_ttft_ape_pct": 74.25170567618721, + "after_ttft_ape_pct": 74.25170567618721, + "silicon_tpot_ms": 22.60126418281943, + "before_tpot_ms": 16.285716581531158, + "after_tpot_ms": 16.285716581531158, + "before_tpot_ape_pct": 27.943337815984194, + "after_tpot_ape_pct": 27.943337815984194 + }, + { + "index": 69, + "config_id": 180, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "1c9203d53a05ef7a4f8e0923c277f84d11941a2cd92f8363d53ddbd3886ed6bb", + "completed": 80, + "silicon_ttft_ms": 1103.126850060653, + "before_ttft_ms": 231.0995412874996, + "after_ttft_ms": 231.0995412874996, + "before_ttft_ape_pct": 79.05050164676953, + "after_ttft_ape_pct": 79.05050164676953, + "silicon_tpot_ms": 12.17838616621369, + "before_tpot_ms": 8.866218142272166, + "after_tpot_ms": 8.866218142272166, + "before_tpot_ape_pct": 27.19710131323, + "after_tpot_ape_pct": 27.19710131323 + }, + { + "index": 70, + "config_id": 180, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "8890b5f1aedce6cef37bc21455514939cfa29821b0a22880fcc9c74297b03abe", + "completed": 640, + "silicon_ttft_ms": 2239.5733038705657, + "before_ttft_ms": 661.4477982203132, + "after_ttft_ms": 661.4477982203132, + "before_ttft_ape_pct": 70.4654544203952, + "after_ttft_ape_pct": 70.4654544203952, + "silicon_tpot_ms": 31.76685732874561, + "before_tpot_ms": 25.275042676708402, + "after_tpot_ms": 25.275042676708402, + "before_tpot_ape_pct": 20.435810142801902, + "after_tpot_ape_pct": 20.435810142801902 + }, + { + "index": 71, + "config_id": 180, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "1b43ebe44f13777813202ca050fdadce2d9629a59c46eb28be188e6b554808cf", + "completed": 320, + "silicon_ttft_ms": 606.1791965999873, + "before_ttft_ms": 82.8731125562504, + "after_ttft_ms": 82.8731125562504, + "before_ttft_ape_pct": 86.32861156881012, + "after_ttft_ape_pct": 86.32861156881012, + "silicon_tpot_ms": 14.418717271507207, + "before_tpot_ms": 11.390050095745753, + "after_tpot_ms": 11.390050095745753, + "before_tpot_ape_pct": 21.005108282041117, + "after_tpot_ape_pct": 21.005108282041117 + }, + { + "index": 72, + "config_id": 180, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "104afcb5c298ef9941e4b80bf1f4508e6c08163203a8935c33291be211863556", + "completed": 160, + "silicon_ttft_ms": 386.1593430337962, + "before_ttft_ms": 63.91118361874985, + "after_ttft_ms": 63.91118361874985, + "before_ttft_ape_pct": 83.44953067388133, + "after_ttft_ape_pct": 83.44953067388133, + "silicon_tpot_ms": 10.717993882375445, + "before_tpot_ms": 8.935841246440827, + "after_tpot_ms": 8.935841246440827, + "before_tpot_ape_pct": 16.62766983721806, + "after_tpot_ape_pct": 16.62766983721806 + }, + { + "index": 73, + "config_id": 180, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "061075d916ba23df56f7fa6b4979e02fa5cfaef13f0dc7698c33ce9108475b2f", + "completed": 40, + "silicon_ttft_ms": 490.79990547616035, + "before_ttft_ms": 42.78188332500022, + "after_ttft_ms": 42.78188332500022, + "before_ttft_ape_pct": 91.28323317758294, + "after_ttft_ape_pct": 91.28323317758294, + "silicon_tpot_ms": 6.751249126130654, + "before_tpot_ms": 5.679537892088526, + "after_tpot_ms": 5.679537892088526, + "before_tpot_ape_pct": 15.874265843547054, + "after_tpot_ape_pct": 15.874265843547054 + }, + { + "index": 74, + "config_id": 180, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "18a9c0a8e65bf7c289648843334d5872e3428f272081e03d9ddbc77687826286", + "completed": 640, + "silicon_ttft_ms": 706.6360079450533, + "before_ttft_ms": 119.0486964718754, + "after_ttft_ms": 119.0486964718754, + "before_ttft_ape_pct": 83.1527554308933, + "after_ttft_ape_pct": 83.1527554308933, + "silicon_tpot_ms": 18.907028631858662, + "before_tpot_ms": 15.476018990295312, + "after_tpot_ms": 15.476018990295312, + "before_tpot_ape_pct": 18.14674166083422, + "after_tpot_ape_pct": 18.14674166083422 + }, + { + "index": 75, + "config_id": 180, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "78216f1ef53b5852e45307e31a024d1e4f970a5d56d6ced0b5264f8756d6778b", + "completed": 80, + "silicon_ttft_ms": 510.46166033738706, + "before_ttft_ms": 51.482904124999685, + "after_ttft_ms": 51.482904124999685, + "before_ttft_ape_pct": 89.91444252816709, + "after_ttft_ape_pct": 89.91444252816709, + "silicon_tpot_ms": 8.420129438739623, + "before_tpot_ms": 7.348196075202719, + "after_tpot_ms": 7.348196075202719, + "before_tpot_ape_pct": 12.730604337329021, + "after_tpot_ape_pct": 12.730604337329021 + }, + { + "index": 76, + "config_id": 185, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "97adecf37b212d40c0823c53d394997086ea1bf216f59c132c8aac612fd54760", + "completed": 80, + "silicon_ttft_ms": 234.8648586892523, + "before_ttft_ms": 123.72634751250021, + "after_ttft_ms": 123.72634751250021, + "before_ttft_ape_pct": 47.32019587647146, + "after_ttft_ape_pct": 47.32019587647146, + "silicon_tpot_ms": 6.461983579439623, + "before_tpot_ms": 5.995684992282622, + "after_tpot_ms": 5.995684992282622, + "before_tpot_ape_pct": 7.21602866093074, + "after_tpot_ape_pct": 7.21602866093074 + }, + { + "index": 77, + "config_id": 185, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "b4f8e036829bd21f5ea852d5c580a6166cc14d4dae789656b0d6a1011d171a96", + "completed": 40, + "silicon_ttft_ms": 1045.5109692411497, + "before_ttft_ms": 104.44806577499996, + "after_ttft_ms": 104.44806577499996, + "before_ttft_ape_pct": 90.00985462153398, + "after_ttft_ape_pct": 90.00985462153398, + "silicon_tpot_ms": 7.342476760421171, + "before_tpot_ms": 5.055986550288306, + "after_tpot_ms": 5.055986550288306, + "before_tpot_ape_pct": 31.140584910774837, + "after_tpot_ape_pct": 31.140584910774837 + }, + { + "index": 78, + "config_id": 185, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "e231a9b7f1b7f176cb87042967420a6c36ddcb769d174ca1ae2169317d082d2e", + "completed": 80, + "silicon_ttft_ms": 429.186265647877, + "before_ttft_ms": 34.12185038750039, + "after_ttft_ms": 34.12185038750039, + "before_ttft_ape_pct": 92.04964065287787, + "after_ttft_ape_pct": 92.04964065287787, + "silicon_tpot_ms": 7.283586425165536, + "before_tpot_ms": 5.3764126720540535, + "after_tpot_ms": 5.3764126720540535, + "before_tpot_ape_pct": 26.184542089347662, + "after_tpot_ape_pct": 26.184542089347662 + }, + { + "index": 79, + "config_id": 185, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "1fa5aea2ede5cfd3d5a8452ed58a8609fcabe65a3916b092d8ba26639eec0906", + "completed": 40, + "silicon_ttft_ms": 361.0080464510247, + "before_ttft_ms": 30.153390075000356, + "after_ttft_ms": 30.153390075000356, + "before_ttft_ape_pct": 91.64744653992331, + "after_ttft_ape_pct": 91.64744653992331, + "silicon_tpot_ms": 5.876064847433309, + "before_tpot_ms": 4.867787875293853, + "after_tpot_ms": 4.867787875293853, + "before_tpot_ape_pct": 17.159051139128856, + "after_tpot_ape_pct": 17.159051139128856 + }, + { + "index": 80, + "config_id": 183, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "74bbe3fd6de1f969c38c22cea57cbc206d238f4cc7a293ebd832f12554da45b8", + "completed": 640, + "silicon_ttft_ms": 4606.413363380125, + "before_ttft_ms": 2285.250759109376, + "after_ttft_ms": 2285.250759109376, + "before_ttft_ape_pct": 50.389802676490824, + "after_ttft_ape_pct": 50.389802676490824, + "silicon_tpot_ms": 43.31710646127792, + "before_tpot_ms": 122.62922839486582, + "after_tpot_ms": 122.62922839486582, + "before_tpot_ape_pct": 183.09653717172148, + "after_tpot_ape_pct": 183.09653717172148 + }, + { + "index": 81, + "config_id": 191, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 128, + "spec_sha256": "14eabc394f1649258551e8a50a43eea6246d69de1fc7c704f68e04593690d0a6", + "completed": 1280, + "silicon_ttft_ms": 2245.1532942250196, + "before_ttft_ms": 1254.5940279749964, + "after_ttft_ms": 1254.5940279749964, + "before_ttft_ape_pct": 44.11989456568237, + "after_ttft_ape_pct": 44.11989456568237, + "silicon_tpot_ms": 26.63461367008041, + "before_tpot_ms": 94.04044973903737, + "after_tpot_ms": 94.04044973903737, + "before_tpot_ape_pct": 253.0760795103114, + "after_tpot_ape_pct": 253.0760795103114 + }, + { + "index": 82, + "config_id": 191, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 256, + "spec_sha256": "c402c8f1781b1fc5e04995820e4b47321b86152142a52bb530189fabaccbd045", + "completed": 2560, + "silicon_ttft_ms": 3048.6806552130474, + "before_ttft_ms": 1867.4909992453133, + "after_ttft_ms": 1867.4909992453133, + "before_ttft_ape_pct": 38.744289400989764, + "after_ttft_ape_pct": 38.744289400989764, + "silicon_tpot_ms": 36.068337505009254, + "before_tpot_ms": 157.22783261618162, + "after_tpot_ms": 157.22783261618162, + "before_tpot_ape_pct": 335.91649488792063, + "after_tpot_ape_pct": 335.91649488792063 + }, + { + "index": 83, + "config_id": 191, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 256, + "spec_sha256": "fdef0811cc8110cd54ce5bf6ab64c59f36c3e2e16d19713e3a64d35330b2abd1", + "completed": 2560, + "silicon_ttft_ms": 773.6551952812079, + "before_ttft_ms": 271.7718375367169, + "after_ttft_ms": 271.7718375367169, + "before_ttft_ape_pct": 64.87171039574892, + "after_ttft_ape_pct": 64.87171039574892, + "silicon_tpot_ms": 23.006045199552638, + "before_tpot_ms": 44.503371063312294, + "after_tpot_ms": 44.503371063312294, + "before_tpot_ape_pct": 93.4420743647747, + "after_tpot_ape_pct": 93.4420743647747 + }, + { + "index": 84, + "config_id": 191, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 128, + "spec_sha256": "1b01476f8b830dfc79e80ddec344b2e5aa3e6415aeae6234a9bde3f335d7feaa", + "completed": 1280, + "silicon_ttft_ms": 734.3053470784071, + "before_ttft_ms": 200.01187842968827, + "after_ttft_ms": 200.01187842968827, + "before_ttft_ape_pct": 72.76175650559009, + "after_ttft_ape_pct": 72.76175650559009, + "silicon_tpot_ms": 20.37124878369584, + "before_tpot_ms": 31.05881124551152, + "after_tpot_ms": 31.05881124551152, + "before_tpot_ape_pct": 52.46395336534051, + "after_tpot_ape_pct": 52.46395336534051 + }, + { + "index": 85, + "config_id": 186, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "9d6a5e0f6ea73e6b95ff7bcbfce508ca0f183912836cf25b37486edd50ffb062", + "completed": 320, + "silicon_ttft_ms": 600.1128721906298, + "before_ttft_ms": 523.595259462501, + "after_ttft_ms": 523.595259462501, + "before_ttft_ape_pct": 12.75053681965057, + "after_ttft_ape_pct": 12.75053681965057, + "silicon_tpot_ms": 20.763165045228813, + "before_tpot_ms": 21.98830108507814, + "after_tpot_ms": 21.98830108507814, + "before_tpot_ape_pct": 5.900526423503316, + "after_tpot_ape_pct": 5.900526423503316 + }, + { + "index": 86, + "config_id": 181, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "cefe865d6a0237f47d53f79f42c19149a4344c38d88fdbbe2df886b31f241123", + "completed": 40, + "silicon_ttft_ms": 432.3558239033446, + "before_ttft_ms": 336.1681141250013, + "after_ttft_ms": 336.1681141250013, + "before_ttft_ape_pct": 22.247349164850515, + "after_ttft_ape_pct": 22.247349164850515, + "silicon_tpot_ms": 12.744894321848943, + "before_tpot_ms": 11.630069742041902, + "after_tpot_ms": 11.630069742041902, + "before_tpot_ape_pct": 8.74722498009155, + "after_tpot_ape_pct": 8.74722498009155 + }, + { + "index": 87, + "config_id": 181, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "7f343cf93439fb5d55aa5a18df21e37e324107df66d9d07da3c1632a128e8967", + "completed": 320, + "silicon_ttft_ms": 283.37134614703245, + "before_ttft_ms": 145.42489761249956, + "after_ttft_ms": 145.42489761249956, + "before_ttft_ape_pct": 48.680450726643635, + "after_ttft_ape_pct": 48.680450726643635, + "silicon_tpot_ms": 25.03310849733815, + "before_tpot_ms": 23.496811877106445, + "after_tpot_ms": 23.496811877106445, + "before_tpot_ape_pct": 6.137058928958283, + "after_tpot_ape_pct": 6.137058928958283 + }, + { + "index": 88, + "config_id": 181, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "b50260f204dcb2b87cfee8a9b377a516739a26cc7268bc76a6956090a1f50c88", + "completed": 80, + "silicon_ttft_ms": 531.2731346522924, + "before_ttft_ms": 414.4809144374999, + "after_ttft_ms": 414.4809144374999, + "before_ttft_ape_pct": 21.983460596258197, + "after_ttft_ape_pct": 21.983460596258197, + "silicon_tpot_ms": 16.52664239089253, + "before_tpot_ms": 16.734132112828654, + "after_tpot_ms": 16.734132112828654, + "before_tpot_ape_pct": 1.2554862447466508, + "after_tpot_ape_pct": 1.2554862447466508 + }, + { + "index": 89, + "config_id": 181, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "c89b364dd3961b7338994b7a79ceb996ba1e5a1edfb458f395b139b785ce3138", + "completed": 160, + "silicon_ttft_ms": 725.3111328172963, + "before_ttft_ms": 524.0097046500005, + "after_ttft_ms": 524.0097046500005, + "before_ttft_ape_pct": 27.753803720809987, + "after_ttft_ape_pct": 27.753803720809987, + "silicon_tpot_ms": 24.871235152725305, + "before_tpot_ms": 22.229874807483224, + "after_tpot_ms": 22.229874807483224, + "before_tpot_ape_pct": 10.62014141646942, + "after_tpot_ape_pct": 10.62014141646942 + }, + { + "index": 90, + "config_id": 181, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "3e11aa83cf6f5bab19a1f0aeb951ed6483cf4ed9975d3080d617fd9a18da8f08", + "completed": 320, + "silicon_ttft_ms": 1001.6335342166714, + "before_ttft_ms": 774.0153813343748, + "after_ttft_ms": 774.0153813343748, + "before_ttft_ape_pct": 22.724693723469002, + "after_ttft_ape_pct": 22.724693723469002, + "silicon_tpot_ms": 33.34223491827833, + "before_tpot_ms": 32.85620264412994, + "after_tpot_ms": 32.85620264412994, + "before_tpot_ape_pct": 1.4577075452190114, + "after_tpot_ape_pct": 1.4577075452190114 + }, + { + "index": 91, + "config_id": 181, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "6e9f7efb4cc4f2a4eaec4b5723692d195c572ef220d5dae0d94b986bbec72bce", + "completed": 640, + "silicon_ttft_ms": 331.6437606518775, + "before_ttft_ms": 206.9225214890625, + "after_ttft_ms": 206.9225214890625, + "before_ttft_ape_pct": 37.606990982632546, + "after_ttft_ape_pct": 37.606990982632546, + "silicon_tpot_ms": 30.417027538952713, + "before_tpot_ms": 30.42413382587153, + "after_tpot_ms": 30.42413382587153, + "before_tpot_ape_pct": 0.023362857891728872, + "after_tpot_ape_pct": 0.023362857891728872 + }, + { + "index": 92, + "config_id": 181, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "b0208e0c6cf88df36edcbaf8f190e4dfd51462154bbc05e5a27b1fd003e390cc", + "completed": 40, + "silicon_ttft_ms": 184.02190948836505, + "before_ttft_ms": 79.24857170000087, + "after_ttft_ms": 79.24857170000087, + "before_ttft_ape_pct": 56.93525193802458, + "after_ttft_ape_pct": 56.93525193802458, + "silicon_tpot_ms": 11.86258790742677, + "before_tpot_ms": 10.445483570473504, + "after_tpot_ms": 10.445483570473504, + "before_tpot_ape_pct": 11.94599650609176, + "after_tpot_ape_pct": 11.94599650609176 + }, + { + "index": 93, + "config_id": 181, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "aa6a6adc3552814c1cfe025ac3380829bd879949e03bd1bdc39760b2c72d9213", + "completed": 160, + "silicon_ttft_ms": 233.8934774743393, + "before_ttft_ms": 110.75285956874973, + "after_ttft_ms": 110.75285956874973, + "before_ttft_ape_pct": 52.648162417910726, + "after_ttft_ape_pct": 52.648162417910726, + "silicon_tpot_ms": 20.96893535039546, + "before_tpot_ms": 17.578949364323005, + "after_tpot_ms": 17.578949364323005, + "before_tpot_ape_pct": 16.166705316340824, + "after_tpot_ape_pct": 16.166705316340824 + }, + { + "index": 94, + "config_id": 181, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "303f65f3fc412abd6d471d0e4d6661c58ae3bc4d2a0fbb93de0937be098c7f92", + "completed": 80, + "silicon_ttft_ms": 207.42877998854965, + "before_ttft_ms": 95.35389578749951, + "after_ttft_ms": 95.35389578749951, + "before_ttft_ape_pct": 54.03053723173652, + "after_ttft_ape_pct": 54.03053723173652, + "silicon_tpot_ms": 14.822894272130894, + "before_tpot_ms": 14.422718780066884, + "after_tpot_ms": 14.422718780066884, + "before_tpot_ape_pct": 2.6997122472660173, + "after_tpot_ape_pct": 2.6997122472660173 + }, + { + "index": 95, + "config_id": 192, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 256, + "spec_sha256": "557e297f67ea87adf79b4f210aa9e21095f12aeea2f1ef76024df0188fe3d02d", + "completed": 2560, + "silicon_ttft_ms": 3595.6301899044774, + "before_ttft_ms": 2348.8296846347675, + "after_ttft_ms": 2348.8296846347675, + "before_ttft_ape_pct": 34.67543766793305, + "after_ttft_ape_pct": 34.67543766793305, + "silicon_tpot_ms": 56.31141828341263, + "before_tpot_ms": 140.0112185163047, + "after_tpot_ms": 140.0112185163047, + "before_tpot_ape_pct": 148.63735061978912, + "after_tpot_ape_pct": 148.63735061978912 + }, + { + "index": 96, + "config_id": 192, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 256, + "spec_sha256": "4ef18de97ad9c81084876d604b1caef25a9b8668037cf5442b13fee0fa79da54", + "completed": 2560, + "silicon_ttft_ms": 634.4021987129963, + "before_ttft_ms": 336.1148590371094, + "after_ttft_ms": 336.1148590371094, + "before_ttft_ape_pct": 47.01864846638594, + "after_ttft_ape_pct": 47.01864846638594, + "silicon_tpot_ms": 30.263881936572957, + "before_tpot_ms": 54.91752841189877, + "after_tpot_ms": 54.91752841189877, + "before_tpot_ape_pct": 81.46227416229988, + "after_tpot_ape_pct": 81.46227416229988 + }, + { + "index": 97, + "config_id": 178, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "e79469298e1ea967ca3b2edd7842a7e1e46f06b04d0f5d9a60a9a40230fa7ba6", + "completed": 640, + "silicon_ttft_ms": 141.0098649939755, + "before_ttft_ms": 113.17266760312495, + "after_ttft_ms": 113.17266760312495, + "before_ttft_ape_pct": 19.741311994050815, + "after_ttft_ape_pct": 19.741311994050815, + "silicon_tpot_ms": 15.610077141126371, + "before_tpot_ms": 15.062078526833583, + "after_tpot_ms": 15.062078526833583, + "before_tpot_ape_pct": 3.5105439219709544, + "after_tpot_ape_pct": 3.5105439219709544 + }, + { + "index": 98, + "config_id": 262, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 256, + "spec_sha256": "6029b27d649a0a76268139abf7b2e9f144233ee3edc7f991e1fd4b95e83409e5", + "completed": 2560, + "silicon_ttft_ms": 247.1074982335267, + "before_ttft_ms": 315.0723286769511, + "after_ttft_ms": 315.237952806251, + "before_ttft_ape_pct": 27.504155450270808, + "after_ttft_ape_pct": 27.571180583252964, + "silicon_tpot_ms": 11.104027085770108, + "before_tpot_ms": 39.37789883595387, + "after_tpot_ms": 38.24220392650322, + "before_tpot_ape_pct": 254.62718644136717, + "after_tpot_ape_pct": 244.3994114127377 + }, + { + "index": 99, + "config_id": 283, + "model": "gpt-oss-120b", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "343c077a0a51020acef7e16d5cfcfb1af246e4066c1e488fa665a04a6e64f8f7", + "completed": 640, + "silicon_ttft_ms": 165.94180535576015, + "before_ttft_ms": 117.77185318593754, + "after_ttft_ms": 118.65773493906235, + "before_ttft_ape_pct": 29.02821990308697, + "after_ttft_ape_pct": 28.49436904421173, + "silicon_tpot_ms": 5.2294283091341205, + "before_tpot_ms": 15.25464815278104, + "after_tpot_ms": 15.695810691932365, + "before_tpot_ape_pct": 191.7077594530955, + "after_tpot_ape_pct": 200.14391180230655 + }, + { + "index": 100, + "config_id": 473, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "49b1ea942a1b70641967292141cc328279ec00478a76a4a7e467c02dfa8911bb", + "completed": 320, + "silicon_ttft_ms": 8292.347574694577, + "before_ttft_ms": 30731.560811728163, + "after_ttft_ms": 30731.560811728163, + "before_ttft_ape_pct": 270.6014555577775, + "after_ttft_ape_pct": 270.6014555577775, + "silicon_tpot_ms": 24.692002606749238, + "before_tpot_ms": 24.313442471826082, + "after_tpot_ms": 24.313442471826082, + "before_tpot_ape_pct": 1.5331285232396685, + "after_tpot_ape_pct": 1.5331285232396685 + }, + { + "index": 101, + "config_id": 473, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "e8f501530d21ecbd08301cd98c0bb1b327ac707940d3266f315125e4b4cd8bed", + "completed": 160, + "silicon_ttft_ms": 626.5326763968915, + "before_ttft_ms": 4970.566047712493, + "after_ttft_ms": 4970.566047712493, + "before_ttft_ape_pct": 693.3450616331102, + "after_ttft_ape_pct": 693.3450616331102, + "silicon_tpot_ms": 20.074626911880017, + "before_tpot_ms": 24.15142487617139, + "after_tpot_ms": 24.15142487617139, + "before_tpot_ape_pct": 20.308212860876406, + "after_tpot_ape_pct": 20.308212860876406 + }, + { + "index": 102, + "config_id": 473, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "c6b83626fd9898bd05bda42b4d61302c9131932214b22f48c51ef1d4ba4c4179", + "completed": 160, + "silicon_ttft_ms": 218.00891533116555, + "before_ttft_ms": 105.25585718749896, + "after_ttft_ms": 105.25585718749896, + "before_ttft_ape_pct": 51.71947118418966, + "after_ttft_ape_pct": 51.71947118418966, + "silicon_tpot_ms": 16.923118889238523, + "before_tpot_ms": 22.492551213127662, + "after_tpot_ms": 22.492551213127662, + "before_tpot_ape_pct": 32.910200302562195, + "after_tpot_ape_pct": 32.910200302562195 + }, + { + "index": 103, + "config_id": 473, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "9582ef2d1031233916093d127d556692b195779be997dc45d55ddb95e75d1012", + "completed": 80, + "silicon_ttft_ms": 473.5316864476772, + "before_ttft_ms": 329.79144853750086, + "after_ttft_ms": 329.79144853750086, + "before_ttft_ape_pct": 30.35493548245559, + "after_ttft_ape_pct": 30.35493548245559, + "silicon_tpot_ms": 14.550238891471366, + "before_tpot_ms": 20.10812375391757, + "after_tpot_ms": 20.10812375391757, + "before_tpot_ape_pct": 38.19789423322777, + "after_tpot_ape_pct": 38.19789423322777 + }, + { + "index": 104, + "config_id": 473, + "model": "DeepSeek-R1-0528", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "83eb28ff9d9d0c98417acfc5ca21ded346a02bd2947fd7d30afbef4ba267e9d3", + "completed": 80, + "silicon_ttft_ms": 194.49161399388686, + "before_ttft_ms": 91.19394122499946, + "after_ttft_ms": 91.19394122499946, + "before_ttft_ape_pct": 53.111633271825376, + "after_ttft_ape_pct": 53.111633271825376, + "silicon_tpot_ms": 13.057724642449031, + "before_tpot_ms": 18.12601714198523, + "after_tpot_ms": 18.12601714198523, + "before_tpot_ape_pct": 38.81451507301519, + "after_tpot_ape_pct": 38.81451507301519 + }, + { + "index": 105, + "config_id": 587, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "52dc204f2523e5acb0ff9ad1bd9c1f73bd4f7491f3f7c7e947ca4d9131bbbada", + "completed": 640, + "silicon_ttft_ms": 538.1686374451192, + "before_ttft_ms": 76.59164550312524, + "after_ttft_ms": 76.59164550312524, + "before_ttft_ape_pct": 85.76809569083524, + "after_ttft_ape_pct": 85.76809569083524, + "silicon_tpot_ms": 16.693422722506106, + "before_tpot_ms": 16.89072068556202, + "after_tpot_ms": 16.89072068556202, + "before_tpot_ape_pct": 1.1818904147794518, + "after_tpot_ape_pct": 1.1818904147794518 + }, + { + "index": 106, + "config_id": 587, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "713a0455eb32d2a0447ac6589febaebe23694f6ed3bac150086945509ab42905", + "completed": 80, + "silicon_ttft_ms": 420.91323638760514, + "before_ttft_ms": 162.90131493749988, + "after_ttft_ms": 162.90131493749988, + "before_ttft_ape_pct": 61.29812492104918, + "after_ttft_ape_pct": 61.29812492104918, + "silicon_tpot_ms": 7.887804626559686, + "before_tpot_ms": 8.941842657249069, + "after_tpot_ms": 8.941842657249069, + "before_tpot_ape_pct": 13.36288207672187, + "after_tpot_ape_pct": 13.36288207672187 + }, + { + "index": 107, + "config_id": 587, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "f74d0682cbedaffee50f779f4df93b9dfc885f5340854d8faa20ae9fe7e8d667", + "completed": 160, + "silicon_ttft_ms": 391.1163377437333, + "before_ttft_ms": 44.7909827999998, + "after_ttft_ms": 44.7909827999998, + "before_ttft_ape_pct": 88.54791312007332, + "after_ttft_ape_pct": 88.54791312007332, + "silicon_tpot_ms": 9.98011719707241, + "before_tpot_ms": 9.006044343208208, + "after_tpot_ms": 9.006044343208208, + "before_tpot_ape_pct": 9.760134421567102, + "after_tpot_ape_pct": 9.760134421567102 + }, + { + "index": 108, + "config_id": 587, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "ca29bd6c6a11a627a67fd50b481c5783789b1cb2aad0b89f32e7c9b1ff959a9e", + "completed": 320, + "silicon_ttft_ms": 772.6545190812885, + "before_ttft_ms": 308.2671786093753, + "after_ttft_ms": 308.2671786093753, + "before_ttft_ape_pct": 60.102844027119, + "after_ttft_ape_pct": 60.102844027119, + "silicon_tpot_ms": 14.564830011653703, + "before_tpot_ms": 16.161165669841196, + "after_tpot_ms": 16.161165669841196, + "before_tpot_ape_pct": 10.960207959243085, + "after_tpot_ape_pct": 10.960207959243085 + }, + { + "index": 109, + "config_id": 587, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "331e28e23ed822b036c1e895216d783b3c3e67b1302476c98dab5b2bde09f09e", + "completed": 80, + "silicon_ttft_ms": 298.44391730021016, + "before_ttft_ms": 39.932019575000105, + "after_ttft_ms": 39.932019575000105, + "before_ttft_ape_pct": 86.61992513158452, + "after_ttft_ape_pct": 86.61992513158452, + "silicon_tpot_ms": 7.862424709146381, + "before_tpot_ms": 7.332061339806498, + "after_tpot_ms": 7.332061339806498, + "before_tpot_ape_pct": 6.745544650149837, + "after_tpot_ape_pct": 6.745544650149837 + }, + { + "index": 110, + "config_id": 587, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "00d8efa27eb2cf6e59c8f060c82fbc63eda522a3b7128ef546dc7b8de368093c", + "completed": 40, + "silicon_ttft_ms": 261.65758902534435, + "before_ttft_ms": 35.955596975000034, + "after_ttft_ms": 35.955596975000034, + "before_ttft_ape_pct": 86.25853081161068, + "after_ttft_ape_pct": 86.25853081161068, + "silicon_tpot_ms": 5.936045077144975, + "before_tpot_ms": 5.746876883350904, + "after_tpot_ms": 5.746876883350904, + "before_tpot_ape_pct": 3.1867715176626343, + "after_tpot_ape_pct": 3.1867715176626343 + }, + { + "index": 111, + "config_id": 587, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "d0e91bf7be04a716d18a60361d9ef4e638bde92f52f0c267f7605d52cf209df4", + "completed": 40, + "silicon_ttft_ms": 318.3010235494294, + "before_ttft_ms": 136.8682856499998, + "after_ttft_ms": 136.8682856499998, + "before_ttft_ape_pct": 57.00036269951065, + "after_ttft_ape_pct": 57.00036269951065, + "silicon_tpot_ms": 5.985981659775782, + "before_tpot_ms": 6.912556058408986, + "after_tpot_ms": 6.912556058408986, + "before_tpot_ape_pct": 15.479071793011645, + "after_tpot_ape_pct": 15.479071793011645 + }, + { + "index": 112, + "config_id": 587, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "e0d1cbecc403a03db706c0caebe4a5b575abea7de53d44155e3b1d9818684143", + "completed": 320, + "silicon_ttft_ms": 431.4421721908275, + "before_ttft_ms": 55.57467857812469, + "after_ttft_ms": 55.57467857812469, + "before_ttft_ape_pct": 87.11885806250208, + "after_ttft_ape_pct": 87.11885806250208, + "silicon_tpot_ms": 12.919420608158626, + "before_tpot_ms": 11.508552563504585, + "after_tpot_ms": 11.508552563504585, + "before_tpot_ape_pct": 10.920521031439112, + "after_tpot_ape_pct": 10.920521031439112 + }, + { + "index": 113, + "config_id": 587, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "0b1411f01e0f606c7d3eb03acba6981be55fcad25c08719797f10d01bbe04d1e", + "completed": 640, + "silicon_ttft_ms": 1134.2155090626875, + "before_ttft_ms": 501.5543165125003, + "after_ttft_ms": 501.5543165125003, + "before_ttft_ape_pct": 55.77962807729693, + "after_ttft_ape_pct": 55.77962807729693, + "silicon_tpot_ms": 21.30548264528225, + "before_tpot_ms": 24.512838028116875, + "after_tpot_ms": 24.512838028116875, + "before_tpot_ape_pct": 15.054131540854065, + "after_tpot_ape_pct": 15.054131540854065 + }, + { + "index": 114, + "config_id": 587, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "d99e391158ea2e784503a25a239271ed2a9b8ce616c6d873170a81ed8683388a", + "completed": 160, + "silicon_ttft_ms": 550.4462199181035, + "before_ttft_ms": 208.44621942500035, + "after_ttft_ms": 208.44621942500035, + "before_ttft_ape_pct": 62.13141050255311, + "after_ttft_ape_pct": 62.13141050255311, + "silicon_tpot_ms": 10.555219470144367, + "before_tpot_ms": 12.08351238693996, + "after_tpot_ms": 12.08351238693996, + "before_tpot_ape_pct": 14.479025482306618, + "after_tpot_ape_pct": 14.479025482306618 + }, + { + "index": 115, + "config_id": 615, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "3739067d36c9bb4e357a322485396416099b6aadf9efc258cdbc47802acafd15", + "completed": 320, + "silicon_ttft_ms": 1385.983195361041, + "before_ttft_ms": 404.8246558937498, + "after_ttft_ms": 404.8246558937498, + "before_ttft_ape_pct": 70.79151772916732, + "after_ttft_ape_pct": 70.79151772916732, + "silicon_tpot_ms": 22.72082891400109, + "before_tpot_ms": 18.577495850348136, + "after_tpot_ms": 18.577495850348136, + "before_tpot_ape_pct": 18.235835846198977, + "after_tpot_ape_pct": 18.235835846198977 + }, + { + "index": 116, + "config_id": 615, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "1d944f4fcba717791e95c3b65e3a7598e8eb2019edf7546c56a1b28419c02543", + "completed": 640, + "silicon_ttft_ms": 2178.391830824694, + "before_ttft_ms": 658.6408727046869, + "after_ttft_ms": 658.6408727046869, + "before_ttft_ape_pct": 69.7648116658912, + "after_ttft_ape_pct": 69.7648116658912, + "silicon_tpot_ms": 33.17059593289462, + "before_tpot_ms": 29.28860816871578, + "after_tpot_ms": 29.28860816871578, + "before_tpot_ape_pct": 11.703099251012105, + "after_tpot_ape_pct": 11.703099251012105 + }, + { + "index": 117, + "config_id": 615, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "61d258fb84f4547a126bb62bef76c27d548c7ee6ef98b51f4805f22337aa2596", + "completed": 160, + "silicon_ttft_ms": 978.4397798175632, + "before_ttft_ms": 273.76166561249966, + "after_ttft_ms": 273.76166561249966, + "before_ttft_ape_pct": 72.02059122498635, + "after_ttft_ape_pct": 72.02059122498635, + "silicon_tpot_ms": 16.376655136265775, + "before_tpot_ms": 13.145709689652666, + "after_tpot_ms": 13.145709689652666, + "before_tpot_ape_pct": 19.72897041385604, + "after_tpot_ape_pct": 19.72897041385604 + }, + { + "index": 118, + "config_id": 615, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "847013ad5e2497913a4506ed180470b6002e6a1813163f5219e45c35db44bf9c", + "completed": 640, + "silicon_ttft_ms": 822.546900670568, + "before_ttft_ms": 93.8757438062494, + "after_ttft_ms": 93.8757438062494, + "before_ttft_ape_pct": 88.58718648994741, + "after_ttft_ape_pct": 88.58718648994741, + "silicon_tpot_ms": 23.54379501731437, + "before_tpot_ms": 18.529749182973582, + "after_tpot_ms": 18.529749182973582, + "before_tpot_ape_pct": 21.296676388209303, + "after_tpot_ape_pct": 21.296676388209303 + }, + { + "index": 119, + "config_id": 615, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "f50ed49cb88f92f0cff849c46ca8f70b0ad0fe7dc321e24aeb5104719c243c43", + "completed": 160, + "silicon_ttft_ms": 481.6890717484057, + "before_ttft_ms": 51.09307547499991, + "after_ttft_ms": 51.09307547499991, + "before_ttft_ape_pct": 89.39293447336362, + "after_ttft_ape_pct": 89.39293447336362, + "silicon_tpot_ms": 14.328304411027087, + "before_tpot_ms": 10.570708743287984, + "after_tpot_ms": 10.570708743287984, + "before_tpot_ape_pct": 26.224984896658476, + "after_tpot_ape_pct": 26.224984896658476 + }, + { + "index": 120, + "config_id": 615, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b200", + "isl": 1024, + "osl": 8192, + "concurrency": 32, + "spec_sha256": "7532b17d47af4d71b0b0c7a38a9b53c259c8790330fa3f3f6e63d7dfdff2e168", + "completed": 320, + "silicon_ttft_ms": 587.7308338873263, + "before_ttft_ms": 64.71188626562623, + "after_ttft_ms": 64.71188626562623, + "before_ttft_ape_pct": 88.98953695561391, + "after_ttft_ape_pct": 88.98953695561391, + "silicon_tpot_ms": 15.793447750195355, + "before_tpot_ms": 12.952644470611215, + "after_tpot_ms": 12.952644470611215, + "before_tpot_ape_pct": 17.987226883686624, + "after_tpot_ape_pct": 17.987226883686624 + }, + { + "index": 121, + "config_id": 615, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b200", + "isl": 1024, + "osl": 8192, + "concurrency": 16, + "spec_sha256": "1967084753a515c44475f2888384bb20e584d818dda99767a45a3dd2c791c1a9", + "completed": 160, + "silicon_ttft_ms": 501.8999164283742, + "before_ttft_ms": 51.09307547500155, + "after_ttft_ms": 51.09307547500155, + "before_ttft_ape_pct": 89.82006694908605, + "after_ttft_ape_pct": 89.82006694908605, + "silicon_tpot_ms": 12.568650775742281, + "before_tpot_ms": 10.440558948192445, + "after_tpot_ms": 10.440558948192445, + "before_tpot_ape_pct": 16.93174442921981, + "after_tpot_ape_pct": 16.93174442921981 + }, + { + "index": 122, + "config_id": 615, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b200", + "isl": 1024, + "osl": 8192, + "concurrency": 64, + "spec_sha256": "e43c07735809537e540aeb44ac7f693268860e87be6f0bc54935fd26b46d5394", + "completed": 640, + "silicon_ttft_ms": 797.130254309377, + "before_ttft_ms": 92.0937715624973, + "after_ttft_ms": 92.0937715624973, + "before_ttft_ape_pct": 88.44683524874036, + "after_ttft_ape_pct": 88.44683524874036, + "silicon_tpot_ms": 20.397601442742875, + "before_tpot_ms": 18.158160954459962, + "after_tpot_ms": 18.158160954459962, + "before_tpot_ape_pct": 10.978940315944197, + "after_tpot_ape_pct": 10.978940315944197 + }, + { + "index": 123, + "config_id": 615, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 128, + "spec_sha256": "897c087b55461bb46ef731e3eb0e30fcf6118a1283b55d0e9274fc520fd343f1", + "completed": 1280, + "silicon_ttft_ms": 1040.0363928065417, + "before_ttft_ms": 149.38777223828131, + "after_ttft_ms": 149.38777223828131, + "before_ttft_ape_pct": 85.63629376130216, + "after_ttft_ape_pct": 85.63629376130216, + "silicon_tpot_ms": 30.104162261016178, + "before_tpot_ms": 23.927052084732242, + "after_tpot_ms": 23.927052084732242, + "before_tpot_ape_pct": 20.519123311672672, + "after_tpot_ape_pct": 20.519123311672672 + }, + { + "index": 124, + "config_id": 615, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "89c5c058c002fd15000d05fc6ef57f78b804eb1fcdb30714f287f5db47ebcc92", + "completed": 320, + "silicon_ttft_ms": 661.896740739212, + "before_ttft_ms": 65.60368626562477, + "after_ttft_ms": 65.60368626562477, + "before_ttft_ape_pct": 90.0885316050419, + "after_ttft_ape_pct": 90.0885316050419, + "silicon_tpot_ms": 18.854730144085675, + "before_tpot_ms": 13.277923697290472, + "after_tpot_ms": 13.277923697290472, + "before_tpot_ape_pct": 29.577757964064666, + "after_tpot_ape_pct": 29.577757964064666 + }, + { + "index": 125, + "config_id": 615, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 128, + "spec_sha256": "b37f6ab44946ef1bf468aae0f98f7af17ef4ad4703589464b64fa5dd852e7702", + "completed": 1280, + "silicon_ttft_ms": 3334.467003904683, + "before_ttft_ms": 1164.0324984273418, + "after_ttft_ms": 1164.0324984273418, + "before_ttft_ape_pct": 65.09089767377358, + "after_ttft_ape_pct": 65.09089767377358, + "silicon_tpot_ms": 51.374326749208905, + "before_tpot_ms": 44.21947227458301, + "after_tpot_ms": 44.21947227458301, + "before_tpot_ape_pct": 13.926906545273743, + "after_tpot_ape_pct": 13.926906545273743 + }, + { + "index": 126, + "config_id": 615, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b200", + "isl": 1024, + "osl": 8192, + "concurrency": 128, + "spec_sha256": "9b0c44b0d9f541ee33354de7e71bde4d9294693aaa93cf05826a639844298e24", + "completed": 1280, + "silicon_ttft_ms": 891.160001078606, + "before_ttft_ms": 146.2943459140616, + "after_ttft_ms": 146.2943459140616, + "before_ttft_ape_pct": 83.5838294204188, + "after_ttft_ape_pct": 83.5838294204188, + "silicon_tpot_ms": 25.868043287503244, + "before_tpot_ms": 22.563468558020283, + "after_tpot_ms": 22.563468558020283, + "before_tpot_ape_pct": 12.77473789863105, + "after_tpot_ape_pct": 12.77473789863105 + }, + { + "index": 127, + "config_id": 621, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "ccf95796356ed9fb1daa2ce2f111aa097571e0430ec5a796c6c3ed619932576d", + "completed": 160, + "silicon_ttft_ms": 354.50660856440663, + "before_ttft_ms": 56.169251225000245, + "after_ttft_ms": 56.169251225000245, + "before_ttft_ape_pct": 84.15565468512403, + "after_ttft_ape_pct": 84.15565468512403, + "silicon_tpot_ms": 14.238929644323706, + "before_tpot_ms": 11.21734670084463, + "after_tpot_ms": 11.21734670084463, + "before_tpot_ape_pct": 21.220576398336366, + "after_tpot_ape_pct": 21.220576398336366 + }, + { + "index": 128, + "config_id": 621, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "1ee1092c54c9a73e20e2263b30fd2e33eb2c07744255337b7831c5848bdd0088", + "completed": 320, + "silicon_ttft_ms": 756.1991440772545, + "before_ttft_ms": 424.5454989125001, + "after_ttft_ms": 424.5454989125001, + "before_ttft_ape_pct": 43.857976799147515, + "after_ttft_ape_pct": 43.857976799147515, + "silicon_tpot_ms": 22.208625058021603, + "before_tpot_ms": 18.62099265493812, + "after_tpot_ms": 18.62099265493812, + "before_tpot_ape_pct": 16.154230141265113, + "after_tpot_ape_pct": 16.154230141265113 + }, + { + "index": 129, + "config_id": 621, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "41bafaa4cfe484e9b42642508d101d172e14a9c6034107a571b3d683b9bd5254", + "completed": 80, + "silicon_ttft_ms": 326.78975421004, + "before_ttft_ms": 48.91725388749992, + "after_ttft_ms": 48.91725388749992, + "before_ttft_ape_pct": 85.0309707518985, + "after_ttft_ape_pct": 85.0309707518985, + "silicon_tpot_ms": 10.020957003187792, + "before_tpot_ms": 8.116710808370843, + "after_tpot_ms": 8.116710808370843, + "before_tpot_ape_pct": 19.002638113417557, + "after_tpot_ape_pct": 19.002638113417557 + }, + { + "index": 130, + "config_id": 621, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "4c4ccf3ce1638a57d8519898e50bf578fa16809894d4a9f41f349674a3d469ae", + "completed": 40, + "silicon_ttft_ms": 313.800502801314, + "before_ttft_ms": 41.90861002500013, + "after_ttft_ms": 41.90861002500013, + "before_ttft_ape_pct": 86.64482381293857, + "after_ttft_ape_pct": 86.64482381293857, + "silicon_tpot_ms": 8.510407949882282, + "before_tpot_ms": 7.1978591961444405, + "after_tpot_ms": 7.1978591961444405, + "before_tpot_ape_pct": 15.42286528997705, + "after_tpot_ape_pct": 15.42286528997705 + }, + { + "index": 131, + "config_id": 621, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "aff573124a77a7567bafb56140b2089d20c27890f65da7e95c8c58210d685468", + "completed": 40, + "silicon_ttft_ms": 399.6235067024827, + "before_ttft_ms": 188.49616180000024, + "after_ttft_ms": 188.49616180000024, + "before_ttft_ape_pct": 52.831563049083975, + "after_ttft_ape_pct": 52.831563049083975, + "silicon_tpot_ms": 8.604319872660259, + "before_tpot_ms": 7.654038194687553, + "after_tpot_ms": 7.654038194687553, + "before_tpot_ape_pct": 11.0442393127686, + "after_tpot_ape_pct": 11.0442393127686 + }, + { + "index": 132, + "config_id": 621, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "0194a9aaf87a67dcc2136f8a8d4d3d986ad0cfadfa314d0190e6dbbf8f5474fe", + "completed": 160, + "silicon_ttft_ms": 570.7883091643453, + "before_ttft_ms": 287.0750369062503, + "after_ttft_ms": 287.0750369062503, + "before_ttft_ape_pct": 49.70551563564107, + "after_ttft_ape_pct": 49.70551563564107, + "silicon_tpot_ms": 15.042995552153101, + "before_tpot_ms": 13.477254836946585, + "after_tpot_ms": 13.477254836946585, + "before_tpot_ape_pct": 10.408437001647663, + "after_tpot_ape_pct": 10.408437001647663 + }, + { + "index": 133, + "config_id": 621, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "adf300d5ee2ed7dbb693ce47a963893101ccd57af2056975a834c64f197a3eb0", + "completed": 80, + "silicon_ttft_ms": 547.6442037150264, + "before_ttft_ms": 224.34727936249993, + "after_ttft_ms": 224.34727936249993, + "before_ttft_ape_pct": 59.03411780119161, + "after_ttft_ape_pct": 59.03411780119161, + "silicon_tpot_ms": 10.855663102974487, + "before_tpot_ms": 9.155429799415185, + "after_tpot_ms": 9.155429799415185, + "before_tpot_ape_pct": 15.662178232976231, + "after_tpot_ape_pct": 15.662178232976231 + }, + { + "index": 134, + "config_id": 621, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "a2c5e0776945192c11a2bc33431688afcd254284d0a814ddfefb864a9a7c95db", + "completed": 320, + "silicon_ttft_ms": 363.68632859084755, + "before_ttft_ms": 72.89156069375001, + "after_ttft_ms": 72.89156069375001, + "before_ttft_ape_pct": 79.95757471110385, + "after_ttft_ape_pct": 79.95757471110385, + "silicon_tpot_ms": 19.997111059442137, + "before_tpot_ms": 13.983771854504408, + "after_tpot_ms": 13.983771854504408, + "before_tpot_ape_pct": 30.071039696998536, + "after_tpot_ape_pct": 30.071039696998536 + }, + { + "index": 135, + "config_id": 621, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "h200", + "isl": 1024, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "1ccefa9126ba3b5fc1d9e3ab716885f4367ceb973a7ea8f9861c9fa752c3e352", + "completed": 640, + "silicon_ttft_ms": 404.9074711392677, + "before_ttft_ms": 105.24919703437499, + "after_ttft_ms": 105.24919703437499, + "before_ttft_ape_pct": 74.00660532683168, + "after_ttft_ape_pct": 74.00660532683168, + "silicon_tpot_ms": 28.760112083694533, + "before_tpot_ms": 19.378015846215167, + "after_tpot_ms": 19.378015846215167, + "before_tpot_ape_pct": 32.621904289443016, + "after_tpot_ape_pct": 32.621904289443016 + }, + { + "index": 136, + "config_id": 621, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "h200", + "isl": 8192, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "620c94e42ffd55c985374bc44aca117bfe6f94e4ae66a935632019bc6592a739", + "completed": 640, + "silicon_ttft_ms": 1129.3540400765778, + "before_ttft_ms": 690.7379062562497, + "after_ttft_ms": 690.7379062562497, + "before_ttft_ape_pct": 38.83778852826231, + "after_ttft_ape_pct": 38.83778852826231, + "silicon_tpot_ms": 34.481202747945666, + "before_tpot_ms": 28.776245518451656, + "after_tpot_ms": 28.776245518451656, + "before_tpot_ape_pct": 16.54512248658119, + "after_tpot_ape_pct": 16.54512248658119 + }, + { + "index": 137, + "config_id": 716, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "3e60fa9d2df7bd3d6e151d34f1b333631815d80fe46de28844ca8e9d4b38f660", + "completed": 40, + "silicon_ttft_ms": 285.9091628633905, + "before_ttft_ms": 168.2029210749999, + "after_ttft_ms": 168.2029210749999, + "before_ttft_ape_pct": 41.16910441398883, + "after_ttft_ape_pct": 41.16910441398883, + "silicon_tpot_ms": 6.980331794164076, + "before_tpot_ms": 7.893085376009992, + "after_tpot_ms": 7.893085376009992, + "before_tpot_ape_pct": 13.076077309233725, + "after_tpot_ape_pct": 13.076077309233725 + }, + { + "index": 138, + "config_id": 716, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "d5dfd8a12702d362819bec38536f118dbdd6477e8bd94398b43cd79c94b4d801", + "completed": 40, + "silicon_ttft_ms": 171.46362470230088, + "before_ttft_ms": 43.546054174999725, + "after_ttft_ms": 43.546054174999725, + "before_ttft_ape_pct": 74.6033281107842, + "after_ttft_ape_pct": 74.6033281107842, + "silicon_tpot_ms": 6.759255353774809, + "before_tpot_ms": 6.949573693799738, + "after_tpot_ms": 6.949573693799738, + "before_tpot_ape_pct": 2.815670219037414, + "after_tpot_ape_pct": 2.815670219037414 + }, + { + "index": 139, + "config_id": 716, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "e0639a2a55a7982e2ae1cbc36883910702fac1949ebb3013e41362fe9d246954", + "completed": 160, + "silicon_ttft_ms": 210.4498686916486, + "before_ttft_ms": 54.86093252499997, + "after_ttft_ms": 54.86093252499997, + "before_ttft_ape_pct": 73.93159099311092, + "after_ttft_ape_pct": 73.93159099311092, + "silicon_tpot_ms": 10.561569413542145, + "before_tpot_ms": 10.21954596568776, + "after_tpot_ms": 10.21954596568776, + "before_tpot_ape_pct": 3.2383771242921577, + "after_tpot_ape_pct": 3.2383771242921577 + }, + { + "index": 140, + "config_id": 716, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "6b7b1b3fd3ee2b21ffd37e5d5c00c30ca765f644eff70f3b694da30a56eb3a33", + "completed": 320, + "silicon_ttft_ms": 286.8199783817545, + "before_ttft_ms": 68.38629075937504, + "after_ttft_ms": 68.38629075937504, + "before_ttft_ape_pct": 76.15706857478611, + "after_ttft_ape_pct": 76.15706857478611, + "silicon_tpot_ms": 14.002294376458131, + "before_tpot_ms": 13.205829499782363, + "after_tpot_ms": 13.205829499782363, + "before_tpot_ape_pct": 5.688102644198468, + "after_tpot_ape_pct": 5.688102644198468 + }, + { + "index": 141, + "config_id": 716, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 256, + "spec_sha256": "75997626a74e2d9a82c35fb14c42fcb84d3f719ec433a5050c95fbb06f4e5f7e", + "completed": 2560, + "silicon_ttft_ms": 645.0889428452228, + "before_ttft_ms": 250.38021992070156, + "after_ttft_ms": 250.38021992070156, + "before_ttft_ape_pct": 61.18671344506743, + "after_ttft_ape_pct": 61.18671344506743, + "silicon_tpot_ms": 33.44910999986752, + "before_tpot_ms": 34.545418978706984, + "after_tpot_ms": 34.545418978706984, + "before_tpot_ape_pct": 3.277543046268816, + "after_tpot_ape_pct": 3.277543046268816 + }, + { + "index": 142, + "config_id": 716, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 256, + "spec_sha256": "c74140a3e952ea01c8b229ea6b3a9231510718e0743004537fab1ce52dd37f81", + "completed": 2560, + "silicon_ttft_ms": 2396.4617931235807, + "before_ttft_ms": 2036.5145508566402, + "after_ttft_ms": 2036.5145508566402, + "before_ttft_ape_pct": 15.019944958011633, + "after_ttft_ape_pct": 15.019944958011633, + "silicon_tpot_ms": 61.17976870420999, + "before_tpot_ms": 71.66310231480337, + "after_tpot_ms": 71.66310231480337, + "before_tpot_ape_pct": 17.135294612305362, + "after_tpot_ape_pct": 17.135294612305362 + }, + { + "index": 143, + "config_id": 716, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 128, + "spec_sha256": "fdc281d121e60a46648362f8ce7c82a18c4ef8e925207a0738740f4649e41ce8", + "completed": 1280, + "silicon_ttft_ms": 1343.3636750400183, + "before_ttft_ms": 1089.122327953124, + "after_ttft_ms": 1089.122327953124, + "before_ttft_ape_pct": 18.92572739688829, + "after_ttft_ape_pct": 18.92572739688829, + "silicon_tpot_ms": 37.53467150879646, + "before_tpot_ms": 43.26574754890312, + "after_tpot_ms": 43.26574754890312, + "before_tpot_ape_pct": 15.268752355441695, + "after_tpot_ape_pct": 15.268752355441695 + }, + { + "index": 144, + "config_id": 716, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "ef428a6cb8258e103af634ff580bcbed557185436182f142cee1337db60a6399", + "completed": 640, + "silicon_ttft_ms": 897.6363394020154, + "before_ttft_ms": 616.2323174656258, + "after_ttft_ms": 616.2323174656258, + "before_ttft_ape_pct": 31.349446271733438, + "after_ttft_ape_pct": 31.349446271733438, + "silicon_tpot_ms": 24.831084729792124, + "before_tpot_ms": 28.490697530610397, + "after_tpot_ms": 28.490697530610397, + "before_tpot_ape_pct": 14.73803033835046, + "after_tpot_ape_pct": 14.73803033835046 + }, + { + "index": 145, + "config_id": 716, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "54a4e2c3caca830b47273af5deb6ecdb6a64de1e707abc98970e89ee8fab8d92", + "completed": 80, + "silicon_ttft_ms": 182.5718104999396, + "before_ttft_ms": 48.81122264999959, + "after_ttft_ms": 48.81122264999959, + "before_ttft_ape_pct": 73.26464446162913, + "after_ttft_ape_pct": 73.26464446162913, + "silicon_tpot_ms": 8.145019093337185, + "before_tpot_ms": 8.356968017637504, + "after_tpot_ms": 8.356968017637504, + "before_tpot_ape_pct": 2.60219063787952, + "after_tpot_ape_pct": 2.60219063787952 + }, + { + "index": 146, + "config_id": 716, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "1d96eef3c7430a4541ef8f4df23548cd978d83297a8eaa50ac49af09d620b15b", + "completed": 640, + "silicon_ttft_ms": 334.72175415263337, + "before_ttft_ms": 94.6752814249999, + "after_ttft_ms": 94.6752814249999, + "before_ttft_ape_pct": 71.71522906699757, + "after_ttft_ape_pct": 71.71522906699757, + "silicon_tpot_ms": 18.668616088518743, + "before_tpot_ms": 18.734806275773796, + "after_tpot_ms": 18.734806275773796, + "before_tpot_ape_pct": 0.3545532616944236, + "after_tpot_ape_pct": 0.3545532616944236 + }, + { + "index": 147, + "config_id": 716, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "8e8226900d5da0cefb60862a6ab1dcd272e5613e724a932dff6e4f57a8a07d28", + "completed": 80, + "silicon_ttft_ms": 405.2475653224974, + "before_ttft_ms": 200.14161352500025, + "after_ttft_ms": 200.14161352500025, + "before_ttft_ape_pct": 50.61250685967062, + "after_ttft_ape_pct": 50.61250685967062, + "silicon_tpot_ms": 8.757566512420754, + "before_tpot_ms": 9.759310948752104, + "after_tpot_ms": 9.759310948752104, + "before_tpot_ape_pct": 11.438616365751697, + "after_tpot_ape_pct": 11.438616365751697 + }, + { + "index": 148, + "config_id": 716, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "318fdba63b1005d1dac73a32ec245e4bc71fbe6160408c2e03f5eaeaf24fccad", + "completed": 320, + "silicon_ttft_ms": 601.9223929692089, + "before_ttft_ms": 378.7562931187505, + "after_ttft_ms": 378.7562931187505, + "before_ttft_ape_pct": 37.07556031428031, + "after_ttft_ape_pct": 37.07556031428031, + "silicon_tpot_ms": 16.916263121752152, + "before_tpot_ms": 18.004658908799662, + "after_tpot_ms": 18.004658908799662, + "before_tpot_ape_pct": 6.434020204190194, + "after_tpot_ape_pct": 6.434020204190194 + }, + { + "index": 149, + "config_id": 716, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 128, + "spec_sha256": "431575955e7034cf7db89f224982b30acf239005aae8f20707adc3ad33542dea", + "completed": 1280, + "silicon_ttft_ms": 395.51614410829643, + "before_ttft_ms": 146.35721746406298, + "after_ttft_ms": 146.35721746406298, + "before_ttft_ape_pct": 62.99589292517251, + "after_ttft_ape_pct": 62.99589292517251, + "silicon_tpot_ms": 24.90569596156061, + "before_tpot_ms": 24.96923494384972, + "after_tpot_ms": 24.96923494384972, + "before_tpot_ape_pct": 0.2551182765066162, + "after_tpot_ape_pct": 0.2551182765066162 + }, + { + "index": 150, + "config_id": 716, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "885fd9c6ff571a0babbf822c9f0986cfb83b9d57f6d4d93239759d8f9f378f5e", + "completed": 160, + "silicon_ttft_ms": 517.0539377300884, + "before_ttft_ms": 256.1257202750003, + "after_ttft_ms": 256.1257202750003, + "before_ttft_ape_pct": 50.46440969013515, + "after_ttft_ape_pct": 50.46440969013515, + "silicon_tpot_ms": 11.790648883833109, + "before_tpot_ms": 12.513883249441502, + "after_tpot_ms": 12.513883249441502, + "before_tpot_ape_pct": 6.133965761630522, + "after_tpot_ape_pct": 6.133965761630522 + }, + { + "index": 151, + "config_id": 736, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "8acd0a16c97a815f155d7e000d7bb1a83c246b133b6ef8d888c245500442b69e", + "completed": 320, + "silicon_ttft_ms": 366.27362702238315, + "before_ttft_ms": 63.07031969687496, + "after_ttft_ms": 63.07031969687496, + "before_ttft_ape_pct": 82.78054573308913, + "after_ttft_ape_pct": 82.78054573308913, + "silicon_tpot_ms": 13.534719556921504, + "before_tpot_ms": 13.731579102514393, + "after_tpot_ms": 13.731579102514393, + "before_tpot_ape_pct": 1.454478201524445, + "after_tpot_ape_pct": 1.454478201524445 + }, + { + "index": 152, + "config_id": 736, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "4891484846911e2f57e116b7590a0dca8ec5a4ea8ce2554b54d552dfc30c8910", + "completed": 80, + "silicon_ttft_ms": 182.59373290347867, + "before_ttft_ms": 43.47395742500048, + "after_ttft_ms": 43.47395742500048, + "before_ttft_ape_pct": 76.19088194665402, + "after_ttft_ape_pct": 76.19088194665402, + "silicon_tpot_ms": 8.097183168504047, + "before_tpot_ms": 8.471565529867288, + "after_tpot_ms": 8.471565529867288, + "before_tpot_ape_pct": 4.6236123547197305, + "after_tpot_ape_pct": 4.6236123547197305 + }, + { + "index": 153, + "config_id": 736, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 128, + "spec_sha256": "843a5e15408ac53074f3bc287ea4b2f758d90307722ac9bcb460462ac2efea66", + "completed": 1280, + "silicon_ttft_ms": 678.2561419668127, + "before_ttft_ms": 141.3182703070316, + "after_ttft_ms": 141.3182703070316, + "before_ttft_ape_pct": 79.16446876585064, + "after_ttft_ape_pct": 79.16446876585064, + "silicon_tpot_ms": 23.126099437547584, + "before_tpot_ms": 26.46049485505882, + "after_tpot_ms": 26.46049485505882, + "before_tpot_ape_pct": 14.418321717053173, + "after_tpot_ape_pct": 14.418321717053173 + }, + { + "index": 154, + "config_id": 736, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "352900e93840d0fcf7e1c06bd43a726e84fdc3cb54d19de43cc719b4bca859e4", + "completed": 640, + "silicon_ttft_ms": 475.9461443085456, + "before_ttft_ms": 89.4883446624999, + "after_ttft_ms": 89.4883446624999, + "before_ttft_ape_pct": 81.19780026950139, + "after_ttft_ape_pct": 81.19780026950139, + "silicon_tpot_ms": 17.435344458861998, + "before_tpot_ms": 19.91296151646649, + "after_tpot_ms": 19.91296151646649, + "before_tpot_ape_pct": 14.21031321434647, + "after_tpot_ape_pct": 14.21031321434647 + }, + { + "index": 155, + "config_id": 736, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "af355554ef86fe763ab66a05943fbdb8deae00ae8bb4996f058efcb97a7f5d2f", + "completed": 160, + "silicon_ttft_ms": 306.74278564360975, + "before_ttft_ms": 49.45917183124982, + "after_ttft_ms": 49.45917183124982, + "before_ttft_ape_pct": 83.87601138606267, + "after_ttft_ape_pct": 83.87601138606267, + "silicon_tpot_ms": 10.576980892255104, + "before_tpot_ms": 10.912511413582868, + "after_tpot_ms": 10.912511413582868, + "before_tpot_ape_pct": 3.1722712250851437, + "after_tpot_ape_pct": 3.1722712250851437 + }, + { + "index": 156, + "config_id": 736, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "27523e07841a4fbcf27e1c4fe1cb7b869910fc7749b7ea08f584fe68712c3ce7", + "completed": 40, + "silicon_ttft_ms": 179.41149852704257, + "before_ttft_ms": 37.99775100000012, + "after_ttft_ms": 37.99775100000012, + "before_ttft_ape_pct": 78.82089425039123, + "after_ttft_ape_pct": 78.82089425039123, + "silicon_tpot_ms": 6.661889342875908, + "before_tpot_ms": 6.813676806695229, + "after_tpot_ms": 6.813676806695229, + "before_tpot_ape_pct": 2.278444687491543, + "after_tpot_ape_pct": 2.278444687491543 + }, + { + "index": 157, + "config_id": 737, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 128, + "spec_sha256": "14a4691cd6fb59732fa75a308dd78a8097a4e3b3c04faff9347072e40053fd96", + "completed": 1280, + "silicon_ttft_ms": 508.68740680816705, + "before_ttft_ms": 112.49686236171867, + "after_ttft_ms": 112.49686236171867, + "before_ttft_ape_pct": 77.88487372478974, + "after_ttft_ape_pct": 77.88487372478974, + "silicon_tpot_ms": 17.157402104472023, + "before_tpot_ms": 19.267872822115585, + "after_tpot_ms": 19.267872822115585, + "before_tpot_ape_pct": 12.300642631051195, + "after_tpot_ape_pct": 12.300642631051195 + }, + { + "index": 158, + "config_id": 737, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "cf216f7291ebf9d2f14070af9e1d22279c5c0098b9d29a85b637bb8777f5ca67", + "completed": 640, + "silicon_ttft_ms": 389.69345494697336, + "before_ttft_ms": 72.16768416562472, + "after_ttft_ms": 72.16768416562472, + "before_ttft_ape_pct": 81.48090935336731, + "after_ttft_ape_pct": 81.48090935336731, + "silicon_tpot_ms": 13.146922588692542, + "before_tpot_ms": 14.12491508468671, + "after_tpot_ms": 14.12491508468671, + "before_tpot_ape_pct": 7.438946182244388, + "after_tpot_ape_pct": 7.438946182244388 + }, + { + "index": 159, + "config_id": 737, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "43354db4fcfc4c1a0d8e1889b14b1b534745b0ab9058ae1706eaf3f23a8abc68", + "completed": 320, + "silicon_ttft_ms": 320.4598238138715, + "before_ttft_ms": 51.633785953124914, + "after_ttft_ms": 51.633785953124914, + "before_ttft_ape_pct": 83.88759460121445, + "after_ttft_ape_pct": 83.88759460121445, + "silicon_tpot_ms": 10.291020196825341, + "before_tpot_ms": 10.020960583899111, + "after_tpot_ms": 10.020960583899111, + "before_tpot_ape_pct": 2.6242258567283794, + "after_tpot_ape_pct": 2.6242258567283794 + }, + { + "index": 160, + "config_id": 737, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "c5b3271d1bb7b18092672a3739fee865e8ff0c4b24317f06fc6a59a717d70743", + "completed": 160, + "silicon_ttft_ms": 438.40306743113615, + "before_ttft_ms": 199.31829204374986, + "after_ttft_ms": 199.31829204374986, + "before_ttft_ape_pct": 54.53537923179368, + "after_ttft_ape_pct": 54.53537923179368, + "silicon_tpot_ms": 8.880492413556823, + "before_tpot_ms": 10.108042822996135, + "after_tpot_ms": 10.108042822996135, + "before_tpot_ape_pct": 13.822999359420107, + "after_tpot_ape_pct": 13.822999359420107 + }, + { + "index": 161, + "config_id": 737, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "36e1aa6915eb60c06b8913f51065cf29ce06ea7722fc5da0e7f379dd382efdcf", + "completed": 40, + "silicon_ttft_ms": 251.5187936834991, + "before_ttft_ms": 130.86919222499978, + "after_ttft_ms": 130.86919222499978, + "before_ttft_ape_pct": 47.96842402572899, + "after_ttft_ape_pct": 47.96842402572899, + "silicon_tpot_ms": 5.42763324067806, + "before_tpot_ms": 6.2102624141144265, + "after_tpot_ms": 6.2102624141144265, + "before_tpot_ape_pct": 14.419345205030742, + "after_tpot_ape_pct": 14.419345205030742 + }, + { + "index": 162, + "config_id": 737, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "50314ae970c3cc94957c070eca1437da1b1f8b0910c9896669eb695d5f05cd3a", + "completed": 80, + "silicon_ttft_ms": 311.1168064555386, + "before_ttft_ms": 155.77341899999993, + "after_ttft_ms": 155.77341899999993, + "before_ttft_ape_pct": 49.93088905267439, + "after_ttft_ape_pct": 49.93088905267439, + "silicon_tpot_ms": 6.782397882306337, + "before_tpot_ms": 7.69391215244291, + "after_tpot_ms": 7.69391215244291, + "before_tpot_ape_pct": 13.439410160741193, + "after_tpot_ape_pct": 13.439410160741193 + }, + { + "index": 163, + "config_id": 737, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "d68db08de0115a4faf264b9fe6a464ec5f6bd6e1336400ed825ee71a2ae7dacf", + "completed": 320, + "silicon_ttft_ms": 645.9430671551672, + "before_ttft_ms": 294.77471921874974, + "after_ttft_ms": 294.77471921874974, + "before_ttft_ape_pct": 54.365216656479795, + "after_ttft_ape_pct": 54.365216656479795, + "silicon_tpot_ms": 11.964467666973261, + "before_tpot_ms": 14.091384025710926, + "after_tpot_ms": 14.091384025710926, + "before_tpot_ape_pct": 17.776941005146508, + "after_tpot_ape_pct": 17.776941005146508 + }, + { + "index": 164, + "config_id": 737, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "2eaac563c34b142a27e1c6e035c6178b752ce689333ee0dd22d1d861a80dd303", + "completed": 640, + "silicon_ttft_ms": 1061.0571732366225, + "before_ttft_ms": 479.6047919593749, + "after_ttft_ms": 479.6047919593749, + "before_ttft_ape_pct": 54.799345025263776, + "after_ttft_ape_pct": 54.799345025263776, + "silicon_tpot_ms": 17.829738165097424, + "before_tpot_ms": 22.318846259902166, + "after_tpot_ms": 22.318846259902166, + "before_tpot_ape_pct": 25.17764452420501, + "after_tpot_ape_pct": 25.17764452420501 + }, + { + "index": 165, + "config_id": 737, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 128, + "spec_sha256": "9eb64ddbf16c7d9ee30dd66839040321e24e6bb556f6f90abf29fb51222c0917", + "completed": 1280, + "silicon_ttft_ms": 1577.7106872239528, + "before_ttft_ms": 847.7367643468749, + "after_ttft_ms": 847.7367643468749, + "before_ttft_ape_pct": 46.26792027133296, + "after_ttft_ape_pct": 46.26792027133296, + "silicon_tpot_ms": 27.82723028077221, + "before_tpot_ms": 34.51630900224447, + "after_tpot_ms": 34.51630900224447, + "before_tpot_ape_pct": 24.037888981334277, + "after_tpot_ape_pct": 24.037888981334277 + }, + { + "index": 166, + "config_id": 737, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "32644bab60a157b6965f916bd9fd12256db709f97dd0649db733ea9c3350dccf", + "completed": 80, + "silicon_ttft_ms": 165.44376562233083, + "before_ttft_ms": 36.37367012500008, + "after_ttft_ms": 36.37367012500008, + "before_ttft_ape_pct": 78.0144812419027, + "after_ttft_ape_pct": 78.0144812419027, + "silicon_tpot_ms": 6.364390239314593, + "before_tpot_ms": 6.610919572070522, + "after_tpot_ms": 6.610919572070522, + "before_tpot_ape_pct": 3.873573484432935, + "after_tpot_ape_pct": 3.873573484432935 + }, + { + "index": 167, + "config_id": 737, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "1d1c6be336bd01d7e6b2066a3e222672431ffc0712bf0a938910dae606b97fee", + "completed": 160, + "silicon_ttft_ms": 257.56981729355175, + "before_ttft_ms": 41.06643914999982, + "after_ttft_ms": 41.06643914999982, + "before_ttft_ape_pct": 84.05619121777903, + "after_ttft_ape_pct": 84.05619121777903, + "silicon_tpot_ms": 8.004855234240946, + "before_tpot_ms": 8.064528074207846, + "after_tpot_ms": 8.064528074207846, + "before_tpot_ape_pct": 0.7454580778881281, + "after_tpot_ape_pct": 0.7454580778881281 + }, + { + "index": 168, + "config_id": 737, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "7f2d013fcccac7b7e9fd7924651c990b31071b6e565876803992dc8d407d4ee0", + "completed": 40, + "silicon_ttft_ms": 165.1337789138779, + "before_ttft_ms": 32.32348997500001, + "after_ttft_ms": 32.32348997500001, + "before_ttft_ape_pct": 80.42587640905519, + "after_ttft_ape_pct": 80.42587640905519, + "silicon_tpot_ms": 5.387205029004747, + "before_tpot_ms": 5.44922679790243, + "after_tpot_ms": 5.44922679790243, + "before_tpot_ape_pct": 1.151279161712937, + "after_tpot_ape_pct": 1.151279161712937 + }, + { + "index": 169, + "config_id": 738, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "f4b90c45a621ac1e25d2de077828900671a878c24edd7d22acf055053439b7c0", + "completed": 40, + "silicon_ttft_ms": 198.84141764923697, + "before_ttft_ms": 35.634773350000486, + "after_ttft_ms": 35.634773350000486, + "before_ttft_ape_pct": 82.07879737969812, + "after_ttft_ape_pct": 82.07879737969812, + "silicon_tpot_ms": 5.825255988319065, + "before_tpot_ms": 5.598090616151398, + "after_tpot_ms": 5.598090616151398, + "before_tpot_ape_pct": 3.899663338798909, + "after_tpot_ape_pct": 3.899663338798909 + }, + { + "index": 170, + "config_id": 738, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "59addaec455add43f35eeb09c14402749cbcd1f0d5f24142b03a836a73c89dcb", + "completed": 160, + "silicon_ttft_ms": 462.51472886360716, + "before_ttft_ms": 204.89328724999996, + "after_ttft_ms": 204.89328724999996, + "before_ttft_ape_pct": 55.700159483910895, + "after_ttft_ape_pct": 55.700159483910895, + "silicon_tpot_ms": 10.470014455007819, + "before_tpot_ms": 12.058015856979967, + "after_tpot_ms": 12.058015856979967, + "before_tpot_ape_pct": 15.167136672028247, + "after_tpot_ape_pct": 15.167136672028247 + }, + { + "index": 171, + "config_id": 738, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "6fc94c2cbd8a164b61142715356a517efe10724de04a751c12c81cef1cc3ed78", + "completed": 320, + "silicon_ttft_ms": 664.4520292566085, + "before_ttft_ms": 303.010582553125, + "after_ttft_ms": 303.010582553125, + "before_ttft_ape_pct": 54.39692119051327, + "after_ttft_ape_pct": 54.39692119051327, + "silicon_tpot_ms": 14.495475414559214, + "before_tpot_ms": 16.05396939464622, + "after_tpot_ms": 16.05396939464622, + "before_tpot_ape_pct": 10.751589275379402, + "after_tpot_ape_pct": 10.751589275379402 + }, + { + "index": 172, + "config_id": 738, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "51e5d8f0791954d7b263ba189d83364957cc50bb753634c7e6071b02e5b795bf", + "completed": 640, + "silicon_ttft_ms": 1033.6237568008073, + "before_ttft_ms": 493.00091102812513, + "after_ttft_ms": 493.00091102812513, + "before_ttft_ape_pct": 52.30363971567144, + "after_ttft_ape_pct": 52.30363971567144, + "silicon_tpot_ms": 20.83387058278117, + "before_tpot_ms": 24.268112648760045, + "after_tpot_ms": 24.268112648760045, + "before_tpot_ape_pct": 16.483936829372524, + "after_tpot_ape_pct": 16.483936829372524 + }, + { + "index": 173, + "config_id": 738, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "26ece7f25ad17c294fd06b922c2b0b7c25bc86b7ad0036397cae7b04d64d88f4", + "completed": 80, + "silicon_ttft_ms": 333.3351041888818, + "before_ttft_ms": 160.12267846249998, + "after_ttft_ms": 160.12267846249998, + "before_ttft_ape_pct": 51.96345165861449, + "after_ttft_ape_pct": 51.96345165861449, + "silicon_tpot_ms": 7.948760361419675, + "before_tpot_ms": 8.743882777299577, + "after_tpot_ms": 8.743882777299577, + "before_tpot_ape_pct": 10.00309959951906, + "after_tpot_ape_pct": 10.00309959951906 + }, + { + "index": 174, + "config_id": 738, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "e29b3350bed000880dc459ba25a4450b888741655e2452d3d42fb948149df7da", + "completed": 160, + "silicon_ttft_ms": 265.1150878180488, + "before_ttft_ms": 44.377150893749906, + "after_ttft_ms": 44.377150893749906, + "before_ttft_ape_pct": 83.26117488861803, + "after_ttft_ape_pct": 83.26117488861803, + "silicon_tpot_ms": 9.506861863720651, + "before_tpot_ms": 9.040215661636129, + "after_tpot_ms": 9.040215661636129, + "before_tpot_ape_pct": 4.90851985412033, + "after_tpot_ape_pct": 4.90851985412033 + }, + { + "index": 175, + "config_id": 738, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "8b2ff83ea6c2d13907c9728ba4f8f1225e482e83fbc6607dc9beaa5662cfc6a3", + "completed": 40, + "silicon_ttft_ms": 248.42590034822933, + "before_ttft_ms": 134.53784954999986, + "after_ttft_ms": 134.53784954999986, + "before_ttft_ape_pct": 45.84387160863166, + "after_ttft_ape_pct": 45.84387160863166, + "silicon_tpot_ms": 6.069020404617435, + "before_tpot_ms": 6.75925820968511, + "after_tpot_ms": 6.75925820968511, + "before_tpot_ape_pct": 11.373133702805283, + "after_tpot_ape_pct": 11.373133702805283 + }, + { + "index": 176, + "config_id": 738, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "2abe44fcc8b1f95458754bb4ea915a61cd4ee5b0dab250f7735af893825183eb", + "completed": 640, + "silicon_ttft_ms": 422.2142068501853, + "before_ttft_ms": 75.72639115312502, + "after_ttft_ms": 75.72639115312502, + "before_ttft_ape_pct": 82.06446161107148, + "after_ttft_ape_pct": 82.06446161107148, + "silicon_tpot_ms": 16.155771932278498, + "before_tpot_ms": 16.801576399671504, + "after_tpot_ms": 16.801576399671504, + "before_tpot_ape_pct": 3.997360634329805, + "after_tpot_ape_pct": 3.997360634329805 + }, + { + "index": 177, + "config_id": 738, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "12fd053e94672fa15645fe5941a13a954500f07cf7d200f642bb4d3c4f6c045a", + "completed": 320, + "silicon_ttft_ms": 328.2613737241263, + "before_ttft_ms": 55.014537556250005, + "after_ttft_ms": 55.014537556250005, + "before_ttft_ape_pct": 83.24063019291307, + "after_ttft_ape_pct": 83.24063019291307, + "silicon_tpot_ms": 12.338514766011377, + "before_tpot_ms": 11.501607643778168, + "after_tpot_ms": 11.501607643778168, + "before_tpot_ape_pct": 6.782883824385555, + "after_tpot_ape_pct": 6.782883824385555 + }, + { + "index": 178, + "config_id": 738, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "32e70cfcf3c9d9ee044b46987efe4689adb6f19efa9207e1ca5333a91bc106ad", + "completed": 80, + "silicon_ttft_ms": 154.63806286206818, + "before_ttft_ms": 39.579730749999975, + "after_ttft_ms": 39.579730749999975, + "before_ttft_ape_pct": 74.40492333035513, + "after_ttft_ape_pct": 74.40492333035513, + "silicon_tpot_ms": 7.456380759089195, + "before_tpot_ms": 7.2032021939405215, + "after_tpot_ms": 7.2032021939405215, + "before_tpot_ape_pct": 3.395461864525273, + "after_tpot_ape_pct": 3.395461864525273 + }, + { + "index": 179, + "config_id": 739, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "9cbc4387c3bb7a54b2b17550816a15ab1c3dac82542af189ede199fefb04abc3", + "completed": 40, + "silicon_ttft_ms": 162.54768197541125, + "before_ttft_ms": 48.571783125000266, + "after_ttft_ms": 48.571783125000266, + "before_ttft_ape_pct": 70.11843999574982, + "after_ttft_ape_pct": 70.11843999574982, + "silicon_tpot_ms": 6.985425321284584, + "before_tpot_ms": 7.301125084873921, + "after_tpot_ms": 7.301125084873921, + "before_tpot_ape_pct": 4.519406465163978, + "after_tpot_ape_pct": 4.519406465163978 + }, + { + "index": 180, + "config_id": 739, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "96fb338b8f86b8d04f1db6803a203927f1bc6adb6c49bd42c226d94375b06b96", + "completed": 160, + "silicon_ttft_ms": 304.7986523626605, + "before_ttft_ms": 59.300403781250374, + "after_ttft_ms": 59.300403781250374, + "before_ttft_ape_pct": 80.54440092776638, + "after_ttft_ape_pct": 80.54440092776638, + "silicon_tpot_ms": 12.210090825392507, + "before_tpot_ms": 12.780969169584408, + "after_tpot_ms": 12.780969169584408, + "before_tpot_ape_pct": 4.675463535493796, + "after_tpot_ape_pct": 4.675463535493796 + }, + { + "index": 181, + "config_id": 739, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "9e93f52289a1b866ba9ec6c56f7a2468f5cb48817242db15ced9d60f1b3394f1", + "completed": 80, + "silicon_ttft_ms": 163.97242512612138, + "before_ttft_ms": 53.33374758749987, + "after_ttft_ms": 53.33374758749987, + "before_ttft_ape_pct": 67.47395329033064, + "after_ttft_ape_pct": 67.47395329033064, + "silicon_tpot_ms": 8.882484003542555, + "before_tpot_ms": 9.532396111451119, + "after_tpot_ms": 9.532396111451119, + "before_tpot_ape_pct": 7.316783319276032, + "after_tpot_ape_pct": 7.316783319276032 + }, + { + "index": 182, + "config_id": 739, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "45f9e191d9ffc070ca01baf44898b06a0464270312ce5e9bfbe333b309881a65", + "completed": 320, + "silicon_ttft_ms": 863.1632967870246, + "before_ttft_ms": 382.77018075624994, + "after_ttft_ms": 382.77018075624994, + "before_ttft_ape_pct": 55.65495171295565, + "after_ttft_ape_pct": 55.65495171295565, + "silicon_tpot_ms": 19.51718697838765, + "before_tpot_ms": 22.118572312308512, + "after_tpot_ms": 22.118572312308512, + "before_tpot_ape_pct": 13.328689922382297, + "after_tpot_ape_pct": 13.328689922382297 + }, + { + "index": 183, + "config_id": 739, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "dcd57514e6b97eaaa1f9259af85150822ee8c3bb694a4afe525c18c6ee10cd68", + "completed": 40, + "silicon_ttft_ms": 280.09356922702864, + "before_ttft_ms": 169.97114804999987, + "after_ttft_ms": 169.97114804999987, + "before_ttft_ape_pct": 39.31629757903136, + "after_ttft_ape_pct": 39.31629757903136, + "silicon_tpot_ms": 7.25970906460373, + "before_tpot_ms": 8.516484108117846, + "after_tpot_ms": 8.516484108117846, + "before_tpot_ape_pct": 17.311644755046608, + "after_tpot_ape_pct": 17.311644755046608 + }, + { + "index": 184, + "config_id": 739, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "d9c31adca4aee7cbe29ed9a35a937145890599a9da81099cccef6a2a2fbd410c", + "completed": 160, + "silicon_ttft_ms": 576.0338567433791, + "before_ttft_ms": 258.8332145875001, + "after_ttft_ms": 258.8332145875001, + "before_ttft_ape_pct": 55.06631918290017, + "after_ttft_ape_pct": 55.06631918290017, + "silicon_tpot_ms": 13.13432572622673, + "before_tpot_ms": 16.06549607319735, + "after_tpot_ms": 16.06549607319735, + "before_tpot_ape_pct": 22.316869613775726, + "after_tpot_ape_pct": 22.316869613775726 + }, + { + "index": 185, + "config_id": 739, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "a305c534fa33c2e22c4e7b83dedbea198182a1dfb33a379f3b943a0151ef740c", + "completed": 640, + "silicon_ttft_ms": 548.0291704319825, + "before_ttft_ms": 98.4225114781249, + "after_ttft_ms": 98.4225114781249, + "before_ttft_ape_pct": 82.04064367585696, + "after_ttft_ape_pct": 82.04064367585696, + "silicon_tpot_ms": 22.565226415210848, + "before_tpot_ms": 25.904399016004998, + "after_tpot_ms": 25.904399016004998, + "before_tpot_ape_pct": 14.797868806418313, + "after_tpot_ape_pct": 14.797868806418313 + }, + { + "index": 186, + "config_id": 739, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "52ff1ee850898a3419845a55c7c661f063d20bc3dfb0551891c05a551af37f23", + "completed": 320, + "silicon_ttft_ms": 436.85279777510004, + "before_ttft_ms": 72.57593102499996, + "after_ttft_ms": 72.57593102499996, + "before_ttft_ape_pct": 83.38663929940918, + "after_ttft_ape_pct": 83.38663929940918, + "silicon_tpot_ms": 16.93736417438413, + "before_tpot_ms": 16.944865055759607, + "after_tpot_ms": 16.944865055759607, + "before_tpot_ape_pct": 0.04428600163668417, + "after_tpot_ape_pct": 0.04428600163668417 + }, + { + "index": 187, + "config_id": 739, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "4a6cc38a6d8cf669bb7cc2a74742e481591195b78d73b00df4af7fa90454ada0", + "completed": 640, + "silicon_ttft_ms": 1288.7369485906675, + "before_ttft_ms": 622.7669709093749, + "after_ttft_ms": 622.7669709093749, + "before_ttft_ape_pct": 51.676176306544306, + "after_ttft_ape_pct": 51.676176306544306, + "silicon_tpot_ms": 28.473599584392808, + "before_tpot_ms": 34.64347413883447, + "after_tpot_ms": 34.64347413883447, + "before_tpot_ape_pct": 21.668755073115342, + "after_tpot_ape_pct": 21.668755073115342 + }, + { + "index": 188, + "config_id": 739, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "2c65de8a5a7a5b1004f7eb43789dc7946fd7b2ee6c13816aa17e009ed2d9b01e", + "completed": 80, + "silicon_ttft_ms": 346.05187023989856, + "before_ttft_ms": 202.26608882499949, + "after_ttft_ms": 202.26608882499949, + "before_ttft_ape_pct": 41.550355244495684, + "after_ttft_ape_pct": 41.550355244495684, + "silicon_tpot_ms": 9.462874820323622, + "before_tpot_ms": 11.23977913374598, + "after_tpot_ms": 11.23977913374598, + "before_tpot_ape_pct": 18.777637315946127, + "after_tpot_ape_pct": 18.777637315946127 + }, + { + "index": 189, + "config_id": 1156, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "h100", + "isl": 1024, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "13d52b8a03de792b1f5acec470eb3fae7fb1f9c4f6ac4c9d208316ec180a3d85", + "completed": 320, + "silicon_ttft_ms": 3069.391084340168, + "before_ttft_ms": 83.48198748124989, + "after_ttft_ms": 83.48198748124989, + "before_ttft_ape_pct": 97.28017756006494, + "after_ttft_ape_pct": 97.28017756006494, + "silicon_tpot_ms": 23.704015480788453, + "before_tpot_ms": 15.44550077887547, + "after_tpot_ms": 15.44550077887547, + "before_tpot_ape_pct": 34.8401506428576, + "after_tpot_ape_pct": 34.8401506428576 + }, + { + "index": 190, + "config_id": 1156, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "h100", + "isl": 1024, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "a057f17c381329275781114f7bc15a8fee091bb9813f55e3b93d850bfe21a9b8", + "completed": 160, + "silicon_ttft_ms": 631.0725222690962, + "before_ttft_ms": 64.59932298750019, + "after_ttft_ms": 64.59932298750019, + "before_ttft_ape_pct": 89.76356588063355, + "after_ttft_ape_pct": 89.76356588063355, + "silicon_tpot_ms": 14.500639421925943, + "before_tpot_ms": 12.49393253385125, + "after_tpot_ms": 12.49393253385125, + "before_tpot_ape_pct": 13.83874758681618, + "after_tpot_ape_pct": 13.83874758681618 + }, + { + "index": 191, + "config_id": 1156, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "h100", + "isl": 1024, + "osl": 1024, + "concurrency": 128, + "spec_sha256": "a327cde93defe72259a9a7ce2044d08bfdbf987a27872e81c6dde11cb729c774", + "completed": 1280, + "silicon_ttft_ms": 3354.2400001471833, + "before_ttft_ms": 191.829944835157, + "after_ttft_ms": 191.829944835157, + "before_ttft_ape_pct": 94.28097140256095, + "after_ttft_ape_pct": 94.28097140256095, + "silicon_tpot_ms": 25.27900800601268, + "before_tpot_ms": 29.1634293633579, + "after_tpot_ms": 29.1634293633579, + "before_tpot_ape_pct": 15.366193785853067, + "after_tpot_ape_pct": 15.366193785853067 + }, + { + "index": 192, + "config_id": 1156, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "h100", + "isl": 1024, + "osl": 1024, + "concurrency": 256, + "spec_sha256": "719a20920fbfac6136f8065c7a0e6dd0d9126a3e9ba88e5bae06136a0286846f", + "completed": 2560, + "silicon_ttft_ms": 11099.651461491885, + "before_ttft_ms": 335.92769619179614, + "after_ttft_ms": 335.92769619179614, + "before_ttft_ape_pct": 96.97352932786013, + "after_ttft_ape_pct": 96.97352932786013, + "silicon_tpot_ms": 34.270860079777684, + "before_tpot_ms": 39.408468308050274, + "after_tpot_ms": 39.408468308050274, + "before_tpot_ape_pct": 14.991185562057584, + "after_tpot_ape_pct": 14.991185562057584 + }, + { + "index": 193, + "config_id": 1156, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "h100", + "isl": 1024, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "ce136c870181623c0424564fdc16b45cafbcc1d208a6a5984afd73af20c19b7f", + "completed": 640, + "silicon_ttft_ms": 2225.653535636957, + "before_ttft_ms": 120.07275689843725, + "after_ttft_ms": 120.07275689843725, + "before_ttft_ape_pct": 94.60505622389813, + "after_ttft_ape_pct": 94.60505622389813, + "silicon_tpot_ms": 18.70754199081889, + "before_tpot_ms": 21.745710099101707, + "after_tpot_ms": 21.745710099101707, + "before_tpot_ape_pct": 16.240338307265922, + "after_tpot_ape_pct": 16.240338307265922 + }, + { + "index": 194, + "config_id": 1359, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "h100", + "isl": 1024, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "a18bb72364d2b8f52e6b6e7a88c2a7ce61f79859236f0c3f8cc89bda3ac40a5f", + "completed": 40, + "silicon_ttft_ms": 337.74865095037967, + "before_ttft_ms": 47.738804150000014, + "after_ttft_ms": 47.738804150000014, + "before_ttft_ape_pct": 85.86558258170109, + "after_ttft_ape_pct": 85.86558258170109, + "silicon_tpot_ms": 8.066625887848035, + "before_tpot_ms": 6.708544981454681, + "after_tpot_ms": 6.708544981454681, + "before_tpot_ape_pct": 16.835798824378788, + "after_tpot_ape_pct": 16.835798824378788 + }, + { + "index": 195, + "config_id": 1359, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "h100", + "isl": 8192, + "osl": 1024, + "concurrency": 2, + "spec_sha256": "65be6d5ace60c593a46da2a3a27711ae8c9b6be1398617d5523d0dcb0f6d6a39", + "completed": 20, + "silicon_ttft_ms": 468.596677982714, + "before_ttft_ms": 182.30903275000017, + "after_ttft_ms": 182.30903275000017, + "before_ttft_ape_pct": 61.09468092372492, + "after_ttft_ape_pct": 61.09468092372492, + "silicon_tpot_ms": 6.699531188578497, + "before_tpot_ms": 5.790343494890635, + "after_tpot_ms": 5.790343494890635, + "before_tpot_ape_pct": 13.570915159524366, + "after_tpot_ape_pct": 13.570915159524366 + }, + { + "index": 196, + "config_id": 1359, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "h100", + "isl": 1024, + "osl": 1024, + "concurrency": 2, + "spec_sha256": "1d2d2f974fea90a2f31c290fb6a73b4f267d8985d310ad3887a0c80e7a6e2691", + "completed": 20, + "silicon_ttft_ms": 352.14359178207815, + "before_ttft_ms": 44.12571674999988, + "after_ttft_ms": 44.12571674999988, + "before_ttft_ape_pct": 87.4693966382592, + "after_ttft_ape_pct": 87.4693966382592, + "silicon_tpot_ms": 6.645712035130703, + "before_tpot_ms": 5.61753601088552, + "after_tpot_ms": 5.61753601088552, + "before_tpot_ape_pct": 15.471269576683689, + "after_tpot_ape_pct": 15.471269576683689 + }, + { + "index": 197, + "config_id": 1359, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "h100", + "isl": 8192, + "osl": 1024, + "concurrency": 1, + "spec_sha256": "33340eada4a04c6346df57cab435929a67994f8d0c8f1a3ed49bbc5cda96c302", + "completed": 10, + "silicon_ttft_ms": 568.7953190878034, + "before_ttft_ms": 175.23694879999965, + "after_ttft_ms": 175.23694879999965, + "before_ttft_ape_pct": 69.19156277849945, + "after_ttft_ape_pct": 69.19156277849945, + "silicon_tpot_ms": 6.534701965251364, + "before_tpot_ms": 5.254564569423472, + "after_tpot_ms": 5.254564569423472, + "before_tpot_ape_pct": 19.589835965512936, + "after_tpot_ape_pct": 19.589835965512936 + }, + { + "index": 198, + "config_id": 1359, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "h100", + "isl": 1024, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "7e0cfe23036752bf1d9c2d070227e22add9e972197788ff941c0abd5fc80468a", + "completed": 80, + "silicon_ttft_ms": 556.8333287010319, + "before_ttft_ms": 55.10213544999995, + "after_ttft_ms": 55.10213544999995, + "before_ttft_ape_pct": 90.10437547290121, + "after_ttft_ape_pct": 90.10437547290121, + "silicon_tpot_ms": 9.974554717622574, + "before_tpot_ms": 8.402878545161723, + "after_tpot_ms": 8.402878545161723, + "before_tpot_ape_pct": 15.756855488336608, + "after_tpot_ape_pct": 15.756855488336608 + }, + { + "index": 199, + "config_id": 1359, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "h100", + "isl": 8192, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "bbb00d8f113db0b7b1965797f664a0cbd76175d2de78340ab5f9a3b10517f615", + "completed": 40, + "silicon_ttft_ms": 442.4172525992617, + "before_ttft_ms": 201.22849380000005, + "after_ttft_ms": 201.22849380000005, + "before_ttft_ape_pct": 54.51612869576057, + "after_ttft_ape_pct": 54.51612869576057, + "silicon_tpot_ms": 8.235025912457811, + "before_tpot_ms": 7.1821904717672975, + "after_tpot_ms": 7.1821904717672975, + "before_tpot_ape_pct": 12.784846725227684, + "after_tpot_ape_pct": 12.784846725227684 + }, + { + "index": 200, + "config_id": 1359, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "h100", + "isl": 1024, + "osl": 1024, + "concurrency": 1, + "spec_sha256": "c09a808744a9e75a2b582db27f7b4086c9394e36f1abfc0e1609db2236de9d50", + "completed": 10, + "silicon_ttft_ms": 350.72518726810813, + "before_ttft_ms": 42.28632820000044, + "after_ttft_ms": 42.28632820000044, + "before_ttft_ape_pct": 87.9431732492953, + "after_ttft_ape_pct": 87.9431732492953, + "silicon_tpot_ms": 6.350312597687807, + "before_tpot_ms": 5.225528323902533, + "after_tpot_ms": 5.225528323902533, + "before_tpot_ape_pct": 17.712266230717777, + "after_tpot_ape_pct": 17.712266230717777 + }, + { + "index": 201, + "config_id": 1359, + "model": "Qwen-3.5-397B-A17B", + "backend": "sglang", + "hardware": "h100", + "isl": 8192, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "c7fadbbebf18bd2906e788df828c76025a7e1b92b765388c225740d86c6efb5a", + "completed": 80, + "silicon_ttft_ms": 794.8539138829801, + "before_ttft_ms": 239.56702009999998, + "after_ttft_ms": 239.56702009999998, + "before_ttft_ape_pct": 69.86024527077193, + "after_ttft_ape_pct": 69.86024527077193, + "silicon_tpot_ms": 10.596815401166522, + "before_tpot_ms": 9.505707232687692, + "after_tpot_ms": 9.505707232687692, + "before_tpot_ape_pct": 10.29656672474183, + "after_tpot_ape_pct": 10.29656672474183 + }, + { + "index": 202, + "config_id": 1385, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "a9cbbab50544ebcbf8f3eb63eb78454657f984ac36665e9351ffa89938eda2bd", + "completed": 40, + "silicon_ttft_ms": 112.74425442097709, + "before_ttft_ms": 92.4347140750003, + "after_ttft_ms": 92.4347140750003, + "before_ttft_ape_pct": 18.013814052237866, + "after_ttft_ape_pct": 18.013814052237866, + "silicon_tpot_ms": 5.9060526543555465, + "before_tpot_ms": 7.488256396583752, + "after_tpot_ms": 7.488256396583752, + "before_tpot_ape_pct": 26.789529908125264, + "after_tpot_ape_pct": 26.789529908125264 + }, + { + "index": 203, + "config_id": 1385, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "54fba00f95042725d9779d4cda2f2d0166ad80ce1db23e565f7fdb80dae7bd34", + "completed": 40, + "silicon_ttft_ms": 41.96820032375399, + "before_ttft_ms": 29.988081775000104, + "after_ttft_ms": 29.988081775000104, + "before_ttft_ape_pct": 28.545704739150178, + "after_ttft_ape_pct": 28.545704739150178, + "silicon_tpot_ms": 5.504830022083077, + "before_tpot_ms": 7.217820378399102, + "after_tpot_ms": 7.217820378399102, + "before_tpot_ape_pct": 31.11795186126045, + "after_tpot_ape_pct": 31.11795186126045 + }, + { + "index": 204, + "config_id": 1386, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 256, + "spec_sha256": "d1f2b0acd03204de5724094126abcb72941739f124e99d391e0efcaa90a02105", + "completed": 2560, + "silicon_ttft_ms": 435.79243385697737, + "before_ttft_ms": 118.58931899218757, + "after_ttft_ms": 118.58931899218757, + "before_ttft_ape_pct": 72.78765995485195, + "after_ttft_ape_pct": 72.78765995485195, + "silicon_tpot_ms": 12.881860738913735, + "before_tpot_ms": 21.111673840930692, + "after_tpot_ms": 21.111673840930692, + "before_tpot_ape_pct": 63.88683489766507, + "after_tpot_ape_pct": 63.88683489766507 + }, + { + "index": 205, + "config_id": 1386, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 128, + "spec_sha256": "d3344ffbb0b66a59f8b049084cecce260ca113cff9a406c718e5d694b48b06fb", + "completed": 1280, + "silicon_ttft_ms": 361.5973926141123, + "before_ttft_ms": 94.71827286875023, + "after_ttft_ms": 94.71827286875023, + "before_ttft_ape_pct": 73.80559849063093, + "after_ttft_ape_pct": 73.80559849063093, + "silicon_tpot_ms": 10.607821262338552, + "before_tpot_ms": 16.510887417601236, + "after_tpot_ms": 16.510887417601236, + "before_tpot_ape_pct": 55.64824302065323, + "after_tpot_ape_pct": 55.64824302065323 + }, + { + "index": 206, + "config_id": 1386, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 1024, + "spec_sha256": "6b062ef4a0c91b5a303fb784a1f3ab7b99cc2a00faaf23b38c5fb6718256e6f3", + "completed": 10240, + "silicon_ttft_ms": 611.6887295196989, + "before_ttft_ms": 259.90062060957246, + "after_ttft_ms": 259.90062060957246, + "before_ttft_ape_pct": 57.51096790459948, + "after_ttft_ape_pct": 57.51096790459948, + "silicon_tpot_ms": 19.852621297494675, + "before_tpot_ms": 37.588164622623665, + "after_tpot_ms": 37.588164622623665, + "before_tpot_ape_pct": 89.33602802047682, + "after_tpot_ape_pct": 89.33602802047682 + }, + { + "index": 207, + "config_id": 1386, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 512, + "spec_sha256": "3c78cd9ba3dcf07d415a7d2cb609844e696743b85f26d6e79e9801b7de39fd4c", + "completed": 5120, + "silicon_ttft_ms": 506.2841166145973, + "before_ttft_ms": 166.00014835839676, + "after_ttft_ms": 166.00014835839676, + "before_ttft_ape_pct": 67.21205684499829, + "after_ttft_ape_pct": 67.21205684499829, + "silicon_tpot_ms": 15.490438045077239, + "before_tpot_ms": 28.746898789319914, + "after_tpot_ms": 28.746898789319914, + "before_tpot_ape_pct": 85.57834649779639, + "after_tpot_ape_pct": 85.57834649779639 + }, + { + "index": 208, + "config_id": 1387, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 512, + "spec_sha256": "352c9cc04fcb2e91f33248c538e6c28e9c2763f554106516c988dd9d29b8b647", + "completed": 5120, + "silicon_ttft_ms": 33919.30846407151, + "before_ttft_ms": 38170.31471226277, + "after_ttft_ms": 38170.31471226277, + "before_ttft_ape_pct": 12.532703173161885, + "after_ttft_ape_pct": 12.532703173161885, + "silicon_tpot_ms": 66.19986704429219, + "before_tpot_ms": 94.46873452987272, + "after_tpot_ms": 94.46873452987272, + "before_tpot_ape_pct": 42.70230250865421, + "after_tpot_ape_pct": 42.70230250865421 + }, + { + "index": 209, + "config_id": 1387, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 256, + "spec_sha256": "f1ed71841995b2fce4afbba6c81bc46c4db2eada4d8f0492a9cef57a87eeb669", + "completed": 2560, + "silicon_ttft_ms": 2991.1311309953817, + "before_ttft_ms": 1135.6410295347666, + "after_ttft_ms": 1135.6410295347666, + "before_ttft_ape_pct": 62.033057736360405, + "after_ttft_ape_pct": 62.033057736360405, + "silicon_tpot_ms": 52.04954308448055, + "before_tpot_ms": 70.5781504367084, + "after_tpot_ms": 70.5781504367084, + "before_tpot_ape_pct": 35.59802114334501, + "after_tpot_ape_pct": 35.59802114334501 + }, + { + "index": 210, + "config_id": 1387, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 2048, + "spec_sha256": "e5f9997225a791bfd7d8a3eec864078458358189586d1077e22805d71bbf979c", + "completed": 20480, + "silicon_ttft_ms": 5668.074341080876, + "before_ttft_ms": 21353.951989783534, + "after_ttft_ms": 21353.951989783534, + "before_ttft_ape_pct": 276.7408595017022, + "after_ttft_ape_pct": 276.7408595017022, + "silicon_tpot_ms": 68.85524894261722, + "before_tpot_ms": 82.25942844378373, + "after_tpot_ms": 82.25942844378373, + "before_tpot_ape_pct": 19.4671861724548, + "after_tpot_ape_pct": 19.4671861724548 + }, + { + "index": 211, + "config_id": 1387, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 128, + "spec_sha256": "7716d7b4e99f0ad7399ca4ae32b507e1ceeeabca32739b61dc4a31797d0d1659", + "completed": 1280, + "silicon_ttft_ms": 1679.5922134642751, + "before_ttft_ms": 646.3421384078122, + "after_ttft_ms": 646.3421384078122, + "before_ttft_ape_pct": 61.517912906092434, + "after_ttft_ape_pct": 61.517912906092434, + "silicon_tpot_ms": 32.21405237839576, + "before_tpot_ms": 44.93655876352357, + "after_tpot_ms": 44.93655876352357, + "before_tpot_ape_pct": 39.49365399821636, + "after_tpot_ape_pct": 39.49365399821636 + }, + { + "index": 212, + "config_id": 1388, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "8968a9264096d5a0da204a4ac082cfeeab286c357bea32f62afd035edf19a64d", + "completed": 80, + "silicon_ttft_ms": 59.756664669839665, + "before_ttft_ms": 62.44154770000025, + "after_ttft_ms": 62.44154770000025, + "before_ttft_ape_pct": 4.493026920084597, + "after_ttft_ape_pct": 4.493026920084597, + "silicon_tpot_ms": 8.994528065062891, + "before_tpot_ms": 12.9001914806603, + "after_tpot_ms": 12.9001914806603, + "before_tpot_ape_pct": 43.42266083718202, + "after_tpot_ape_pct": 43.42266083718202 + }, + { + "index": 213, + "config_id": 1388, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "ded958fffcd740774cd5ae0316262e7258bb44ef668e2f27bc7969300e796bc0", + "completed": 40, + "silicon_ttft_ms": 189.28565162059385, + "before_ttft_ms": 187.0868942500003, + "after_ttft_ms": 187.0868942500003, + "before_ttft_ape_pct": 1.1616080520465144, + "after_ttft_ape_pct": 1.1616080520465144, + "silicon_tpot_ms": 8.10721196439507, + "before_tpot_ms": 11.166373495155655, + "after_tpot_ms": 11.166373495155655, + "before_tpot_ape_pct": 37.73382938790411, + "after_tpot_ape_pct": 37.73382938790411 + }, + { + "index": 214, + "config_id": 1388, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "eb5321cbd5a587c3cb7ceb2811338c47e21f385703ae03c6c0eb7697b67e8aae", + "completed": 40, + "silicon_ttft_ms": 50.79417737724725, + "before_ttft_ms": 54.78348907500058, + "after_ttft_ms": 54.78348907500058, + "before_ttft_ape_pct": 7.853875982919845, + "after_ttft_ape_pct": 7.853875982919845, + "silicon_tpot_ms": 7.277527824909415, + "before_tpot_ms": 10.701222367164785, + "after_tpot_ms": 10.701222367164785, + "before_tpot_ape_pct": 47.044746851215045, + "after_tpot_ape_pct": 47.044746851215045 + }, + { + "index": 215, + "config_id": 1388, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "f89e772582ca276d849c49782a921055043d3d2bc2cfc9b418418a5cd3ffabcf", + "completed": 80, + "silicon_ttft_ms": 233.8298771515838, + "before_ttft_ms": 221.66075866249975, + "after_ttft_ms": 221.66075866249975, + "before_ttft_ape_pct": 5.204261592796899, + "after_ttft_ape_pct": 5.204261592796899, + "silicon_tpot_ms": 10.727939066946444, + "before_tpot_ms": 14.757718913150631, + "after_tpot_ms": 14.757718913150631, + "before_tpot_ape_pct": 37.56341102477203, + "after_tpot_ape_pct": 37.56341102477203 + }, + { + "index": 216, + "config_id": 1388, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "a8b57c69b143f992c35d2a34c0ed4a511ed0b57c8ef838f55c3a15cfe779a7ba", + "completed": 160, + "silicon_ttft_ms": 304.1036628826987, + "before_ttft_ms": 284.50151406249995, + "after_ttft_ms": 284.50151406249995, + "before_ttft_ape_pct": 6.445877249350939, + "after_ttft_ape_pct": 6.445877249350939, + "silicon_tpot_ms": 14.988792869202683, + "before_tpot_ms": 19.589088780389236, + "after_tpot_ms": 19.589088780389236, + "before_tpot_ape_pct": 30.69157037081174, + "after_tpot_ape_pct": 30.69157037081174 + }, + { + "index": 217, + "config_id": 1389, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 2048, + "spec_sha256": "438b271355983f3a64889a6ac82f99eb7bf28577f07583dde31ce508b132a55d", + "completed": 20480, + "silicon_ttft_ms": 1400.476266373272, + "before_ttft_ms": 624.5747223606917, + "after_ttft_ms": 624.5747223606917, + "before_ttft_ape_pct": 55.40269140168188, + "after_ttft_ape_pct": 55.40269140168188, + "silicon_tpot_ms": 43.509074538329685, + "before_tpot_ms": 62.3365257315772, + "after_tpot_ms": 62.3365257315772, + "before_tpot_ape_pct": 43.27246992270845, + "after_tpot_ape_pct": 43.27246992270845 + }, + { + "index": 218, + "config_id": 1389, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 128, + "spec_sha256": "7d7e657a5a60763ee96ccf0ba60a7904e529d928812f4662cbe6513080e10f56", + "completed": 1280, + "silicon_ttft_ms": 1300.8231926745793, + "before_ttft_ms": 491.2053977617197, + "after_ttft_ms": 491.2053977617197, + "before_ttft_ape_pct": 62.23888069278895, + "after_ttft_ape_pct": 62.23888069278895, + "silicon_tpot_ms": 21.466741961306102, + "before_tpot_ms": 37.57837146832593, + "after_tpot_ms": 37.57837146832593, + "before_tpot_ape_pct": 75.05391146947737, + "after_tpot_ape_pct": 75.05391146947737 + }, + { + "index": 219, + "config_id": 1389, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 1024, + "spec_sha256": "b84610f24f9e9fe03e0af56e8da21b888c8efa270829aeb3b5b87c84b64116c5", + "completed": 10240, + "silicon_ttft_ms": 808.4051245516889, + "before_ttft_ms": 338.08190415878744, + "after_ttft_ms": 338.08190415878744, + "before_ttft_ape_pct": 58.17914880904856, + "after_ttft_ape_pct": 58.17914880904856, + "silicon_tpot_ms": 28.157943082799758, + "before_tpot_ms": 42.22771122335592, + "after_tpot_ms": 42.22771122335592, + "before_tpot_ape_pct": 49.96731508115968, + "after_tpot_ape_pct": 49.96731508115968 + }, + { + "index": 220, + "config_id": 1390, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "f2bf9321106df3d8040dc8b2d85975642c6fbd21044b1c375b8399eaae8ca986", + "completed": 40, + "silicon_ttft_ms": 43.77929190522991, + "before_ttft_ms": 31.89533380000031, + "after_ttft_ms": 31.89533380000031, + "before_ttft_ape_pct": 27.14515833411626, + "after_ttft_ape_pct": 27.14515833411626, + "silicon_tpot_ms": 6.022791172023818, + "before_tpot_ms": 7.57100657689323, + "after_tpot_ms": 7.57100657689323, + "before_tpot_ape_pct": 25.705945310887635, + "after_tpot_ape_pct": 25.705945310887635 + }, + { + "index": 221, + "config_id": 1390, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b300", + "isl": 1024, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "9a403bad6c628e747cf60fc7d2b5e6fea8382ad9ae93fb47a1a0d4f01083c784", + "completed": 80, + "silicon_ttft_ms": 52.59004572289996, + "before_ttft_ms": 35.778396812499835, + "after_ttft_ms": 35.778396812499835, + "before_ttft_ape_pct": 31.967359372497395, + "after_ttft_ape_pct": 31.967359372497395, + "silicon_tpot_ms": 6.77803308228198, + "before_tpot_ms": 8.430681412406276, + "after_tpot_ms": 8.430681412406276, + "before_tpot_ape_pct": 24.382417584304484, + "after_tpot_ape_pct": 24.382417584304484 + }, + { + "index": 222, + "config_id": 1390, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b300", + "isl": 8192, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "29b26fb385aa96cd65121e682853da128d2fec8e27f016cd5f3759944e08f83b", + "completed": 40, + "silicon_ttft_ms": 139.5004576748761, + "before_ttft_ms": 99.70132817500021, + "after_ttft_ms": 99.70132817500021, + "before_ttft_ape_pct": 28.529748334326555, + "after_ttft_ape_pct": 28.529748334326555, + "silicon_tpot_ms": 6.618510167623879, + "before_tpot_ms": 7.95813103371321, + "after_tpot_ms": 7.95813103371321, + "before_tpot_ape_pct": 20.24051987775779, + "after_tpot_ape_pct": 20.24051987775779 + }, + { + "index": 223, + "config_id": 1444, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "a72fe8145a6d0ff17606f21b707983d2ab203a087e61d14cbe9074c8f312c2fa", + "completed": 40, + "silicon_ttft_ms": 49.765678624999055, + "before_ttft_ms": 36.36577147499994, + "after_ttft_ms": 36.36577147499994, + "before_ttft_ape_pct": 26.92600105179289, + "after_ttft_ape_pct": 26.92600105179289, + "silicon_tpot_ms": 6.28018352773671, + "before_tpot_ms": 5.988918165739604, + "after_tpot_ms": 5.988918165739604, + "before_tpot_ape_pct": 4.637847934072625, + "after_tpot_ape_pct": 4.637847934072625 + }, + { + "index": 224, + "config_id": 1444, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "3b67054caa060014ebaca134e73e284ad59d0397c064e6a5fd92a6a087a44cea", + "completed": 80, + "silicon_ttft_ms": 60.48430263745104, + "before_ttft_ms": 40.84885097499976, + "after_ttft_ms": 40.84885097499976, + "before_ttft_ape_pct": 32.463715057025844, + "after_ttft_ape_pct": 32.463715057025844, + "silicon_tpot_ms": 6.88658080152626, + "before_tpot_ms": 6.5518639031996715, + "after_tpot_ms": 6.5518639031996715, + "before_tpot_ape_pct": 4.860422145230703, + "after_tpot_ape_pct": 4.860422145230703 + }, + { + "index": 225, + "config_id": 1445, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "448e44ee4b70edd357b4a175f7da7bc02977491dff4f52deda24a04f33845655", + "completed": 40, + "silicon_ttft_ms": 59.46157420007694, + "before_ttft_ms": 52.937885800000025, + "after_ttft_ms": 52.937885800000025, + "before_ttft_ape_pct": 10.971267558652142, + "after_ttft_ape_pct": 10.971267558652142, + "silicon_tpot_ms": 7.394059896971411, + "before_tpot_ms": 7.853031742774489, + "after_tpot_ms": 7.853031742774489, + "before_tpot_ape_pct": 6.2073049474629105, + "after_tpot_ape_pct": 6.2073049474629105 + }, + { + "index": 226, + "config_id": 1445, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "f4b5a0f0d8cde83a6af7a3981d5581b8bb6b79c396a0ee1f4ba047b07bd9d814", + "completed": 80, + "silicon_ttft_ms": 260.17300938749486, + "before_ttft_ms": 235.41758889999966, + "after_ttft_ms": 235.41758889999966, + "before_ttft_ape_pct": 9.514984104529123, + "after_ttft_ape_pct": 9.514984104529123, + "silicon_tpot_ms": 10.510441286868552, + "before_tpot_ms": 11.384886181954093, + "after_tpot_ms": 11.384886181954093, + "before_tpot_ape_pct": 8.319773368393646, + "after_tpot_ape_pct": 8.319773368393646 + }, + { + "index": 227, + "config_id": 1445, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "7b50913fd01a5fd10559f71d064fbe9ef027b6e00d9103288c5cb5732ca4f3b0", + "completed": 320, + "silicon_ttft_ms": 479.0752512625602, + "before_ttft_ms": 426.27521230937464, + "after_ttft_ms": 426.27521230937464, + "before_ttft_ape_pct": 11.021241196249598, + "after_ttft_ape_pct": 11.021241196249598, + "silicon_tpot_ms": 22.671785865102137, + "before_tpot_ms": 22.351616655972528, + "after_tpot_ms": 22.351616655972528, + "before_tpot_ape_pct": 1.4121922773734141, + "after_tpot_ape_pct": 1.4121922773734141 + }, + { + "index": 228, + "config_id": 1445, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "8264b11c83d91797fd55e0e82831e7d10f30044499c82759df1ab3b11823e1ea", + "completed": 40, + "silicon_ttft_ms": 226.69318177486275, + "before_ttft_ms": 198.2973792000002, + "after_ttft_ms": 198.2973792000002, + "before_ttft_ape_pct": 12.526094676752763, + "after_ttft_ape_pct": 12.526094676752763, + "silicon_tpot_ms": 8.057361785715633, + "before_tpot_ms": 8.468200373132643, + "after_tpot_ms": 8.468200373132643, + "before_tpot_ape_pct": 5.098921934290668, + "after_tpot_ape_pct": 5.098921934290668 + }, + { + "index": 229, + "config_id": 1445, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "b82c4f351fb138cc9dbca60fbac86dc946de8802fe1f3ebd7a84f3d3a6780172", + "completed": 80, + "silicon_ttft_ms": 65.38645565051411, + "before_ttft_ms": 60.16428394999808, + "after_ttft_ms": 60.16428394999808, + "before_ttft_ape_pct": 7.986626050551138, + "after_ttft_ape_pct": 7.986626050551138, + "silicon_tpot_ms": 9.00313122104784, + "before_tpot_ms": 9.387745447026242, + "after_tpot_ms": 9.387745447026242, + "before_tpot_ape_pct": 4.272005111724231, + "after_tpot_ape_pct": 4.272005111724231 + }, + { + "index": 230, + "config_id": 1445, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "526a2ff19bd2d080fbf2a5b06f7611f78c924995f5baf662cf09c22bebb38e40", + "completed": 160, + "silicon_ttft_ms": 344.1529355685816, + "before_ttft_ms": 303.2780263000002, + "after_ttft_ms": 303.2780263000002, + "before_ttft_ape_pct": 11.87696080553583, + "after_ttft_ape_pct": 11.87696080553583, + "silicon_tpot_ms": 14.9627133764062, + "before_tpot_ms": 15.632856144983837, + "after_tpot_ms": 15.632856144983837, + "before_tpot_ape_pct": 4.4787516255864634, + "after_tpot_ape_pct": 4.4787516255864634 + }, + { + "index": 231, + "config_id": 1446, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 128, + "spec_sha256": "688dfc15c82fd21bb1bf60ace5f4e0aae0a77ccd886956d1f823dc700925490f", + "completed": 1280, + "silicon_ttft_ms": 361.35865468440613, + "before_ttft_ms": 94.74963520625016, + "after_ttft_ms": 94.74963520625016, + "before_ttft_ape_pct": 73.77961369459932, + "after_ttft_ape_pct": 73.77961369459932, + "silicon_tpot_ms": 10.622973095858882, + "before_tpot_ms": 15.745321906637196, + "after_tpot_ms": 15.745321906637196, + "before_tpot_ape_pct": 48.21954046720822, + "after_tpot_ape_pct": 48.21954046720822 + }, + { + "index": 232, + "config_id": 1446, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 256, + "spec_sha256": "083531a2f01e5b7cac830ce7e8db7dbeedb5a1999136f4e94f8dc97bb925f33d", + "completed": 2560, + "silicon_ttft_ms": 443.80060761248285, + "before_ttft_ms": 120.50178959882771, + "after_ttft_ms": 120.50178959882771, + "before_ttft_ape_pct": 72.84776371824005, + "after_ttft_ape_pct": 72.84776371824005, + "silicon_tpot_ms": 12.910107070580786, + "before_tpot_ms": 20.82302189892959, + "after_tpot_ms": 20.82302189892959, + "before_tpot_ape_pct": 61.29240280570984, + "after_tpot_ape_pct": 61.29240280570984 + }, + { + "index": 233, + "config_id": 1446, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 512, + "spec_sha256": "3b9b29a36c3b50a844a1e0382028532630e63193eccf963d8536fc26bc3109fc", + "completed": 5120, + "silicon_ttft_ms": 509.6021156130121, + "before_ttft_ms": 169.91046931601707, + "after_ttft_ms": 169.91046931601707, + "before_ttft_ape_pct": 66.65820958933267, + "after_ttft_ape_pct": 66.65820958933267, + "silicon_tpot_ms": 15.693259981788586, + "before_tpot_ms": 27.982313518984217, + "after_tpot_ms": 27.982313518984217, + "before_tpot_ape_pct": 78.3078439499288, + "after_tpot_ape_pct": 78.3078439499288 + }, + { + "index": 234, + "config_id": 1446, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 1024, + "spec_sha256": "c9c5101ef04d364366261ed534d08f869fcca2cd4147febb034f07b1490d6f46", + "completed": 10240, + "silicon_ttft_ms": 692.714326071657, + "before_ttft_ms": 267.75708221669845, + "after_ttft_ms": 267.75708221669845, + "before_ttft_ape_pct": 61.34668042234186, + "after_ttft_ape_pct": 61.34668042234186, + "silicon_tpot_ms": 20.26178602670827, + "before_tpot_ms": 36.17924061775088, + "after_tpot_ms": 36.17924061775088, + "before_tpot_ape_pct": 78.55899065393773, + "after_tpot_ape_pct": 78.55899065393773 + }, + { + "index": 235, + "config_id": 1447, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 1024, + "spec_sha256": "05cd5a21cbec0f04d8e87d2b3956f16e9a455b1154622b325a5523670908b9b9", + "completed": 10240, + "silicon_ttft_ms": 58333.125580179476, + "before_ttft_ms": 125421.97930730034, + "after_ttft_ms": 125421.97930730034, + "before_ttft_ape_pct": 115.00987313787351, + "after_ttft_ape_pct": 115.00987313787351, + "silicon_tpot_ms": 54.03266736235344, + "before_tpot_ms": 130.96025404844409, + "after_tpot_ms": 130.96025404844409, + "before_tpot_ape_pct": 142.37236553620326, + "after_tpot_ape_pct": 142.37236553620326 + }, + { + "index": 236, + "config_id": 1447, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 256, + "spec_sha256": "2e7a0370494a499e02ae70bc0f162e0ce19db43c218bcd42551c8a9d6f3da26a", + "completed": 2560, + "silicon_ttft_ms": 2128.929119654269, + "before_ttft_ms": 1044.8854300496118, + "after_ttft_ms": 1044.8854300496118, + "before_ttft_ape_pct": 50.91967034490572, + "after_ttft_ape_pct": 50.91967034490572, + "silicon_tpot_ms": 33.04418500599972, + "before_tpot_ms": 72.6305979432324, + "after_tpot_ms": 72.6305979432324, + "before_tpot_ape_pct": 119.7984242312077, + "after_tpot_ape_pct": 119.7984242312077 + }, + { + "index": 237, + "config_id": 1447, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 512, + "spec_sha256": "57a9a19ac86fdbab201967568d460bce0a460dbfe8c997ff4db01ac441c964c7", + "completed": 5120, + "silicon_ttft_ms": 5526.293098364841, + "before_ttft_ms": 4992.721051094732, + "after_ttft_ms": 4992.721051094732, + "before_ttft_ape_pct": 9.655152880472206, + "after_ttft_ape_pct": 9.655152880472206, + "silicon_tpot_ms": 53.34822143784371, + "before_tpot_ms": 117.2873605787203, + "after_tpot_ms": 117.2873605787203, + "before_tpot_ape_pct": 119.85242884877127, + "after_tpot_ape_pct": 119.85242884877127 + }, + { + "index": 238, + "config_id": 1447, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 1024, + "spec_sha256": "ab84fdf7cbc02a4101dbe05dddc8f240fd4a92a8a23b1bb9807f227c0f4cf5d4", + "completed": 10240, + "silicon_ttft_ms": 865.6363492955294, + "before_ttft_ms": 441.3721656742151, + "after_ttft_ms": 441.3721656742151, + "before_ttft_ape_pct": 49.01182626706794, + "after_ttft_ape_pct": 49.01182626706794, + "silicon_tpot_ms": 28.71276834217647, + "before_tpot_ms": 49.106976978578864, + "after_tpot_ms": 49.106976978578864, + "before_tpot_ape_pct": 71.02836060027391, + "after_tpot_ape_pct": 71.02836060027391 + }, + { + "index": 239, + "config_id": 1448, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "0cfaf995c80ccf9a38b3332cad25380e6892928e167ded9cf3bd125143321406", + "completed": 40, + "silicon_ttft_ms": 158.9497644498806, + "before_ttft_ms": 112.2727627749999, + "after_ttft_ms": 112.2727627749999, + "before_ttft_ape_pct": 29.36588288534312, + "after_ttft_ape_pct": 29.36588288534312, + "silicon_tpot_ms": 6.739255458020658, + "before_tpot_ms": 6.475992235276865, + "after_tpot_ms": 6.475992235276865, + "before_tpot_ape_pct": 3.906414059886576, + "after_tpot_ape_pct": 3.906414059886576 + }, + { + "index": 240, + "config_id": 1449, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 256, + "spec_sha256": "4858f83c608f5f580fe622530cec7b695405de850f5ee62a6c59a3cadcb71b4b", + "completed": 2560, + "silicon_ttft_ms": 14293.105692189845, + "before_ttft_ms": 18605.088466374345, + "after_ttft_ms": 18605.088466374345, + "before_ttft_ape_pct": 30.168270402846666, + "after_ttft_ape_pct": 30.168270402846666, + "silicon_tpot_ms": 45.97813886522954, + "before_tpot_ms": 62.99972737060765, + "after_tpot_ms": 62.99972737060765, + "before_tpot_ape_pct": 37.021047231319095, + "after_tpot_ape_pct": 37.021047231319095 + }, + { + "index": 241, + "config_id": 1449, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 1024, + "spec_sha256": "df265ca0787192cdb78f7680dedacbae69951e0507b9b1f1286f950fa55a5f89", + "completed": 10240, + "silicon_ttft_ms": 1735.4954449806842, + "before_ttft_ms": 8994.023264327321, + "after_ttft_ms": 8994.023264327321, + "before_ttft_ape_pct": 418.2395200367364, + "after_ttft_ape_pct": 418.2395200367364, + "silicon_tpot_ms": 46.19516065147828, + "before_tpot_ms": 55.584524249954995, + "after_tpot_ms": 55.584524249954995, + "before_tpot_ape_pct": 20.325426876021147, + "after_tpot_ape_pct": 20.325426876021147 + }, + { + "index": 242, + "config_id": 1450, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "fd25a13788a127642e0e6ca00664463f1f55cfb93aa124d1f29530866b115f0a", + "completed": 40, + "silicon_ttft_ms": 39.87175045017466, + "before_ttft_ms": 32.30652659999959, + "after_ttft_ms": 32.30652659999959, + "before_ttft_ape_pct": 18.97389446101413, + "after_ttft_ape_pct": 18.97389446101413, + "silicon_tpot_ms": 5.631770200088282, + "before_tpot_ms": 5.680733654341742, + "after_tpot_ms": 5.680733654341742, + "before_tpot_ape_pct": 0.8694149887843672, + "after_tpot_ape_pct": 0.8694149887843672 + }, + { + "index": 243, + "config_id": 1450, + "model": "MiniMax-M2.5", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "acc0fa52d9239def939dc0006882bc620a677d70a14a91cee3875e1da7653431", + "completed": 40, + "silicon_ttft_ms": 131.83356732461107, + "before_ttft_ms": 99.15038572500018, + "after_ttft_ms": 99.15038572500018, + "before_ttft_ape_pct": 24.791244189831996, + "after_ttft_ape_pct": 24.791244189831996, + "silicon_tpot_ms": 6.037086844899578, + "before_tpot_ms": 5.981544310239234, + "after_tpot_ms": 5.981544310239234, + "before_tpot_ape_pct": 0.920022124698594, + "after_tpot_ape_pct": 0.920022124698594 + }, + { + "index": 244, + "config_id": 1552, + "model": "Qwen-3.5-397B-A17B", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "8a6c627c192ca6921287127acd539782c52dd6183b821b2af80e0240b649fba1", + "completed": 40, + "silicon_ttft_ms": 436.8923836518661, + "before_ttft_ms": 134.72857442499972, + "after_ttft_ms": 134.72857442499972, + "before_ttft_ape_pct": 69.16206840255722, + "after_ttft_ape_pct": 69.16206840255722, + "silicon_tpot_ms": 5.5460571458534735, + "before_tpot_ms": 7.47041421536787, + "after_tpot_ms": 7.47041421536787, + "before_tpot_ape_pct": 34.697750472209755, + "after_tpot_ape_pct": 34.697750472209755 + }, + { + "index": 245, + "config_id": 1552, + "model": "Qwen-3.5-397B-A17B", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "234ed3f6c01701fdfcf6a02d1cebb8e72799a58a278f4e281932a3f95c8113b1", + "completed": 640, + "silicon_ttft_ms": 394.23737491551947, + "before_ttft_ms": 80.01020831874982, + "after_ttft_ms": 80.01020831874982, + "before_ttft_ape_pct": 79.70506770549213, + "after_ttft_ape_pct": 79.70506770549213, + "silicon_tpot_ms": 10.165825008399915, + "before_tpot_ms": 13.420419143638451, + "after_tpot_ms": 13.420419143638451, + "before_tpot_ape_pct": 32.01505173017731, + "after_tpot_ape_pct": 32.01505173017731 + }, + { + "index": 246, + "config_id": 1552, + "model": "Qwen-3.5-397B-A17B", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "b2d903d4cf99905a3edaaff5c8c9e3a3b63cafbc9c3cd8ed5a4e10d1a5f6692d", + "completed": 40, + "silicon_ttft_ms": 391.24625302647473, + "before_ttft_ms": 37.38472784999993, + "after_ttft_ms": 37.38472784999993, + "before_ttft_ape_pct": 90.44470648324133, + "after_ttft_ape_pct": 90.44470648324133, + "silicon_tpot_ms": 5.3555726894542, + "before_tpot_ms": 7.204791513214827, + "after_tpot_ms": 7.204791513214827, + "before_tpot_ape_pct": 34.52887171902219, + "after_tpot_ape_pct": 34.52887171902219 + }, + { + "index": 247, + "config_id": 1553, + "model": "Qwen-3.5-397B-A17B", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 1024, + "spec_sha256": "2bca8f97115fc34fa43f1b404f99c507152473b8fcfeeebb02adae8dff71b113", + "completed": 10240, + "silicon_ttft_ms": 1061.8200103955644, + "before_ttft_ms": 664.4044430442417, + "after_ttft_ms": 664.4044430442417, + "before_ttft_ape_pct": 37.42777151122551, + "after_ttft_ape_pct": 37.42777151122551, + "silicon_tpot_ms": 26.723510568227518, + "before_tpot_ms": 49.73672805066596, + "after_tpot_ms": 49.73672805066596, + "before_tpot_ape_pct": 86.1159967126461, + "after_tpot_ape_pct": 86.1159967126461 + }, + { + "index": 248, + "config_id": 1553, + "model": "Qwen-3.5-397B-A17B", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 256, + "spec_sha256": "9cebcbc6a08b94cddd511588ed2a7e3ad8d52a42ec5dbb5a206c4b34af40827a", + "completed": 2560, + "silicon_ttft_ms": 2243.5807363432104, + "before_ttft_ms": 878.8255827765624, + "after_ttft_ms": 878.8255827765624, + "before_ttft_ape_pct": 60.829331053673094, + "after_ttft_ape_pct": 60.829331053673094, + "silicon_tpot_ms": 23.606296225803963, + "before_tpot_ms": 77.64515405667075, + "after_tpot_ms": 77.64515405667075, + "before_tpot_ape_pct": 228.91713852085397, + "after_tpot_ape_pct": 228.91713852085397 + }, + { + "index": 249, + "config_id": 1553, + "model": "Qwen-3.5-397B-A17B", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 512, + "spec_sha256": "3e1bb3f11f9b2d9aff8886a6e947b0b22578c924c02f535e51f9143e8fadd0a7", + "completed": 5120, + "silicon_ttft_ms": 2915.429193386558, + "before_ttft_ms": 1399.8037921550763, + "after_ttft_ms": 1399.8037921550763, + "before_ttft_ape_pct": 51.98635606275636, + "after_ttft_ape_pct": 51.98635606275636, + "silicon_tpot_ms": 35.09371775002287, + "before_tpot_ms": 117.62597871634401, + "after_tpot_ms": 117.62597871634401, + "before_tpot_ape_pct": 235.1767389086822, + "after_tpot_ape_pct": 235.1767389086822 + }, + { + "index": 250, + "config_id": 1553, + "model": "Qwen-3.5-397B-A17B", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 512, + "spec_sha256": "5a074524f04dfbec43481b263dd3dfeeee0981b0074d49473d3f9d4e99708141", + "completed": 5120, + "silicon_ttft_ms": 854.6951423828773, + "before_ttft_ms": 229.3790417873047, + "after_ttft_ms": 229.3790417873047, + "before_ttft_ape_pct": 73.16247274463275, + "after_ttft_ape_pct": 73.16247274463275, + "silicon_tpot_ms": 19.264348265786083, + "before_tpot_ms": 38.15214491502981, + "after_tpot_ms": 38.15214491502981, + "before_tpot_ape_pct": 98.04534463690567, + "after_tpot_ape_pct": 98.04534463690567 + }, + { + "index": 251, + "config_id": 1553, + "model": "Qwen-3.5-397B-A17B", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 1024, + "spec_sha256": "8348471205108b37c728fe8e37cc14d6f77e85fb6112ba19e4a678a95d4f035f", + "completed": 10240, + "silicon_ttft_ms": 4516.75263256372, + "before_ttft_ms": 3750.1229240928815, + "after_ttft_ms": 3750.1229240928815, + "before_ttft_ape_pct": 16.973028430731162, + "after_ttft_ape_pct": 16.973028430731162, + "silicon_tpot_ms": 57.814974857846956, + "before_tpot_ms": 163.1528824184482, + "after_tpot_ms": 163.1528824184482, + "before_tpot_ape_pct": 182.19831076568255, + "after_tpot_ape_pct": 182.19831076568255 + }, + { + "index": 252, + "config_id": 1554, + "model": "Qwen-3.5-397B-A17B", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 32, + "spec_sha256": "a4d7a11a68758d5b31789d618f81367e377d9a450e0739d61298d0b343e34d59", + "completed": 320, + "silicon_ttft_ms": 834.2485467454026, + "before_ttft_ms": 432.3503502062502, + "after_ttft_ms": 432.3503502062502, + "before_ttft_ape_pct": 48.17487523437117, + "after_ttft_ape_pct": 48.17487523437117, + "silicon_tpot_ms": 16.27184513595418, + "before_tpot_ms": 22.711589490094447, + "after_tpot_ms": 22.711589490094447, + "before_tpot_ape_pct": 39.57599338203535, + "after_tpot_ape_pct": 39.57599338203535 + }, + { + "index": 253, + "config_id": 1554, + "model": "Qwen-3.5-397B-A17B", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "7deab3ec8dadc01ccfdb29216f8ddf5df497399284e08bf5e44ad1e6aa3dc1ce", + "completed": 80, + "silicon_ttft_ms": 610.5244070626213, + "before_ttft_ms": 258.6699619375004, + "after_ttft_ms": 258.6699619375004, + "before_ttft_ape_pct": 57.63151170613746, + "after_ttft_ape_pct": 57.63151170613746, + "silicon_tpot_ms": 9.024194664205083, + "before_tpot_ms": 13.515001405545467, + "after_tpot_ms": 13.515001405545467, + "before_tpot_ape_pct": 49.76407212439013, + "after_tpot_ape_pct": 49.76407212439013 + }, + { + "index": 254, + "config_id": 1555, + "model": "Qwen-3.5-397B-A17B", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 1024, + "spec_sha256": "1311f7d5cb5363d105de1ebb14aed46eaddd373767a145350f0b0ac8705a3b38", + "completed": 10240, + "silicon_ttft_ms": 6920.29263017825, + "before_ttft_ms": 4654.914700451452, + "after_ttft_ms": 4654.914700451452, + "before_ttft_ape_pct": 32.73529098824319, + "after_ttft_ape_pct": 32.73529098824319, + "silicon_tpot_ms": 102.82562778065234, + "before_tpot_ms": 171.322507606962, + "after_tpot_ms": 171.322507606962, + "before_tpot_ape_pct": 66.6145992051974, + "after_tpot_ape_pct": 66.6145992051974 + }, + { + "index": 255, + "config_id": 1555, + "model": "Qwen-3.5-397B-A17B", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 1024, + "spec_sha256": "0e3bb6dda06338bb24788f6ae1995563767068bfa235fae28de7357b5d03ae42", + "completed": 10240, + "silicon_ttft_ms": 1383.014349500027, + "before_ttft_ms": 776.3915716720743, + "after_ttft_ms": 776.3915716720743, + "before_ttft_ape_pct": 43.862363253660575, + "after_ttft_ape_pct": 43.862363253660575, + "silicon_tpot_ms": 43.71394129702334, + "before_tpot_ms": 62.57214944162358, + "after_tpot_ms": 62.57214944162358, + "before_tpot_ape_pct": 43.140031726867825, + "after_tpot_ape_pct": 43.140031726867825 + }, + { + "index": 256, + "config_id": 1556, + "model": "Qwen-3.5-397B-A17B", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 8, + "spec_sha256": "62757ecf941040fd38cb052809bd8a8f72f62fe431683290b23d1cb1618ad409", + "completed": 80, + "silicon_ttft_ms": 429.9583063722821, + "before_ttft_ms": 69.06432680000013, + "after_ttft_ms": 69.06432680000013, + "before_ttft_ape_pct": 83.93697114896523, + "after_ttft_ape_pct": 83.93697114896523, + "silicon_tpot_ms": 6.842830438749364, + "before_tpot_ms": 12.883911933724267, + "after_tpot_ms": 12.883911933724267, + "before_tpot_ape_pct": 88.28337263430726, + "after_tpot_ape_pct": 88.28337263430726 + }, + { + "index": 257, + "config_id": 1556, + "model": "Qwen-3.5-397B-A17B", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "5ba8bb4cf42586eddfa85b4b46a6dc2ea142cb53d00dd1267bc60674a34ce9fc", + "completed": 160, + "silicon_ttft_ms": 438.6986423352937, + "before_ttft_ms": 83.63057056875016, + "after_ttft_ms": 83.63057056875016, + "before_ttft_ape_pct": 80.93666984616925, + "after_ttft_ape_pct": 80.93666984616925, + "silicon_tpot_ms": 8.290354671325023, + "before_tpot_ms": 15.297529432370695, + "after_tpot_ms": 15.297529432370695, + "before_tpot_ape_pct": 84.52201430274557, + "after_tpot_ape_pct": 84.52201430274557 + }, + { + "index": 258, + "config_id": 1556, + "model": "Qwen-3.5-397B-A17B", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "b4a1f074e85cee8ae26e0fbce1075890727e682d66fbbebed11a6ef6a421d591", + "completed": 40, + "silicon_ttft_ms": 462.16799849498784, + "before_ttft_ms": 215.2586683999995, + "after_ttft_ms": 215.2586683999995, + "before_ttft_ape_pct": 53.42415115261729, + "after_ttft_ape_pct": 53.42415115261729, + "silicon_tpot_ms": 5.982946747053105, + "before_tpot_ms": 11.613662487920676, + "after_tpot_ms": 11.613662487920676, + "before_tpot_ape_pct": 94.11275043758287, + "after_tpot_ape_pct": 94.11275043758287 + }, + { + "index": 259, + "config_id": 1557, + "model": "Qwen-3.5-397B-A17B", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 64, + "spec_sha256": "caa504644453affa66ab706e42177583f80f327473bc257e9e5b80f069663878", + "completed": 640, + "silicon_ttft_ms": 442.7471009533292, + "before_ttft_ms": 99.74891588437448, + "after_ttft_ms": 99.74891588437448, + "before_ttft_ape_pct": 77.47045307138235, + "after_ttft_ape_pct": 77.47045307138235, + "silicon_tpot_ms": 12.243145263224786, + "before_tpot_ms": 16.89855976858599, + "after_tpot_ms": 16.89855976858599, + "before_tpot_ape_pct": 38.02466119016699, + "after_tpot_ape_pct": 38.02466119016699 + }, + { + "index": 260, + "config_id": 1557, + "model": "Qwen-3.5-397B-A17B", + "backend": "trtllm", + "hardware": "b200", + "isl": 1024, + "osl": 1024, + "concurrency": 128, + "spec_sha256": "5180c8aed12b5d485ba2f73133a10a93d014853c70f860c535ecbdb251911529", + "completed": 1280, + "silicon_ttft_ms": 469.13041841276026, + "before_ttft_ms": 148.2634121124999, + "after_ttft_ms": 148.2634121124999, + "before_ttft_ape_pct": 68.39612050437292, + "after_ttft_ape_pct": 68.39612050437292, + "silicon_tpot_ms": 15.775930868241035, + "before_tpot_ms": 21.25532832103029, + "after_tpot_ms": 21.25532832103029, + "before_tpot_ape_pct": 34.73264112623606, + "after_tpot_ape_pct": 34.73264112623606 + }, + { + "index": 261, + "config_id": 1558, + "model": "Qwen-3.5-397B-A17B", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 16, + "spec_sha256": "f0e363d928d0c6b5b81f2eebe7bcb602bda624e0205decee8da7d9a7d14fcfd5", + "completed": 160, + "silicon_ttft_ms": 741.738339108997, + "before_ttft_ms": 357.2644937625002, + "after_ttft_ms": 357.2644937625002, + "before_ttft_ape_pct": 51.83416106120937, + "after_ttft_ape_pct": 51.83416106120937, + "silicon_tpot_ms": 11.916830327206366, + "before_tpot_ms": 20.53006921070068, + "after_tpot_ms": 20.53006921070068, + "before_tpot_ape_pct": 72.27793504645372, + "after_tpot_ape_pct": 72.27793504645372 + }, + { + "index": 262, + "config_id": 1558, + "model": "Qwen-3.5-397B-A17B", + "backend": "trtllm", + "hardware": "b200", + "isl": 8192, + "osl": 1024, + "concurrency": 4, + "spec_sha256": "c85e0bccd3c1d57d3acfe4c789e5bbe958ed0a6e9baaf062a0b39fbb60182217", + "completed": 40, + "silicon_ttft_ms": 573.8227869020193, + "before_ttft_ms": 257.99375477500104, + "after_ttft_ms": 257.99375477500104, + "before_ttft_ape_pct": 55.03947200008046, + "after_ttft_ape_pct": 55.03947200008046, + "silicon_tpot_ms": 7.337073735527814, + "before_tpot_ms": 13.045057350815759, + "after_tpot_ms": 13.045057350815759, + "before_tpot_ape_pct": 77.79645974727723, + "after_tpot_ape_pct": 77.79645974727723 + } + ], + "package_audit": { + "files_checked": 1773, + "changed": [ + "aiconfigurator_core/systems/data/b200_sxm/moe/trtllm/1.3.0rc20/moe_perf.parquet" + ], + "before": "0bc472446e0993221f28c454645fc4054eb73162270868c2ea1332b5572d7803", + "after": "1ff05df3a5715ca5d80e2435ba5ec8221e25da7f414ae7834a87af84c46b9b87", + "runtime_sha256": "7c0ccae3937ef48224f1a35cbab04735ea13914a395bc0853435c63425ef62b8" + }, + "artifacts_sha256": { + "production/summary.json": "941b8f092b83ce138fdb0c852076d66dbb2dc97c4ca779f66a16125304e66606", + "production/plan.json": "fef1029217f6421a2c75aa9c22f681ee076dd93af31cacf8aff2930add96a9fe", + "production/source.sha256": "28c9164548c3d69741cf1b9195651a7d8193b62b06ce4c2cdfbe83be7b9cc905", + "production/gpu.csv": "a640ed1d07f0e9a03a34a13bb11917a84666b7003f8c4b3003daa7d4cf9957a9", + "production/shard-0/timing-mode.jsonl": "ec0bedbffb6e3c8043d7ac3618289c8daab46aea6a9cfe19b136a84363147bcb", + "production/shard-0/moe_perf.txt": "9a1bc9aebe0db100b5eb293fc6810c4818401e43ea9f30165d32f66b2b036a00", + "production/shard-0/outcomes.jsonl": "5d0b2ee13a917a48617bb77846891e287e372c34b9b1611bfba3b0612c3b68e6", + "production/shard-1/timing-mode.jsonl": "9a7c7275c6555b79b1332cca7de38efb3512f29b9658f3b2f25f23156c397b8b", + "production/shard-1/moe_perf.txt": "05639ded5f4cebe3eed1f07fad998b7ba55a3b19014a48080876650a09956acb", + "production/shard-1/outcomes.jsonl": "64cd1a34033d8cd9f661e172ad418db421d8b7dcf5e74267009572c6f9ff7187", + "contract-probe/serving.json": "ee4392d1c02a98e8d84a3d0f51a84d0d941ebf899013bc4e0f728259200c9b7d", + "contract-probe/tuned.json": "50d80667e73cdbbd796f42d971e23d3b1b29f91a00d3c423ff62ff20443b8384", + "contract-probe/tuned.log": "30195cf0565441f28fa95d6c7967aa7b0356cdb1c51e2c859dcc993098d3de32", + "contract-probe/tuned.csv": "856c397f421bab8af7b4fe744585f5b63f7971ebe529f9970ac3cb1c37f04ebb", + "contract-probe/baseline.log": "531f963b3825ece873ff18252f6514b0fa4b066ada25ec72754841a3790c4ba2", + "contract-probe/baseline.json": "763c0b1aae87960dc37ec788ebebf4cfb9ffcbe6e817d18d3c6dd3f5340dc1e5", + "contract-probe/baseline.csv": "8b3435d3952ac033dee2cf54c4df014382d6ffa171782717da009e6c8a59e51b", + "contract-probe/serving.csv": "59209ffe7b46733fd0a23815eb3df2698bd2c799b15110630663d9ca489dad78", + "contract-probe/serving.log": "6a9ff6d9c6a92e31a99506b148e4e52e136c1ca51ff577cec0e741c0393da0b5", + "contract-probe/bias.log": "3f4134d9653f507d1c291e9c19c92bd5cbf581e518a4bf55820608a15296ac52", + "contract-probe/bias.json": "7c0743eba296a05f26358a4d7c11cef4714114bd76468139286f2ca2b730e33d", + "contract-probe/bias.csv": "5c2a544d4ed41b45298e31917c5b6faa384723d4f83a0a96d46638949c19635c", + "ep-probe/serving.json": "68877359d85262df591f761e41f387898e918c5b641fd7b0cbf7fef5b5d4bf38", + "ep-probe/baseline.log": "1114b411769ff5b96f7a63c162efc2e8b6492f4362b6714881799f47881fd522", + "ep-probe/baseline.json": "d28f57b7a88e6bcbc805e135b1360ba2db7a50d2cfababc51aac0f4b9cbd7519", + "ep-probe/baseline.csv": "27e82ed08212011f19d7523eedbe76abc46d1f5a2b6fd1510867a5932095b617", + "ep-probe/serving.csv": "523451b889206bc88551b8ffe0d8197f35d0b7637c41c02f77aafeabe6aad689", + "ep-probe/serving.log": "7d8009b0fec2b1b73d7c995711b3b9a079075610b6039b0fffc745d456406e25", + "rc14-probe/baseline.csv": "398a8fa0652c47cc3d0d9b8d2be7bb8050f99d135181b52856b93ac9f05f3082", + "rc14-probe/baseline.json": "850cc6289bde79e57025e5b2f46dd91bfe84c2e0a176b95b22f1da9ea01aa327", + "rc14-probe/baseline.log": "b0962e7a5a15cd334e15734c70a35b7a1c978e4b0d2b07f8c089a245148c1abf", + "rc14-probe/serving.csv": "fa9bcd10256581aae158fc14fb844b25dca91fd7886aabb75d0d603490b86b1b", + "rc14-probe/serving.json": "4743a4b1f2d55f6b1f38834d7893e823316b6ce8adbc702ec1ff83392548f0ba", + "rc14-probe/serving.log": "428fba1eaadcc1a261d52dfb1d81a89d9c3d1e59c2e6016294d96dc6a2c5ed80", + "prepare-rc14.sh": "dfe53e4fb8539b4538213566607d8ecaf963e738a3a500fd8b4ae05610cfb853", + "rc14-probe.sh": "eb3a85d886d2b32708a86f75c18c8925867b3c1bd752bbd9301bc250c1e3528d", + "version_probe.py": "d656307c22587799e5f2d4f9fd8be4b34d325f9151209823bbb44c7ad1dc4d13", + "job-status.txt": "7f18fbc87728ec3dc58b66bd5bc10db8140ff8fa7a4b37d6fec77a4421ade430", + "rc14-image.sha256": "9e384cda346c4a7e19b37010539342af697d6c355c06fcbd7407bee218af7cae" + }, + "ep_recheck_job": 1994626, + "original_server_artifacts": { + "run_id": 26016885799, + "status": "expired", + "listing_sha256": "83335f73ee93b034b16f5d8cf07a0d5d2ca5b405b0df0d517a1e8d184ee45b78" + }, + "rc14_probe": { + "import_job": 1994625, + "probe_job": 1994650, + "state": "COMPLETED", + "exit_code": "0:0", + "image_digest": "sha256:fe2f17d0c9698bafeb9f437929003c6badc56d17e8382a02505233145a07f61f", + "squashfs_sha256": "9f0d576d030724c8b16897d3d2259ea0f19eadf5a6f333902b01e40ff917e9d4", + "graph_measurements": 18, + "forwards_per_timed_closure": 5, + "paired_medians": [ + { + "arm": "baseline", + "tokens": 64, + "rc14_per_forward_ms": 0.9547574615478516, + "rc20_per_forward_ms": 0.8983990478515625, + "rc14_vs_rc20_pct": 6.273204967332147, + "routing_sha256": "db1dfa1338b9093f8798a2cc318cd914a3e3521396ff81978565c065bd4d0f92" + }, + { + "arm": "baseline", + "tokens": 256, + "rc14_per_forward_ms": 1.928819580078125, + "rc20_per_forward_ms": 1.8201618957519532, + "rc14_vs_rc20_pct": 5.9696714110852644, + "routing_sha256": "b9700c40021e87ecccb2f7802b1536542c8e6bb13b761e84d6c59d735159afc8" + }, + { + "arm": "baseline", + "tokens": 16384, + "rc14_per_forward_ms": 12.520361328125, + "rc20_per_forward_ms": 11.35655029296875, + "rc14_vs_rc20_pct": 10.247927452730154, + "routing_sha256": "9a32ee5879e3e816fa2a5746024f143f61265362f070f8f2ee94cdb27560b98f" + }, + { + "arm": "serving", + "tokens": 64, + "rc14_per_forward_ms": 0.9393708801269531, + "rc20_per_forward_ms": 0.8910678100585937, + "rc14_vs_rc20_pct": 5.420807431612107, + "routing_sha256": "db1dfa1338b9093f8798a2cc318cd914a3e3521396ff81978565c065bd4d0f92" + }, + { + "arm": "serving", + "tokens": 256, + "rc14_per_forward_ms": 1.4044166564941407, + "rc20_per_forward_ms": 1.3499250793457032, + "rc14_vs_rc20_pct": 4.036637142473798, + "routing_sha256": "b9700c40021e87ecccb2f7802b1536542c8e6bb13b761e84d6c59d735159afc8" + }, + { + "arm": "serving", + "tokens": 16384, + "rc14_per_forward_ms": 12.560281982421875, + "rc20_per_forward_ms": 11.262667236328124, + "rc14_vs_rc20_pct": 11.521380494207012, + "routing_sha256": "9a32ee5879e3e816fa2a5746024f143f61265362f070f8f2ee94cdb27560b98f" + } + ], + "limitations": [ + "Separate B200 nodes and image dependencies; not a version-only causal effect.", + "Historical serving artifacts expired; resolved tag digest does not prove original serving digest.", + "Diagnostic only; no rc14 rows published in the rc20 corpus." + ] + }, + "pdl_source_check": { + "recipe_commit": "2baba8e27be8529b4453afc953f9859ec092c73d", + "recipe_value": "1", + "rc14_default": true, + "rc20_default": true, + "path": "cpp/tensorrt_llm/common/envUtils.cpp" + } +} diff --git a/python/aisimulate/collector/trtllm/gym-tpot-serving-20260917.json b/python/aisimulate/collector/trtllm/gym-tpot-serving-20260917.json new file mode 100644 index 000000000..3111e2481 --- /dev/null +++ b/python/aisimulate/collector/trtllm/gym-tpot-serving-20260917.json @@ -0,0 +1,1019 @@ +{ + "manifest": { + "runtime_commit": "89d2051137b772944a3e303c452108811790bef4", + "prediction_data_commit": "a85b9e10f5117439eafd7a5d113118b781229920", + "recipe_commit": "2baba8e27be8529b4453afc953f9859ec092c73d", + "trtllm_source_commit": "93cb6518b6d6dbd6095748189e626db731f44545", + "image_digest": "sha256:fe2f17d0c9698bafeb9f437929003c6badc56d17e8382a02505233145a07f61f", + "model_revision": "b5c939de8f754692c1647ca79fbf85e8c1e70f8a", + "model_prepare_job": 1994868, + "baseline_job": 1994872, + "diagnostic_job": 1994898, + "default_resolution": { + "source": "tensorrt_llm/_torch/model_config.py", + "source_sha256": "789291392c2c7d3683856b6fa9e56a9a6b4e45786cfdb0cf1b929bc81c117624", + "symbol": "ModelConfig.get_mxfp4_quant_algo", + "rule": "On SM >= 100, non-TRITON static MXFP4 checkpoints default to W4A8_MXFP4_MXFP8 unless OVERRIDE_QUANT_ALGO is set." + }, + "scripts_sha256": { + "baseline-inner.sh": "4eca7d0152a133183485ab0e15edaed358b33ae180df7b7c8ffcbda8dbc07bad", + "benchmark_lib.sh": "9b8a17a79925b730149a99fd0dfa01d2ac81c313fec4665cecb04356fe740999", + "diagnostic-inner.sh": "3a357b1e6283d185a33f8615717731077c0c64eb214cf3d3f8f25ac7f3e94a28", + "hooks/sitecustomize.py": "33bbd70ab55ca5c26c47addff9d9e7f67b96399b11f73c949a3cc9b1565e9722", + "hooks-v2/sitecustomize.py": "bd51deb5bf93f573a9f65113864985126b07576d926594e2f0db7c78f5566606", + "inferencex/utils/bench_serving/backend_request_func.py": "4ebf482eb32685e59ed69bd39c51e9f1fd2134f5e9b8fe7ad10550d40ecca16c", + "inferencex/utils/bench_serving/benchmark_serving.py": "67077f1e7abef278faa367237e33143135032c65c27032100cf26e82de37d086", + "inferencex/utils/bench_serving/benchmark_utils.py": "b99a18940871b701e9e65c2cc098da1c5d33fc3f9d3890f8d2cddc539a9e3d51", + "inferencex/utils/bench_serving/encoding_dsv4.py": "bdbd57c132a1b3725042323d02b98b9d1df28e5f388f134399555d041f5055e0", + "prepare-model.sh": "f43f7d0bf16ad6f4e4a637227bdd8f7bbc0ca2a68387520009197b579d9c0fe9", + "replay-client-lengths.py": "bb682b61d25202cd7f4683e6efe1344ce37f3cb128b556d99897c7372078c671", + "replay-quantization.py": "c5e5ad82cbf46dca0a18921fe987a401ffba2fada1c9af4ae523f814df603ef1", + "routing-inner.sh": "05ce9a843a0b711d8631c0c062085a30832cfcf4ee52212aeef3c4f89b0b182f", + "run-baseline.sh": "49a8186c879b89d41b87aa5dac32fed64ac8a16832eb464e92141da3e02704ef", + "run-diagnostic.sh": "a9ccb10f5c347a6d127dbe46873a6e5f72301bcf4c4dbecb1b5bfeb309174470", + "run-routing.sh": "1afec2cd34b920dc45b86e947e22e0436e3cfddd6198b5043f95a9ea3a8e2367", + "upstream/cuda_graph_runner.py": "de98a824218958725db8705af01b4e55d9425c86e38b8fb34f9652cc309d4cda", + "upstream/fused_moe_trtllm_gen.py": "4c83eb8113f1c007d761718df8acaa05e4d6de22bbd0430281e1a9210c2d9751", + "upstream/image-cuda_graph_runner.py": "de98a824218958725db8705af01b4e55d9425c86e38b8fb34f9652cc309d4cda", + "upstream/image-fused_moe_trtllm_gen.py": "4c83eb8113f1c007d761718df8acaa05e4d6de22bbd0430281e1a9210c2d9751", + "upstream/interface.py": "1b4ea1e275f1fa998c8cc3e2a529da132dfd65891c0a2403cf367cf7b6f80be8", + "upstream/openai_server.py": "60ff86b178b1142f151ec59cf14f17121238c625ca7e4cfd3e1decfb2f99285f", + "upstream/profiler.py": "58e3e5f2e06e1980438ff96182216ffc0ea433f52163159bc7c3d272387959b4", + "upstream/py_executor.py": "4d3929d295a2bc65380cf3c4a463b3eb1a0e14d80c3acb95b0b78485f36ef1fe", + "upstream/serve.py": "2ad5bdd864fea525aa42b3d2310ab4f781100f037af31413188dd251da568dec" + }, + "limits": [ + "Historical model revision was not pinned by the original recipe. This run pins the revision used to resolve the replay config.", + "Original image digest is unavailable; current resolved rc14 tag digest is pinned.", + "CPU allocation, GPU node and serving dependencies are recorded but do not recreate the historical host.", + "Diagnostic graph copies and profiling are not used as the uninstrumented E2E truth." + ], + "routing_retry_job": 1995012, + "runtime_source_verified": { + "image-cuda_graph_runner.py": "de98a824218958725db8705af01b4e55d9425c86e38b8fb34f9652cc309d4cda", + "image-fused_moe_trtllm_gen.py": "4c83eb8113f1c007d761718df8acaa05e4d6de22bbd0430281e1a9210c2d9751" + } + }, + "comparison": { + "completed": 2560, + "measured_ttft_ms": 189.62700960501024, + "measured_tpot_ms": 16.549248157714064, + "historical_ttft_ms": 183.61345084202867, + "historical_tpot_ms": 16.792121521213446, + "observed_vs_historical_tpot_pct": -1.4463530602286334, + "pure_decode_batch256_steps": 9233, + "previous_device_aligned_steps": 8246, + "pure_decode_device_median_ms": 14.058015823364258, + "pure_decode_host_median_ms": 13.551712036132812, + "quant_arms": [ + { + "quant": "w4a16_mxfp4", + "ttft_ms": 487.35847630390583, + "tpot_ms": 65.16758265270599, + "ttft_ape_pct": 157.00899746247384, + "tpot_ape_pct": 293.7797175536918, + "lengths_match_client": true + }, + { + "quant": "w4a8_mxfp4_mxfp8", + "ttft_ms": 120.92177186796793, + "tpot_ms": 17.17212172635788, + "ttft_ape_pct": 36.23177831056563, + "tpot_ape_pct": 3.76375749948179, + "lengths_match_client": true + } + ] + }, + "trace": { + "graph_replays": 11, + "kernel_count": 5522, + "moe_layer_instances": 396, + "kernels_by_group": { + "other": { + "count": 352, + "summed_duration_per_step_ms": 0.28767754545454544 + }, + "norm": { + "count": 803, + "summed_duration_per_step_ms": 0.4094055454545455 + }, + "dense_gemm": { + "count": 1595, + "summed_duration_per_step_ms": 1.294778090909091 + }, + "attention": { + "count": 792, + "summed_duration_per_step_ms": 1.7492004545454547 + }, + "moe": { + "count": 1980, + "summed_duration_per_step_ms": 10.013167909090908 + } + }, + "moe_per_layer_ms": 0.278143553030303, + "kernel_names": [ + "bmm_Bfloat16_MxE2m1MxE4m3_Fp32_bA32_bB32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_16dp256b_rM_TN_transOut_schPd2x1x2x3_biasM_bN_rgTma_clmp_dynB_sm100f", + "bmm_MxE4m3_MxE2m1MxE4m3_Fp32_bA32_bB32_bC32_t128x16x256_s6_et128x16_m256x16x32_c2x1x1_16dp256b_rM_TN_transOut_schPd2x1x2x3_biasM_bN_tma_ldgstsSf_rgTma_clmp_swiGlu_dynB_sm100f", + "void moe::dev::finalize::finalizeKernelVecLoad >(moe::dev::finalize::KernelParams)", + "void moe::dev::routing::routingCustom::routingIndicesClusterKernel > >(moe::dev::routing::routingCustom::KernelParams<__nv_bfloat16, __nv_bfloat16, 128, 4, moe::dev::routing::TopKExpertSelect >)", + "void tensorrt_llm::_v1::kernels::quantize_with_block_size<(tensorrt_llm::_v1::BlockScaleQuantizationType)2, __nv_bfloat16, 32, true>(int, int, int, int, __nv_bfloat16 const*, float const*, unsigned int*, unsigned int*, tensorrt_llm::_v1::QuantizationSFLayout)" + ], + "streams": { + "13": 5357, + "7": 165 + }, + "limitation": "Summed GPU kernel durations; not a full critical-path or profiler-free E2E measurement." + }, + "routing": { + "job": 1995012, + "sampled_steps": 3, + "layers_per_step": 36, + "layer_observations": 108, + "tokens_per_layer": 256, + "experts": 128, + "topk": 4, + "max_tokens_per_expert": { + "min": 45, + "median": 84.5, + "max": 198 + }, + "active_experts": { + "min": 77, + "median": 105.0, + "max": 116 + }, + "model": { + "quant_algo": "W4A8_MXFP4_MXFP8", + "moe_class": "", + "pid": 1236947 + }, + "layers": [ + { + "step": 100, + "layer": 0, + "active_experts": 116, + "max_tokens_per_expert": 46 + }, + { + "step": 100, + "layer": 1, + "active_experts": 109, + "max_tokens_per_expert": 49 + }, + { + "step": 100, + "layer": 2, + "active_experts": 111, + "max_tokens_per_expert": 84 + }, + { + "step": 100, + "layer": 3, + "active_experts": 105, + "max_tokens_per_expert": 89 + }, + { + "step": 100, + "layer": 4, + "active_experts": 104, + "max_tokens_per_expert": 100 + }, + { + "step": 100, + "layer": 5, + "active_experts": 107, + "max_tokens_per_expert": 86 + }, + { + "step": 100, + "layer": 6, + "active_experts": 105, + "max_tokens_per_expert": 102 + }, + { + "step": 100, + "layer": 7, + "active_experts": 103, + "max_tokens_per_expert": 83 + }, + { + "step": 100, + "layer": 8, + "active_experts": 109, + "max_tokens_per_expert": 83 + }, + { + "step": 100, + "layer": 9, + "active_experts": 100, + "max_tokens_per_expert": 73 + }, + { + "step": 100, + "layer": 10, + "active_experts": 96, + "max_tokens_per_expert": 144 + }, + { + "step": 100, + "layer": 11, + "active_experts": 88, + "max_tokens_per_expert": 198 + }, + { + "step": 100, + "layer": 12, + "active_experts": 95, + "max_tokens_per_expert": 156 + }, + { + "step": 100, + "layer": 13, + "active_experts": 91, + "max_tokens_per_expert": 182 + }, + { + "step": 100, + "layer": 14, + "active_experts": 99, + "max_tokens_per_expert": 57 + }, + { + "step": 100, + "layer": 15, + "active_experts": 102, + "max_tokens_per_expert": 82 + }, + { + "step": 100, + "layer": 16, + "active_experts": 106, + "max_tokens_per_expert": 80 + }, + { + "step": 100, + "layer": 17, + "active_experts": 97, + "max_tokens_per_expert": 159 + }, + { + "step": 100, + "layer": 18, + "active_experts": 110, + "max_tokens_per_expert": 79 + }, + { + "step": 100, + "layer": 19, + "active_experts": 100, + "max_tokens_per_expert": 69 + }, + { + "step": 100, + "layer": 20, + "active_experts": 109, + "max_tokens_per_expert": 72 + }, + { + "step": 100, + "layer": 21, + "active_experts": 91, + "max_tokens_per_expert": 164 + }, + { + "step": 100, + "layer": 22, + "active_experts": 104, + "max_tokens_per_expert": 63 + }, + { + "step": 100, + "layer": 23, + "active_experts": 106, + "max_tokens_per_expert": 83 + }, + { + "step": 100, + "layer": 24, + "active_experts": 102, + "max_tokens_per_expert": 131 + }, + { + "step": 100, + "layer": 25, + "active_experts": 107, + "max_tokens_per_expert": 74 + }, + { + "step": 100, + "layer": 26, + "active_experts": 109, + "max_tokens_per_expert": 72 + }, + { + "step": 100, + "layer": 27, + "active_experts": 106, + "max_tokens_per_expert": 89 + }, + { + "step": 100, + "layer": 28, + "active_experts": 105, + "max_tokens_per_expert": 110 + }, + { + "step": 100, + "layer": 29, + "active_experts": 103, + "max_tokens_per_expert": 83 + }, + { + "step": 100, + "layer": 30, + "active_experts": 103, + "max_tokens_per_expert": 80 + }, + { + "step": 100, + "layer": 31, + "active_experts": 107, + "max_tokens_per_expert": 93 + }, + { + "step": 100, + "layer": 32, + "active_experts": 106, + "max_tokens_per_expert": 111 + }, + { + "step": 100, + "layer": 33, + "active_experts": 97, + "max_tokens_per_expert": 81 + }, + { + "step": 100, + "layer": 34, + "active_experts": 83, + "max_tokens_per_expert": 73 + }, + { + "step": 100, + "layer": 35, + "active_experts": 82, + "max_tokens_per_expert": 118 + }, + { + "step": 150, + "layer": 0, + "active_experts": 115, + "max_tokens_per_expert": 50 + }, + { + "step": 150, + "layer": 1, + "active_experts": 116, + "max_tokens_per_expert": 48 + }, + { + "step": 150, + "layer": 2, + "active_experts": 105, + "max_tokens_per_expert": 59 + }, + { + "step": 150, + "layer": 3, + "active_experts": 106, + "max_tokens_per_expert": 67 + }, + { + "step": 150, + "layer": 4, + "active_experts": 100, + "max_tokens_per_expert": 87 + }, + { + "step": 150, + "layer": 5, + "active_experts": 106, + "max_tokens_per_expert": 74 + }, + { + "step": 150, + "layer": 6, + "active_experts": 108, + "max_tokens_per_expert": 98 + }, + { + "step": 150, + "layer": 7, + "active_experts": 107, + "max_tokens_per_expert": 101 + }, + { + "step": 150, + "layer": 8, + "active_experts": 106, + "max_tokens_per_expert": 102 + }, + { + "step": 150, + "layer": 9, + "active_experts": 97, + "max_tokens_per_expert": 83 + }, + { + "step": 150, + "layer": 10, + "active_experts": 96, + "max_tokens_per_expert": 136 + }, + { + "step": 150, + "layer": 11, + "active_experts": 89, + "max_tokens_per_expert": 186 + }, + { + "step": 150, + "layer": 12, + "active_experts": 94, + "max_tokens_per_expert": 113 + }, + { + "step": 150, + "layer": 13, + "active_experts": 87, + "max_tokens_per_expert": 159 + }, + { + "step": 150, + "layer": 14, + "active_experts": 103, + "max_tokens_per_expert": 77 + }, + { + "step": 150, + "layer": 15, + "active_experts": 107, + "max_tokens_per_expert": 105 + }, + { + "step": 150, + "layer": 16, + "active_experts": 109, + "max_tokens_per_expert": 85 + }, + { + "step": 150, + "layer": 17, + "active_experts": 104, + "max_tokens_per_expert": 124 + }, + { + "step": 150, + "layer": 18, + "active_experts": 105, + "max_tokens_per_expert": 79 + }, + { + "step": 150, + "layer": 19, + "active_experts": 100, + "max_tokens_per_expert": 65 + }, + { + "step": 150, + "layer": 20, + "active_experts": 116, + "max_tokens_per_expert": 55 + }, + { + "step": 150, + "layer": 21, + "active_experts": 103, + "max_tokens_per_expert": 144 + }, + { + "step": 150, + "layer": 22, + "active_experts": 103, + "max_tokens_per_expert": 45 + }, + { + "step": 150, + "layer": 23, + "active_experts": 108, + "max_tokens_per_expert": 71 + }, + { + "step": 150, + "layer": 24, + "active_experts": 102, + "max_tokens_per_expert": 99 + }, + { + "step": 150, + "layer": 25, + "active_experts": 106, + "max_tokens_per_expert": 64 + }, + { + "step": 150, + "layer": 26, + "active_experts": 103, + "max_tokens_per_expert": 62 + }, + { + "step": 150, + "layer": 27, + "active_experts": 106, + "max_tokens_per_expert": 61 + }, + { + "step": 150, + "layer": 28, + "active_experts": 105, + "max_tokens_per_expert": 90 + }, + { + "step": 150, + "layer": 29, + "active_experts": 103, + "max_tokens_per_expert": 72 + }, + { + "step": 150, + "layer": 30, + "active_experts": 105, + "max_tokens_per_expert": 60 + }, + { + "step": 150, + "layer": 31, + "active_experts": 115, + "max_tokens_per_expert": 64 + }, + { + "step": 150, + "layer": 32, + "active_experts": 103, + "max_tokens_per_expert": 92 + }, + { + "step": 150, + "layer": 33, + "active_experts": 101, + "max_tokens_per_expert": 54 + }, + { + "step": 150, + "layer": 34, + "active_experts": 85, + "max_tokens_per_expert": 90 + }, + { + "step": 150, + "layer": 35, + "active_experts": 78, + "max_tokens_per_expert": 150 + }, + { + "step": 200, + "layer": 0, + "active_experts": 114, + "max_tokens_per_expert": 51 + }, + { + "step": 200, + "layer": 1, + "active_experts": 113, + "max_tokens_per_expert": 57 + }, + { + "step": 200, + "layer": 2, + "active_experts": 107, + "max_tokens_per_expert": 93 + }, + { + "step": 200, + "layer": 3, + "active_experts": 106, + "max_tokens_per_expert": 56 + }, + { + "step": 200, + "layer": 4, + "active_experts": 106, + "max_tokens_per_expert": 121 + }, + { + "step": 200, + "layer": 5, + "active_experts": 108, + "max_tokens_per_expert": 81 + }, + { + "step": 200, + "layer": 6, + "active_experts": 109, + "max_tokens_per_expert": 88 + }, + { + "step": 200, + "layer": 7, + "active_experts": 116, + "max_tokens_per_expert": 126 + }, + { + "step": 200, + "layer": 8, + "active_experts": 108, + "max_tokens_per_expert": 101 + }, + { + "step": 200, + "layer": 9, + "active_experts": 106, + "max_tokens_per_expert": 76 + }, + { + "step": 200, + "layer": 10, + "active_experts": 101, + "max_tokens_per_expert": 137 + }, + { + "step": 200, + "layer": 11, + "active_experts": 98, + "max_tokens_per_expert": 176 + }, + { + "step": 200, + "layer": 12, + "active_experts": 97, + "max_tokens_per_expert": 93 + }, + { + "step": 200, + "layer": 13, + "active_experts": 93, + "max_tokens_per_expert": 153 + }, + { + "step": 200, + "layer": 14, + "active_experts": 106, + "max_tokens_per_expert": 88 + }, + { + "step": 200, + "layer": 15, + "active_experts": 105, + "max_tokens_per_expert": 102 + }, + { + "step": 200, + "layer": 16, + "active_experts": 105, + "max_tokens_per_expert": 114 + }, + { + "step": 200, + "layer": 17, + "active_experts": 104, + "max_tokens_per_expert": 96 + }, + { + "step": 200, + "layer": 18, + "active_experts": 107, + "max_tokens_per_expert": 90 + }, + { + "step": 200, + "layer": 19, + "active_experts": 107, + "max_tokens_per_expert": 87 + }, + { + "step": 200, + "layer": 20, + "active_experts": 104, + "max_tokens_per_expert": 67 + }, + { + "step": 200, + "layer": 21, + "active_experts": 95, + "max_tokens_per_expert": 152 + }, + { + "step": 200, + "layer": 22, + "active_experts": 108, + "max_tokens_per_expert": 65 + }, + { + "step": 200, + "layer": 23, + "active_experts": 105, + "max_tokens_per_expert": 93 + }, + { + "step": 200, + "layer": 24, + "active_experts": 97, + "max_tokens_per_expert": 92 + }, + { + "step": 200, + "layer": 25, + "active_experts": 104, + "max_tokens_per_expert": 58 + }, + { + "step": 200, + "layer": 26, + "active_experts": 108, + "max_tokens_per_expert": 72 + }, + { + "step": 200, + "layer": 27, + "active_experts": 100, + "max_tokens_per_expert": 69 + }, + { + "step": 200, + "layer": 28, + "active_experts": 106, + "max_tokens_per_expert": 107 + }, + { + "step": 200, + "layer": 29, + "active_experts": 106, + "max_tokens_per_expert": 62 + }, + { + "step": 200, + "layer": 30, + "active_experts": 103, + "max_tokens_per_expert": 60 + }, + { + "step": 200, + "layer": 31, + "active_experts": 109, + "max_tokens_per_expert": 55 + }, + { + "step": 200, + "layer": 32, + "active_experts": 105, + "max_tokens_per_expert": 95 + }, + { + "step": 200, + "layer": 33, + "active_experts": 94, + "max_tokens_per_expert": 61 + }, + { + "step": 200, + "layer": 34, + "active_experts": 82, + "max_tokens_per_expert": 74 + }, + { + "step": 200, + "layer": 35, + "active_experts": 77, + "max_tokens_per_expert": 156 + } + ], + "limitations": [ + "Separate diagnostic run without the 512-request client warmup; not used for E2E accuracy.", + "Top-k reconstructed from observed logits with torch.topk; native ties can differ.", + "No causal routing-distribution ablation is claimed." + ] + }, + "representative_decode": { + "batch": 256, + "context_length": 1024, + "generation_length": 512, + "quant": "w4a8_mxfp4_mxfp8", + "total_ms": 12.961707304601322, + "ops_ms": { + "generation_attention": 1.9882100820599273, + "generation_embedding": 0.0032393766233766233, + "generation_add_norm_1": 0.14247023376623377, + "generation_qkv_gemm": 0.3419581866613497, + "generation_proj_gemm": 0.3183382330770096, + "generation_add_norm_2": 0.14247023376623377, + "generation_router_gemm": 0.1929009641392992, + "generation_moe_pre_dispatch": 0.0, + "generation_moe": 9.568742465972901, + "generation_moe_post_dispatch": 0.0, + "generation_logits_gemm": 0.26337752853499063, + "generation_embedding_ar": 0.0, + "generation_p2p": 0.0 + }, + "raw_ops": [ + [ + "generation_attention", + 1.9882100820599273, + 0.0, + "silicon", + null + ], + [ + "generation_embedding", + 0.0032393766233766233, + 0.0, + "empirical", + null + ], + [ + "generation_add_norm_1", + 0.14247023376623377, + 0.0, + "empirical", + null + ], + [ + "generation_qkv_gemm", + 0.3419581866613497, + 0.0, + "silicon", + null + ], + [ + "generation_proj_gemm", + 0.3183382330770096, + 0.0, + "silicon", + null + ], + [ + "generation_add_norm_2", + 0.14247023376623377, + 0.0, + "empirical", + null + ], + [ + "generation_router_gemm", + 0.1929009641392992, + 0.0, + "silicon", + null + ], + [ + "generation_moe_pre_dispatch", + 0.0, + 0.0, + "silicon", + null + ], + [ + "generation_moe", + 9.568742465972901, + 9454.030786500407, + "silicon", + null + ], + [ + "generation_moe_post_dispatch", + 0.0, + 0.0, + "silicon", + null + ], + [ + "generation_logits_gemm", + 0.26337752853499063, + 0.0, + "silicon", + null + ], + [ + "generation_embedding_ar", + 0.0, + 0.0, + "empirical", + null + ], + [ + "generation_p2p", + 0.0, + 0.0, + "silicon", + null + ] + ], + "limitation": "Representative length query, not reconstruction of every profiled request KV length." + }, + "model_manifest": { + "revision": "b5c939de8f754692c1647ca79fbf85e8c1e70f8a", + "shards": { + "model-00000-of-00014.safetensors": 4625017896, + "model-00001-of-00014.safetensors": 4115586736, + "model-00002-of-00014.safetensors": 4625017888, + "model-00003-of-00014.safetensors": 4115586752, + "model-00004-of-00014.safetensors": 4625017896, + "model-00005-of-00014.safetensors": 4115586696, + "model-00006-of-00014.safetensors": 4625017856, + "model-00007-of-00014.safetensors": 4060267176, + "model-00008-of-00014.safetensors": 4625017896, + "model-00009-of-00014.safetensors": 4170906304, + "model-00010-of-00014.safetensors": 4625017896, + "model-00011-of-00014.safetensors": 4115586752, + "model-00012-of-00014.safetensors": 4064660808, + "model-00013-of-00014.safetensors": 4625017896, + "model-00014-of-00014.safetensors": 4115586736 + }, + "metadata_sha256": { + "config.json": "933aeb666a3fd851133ddd7686414f369bc564c4185fb5704416550879f10566", + "model.safetensors.index.json": "ede2655fdc05008561983b6e0829c600727c28d591e071077377059f03a6c00e", + "tokenizer.json": "0614fe83cadab421296e664e1f48f4261fa8fef6e03e63bb75c20f38e37d07d3", + "tokenizer_config.json": "9279e942392b742d633c7adbb89ebe002c98399db8926a7af5125c726f404070" + } + }, + "job_accounting": "JobID|JobName|NodeList|State|Elapsed|ExitCode\n1994868|trt-serving-model|nsc-svg-slurm-1-cpu-28|COMPLETED|00:03:14|0:0\n1994868.batch|batch|nsc-svg-slurm-1-cpu-28|COMPLETED|00:03:14|0:0\n1994868.extern|extern|nsc-svg-slurm-1-cpu-28|COMPLETED|00:03:14|0:0\n1994868.0|siowrapper|nsc-svg-slurm-1-cpu-28|COMPLETED|00:03:10|0:0\n1994872|trt-serving-baseline|nsc-svg-slurm-1-gpu-78|COMPLETED|00:06:30|0:0\n1994872.batch|batch|nsc-svg-slurm-1-gpu-78|COMPLETED|00:06:30|0:0\n1994872.extern|extern|nsc-svg-slurm-1-gpu-78|COMPLETED|00:06:31|0:0\n1994872.0|siowrapper|nsc-svg-slurm-1-gpu-78|COMPLETED|00:05:58|0:0\n1994898|trt-serving-diagnostic|nsc-svg-slurm-1-gpu-78|COMPLETED|00:05:59|0:0\n1994898.batch|batch|nsc-svg-slurm-1-gpu-78|COMPLETED|00:05:59|0:0\n1994898.extern|extern|nsc-svg-slurm-1-gpu-78|COMPLETED|00:05:59|0:0\n1994898.0|siowrapper|nsc-svg-slurm-1-gpu-78|COMPLETED|00:05:56|0:0\n1995012|trt-serving-routing|nsc-svg-slurm-1-gpu-162|COMPLETED|00:06:16|0:0\n1995012.batch|batch|nsc-svg-slurm-1-gpu-162|COMPLETED|00:06:16|0:0\n1995012.extern|extern|nsc-svg-slurm-1-gpu-162|COMPLETED|00:06:16|0:0\n1995012.0|siowrapper|nsc-svg-slurm-1-gpu-162|COMPLETED|00:06:13|0:0\n", + "artifacts_sha256": { + "baseline/client.log": "31c26290458da6ba6ec4e5b4e780811d9855735886a83b7cba5fda48812b4a6b", + "baseline/gpu-before.txt": "e5fb037fa3fe98bba25ce41dea4ed15de8001602b8965e85d09c1a3c1288a52c", + "baseline/result.json": "153f2744e915400203c29007840644cf6a9fabd91a0434347df3ec6e48a9597e", + "baseline/runtime.json": "870e0ebe2f083bf2fc1656672411f422199a92456e80460480532e825c24cc42", + "baseline/server.log": "182229cfce830ce9dc5040073cf6bfb572519818bdec92084b5fbe27bec7985d", + "baseline/telemetry.csv": "e57cb11f70daa8320da6118a4e8ed41d40acc948bce458cd1ed54a9534f9a53a", + "baseline-inner.sh": "4eca7d0152a133183485ab0e15edaed358b33ae180df7b7c8ffcbda8dbc07bad", + "benchmark_lib.sh": "9b8a17a79925b730149a99fd0dfa01d2ac81c313fec4665cecb04356fe740999", + "client-lengths-replay.json": "64ae4fe13e850d3a031a61deb2258796ead09d3514fa6f2bfec2dff8d01bd85f", + "client-lengths.jsonl": "5eba733ff20b8039910d10b248d46676506814658fa59f9c8d8395e603ae4bd7", + "client-w4a16_mxfp4.report.json.gz": "0b21be3052908406a9c855af1065d0de611ad5af71de2dc567feb199832459ce", + "client-w4a8_mxfp4_mxfp8.report.json.gz": "c34c8076a85d7bb87f2048ec7fc8520c12d228088f5d2578296640010237b7c0", + "comparison.json": "5bf7bc492c1b831b2c126736f84fe631c7d6a32344b6f5e308374f9140f1eff8", + "diagnostic/client.log": "3de3c9248a0530da9c022ae152d9dad0462a806e34bd14990c88eebc9974d379", + "diagnostic/gpu-before.txt": "247d5149c53245c9430b74073e82737713779907b2b82b3af9a86489f7350824", + "diagnostic/result.json": "b5ef8da52ab6987e25d6613e4f8004053d92bf3bc0cf44a33ca0d5d00fbdffbb", + "diagnostic/routing/hooks-loaded-3795361.json": "ac457acb83088743ec88696207f7201016790a8adbdf44293554e06c6103d037", + "diagnostic/routing/hooks-loaded-3795575.json": "3d841e0af540b58a95112059cae199b359a3e39e6aab7c624b615920aafe0cd4", + "diagnostic/runtime.json": "870e0ebe2f083bf2fc1656672411f422199a92456e80460480532e825c24cc42", + "diagnostic/server.log": "ce85439c24ff8717f30bcad536adcb215c355e47e2b1e9d880503abc330fd269", + "diagnostic/telemetry.csv": "a5115af9912281264ffa70fb1165485979c9b173ec49d0bc176eec3ba47d803b", + "diagnostic/trace-rank-0.json": "1982fe62cf41ed26ffe8c55aa1b4d2f83af0f63238532c521a9aa6af8e350c88", + "diagnostic-inner.sh": "3a357b1e6283d185a33f8615717731077c0c64eb214cf3d3f8f25ac7f3e94a28", + "hooks/sitecustomize.py": "33bbd70ab55ca5c26c47addff9d9e7f67b96399b11f73c949a3cc9b1565e9722", + "hooks-v2/sitecustomize.py": "bd51deb5bf93f573a9f65113864985126b07576d926594e2f0db7c78f5566606", + "inferencex/utils/bench_serving/backend_request_func.py": "4ebf482eb32685e59ed69bd39c51e9f1fd2134f5e9b8fe7ad10550d40ecca16c", + "inferencex/utils/bench_serving/benchmark_serving.py": "67077f1e7abef278faa367237e33143135032c65c27032100cf26e82de37d086", + "inferencex/utils/bench_serving/benchmark_utils.py": "b99a18940871b701e9e65c2cc098da1c5d33fc3f9d3890f8d2cddc539a9e3d51", + "inferencex/utils/bench_serving/encoding_dsv4.py": "bdbd57c132a1b3725042323d02b98b9d1df28e5f388f134399555d041f5055e0", + "job-status.txt": "b6c196c1592a00d469ae67bd84ae6556751145102cc84c06e498fa7ce2d3d3d5", + "manifest.json": "4f903fa8e54e4b5d734ebc72dafe94cb7be8c0257f0b36ebfbaefb0862990f10", + "model-manifest.json": "823cffec3f83f3bc22e766ba335376a33a8dd466897ee070e57508a99ecfbf51", + "model-metadata.json": "cc6570312b639a5b13b6997c6a333deb696a628e7e8326c3f818a969d985e1d3", + "prepare-model.sh": "f43f7d0bf16ad6f4e4a637227bdd8f7bbc0ca2a68387520009197b579d9c0fe9", + "replay-client-lengths.log": "32ad3b4ed26bbaf0379b4806df752fba4cd62d28212ed75ab2598d0108edf2d8", + "replay-client-lengths.py": "bb682b61d25202cd7f4683e6efe1344ce37f3cb128b556d99897c7372078c671", + "replay-quantization.log": "62f2f5b9d0960157c4466523769da2094954c11e6011770e3e1277e019a39993", + "replay-quantization.py": "c5e5ad82cbf46dca0a18921fe987a401ffba2fada1c9af4ae523f814df603ef1", + "routing-inner.sh": "05ce9a843a0b711d8631c0c062085a30832cfcf4ee52212aeef3c4f89b0b182f", + "routing-snapshots/model-1236947.json": "1453edd57dc0ac5346271f3cd1085790138fe067bc349088fdc5361f67717058", + "routing-snapshots/routing-1236947-100.json": "ac9f179a34f02c46d1c37731adea8f42bb53746004f4ecc87718142ef7ecb277", + "routing-snapshots/routing-1236947-100.pt": "2fd73e3559e54a4898ce165420c53f973678f4b77b981233e31ce85908758369", + "routing-snapshots/routing-1236947-150.json": "2f69ecc74fa932efba1bc120423f7db66e317c4e4e85c3f24a01c4c526317154", + "routing-snapshots/routing-1236947-150.pt": "eb9572487558a278dea9eb1bd8dad70ef3a6133cc40904f574d9f266a09fcdc4", + "routing-snapshots/routing-1236947-200.json": "9c36ed2b06bfe28a8aa1c28704b7bfe0ca3ee81750263fd286cfcaebf5e29316", + "routing-snapshots/routing-1236947-200.pt": "f5791c47cc76105d89a4a33780a260026c3fab8beba12f963de0ebaf388d17c7", + "routing-snapshots/state-1234272.json": "95fb663df229ea4fc9e81a735125b72850a9e9b4bfb39fe81d80792e8a23d5db", + "routing-snapshots/state-1236947.json": "20253369a776e1260fa49838b4b8a1a1fba929a4d8e185f5be22c27560fca103", + "routing-summary.json": "838756949109b5b1b9a194be4dd6c9cfea38311f4572d7369d5ca53e27e2beea", + "run-baseline.sh": "49a8186c879b89d41b87aa5dac32fed64ac8a16832eb464e92141da3e02704ef", + "run-diagnostic.sh": "a9ccb10f5c347a6d127dbe46873a6e5f72301bcf4c4dbecb1b5bfeb309174470", + "run-routing.sh": "1afec2cd34b920dc45b86e947e22e0436e3cfddd6198b5043f95a9ea3a8e2367", + "target.json": "255c9aa06ab68f548849ff3a24379620e564b224f666702ac2680049c4fd6d86", + "trace-rank-0.json": "1982fe62cf41ed26ffe8c55aa1b4d2f83af0f63238532c521a9aa6af8e350c88", + "trace-summary.json": "a7c08c68ec6874151b71e05e7890708471a99f5639a915ee1a06230c83f4e95a", + "upstream/cuda_graph_runner.py": "de98a824218958725db8705af01b4e55d9425c86e38b8fb34f9652cc309d4cda", + "upstream/fused_moe_trtllm_gen.py": "4c83eb8113f1c007d761718df8acaa05e4d6de22bbd0430281e1a9210c2d9751", + "upstream/image-cuda_graph_runner.py": "de98a824218958725db8705af01b4e55d9425c86e38b8fb34f9652cc309d4cda", + "upstream/image-fused_moe_trtllm_gen.py": "4c83eb8113f1c007d761718df8acaa05e4d6de22bbd0430281e1a9210c2d9751", + "upstream/interface.py": "1b4ea1e275f1fa998c8cc3e2a529da132dfd65891c0a2403cf367cf7b6f80be8", + "upstream/openai_server.py": "60ff86b178b1142f151ec59cf14f17121238c625ca7e4cfd3e1decfb2f99285f", + "upstream/profiler.py": "58e3e5f2e06e1980438ff96182216ffc0ea433f52163159bc7c3d272387959b4", + "upstream/py_executor.py": "4d3929d295a2bc65380cf3c4a463b3eb1a0e14d80c3acb95b0b78485f36ef1fe", + "upstream/serve.py": "2ad5bdd864fea525aa42b3d2310ab4f781100f037af31413188dd251da568dec", + "w4a8-decode-breakdown.json": "5eb972a5d19138b3fed6e114189475d33240c68a27dcc02ac4464b0a656fa84f", + "w4a8-replay.json": "04eed6913cc7d452f5745f543ad9ca1a79ded9a595f6e0587d2ef51c526cfda2", + "w4a8.report.json.gz": "891f58841d656dc4ae27278d6f9b8c4160367f7a17fb6f2969e39c6cc40350f5", + "w4a8.spec.json": "86e35e11a163a9c71e1847c4b3b79fd5278294fb8c5d98630777808f8cd6a99f" + } +} diff --git a/python/aisimulate/collector/trtllm/gym-tpot-serving-20260917.md b/python/aisimulate/collector/trtllm/gym-tpot-serving-20260917.md new file mode 100644 index 000000000..1cb0d172e --- /dev/null +++ b/python/aisimulate/collector/trtllm/gym-tpot-serving-20260917.md @@ -0,0 +1,120 @@ +# GPT-OSS/B200 serving reproduction + +## Finding + +The reproduced rc14 server uses **W4A8 MXFP4/MXFP8 MoE**, while the frozen +Gym specification explicitly requests **W4A16 MXFP4**. The checkpoint declares +MXFP4 weights; TRT-LLM chooses activation quantization at runtime. On SM100, +`ModelConfig.get_mxfp4_quant_algo()` selects `W4A8_MXFP4_MXFP8` for the TRTLLM +backend unless `OVERRIDE_QUANT_ALGO` is set. The original recipe sets no such +override. Both the real server log and GPU kernel trace confirm that path. + +The earlier bias/routing/autotuning experiments used the W4A16 operator. +They establish a valid collector improvement, but do not establish full +GPT-OSS/B200 serving parity. This reproduction supersedes that interpretation. + +## Matched point + +- Original InferenceX config131: GPT-OSS-120B, one B200, TP1/EP1, ISL/OSL 1024, + concurrency 256, 2,560 requests, random range ratio 0.8, seed 0. +- Recipe/client: SemiAnalysisAI/InferenceX + `2baba8e27be8529b4453afc953f9859ec092c73d`; unmodified benchmark client. + Preserve 512 warmups, ignore-EOS, streaming interval 20, FP8 KV, max batch 512, + max sequence 2304, max tokens 20000, graph padding through batch 256, and PDL1. +- Model revision: `b5c939de8f754692c1647ca79fbf85e8c1e70f8a`; configuration + exactly matches the frozen replay checkpoint configuration. CPU job 1994868 + verified all 15 checkpoint shards; completed 0:0. +- rc14 amd 64 image: + `sha256:fe2f17d0c9698bafeb9f437929003c6badc56d17e8382a02505233145a07f61f`. + Source `93cb6518b6d6dbd6095748189e626db731f44545`. +- Uninstrumented baseline job 1994872, one B200 on gpu78, completed 0:0 in 6m30s. + All 2,560 requests succeeded; complete per-request token counts retained. + Adaptations are local model/output paths, localhost endpoint, detailed output, + CPU/GPU allocation and storage. No serving algorithm or quantization override. +- The historical model revision and image digest are unavailable. This run + pins the replay's resolved checkpoint and the current rc14 image digest; + it does not prove historical artifact identity. + +## Results + +| Run | TTFT (ms) | TPOT (ms) | TPOT APE versus reproduction | +| --- | ---: | ---: | ---: | +| Historical silicon | 183.61345 | 16.79212 | — | +| Uninstrumented reproduction | 189.62701 | 16.54925 | — | +| Replay, W4A16, exact client lengths | 487.35848 | 65.16758 | 293.78% | +| Replay, W4A8, exact client lengths | 120.92177 | 17.17212 | 3.76% | + +The reproduced TPOT is 1.45% below the historical value. The two replay arms +use the same exact 2,560 client input/output lengths, concurrency and scheduler +settings; only `aic_moe_dtype` changes. Runtime remains +`89d2051137b772944a3e303c452108811790bef4`; performance data remains PR#264 +`a85b9e10f5117439eafd7a5d113118b781229920`. No source/table mutation or +latency scaling is applied. Corrected TTFT still has 36.23% error. + +A separate first ablation preserves the frozen synthetic workload and changes +only precision: TPOT 65.15635→17.17169ms. Its request lengths are identical +between arms but differ from the real client: the frozen runner uses Python +`random.Random` even though the workload contains `length_sampler=numpy_random_state`. +The exact-client-length ablation changes TPOT by only 0.00043ms for W4A8. +This excludes that sampler mismatch as the cause of this point's large gap. + +## Kernel and step checks + +- The original `print_iter_log` output yields 9,233 pure-decode batch 256 + iterations. Restricting to adjacent pure-decode iterations aligns the + previous-device event correctly: 8,246 samples, median 14.05802ms. +- Diagnostic job 1994898 completed 0:0 in 5m59s on the same B200 node. Native + profiler range 3000–3010 produced 11 graph replays and 396 MoE layer instances. + The diagnostic E2E score is not used as silicon truth. +- Each layer uses native `bmm_MxE4m3_MxE2m1MxE4m3...` followed by + `bmm_Bfloat16_MxE2m1MxE4m3...`. The latter's BF16 names the output; + its input is still MXFP8. Quantization kernels are present. +- MoE GEMM, routing, quantization and finalize kernels sum to 10.01317ms + per decode step, or 0.27814ms per layer. These are summed GPU durations, + not an asserted full critical path. The selected W4A8 table row is 0.265798ms; + the W4A16 row is 1.336288ms for the same logical batch 256 shape. +- The initial routing hook produced no snapshots. Its missing result is not + treated as routing agreement; the retry and its outcome are recorded below. + +## Live routing snapshots + +- Retry job `1995012` completed `0:0` in 6m16s. It captures router outputs in the decode CUDA graph and + saves them after live replays 100, 150 and 200. All three samples contain + all 36 layers at batch 256, with 128 experts and top-k 4 (108 layer samples). +- Across the 108 samples, active experts range from 77 to 116 (median 105); + the hottest expert receives 45–198 of 256 tokens (median 84.5). +- The model itself reports `W4A8_MXFP4_MXFP8`. Input/output request lengths + retain the original benchmark sampling; this diagnostic omits the 512 client + warmups and is not used for the E2E accuracy comparison. +- Reconstructed top-k histograms are retained with every snapshot. Native + tie-breaking may differ from `torch.topk`; no independent routing-distribution + latency contribution is claimed. Full logits remain in the raw artifacts. + +## Scope and next correction + +This confirms the activation-precision mapping defect for the reproduced TP1 +point. It does not produce a new 31-point or 263-point MAPE. The automatic Gym +mapping still needs a backend/version/architecture-aware default rule, honoring +explicit runtime overrides. Before the full rerun, collect missing W4A8 TP2/4/8 +profiles under their correct keys; do not relabel the prior W4A16 measurements. +Hopper defaults and non-TRT backends must remain independently resolved. + +## Evidence + +- Source rule: https://github.com/NVIDIA/TensorRT-LLM/blob/93cb6518b6d6dbd6095748189e626db731f44545/tensorrt_llm/_torch/model_config.py#L342-L353 +- Original recipe: https://github.com/SemiAnalysisAI/InferenceX/blob/2baba8e27be8529b4453afc953f9859ec092c73d/benchmarks/single_node/gptoss_fp4_b200_trt.sh +- Raw client/server results, telemetry, profile, replay reports and runners: + `${CAMPAIGN_ARTIFACT_ROOT}/serving-repro/`. +- Remote: `${B200_CLUSTER}`, + `${B200_ARTIFACT_ROOT}/results/trt-62-20260917/serving-repro/`. +- `comparison.json`, `trace-summary.json`, `manifest.json`, `model-manifest.json` + and `client-lengths-replay.json` retain the numeric results and provenance. + +[Machine-readable evidence](gym-tpot-serving-20260917.json) records paired +metrics, routing summaries, source identity, and artifact hashes. + +## Full correction follow-up + +The mapping, W4A8 collection, and complete 263-point rerun are recorded in +[the 2026-09-18 report](gym-w4a8-20260918.md). Earlier metrics above remain +historical ablations. diff --git a/python/aisimulate/collector/trtllm/gym-w4a8-20260918-paired.json.gz b/python/aisimulate/collector/trtllm/gym-w4a8-20260918-paired.json.gz new file mode 100644 index 000000000..6fe72b5f9 Binary files /dev/null and b/python/aisimulate/collector/trtllm/gym-w4a8-20260918-paired.json.gz differ diff --git a/python/aisimulate/collector/trtllm/gym-w4a8-20260918.json b/python/aisimulate/collector/trtllm/gym-w4a8-20260918.json new file mode 100644 index 000000000..8492245d0 --- /dev/null +++ b/python/aisimulate/collector/trtllm/gym-w4a8-20260918.json @@ -0,0 +1,1955 @@ +{ + "points": 263, + "successful_points": 263, + "completed_requests": 293180, + "changed_mapping_points": 31, + "precision_data_unchanged_points": 232, + "context_violations": 0, + "truncated_requests": 0, + "groups": { + "trtllm": { + "points": 166, + "mape_pct": { + "original": { + "ttft": 66.24674189164078, + "tpot": 68.1603347011892 + }, + "before": { + "ttft": 65.99932710369264, + "tpot": 66.71573133021964 + }, + "precision_data": { + "ttft": 58.0793887168306, + "tpot": 36.19155227698191 + }, + "client_lengths": { + "ttft": 57.814434932770986, + "tpot": 36.247247332744635 + } + } + }, + "sglang": { + "points": 97, + "mape_pct": { + "original": { + "ttft": 69.60881280689533, + "tpot": 12.550584383154693 + }, + "before": { + "ttft": 69.60881280689533, + "tpot": 12.550584383154693 + }, + "precision_data": { + "ttft": 69.60881280689533, + "tpot": 12.550584383154693 + }, + "client_lengths": { + "ttft": 69.77207211504447, + "tpot": 12.51701769210102 + } + } + }, + "trtllm/DeepSeek-R1-0528/b200": { + "points": 31, + "mape_pct": { + "original": { + "ttft": 64.00001913374423, + "tpot": 49.31907317174891 + }, + "before": { + "ttft": 64.00001913374423, + "tpot": 49.31907317174891 + }, + "precision_data": { + "ttft": 64.00001913374423, + "tpot": 49.31907317174891 + }, + "client_lengths": { + "ttft": 61.57913787090284, + "tpot": 49.18321237640418 + } + } + }, + "trtllm/gpt-oss-120b/h200": { + "points": 33, + "mape_pct": { + "original": { + "ttft": 79.53530478234222, + "tpot": 21.34985420167682 + }, + "before": { + "ttft": 79.53530478234222, + "tpot": 21.34985420167682 + }, + "precision_data": { + "ttft": 79.53530478234222, + "tpot": 21.34985420167682 + }, + "client_lengths": { + "ttft": 79.6762251104632, + "tpot": 21.41316751159219 + } + } + }, + "trtllm/gpt-oss-120b/b200": { + "points": 31, + "mape_pct": { + "original": { + "ttft": 86.99302235262971, + "tpot": 176.51718188221633 + }, + "before": { + "ttft": 85.6681560687784, + "tpot": 168.78156383121805 + }, + "precision_data": { + "ttft": 43.25816341654939, + "tpot": 5.329508255815983 + }, + "client_lengths": { + "ttft": 43.660482275955694, + "tpot": 5.329564895982209 + } + } + }, + "trtllm/DeepSeek-R1-0528/h200": { + "points": 10, + "mape_pct": { + "original": { + "ttft": 39.500050317882646, + "tpot": 24.21499332147967 + }, + "before": { + "ttft": 39.500050317882646, + "tpot": 24.21499332147967 + }, + "precision_data": { + "ttft": 39.500050317882646, + "tpot": 24.21499332147967 + }, + "client_lengths": { + "ttft": 39.86867900785781, + "tpot": 24.351692850180722 + } + } + }, + "sglang/Qwen-3.5-397B-A17B/b200": { + "points": 22, + "mape_pct": { + "original": { + "ttft": 77.85630573824852, + "tpot": 14.591644774189229 + }, + "before": { + "ttft": 77.85630573824852, + "tpot": 14.591644774189229 + }, + "precision_data": { + "ttft": 77.85630573824852, + "tpot": 14.591644774189229 + }, + "client_lengths": { + "ttft": 77.9703954168008, + "tpot": 14.536437292628792 + } + } + }, + "sglang/Qwen-3.5-397B-A17B/h200": { + "points": 10, + "mape_pct": { + "original": { + "ttft": 65.4062591101223, + "tpot": 18.81532309634113 + }, + "before": { + "ttft": 65.4062591101223, + "tpot": 18.81532309634113 + }, + "precision_data": { + "ttft": 65.4062591101223, + "tpot": 18.81532309634113 + }, + "client_lengths": { + "ttft": 65.62642361458465, + "tpot": 18.881058171851908 + } + } + }, + "sglang/Qwen-3.5-397B-A17B/b300": { + "points": 52, + "mape_pct": { + "original": { + "ttft": 63.58086690231202, + "tpot": 9.441198049816474 + }, + "before": { + "ttft": 63.58086690231202, + "tpot": 9.441198049816474 + }, + "precision_data": { + "ttft": 63.58086690231202, + "tpot": 9.441198049816474 + }, + "client_lengths": { + "ttft": 63.77418852894948, + "tpot": 9.386334397620272 + } + } + }, + "sglang/Qwen-3.5-397B-A17B/h100": { + "points": 13, + "mape_pct": { + "original": { + "ttft": 82.99603430814857, + "tpot": 16.71499773692108 + }, + "before": { + "ttft": 82.99603430814857, + "tpot": 16.71499773692108 + }, + "precision_data": { + "ttft": 82.99603430814857, + "tpot": 16.71499773692108 + }, + "client_lengths": { + "ttft": 83.07848125680587, + "tpot": 16.726855792399387 + } + } + }, + "trtllm/MiniMax-M2.5/b300": { + "points": 21, + "mape_pct": { + "original": { + "ttft": 48.53914152597353, + "tpot": 44.033185511753715 + }, + "before": { + "ttft": 48.53914152597353, + "tpot": 44.033185511753715 + }, + "precision_data": { + "ttft": 48.53914152597353, + "tpot": 44.033185511753715 + }, + "client_lengths": { + "ttft": 48.85445174824268, + "tpot": 44.15013574200505 + } + } + }, + "trtllm/MiniMax-M2.5/b200": { + "points": 21, + "mape_pct": { + "original": { + "ttft": 54.47878535865214, + "tpot": 39.1314238913374 + }, + "before": { + "ttft": 54.47878535865214, + "tpot": 39.1314238913374 + }, + "precision_data": { + "ttft": 54.47878535865214, + "tpot": 39.1314238913374 + }, + "client_lengths": { + "ttft": 54.69027656187981, + "tpot": 39.39819565230827 + } + } + }, + "trtllm/Qwen-3.5-397B-A17B/b200": { + "points": 19, + "mape_pct": { + "original": { + "ttft": 59.638570650606255, + "tpot": 85.29156496786527 + }, + "before": { + "ttft": 59.638570650606255, + "tpot": 85.29156496786527 + }, + "precision_data": { + "ttft": 59.638570650606255, + "tpot": 85.29156496786527 + }, + "client_lengths": { + "ttft": 59.596129528996414, + "tpot": 85.39371295403694 + } + } + } + }, + "regressions_over_0_1pp": { + "ttft": [ + { + "index": 28, + "config_id": 155, + "hardware": "b200", + "isl": 1024, + "concurrency": 4, + "before": 5.97067785474809, + "after": 65.78004166605464, + "delta_pp": 59.809363811306554 + }, + { + "index": 99, + "config_id": 283, + "hardware": "b200", + "isl": 1024, + "concurrency": 64, + "before": 28.49436904421173, + "after": 77.53071927757256, + "delta_pp": 49.03635023336083 + }, + { + "index": 31, + "config_id": 155, + "hardware": "b200", + "isl": 1024, + "concurrency": 32, + "before": 16.071851899473355, + "after": 63.55160077985149, + "delta_pp": 47.47974888037814 + }, + { + "index": 29, + "config_id": 155, + "hardware": "b200", + "isl": 1024, + "concurrency": 16, + "before": 16.16960713363794, + "after": 63.567109315979756, + "delta_pp": 47.39750218234182 + }, + { + "index": 30, + "config_id": 155, + "hardware": "b200", + "isl": 1024, + "concurrency": 8, + "before": 17.848165526977674, + "after": 61.963780923136234, + "delta_pp": 44.11561539615856 + }, + { + "index": 98, + "config_id": 262, + "hardware": "b200", + "isl": 1024, + "concurrency": 256, + "before": 27.57118058325296, + "after": 66.49031010820744, + "delta_pp": 38.91912952495448 + }, + { + "index": 51, + "config_id": 139, + "hardware": "b200", + "isl": 8192, + "concurrency": 4, + "before": 16.422305642198626, + "after": 53.46819905670914, + "delta_pp": 37.045893414510516 + }, + { + "index": 55, + "config_id": 133, + "hardware": "b200", + "isl": 1024, + "concurrency": 4, + "before": 38.842919903403846, + "after": 74.10655186198794, + "delta_pp": 35.26363195858409 + }, + { + "index": 53, + "config_id": 133, + "hardware": "b200", + "isl": 8192, + "concurrency": 8, + "before": 19.79045910971629, + "after": 48.22453930362243, + "delta_pp": 28.434080193906137 + }, + { + "index": 52, + "config_id": 133, + "hardware": "b200", + "isl": 8192, + "concurrency": 4, + "before": 22.858031291043574, + "after": 45.656993527679084, + "delta_pp": 22.79896223663551 + }, + { + "index": 36, + "config_id": 155, + "hardware": "b200", + "isl": 8192, + "concurrency": 4, + "before": 32.49841291453372, + "after": 51.540236151022135, + "delta_pp": 19.041823236488412 + }, + { + "index": 50, + "config_id": 139, + "hardware": "b200", + "isl": 1024, + "concurrency": 4, + "before": 61.592629672996516, + "after": 79.17348395815317, + "delta_pp": 17.580854285156654 + }, + { + "index": 32, + "config_id": 155, + "hardware": "b200", + "isl": 1024, + "concurrency": 64, + "before": 42.58792341930815, + "after": 55.80629605943744, + "delta_pp": 13.218372640129296 + }, + { + "index": 56, + "config_id": 133, + "hardware": "b200", + "isl": 8192, + "concurrency": 16, + "before": 34.346978040469665, + "after": 42.463699021410534, + "delta_pp": 8.116720980940869 + }, + { + "index": 54, + "config_id": 133, + "hardware": "b200", + "isl": 8192, + "concurrency": 32, + "before": 38.85456322589779, + "after": 41.33906797415593, + "delta_pp": 2.4845047482581393 + }, + { + "index": 226, + "config_id": 1445, + "hardware": "b200", + "isl": 8192, + "concurrency": 8, + "before": 9.514984104529125, + "after": 11.490623463546967, + "delta_pp": 1.9756393590178423 + }, + { + "index": 3, + "config_id": 75, + "hardware": "b200", + "isl": 8192, + "concurrency": 8, + "before": 15.48665382242986, + "after": 17.45295426984718, + "delta_pp": 1.9663004474173196 + }, + { + "index": 88, + "config_id": 181, + "hardware": "h200", + "isl": 8192, + "concurrency": 8, + "before": 21.983460596258197, + "after": 23.900993398226003, + "delta_pp": 1.9175328019678055 + }, + { + "index": 215, + "config_id": 1388, + "hardware": "b300", + "isl": 8192, + "concurrency": 8, + "before": 5.204261592796897, + "after": 7.076598798303466, + "delta_pp": 1.872337205506569 + }, + { + "index": 103, + "config_id": 473, + "hardware": "b200", + "isl": 8192, + "concurrency": 8, + "before": 30.3549354824556, + "after": 32.02687987162458, + "delta_pp": 1.6719443891689814 + }, + { + "index": 227, + "config_id": 1445, + "hardware": "b200", + "isl": 8192, + "concurrency": 32, + "before": 11.021241196249596, + "after": 12.384596279487921, + "delta_pp": 1.363355083238325 + }, + { + "index": 230, + "config_id": 1445, + "hardware": "b200", + "isl": 8192, + "concurrency": 16, + "before": 11.876960805535827, + "after": 13.163030096630942, + "delta_pp": 1.2860692910951155 + }, + { + "index": 216, + "config_id": 1388, + "hardware": "b300", + "isl": 8192, + "concurrency": 16, + "before": 6.445877249350938, + "after": 7.694280286167966, + "delta_pp": 1.2484030368170274 + }, + { + "index": 253, + "config_id": 1554, + "hardware": "b200", + "isl": 8192, + "concurrency": 8, + "before": 57.631511706137466, + "after": 58.876459485796026, + "delta_pp": 1.2449477796585597 + }, + { + "index": 76, + "config_id": 185, + "hardware": "h200", + "isl": 8192, + "concurrency": 8, + "before": 47.32019587647146, + "after": 48.41767853708987, + "delta_pp": 1.097482660618411 + }, + { + "index": 228, + "config_id": 1445, + "hardware": "b200", + "isl": 8192, + "concurrency": 4, + "before": 12.526094676752766, + "after": 13.549946566265957, + "delta_pp": 1.0238518895131907 + }, + { + "index": 213, + "config_id": 1388, + "hardware": "b300", + "isl": 8192, + "concurrency": 4, + "before": 1.1616080520465188, + "after": 2.17821852068244, + "delta_pp": 1.0166104686359212 + }, + { + "index": 210, + "config_id": 1387, + "hardware": "b300", + "isl": 1024, + "concurrency": 2048, + "before": 276.7408595017022, + "after": 277.72875886555505, + "delta_pp": 0.9878993638528755 + }, + { + "index": 188, + "config_id": 739, + "hardware": "b300", + "isl": 8192, + "concurrency": 8, + "before": 41.550355244495684, + "after": 42.497034357291234, + "delta_pp": 0.9466791127955503 + }, + { + "index": 17, + "config_id": 91, + "hardware": "b200", + "isl": 8192, + "concurrency": 4, + "before": 3.55785707908402, + "after": 4.49777542532064, + "delta_pp": 0.9399183462366199 + }, + { + "index": 86, + "config_id": 181, + "hardware": "h200", + "isl": 8192, + "concurrency": 4, + "before": 22.247349164850508, + "after": 23.13597536405725, + "delta_pp": 0.8886261992067404 + }, + { + "index": 148, + "config_id": 716, + "hardware": "b300", + "isl": 8192, + "concurrency": 32, + "before": 37.07556031428032, + "after": 37.909274223954036, + "delta_pp": 0.833713909673719 + }, + { + "index": 147, + "config_id": 716, + "hardware": "b300", + "isl": 8192, + "concurrency": 8, + "before": 50.61250685967062, + "after": 51.4186663914128, + "delta_pp": 0.8061595317421819 + }, + { + "index": 162, + "config_id": 737, + "hardware": "b300", + "isl": 8192, + "concurrency": 8, + "before": 49.93088905267438, + "after": 50.72340542798364, + "delta_pp": 0.7925163753092619 + }, + { + "index": 100, + "config_id": 473, + "hardware": "b200", + "isl": 8192, + "concurrency": 32, + "before": 270.60145555777746, + "after": 271.3713442897109, + "delta_pp": 0.7698887319334631 + }, + { + "index": 173, + "config_id": 738, + "hardware": "b300", + "isl": 8192, + "concurrency": 8, + "before": 51.96345165861449, + "after": 52.729780686190416, + "delta_pp": 0.766329027575928 + }, + { + "index": 202, + "config_id": 1385, + "hardware": "b300", + "isl": 8192, + "concurrency": 4, + "before": 18.01381405223787, + "after": 18.758160098349332, + "delta_pp": 0.7443460461114633 + }, + { + "index": 128, + "config_id": 621, + "hardware": "h200", + "isl": 8192, + "concurrency": 32, + "before": 43.85797679914752, + "after": 44.58923733120259, + "delta_pp": 0.7312605320550674 + }, + { + "index": 222, + "config_id": 1390, + "hardware": "b300", + "isl": 8192, + "concurrency": 4, + "before": 28.529748334326552, + "after": 29.21008011661489, + "delta_pp": 0.6803317822883379 + }, + { + "index": 243, + "config_id": 1450, + "hardware": "b200", + "isl": 8192, + "concurrency": 4, + "before": 24.791244189831993, + "after": 25.47156877878274, + "delta_pp": 0.6803245889507465 + }, + { + "index": 5, + "config_id": 75, + "hardware": "b200", + "isl": 8192, + "concurrency": 4, + "before": 28.303764089258287, + "after": 28.971049324440965, + "delta_pp": 0.6672852351826783 + }, + { + "index": 183, + "config_id": 739, + "hardware": "b300", + "isl": 8192, + "concurrency": 4, + "before": 39.31629757903136, + "after": 39.977873585618894, + "delta_pp": 0.6615760065875378 + }, + { + "index": 137, + "config_id": 716, + "hardware": "b300", + "isl": 8192, + "concurrency": 4, + "before": 41.16910441398883, + "after": 41.82262572169621, + "delta_pp": 0.6535213077073792 + }, + { + "index": 133, + "config_id": 621, + "hardware": "h200", + "isl": 8192, + "concurrency": 8, + "before": 59.03411780119161, + "after": 59.683558341395084, + "delta_pp": 0.6494405402034715 + }, + { + "index": 235, + "config_id": 1447, + "hardware": "b200", + "isl": 8192, + "concurrency": 1024, + "before": 115.00987313787351, + "after": 115.65843246958252, + "delta_pp": 0.6485593317090093 + }, + { + "index": 106, + "config_id": 587, + "hardware": "b200", + "isl": 8192, + "concurrency": 8, + "before": 61.29812492104919, + "after": 61.9138435718196, + "delta_pp": 0.6157186507704111 + }, + { + "index": 239, + "config_id": 1448, + "hardware": "b200", + "isl": 8192, + "concurrency": 4, + "before": 29.365882885343126, + "after": 29.970372724208534, + "delta_pp": 0.6044898388654083 + }, + { + "index": 171, + "config_id": 738, + "hardware": "b300", + "isl": 8192, + "concurrency": 32, + "before": 54.396921190513275, + "after": 54.992730146248434, + "delta_pp": 0.5958089557351585 + }, + { + "index": 163, + "config_id": 737, + "hardware": "b300", + "isl": 8192, + "concurrency": 32, + "before": 54.3652166564798, + "after": 54.95830415198421, + "delta_pp": 0.5930874955044061 + }, + { + "index": 182, + "config_id": 739, + "hardware": "b300", + "isl": 8192, + "concurrency": 32, + "before": 55.65495171295565, + "after": 56.239504996503676, + "delta_pp": 0.5845532835480256 + }, + { + "index": 4, + "config_id": 75, + "hardware": "b200", + "isl": 8192, + "concurrency": 64, + "before": 0.5926755631872691, + "after": 1.1694020465583048, + "delta_pp": 0.5767264833710357 + }, + { + "index": 252, + "config_id": 1554, + "hardware": "b200", + "isl": 8192, + "concurrency": 32, + "before": 48.17487523437117, + "after": 48.746897557993044, + "delta_pp": 0.5720223236218729 + }, + { + "index": 25, + "config_id": 125, + "hardware": "h200", + "isl": 8192, + "concurrency": 8, + "before": 71.6330577077511, + "after": 72.20449500294133, + "delta_pp": 0.571437295190222 + }, + { + "index": 175, + "config_id": 738, + "hardware": "b300", + "isl": 8192, + "concurrency": 4, + "before": 45.84387160863165, + "after": 46.4150814233131, + "delta_pp": 0.5712098146814455 + }, + { + "index": 89, + "config_id": 181, + "hardware": "h200", + "isl": 8192, + "concurrency": 16, + "before": 27.753803720809987, + "after": 28.313952853358092, + "delta_pp": 0.5601491325481049 + }, + { + "index": 144, + "config_id": 716, + "hardware": "b300", + "isl": 8192, + "concurrency": 64, + "before": 31.34944627173344, + "after": 31.893088040243363, + "delta_pp": 0.5436417685099215 + }, + { + "index": 161, + "config_id": 737, + "hardware": "b300", + "isl": 8192, + "concurrency": 4, + "before": 47.96842402572899, + "after": 48.50626405358068, + "delta_pp": 0.5378400278516864 + }, + { + "index": 26, + "config_id": 125, + "hardware": "h200", + "isl": 8192, + "concurrency": 32, + "before": 63.59626061423262, + "after": 64.13280006576257, + "delta_pp": 0.5365394515299471 + }, + { + "index": 108, + "config_id": 587, + "hardware": "b200", + "isl": 8192, + "concurrency": 32, + "before": 60.10284402711899, + "after": 60.62321245595197, + "delta_pp": 0.5203684288329811 + }, + { + "index": 254, + "config_id": 1555, + "hardware": "b200", + "isl": 8192, + "concurrency": 1024, + "before": 32.73529098824319, + "after": 33.243635303921856, + "delta_pp": 0.5083443156786629 + }, + { + "index": 131, + "config_id": 621, + "hardware": "h200", + "isl": 8192, + "concurrency": 4, + "before": 52.831563049083975, + "after": 53.32344321679983, + "delta_pp": 0.4918801677158555 + }, + { + "index": 7, + "config_id": 75, + "hardware": "b200", + "isl": 8192, + "concurrency": 256, + "before": 1.0991653602672433, + "after": 1.5907808535124188, + "delta_pp": 0.4916154932451755 + }, + { + "index": 143, + "config_id": 716, + "hardware": "b300", + "isl": 8192, + "concurrency": 128, + "before": 18.925727396888295, + "after": 19.413105016573276, + "delta_pp": 0.48737761968498106 + }, + { + "index": 22, + "config_id": 125, + "hardware": "h200", + "isl": 8192, + "concurrency": 16, + "before": 65.14620503064666, + "after": 65.61944550099611, + "delta_pp": 0.4732404703494524 + }, + { + "index": 229, + "config_id": 1445, + "hardware": "b200", + "isl": 1024, + "concurrency": 8, + "before": 7.98662605055114, + "after": 8.452820095583037, + "delta_pp": 0.4661940450318971 + }, + { + "index": 201, + "config_id": 1359, + "hardware": "h100", + "isl": 8192, + "concurrency": 8, + "before": 69.86024527077191, + "after": 70.32574659722633, + "delta_pp": 0.4655013264544152 + }, + { + "index": 136, + "config_id": 621, + "hardware": "h200", + "isl": 8192, + "concurrency": 64, + "before": 38.83778852826231, + "after": 39.30052733490425, + "delta_pp": 0.4627388066419442 + }, + { + "index": 111, + "config_id": 587, + "hardware": "b200", + "isl": 8192, + "concurrency": 4, + "before": 57.00036269951067, + "after": 57.45096788733133, + "delta_pp": 0.4506051878206634 + }, + { + "index": 199, + "config_id": 1359, + "hardware": "h100", + "isl": 8192, + "concurrency": 4, + "before": 54.51612869576057, + "after": 54.961954014147615, + "delta_pp": 0.445825318387044 + }, + { + "index": 69, + "config_id": 180, + "hardware": "h200", + "isl": 8192, + "concurrency": 8, + "before": 79.05050164676953, + "after": 79.48438529417018, + "delta_pp": 0.43388364740064844 + }, + { + "index": 68, + "config_id": 180, + "hardware": "h200", + "isl": 8192, + "concurrency": 32, + "before": 74.25170567618721, + "after": 74.64776820511126, + "delta_pp": 0.39606252892404825 + }, + { + "index": 224, + "config_id": 1444, + "hardware": "b200", + "isl": 1024, + "concurrency": 8, + "before": 32.463715057025844, + "after": 32.85369731641274, + "delta_pp": 0.38998225938689757 + }, + { + "index": 115, + "config_id": 615, + "hardware": "b200", + "isl": 8192, + "concurrency": 32, + "before": 70.79151772916732, + "after": 71.18113667525729, + "delta_pp": 0.3896189460899677 + }, + { + "index": 97, + "config_id": 178, + "hardware": "b200", + "isl": 1024, + "concurrency": 64, + "before": 19.741311994050815, + "after": 20.12950397335726, + "delta_pp": 0.3881919793064448 + }, + { + "index": 214, + "config_id": 1388, + "hardware": "b300", + "isl": 1024, + "concurrency": 4, + "before": 7.853875982919849, + "after": 8.240597276860294, + "delta_pp": 0.38672129394044497 + }, + { + "index": 187, + "config_id": 739, + "hardware": "b300", + "isl": 8192, + "concurrency": 64, + "before": 51.6761763065443, + "after": 52.053748650051546, + "delta_pp": 0.37757234350724644 + }, + { + "index": 96, + "config_id": 192, + "hardware": "b200", + "isl": 1024, + "concurrency": 256, + "before": 47.01864846638594, + "after": 47.38266250688673, + "delta_pp": 0.3640140405007912 + }, + { + "index": 172, + "config_id": 738, + "hardware": "b300", + "isl": 8192, + "concurrency": 64, + "before": 52.303639715671444, + "after": 52.667536499341644, + "delta_pp": 0.3638967836701994 + }, + { + "index": 87, + "config_id": 181, + "hardware": "h200", + "isl": 1024, + "concurrency": 32, + "before": 48.68045072664363, + "after": 49.04282208964024, + "delta_pp": 0.3623713629966119 + }, + { + "index": 164, + "config_id": 737, + "hardware": "b300", + "isl": 8192, + "concurrency": 64, + "before": 54.799345025263776, + "after": 55.13935345951666, + "delta_pp": 0.34000843425288707 + }, + { + "index": 113, + "config_id": 587, + "hardware": "b200", + "isl": 8192, + "concurrency": 64, + "before": 55.77962807729694, + "after": 56.115454626368994, + "delta_pp": 0.33582654907205267 + }, + { + "index": 221, + "config_id": 1390, + "hardware": "b300", + "isl": 1024, + "concurrency": 8, + "before": 31.967359372497388, + "after": 32.296476290195784, + "delta_pp": 0.32911691769839635 + }, + { + "index": 95, + "config_id": 192, + "hardware": "b200", + "isl": 8192, + "concurrency": 256, + "before": 34.67543766793305, + "after": 35.0035993655575, + "delta_pp": 0.328161697624445 + }, + { + "index": 66, + "config_id": 180, + "hardware": "h200", + "isl": 8192, + "concurrency": 16, + "before": 76.6522619840153, + "after": 76.97247335468657, + "delta_pp": 0.3202113706712737 + }, + { + "index": 6, + "config_id": 75, + "hardware": "b200", + "isl": 1024, + "concurrency": 128, + "before": 58.05010922454539, + "after": 58.36261160107507, + "delta_pp": 0.31250237652968593 + }, + { + "index": 11, + "config_id": 82, + "hardware": "b200", + "isl": 1024, + "concurrency": 8, + "before": 45.16072270688968, + "after": 45.465951542174246, + "delta_pp": 0.30522883528456646 + }, + { + "index": 165, + "config_id": 737, + "hardware": "b300", + "isl": 8192, + "concurrency": 128, + "before": 46.26792027133297, + "after": 46.569081958745045, + "delta_pp": 0.30116168741207616 + }, + { + "index": 94, + "config_id": 181, + "hardware": "h200", + "isl": 1024, + "concurrency": 8, + "before": 54.03053723173651, + "after": 54.29837011757315, + "delta_pp": 0.2678328858366399 + }, + { + "index": 211, + "config_id": 1387, + "hardware": "b300", + "isl": 8192, + "concurrency": 128, + "before": 61.51791290609244, + "after": 61.78574175991519, + "delta_pp": 0.2678288538227491 + }, + { + "index": 218, + "config_id": 1389, + "hardware": "b300", + "isl": 8192, + "concurrency": 128, + "before": 62.23888069278896, + "after": 62.503026135505515, + "delta_pp": 0.26414544271655416 + }, + { + "index": 23, + "config_id": 125, + "hardware": "h200", + "isl": 8192, + "concurrency": 4, + "before": 73.3027810504579, + "after": 73.56633939310883, + "delta_pp": 0.26355834265093847 + }, + { + "index": 60, + "config_id": 179, + "hardware": "h200", + "isl": 8192, + "concurrency": 32, + "before": 83.5588550578515, + "after": 83.81154946613509, + "delta_pp": 0.25269440828358825 + }, + { + "index": 209, + "config_id": 1387, + "hardware": "b300", + "isl": 8192, + "concurrency": 256, + "before": 62.033057736360405, + "after": 62.28545064420839, + "delta_pp": 0.25239290784798385 + }, + { + "index": 57, + "config_id": 179, + "hardware": "h200", + "isl": 8192, + "concurrency": 16, + "before": 83.0934011976221, + "after": 83.3408287496737, + "delta_pp": 0.2474275520516045 + }, + { + "index": 58, + "config_id": 179, + "hardware": "h200", + "isl": 8192, + "concurrency": 8, + "before": 88.47795553979431, + "after": 88.72436115999437, + "delta_pp": 0.24640562020006485 + }, + { + "index": 258, + "config_id": 1556, + "hardware": "b200", + "isl": 8192, + "concurrency": 4, + "before": 53.424151152617306, + "after": 53.669897240338095, + "delta_pp": 0.24574608772078932 + }, + { + "index": 116, + "config_id": 615, + "hardware": "b200", + "isl": 8192, + "concurrency": 64, + "before": 69.7648116658912, + "after": 70.00850294710987, + "delta_pp": 0.24369128121867334 + }, + { + "index": 2, + "config_id": 75, + "hardware": "b200", + "isl": 1024, + "concurrency": 8, + "before": 57.381431738612186, + "after": 57.611497427345206, + "delta_pp": 0.23006568873302058 + }, + { + "index": 262, + "config_id": 1558, + "hardware": "b200", + "isl": 8192, + "concurrency": 4, + "before": 55.03947200008046, + "after": 55.26926676513668, + "delta_pp": 0.22979476505621932 + }, + { + "index": 104, + "config_id": 473, + "hardware": "b200", + "isl": 1024, + "concurrency": 8, + "before": 53.111633271825376, + "after": 53.33147865035794, + "delta_pp": 0.21984537853256114 + }, + { + "index": 125, + "config_id": 615, + "hardware": "b200", + "isl": 8192, + "concurrency": 128, + "before": 65.09089767377358, + "after": 65.3057784307248, + "delta_pp": 0.21488075695121722 + }, + { + "index": 240, + "config_id": 1449, + "hardware": "b200", + "isl": 8192, + "concurrency": 256, + "before": 30.168270402846655, + "after": 30.382007073223807, + "delta_pp": 0.21373667037715194 + }, + { + "index": 67, + "config_id": 180, + "hardware": "h200", + "isl": 8192, + "concurrency": 4, + "before": 81.03362087410416, + "after": 81.22816982555118, + "delta_pp": 0.19454895144701823 + }, + { + "index": 236, + "config_id": 1447, + "hardware": "b200", + "isl": 8192, + "concurrency": 256, + "before": 50.91967034490572, + "after": 51.107552419639006, + "delta_pp": 0.1878820747332881 + }, + { + "index": 64, + "config_id": 179, + "hardware": "h200", + "isl": 1024, + "concurrency": 8, + "before": 66.26084736634235, + "after": 66.43059484336801, + "delta_pp": 0.16974747702566617 + }, + { + "index": 75, + "config_id": 180, + "hardware": "h200", + "isl": 1024, + "concurrency": 8, + "before": 89.91444252816709, + "after": 90.07491172226065, + "delta_pp": 0.1604691940935652 + }, + { + "index": 248, + "config_id": 1553, + "hardware": "b200", + "isl": 8192, + "concurrency": 256, + "before": 60.8293310536731, + "after": 60.98887533717333, + "delta_pp": 0.15954428350023164 + }, + { + "index": 197, + "config_id": 1359, + "hardware": "h100", + "isl": 8192, + "concurrency": 1, + "before": 69.19156277849945, + "after": 69.35104478715147, + "delta_pp": 0.1594820086520201 + }, + { + "index": 152, + "config_id": 736, + "hardware": "b300", + "isl": 1024, + "concurrency": 8, + "before": 76.190881946654, + "after": 76.34631547823663, + "delta_pp": 0.15543353158263074 + }, + { + "index": 244, + "config_id": 1552, + "hardware": "b200", + "isl": 8192, + "concurrency": 4, + "before": 69.16206840255724, + "after": 69.31054420558611, + "delta_pp": 0.1484758030288731 + }, + { + "index": 90, + "config_id": 181, + "hardware": "h200", + "isl": 8192, + "concurrency": 32, + "before": 22.724693723468995, + "after": 22.867431581918048, + "delta_pp": 0.14273785844905262 + }, + { + "index": 18, + "config_id": 125, + "hardware": "h200", + "isl": 1024, + "concurrency": 8, + "before": 76.78102599235402, + "after": 76.91659743689458, + "delta_pp": 0.13557144454055958 + }, + { + "index": 129, + "config_id": 621, + "hardware": "h200", + "isl": 1024, + "concurrency": 8, + "before": 85.0309707518985, + "after": 85.1606453154491, + "delta_pp": 0.12967456355060847 + }, + { + "index": 142, + "config_id": 716, + "hardware": "b300", + "isl": 8192, + "concurrency": 256, + "before": 15.019944958011637, + "after": 15.14729265816176, + "delta_pp": 0.1273477001501231 + }, + { + "index": 181, + "config_id": 739, + "hardware": "b300", + "isl": 1024, + "concurrency": 8, + "before": 67.47395329033063, + "after": 67.586856191448, + "delta_pp": 0.11290290111737988 + }, + { + "index": 166, + "config_id": 737, + "hardware": "b300", + "isl": 1024, + "concurrency": 8, + "before": 78.0144812419027, + "after": 78.12268030992236, + "delta_pp": 0.10819906801965828 + }, + { + "index": 85, + "config_id": 186, + "hardware": "b200", + "isl": 8192, + "concurrency": 32, + "before": 12.750536819650572, + "after": 12.851725542850176, + "delta_pp": 0.10118872319960381 + } + ], + "tpot": [ + { + "index": 251, + "config_id": 1553, + "hardware": "b200", + "isl": 8192, + "concurrency": 1024, + "before": 182.19831076568258, + "after": 185.90875378217322, + "delta_pp": 3.7104430164906432 + }, + { + "index": 206, + "config_id": 1386, + "hardware": "b300", + "isl": 1024, + "concurrency": 1024, + "before": 89.33602802047682, + "after": 92.75414718724014, + "delta_pp": 3.4181191667633186 + }, + { + "index": 234, + "config_id": 1446, + "hardware": "b200", + "isl": 1024, + "concurrency": 1024, + "before": 78.55899065393774, + "after": 81.80185899981885, + "delta_pp": 3.2428683458811065 + }, + { + "index": 247, + "config_id": 1553, + "hardware": "b200", + "isl": 1024, + "concurrency": 1024, + "before": 86.1159967126461, + "after": 88.07907437874694, + "delta_pp": 1.963077666100844 + }, + { + "index": 82, + "config_id": 191, + "hardware": "b200", + "isl": 8192, + "concurrency": 256, + "before": 335.91649488792063, + "after": 337.7463114631881, + "delta_pp": 1.8298165752674436 + }, + { + "index": 238, + "config_id": 1447, + "hardware": "b200", + "isl": 1024, + "concurrency": 1024, + "before": 71.02836060027391, + "after": 72.70280806455413, + "delta_pp": 1.6744474642802203 + }, + { + "index": 83, + "config_id": 191, + "hardware": "b200", + "isl": 1024, + "concurrency": 256, + "before": 93.4420743647747, + "after": 95.0511948988452, + "delta_pp": 1.6091205340704988 + }, + { + "index": 219, + "config_id": 1389, + "hardware": "b300", + "isl": 1024, + "concurrency": 1024, + "before": 49.96731508115967, + "after": 51.32375739659161, + "delta_pp": 1.3564423154319414 + }, + { + "index": 237, + "config_id": 1447, + "hardware": "b200", + "isl": 8192, + "concurrency": 512, + "before": 119.85242884877128, + "after": 120.82434704224234, + "delta_pp": 0.9719181934710548 + }, + { + "index": 80, + "config_id": 183, + "hardware": "h200", + "isl": 8192, + "concurrency": 64, + "before": 183.09653717172145, + "after": 183.98188114263263, + "delta_pp": 0.8853439709111797 + }, + { + "index": 101, + "config_id": 473, + "hardware": "b200", + "isl": 8192, + "concurrency": 16, + "before": 20.308212860876402, + "after": 21.177873927221448, + "delta_pp": 0.8696610663450457 + }, + { + "index": 236, + "config_id": 1447, + "hardware": "b200", + "isl": 8192, + "concurrency": 256, + "before": 119.79842423120772, + "after": 120.56315582414251, + "delta_pp": 0.764731592934794 + }, + { + "index": 204, + "config_id": 1386, + "hardware": "b300", + "isl": 1024, + "concurrency": 256, + "before": 63.88683489766508, + "after": 64.65039065610421, + "delta_pp": 0.763555758439125 + }, + { + "index": 235, + "config_id": 1447, + "hardware": "b200", + "isl": 8192, + "concurrency": 1024, + "before": 142.3723655362033, + "after": 143.13036498913536, + "delta_pp": 0.7579994529320686 + }, + { + "index": 232, + "config_id": 1446, + "hardware": "b200", + "isl": 1024, + "concurrency": 256, + "before": 61.29240280570985, + "after": 62.00140485628994, + "delta_pp": 0.7090020505800894 + }, + { + "index": 211, + "config_id": 1387, + "hardware": "b300", + "isl": 8192, + "concurrency": 128, + "before": 39.49365399821635, + "after": 40.086005577366954, + "delta_pp": 0.592351579150602 + }, + { + "index": 6, + "config_id": 75, + "hardware": "b200", + "isl": 1024, + "concurrency": 128, + "before": 49.57361625510564, + "after": 50.040923162220835, + "delta_pp": 0.4673069071151943 + }, + { + "index": 165, + "config_id": 737, + "hardware": "b300", + "isl": 8192, + "concurrency": 128, + "before": 24.037888981334284, + "after": 24.482646425039714, + "delta_pp": 0.4447574437054307 + }, + { + "index": 90, + "config_id": 181, + "hardware": "h200", + "isl": 8192, + "concurrency": 32, + "before": 1.4577075452190162, + "after": 1.8867016465635922, + "delta_pp": 0.42899410134457594 + }, + { + "index": 143, + "config_id": 716, + "hardware": "b300", + "isl": 8192, + "concurrency": 128, + "before": 15.268752355441704, + "after": 15.693006402771784, + "delta_pp": 0.42425404733008065 + }, + { + "index": 8, + "config_id": 75, + "hardware": "b200", + "isl": 8192, + "concurrency": 128, + "before": 27.295886260674685, + "after": 27.6737299181995, + "delta_pp": 0.37784365752481364 + }, + { + "index": 115, + "config_id": 615, + "hardware": "b200", + "isl": 8192, + "concurrency": 32, + "before": 18.23583584619898, + "after": 18.551038870090164, + "delta_pp": 0.3152030238911827 + }, + { + "index": 260, + "config_id": 1557, + "hardware": "b200", + "isl": 1024, + "concurrency": 128, + "before": 34.73264112623606, + "after": 35.04475387038968, + "delta_pp": 0.312112744153616 + }, + { + "index": 255, + "config_id": 1555, + "hardware": "b200", + "isl": 1024, + "concurrency": 1024, + "before": 43.14003172686781, + "after": 43.44730478495279, + "delta_pp": 0.30727305808498073 + }, + { + "index": 128, + "config_id": 621, + "hardware": "h200", + "isl": 8192, + "concurrency": 32, + "before": 16.154230141265113, + "after": 16.456197867967475, + "delta_pp": 0.3019677267023617 + }, + { + "index": 227, + "config_id": 1445, + "hardware": "b200", + "isl": 8192, + "concurrency": 32, + "before": 1.4121922773734137, + "after": 1.6668205368593834, + "delta_pp": 0.2546282594859697 + }, + { + "index": 201, + "config_id": 1359, + "hardware": "h100", + "isl": 8192, + "concurrency": 8, + "before": 10.296566724741828, + "after": 10.499214718111253, + "delta_pp": 0.20264799336942474 + }, + { + "index": 217, + "config_id": 1389, + "hardware": "b300", + "isl": 1024, + "concurrency": 2048, + "before": 43.272469922708446, + "after": 43.47256030205818, + "delta_pp": 0.2000903793497315 + }, + { + "index": 25, + "config_id": 125, + "hardware": "h200", + "isl": 8192, + "concurrency": 8, + "before": 16.119312205993356, + "after": 16.31124233562153, + "delta_pp": 0.19193012962817235 + }, + { + "index": 26, + "config_id": 125, + "hardware": "h200", + "isl": 8192, + "concurrency": 32, + "before": 16.771396354853568, + "after": 16.951098996760635, + "delta_pp": 0.17970264190706686 + }, + { + "index": 142, + "config_id": 716, + "hardware": "b300", + "isl": 8192, + "concurrency": 256, + "before": 17.135294612305362, + "after": 17.313767797718338, + "delta_pp": 0.17847318541297597 + }, + { + "index": 68, + "config_id": 180, + "hardware": "h200", + "isl": 8192, + "concurrency": 32, + "before": 27.943337815984194, + "after": 28.11169358053284, + "delta_pp": 0.16835576454864665 + }, + { + "index": 69, + "config_id": 180, + "hardware": "h200", + "isl": 8192, + "concurrency": 8, + "before": 27.19710131323, + "after": 27.3545655292152, + "delta_pp": 0.15746421598520044 + }, + { + "index": 60, + "config_id": 179, + "hardware": "h200", + "isl": 8192, + "concurrency": 32, + "before": 39.37476345547735, + "after": 39.5301696091605, + "delta_pp": 0.15540615368315258 + }, + { + "index": 209, + "config_id": 1387, + "hardware": "b300", + "isl": 8192, + "concurrency": 256, + "before": 35.598021143345, + "after": 35.74644254030619, + "delta_pp": 0.14842139696119006 + }, + { + "index": 117, + "config_id": 615, + "hardware": "b200", + "isl": 8192, + "concurrency": 16, + "before": 19.728970413856032, + "after": 19.868169114739555, + "delta_pp": 0.13919870088352226 + }, + { + "index": 89, + "config_id": 181, + "hardware": "h200", + "isl": 8192, + "concurrency": 16, + "before": 10.620141416469417, + "after": 10.757437677321581, + "delta_pp": 0.13729626085216395 + }, + { + "index": 133, + "config_id": 621, + "hardware": "h200", + "isl": 8192, + "concurrency": 8, + "before": 15.662178232976236, + "after": 15.78531500600482, + "delta_pp": 0.12313677302858395 + }, + { + "index": 132, + "config_id": 621, + "hardware": "h200", + "isl": 8192, + "concurrency": 16, + "before": 10.408437001647666, + "after": 10.53089384055615, + "delta_pp": 0.12245683890848369 + }, + { + "index": 27, + "config_id": 125, + "hardware": "h200", + "isl": 8192, + "concurrency": 64, + "before": 9.466921277285723, + "after": 9.585447549847503, + "delta_pp": 0.11852627256178039 + }, + { + "index": 61, + "config_id": 179, + "hardware": "h200", + "isl": 1024, + "concurrency": 32, + "before": 31.78742504577217, + "after": 31.903021379144967, + "delta_pp": 0.11559633337279607 + }, + { + "index": 70, + "config_id": 180, + "hardware": "h200", + "isl": 8192, + "concurrency": 64, + "before": 20.435810142801905, + "after": 20.546047599046116, + "delta_pp": 0.11023745624421011 + }, + { + "index": 92, + "config_id": 181, + "hardware": "h200", + "isl": 1024, + "concurrency": 4, + "before": 11.94599650609176, + "after": 12.05563163121597, + "delta_pp": 0.10963512512421048 + }, + { + "index": 76, + "config_id": 185, + "hardware": "h200", + "isl": 8192, + "concurrency": 8, + "before": 7.21602866093074, + "after": 7.3247042663290705, + "delta_pp": 0.10867560539833043 + }, + { + "index": 95, + "config_id": 192, + "hardware": "b200", + "isl": 8192, + "concurrency": 256, + "before": 148.63735061978912, + "after": 148.7434150619434, + "delta_pp": 0.10606444215429178 + }, + { + "index": 97, + "config_id": 178, + "hardware": "b200", + "isl": 1024, + "concurrency": 64, + "before": 3.510543921970952, + "after": 3.611142997944822, + "delta_pp": 0.10059907597386974 + } + ] + }, + "collection": { + "collector_ref": "3830b5a4", + "summary": { + "planned": 6, + "passed": 6, + "failed": 0, + "unattempted": 0, + "worker_exit_codes": [ + 0, + 0 + ], + "runner_sha256": "4c4c3584c2f69d89895a7cd749350f1854b2247775a62d2e488905f20efd4dc7" + }, + "cuda_graph_measurements": 162, + "replaced_rows": 81, + "added_rows": 81, + "unchanged_rows": 218295, + "old_parquet_sha256": "1ff05df3a5715ca5d80e2435ba5ec8221e25da7f414ae7834a87af84c46b9b87", + "new_parquet_sha256": "db1930fde20b419284bf1f038fadd4b398d29701b12867fd4ac9da878808e781", + "collector_archive_sha256": "d2faeb65a3b63b714ebf6b67c342a3638c174f1342d1bec071a946d01a3cba6f", + "min_ratio": 0.6115285705012946, + "median_ratio": 0.9825005890555982, + "max_ratio": 1.1830778850944927 + }, + "provenance": { + "frozen_predictions_sha256": "c4d9e9f777216218459fb68001b2a989dacd09b9178031a8b55b6cde439be121", + "runtime_base": "89d2051137b772944a3e303c452108811790bef4", + "native_binary_sha256": "7c0ccae3937ef48224f1a35cbab04735ea13914a395bc0853435c63425ef62b8", + "runner_before_sha256": "0fd35eeec5e32f8ee4cfce279d3bc145eb534b5b2d927e83f214be2f4ea12879", + "runner_after_sha256": "f8d3db73c2c80ee8ed45f622955847d8e336ff970dbe2141c6c73ce4abc119ec", + "replay_script_sha256": "b3a73dfa02cc105ef74ff28faf6085902b54510a40e6fb60ba33870e5ed07698", + "slurm_job": "1995371", + "slurm_state": "COMPLETED", + "slurm_exit_code": "0:0" + } +} diff --git a/python/aisimulate/collector/trtllm/gym-w4a8-20260918.md b/python/aisimulate/collector/trtllm/gym-w4a8-20260918.md new file mode 100644 index 000000000..1927f164f --- /dev/null +++ b/python/aisimulate/collector/trtllm/gym-w4a8-20260918.md @@ -0,0 +1,87 @@ +# TRT GPT-OSS activation correction: full 263-point replay + +## Result + +- All **263/263** original Gym points produced finite TTFT and TPOT predictions; + **293,180/293,180** requests completed, with no truncation or context violations. +- Correct TRT GPT-OSS B200 static MXFP4 checkpoints to **W4A8 MXFP4/MXFP8**. + The original mapping forced W4A16 despite the serving default. H200 remains + W4A16. Source recipes were re-fetched at their immutable commits, matched to + their archived SHA256, and contain no `OVERRIDE_QUANT_ALGO`. +- Collect true W4A8 profiles; never relabel W4A16 measurements. B200 rc20 job + **1995371** completed `0:0`: six canonical cases, **162 CUDA-Graph rows**, + **81 new TP2/4/8 rows**, **81 refreshed rows**, **218,295 untouched rows**. +- Honor `length_sampler=numpy_random_state` in engine replay. Preserve + `python_random` as the default and reject unknown selectors. The client draws + its complete input vector before output lengths, with inclusive bounds. + +## Backend-separated MAPE + +MAPE is the mean per-point absolute percentage error against the original +historical silicon observation. No fitting or latency multiplier is applied. +"Before" is the preceding PR state after W4A16 autotuning correction. + +| Backend / slice | Points | TTFT before | TTFT after | TPOT before | TPOT after | +|---|---:|---:|---:|---:|---:| +| trtllm | 166 | 66.00% | 57.81% | 66.72% | 36.25% | +| sglang | 97 | 69.61% | 69.77% | 12.55% | 12.52% | +| trtllm/gpt-oss-120b/b200 | 31 | 85.67% | 43.66% | 168.78% | 5.33% | + +## Controlled ablations and regressions + +- Mapping plus W4A8 data, keeping the original Python-sampled request vectors: + GPT-OSS/B200 TPOT MAPE **168.7816% -> 5.3295%**; TRT TPOT **36.1916%**. + All **232 points outside GPT-OSS/B200 remain bit-identical** in TTFT/TPOT. +- Correcting the sampler separately yields TRT TPOT **36.2472%** and SGLang + TPOT **12.5170%**. Every generated request length was checked against NumPy's + client contract. This is workload alignment, not an accuracy tuning step. +- All 31 GPT-OSS/B200 TPOT APEs improve; final range **0.23%–17.38%**. + **15/31 TTFT APEs worsen** despite mean TTFT improvement. Config155, + ISL1024/concurrency4, worsens **5.97% -> 65.78%**. Correct activation precision + removes an accidental compensation; TTFT scheduling/overhead remains open. +- Across the full cohort, **117 TTFT and 46 TPOT** APEs worsen by more than + **0.1 percentage point**, including the separately changed request sampling. + This reporting threshold is not an acceptance gate. The largest TPOT + regression is config1553/B200/ISL8192/C1024: **182.20% -> 185.91%**. +- Residual TRT TPOT MAPE: DeepSeek B200 **49.18%**, MiniMax B200 **39.40%**, + MiniMax B300 **44.15%**, Qwen B200 **85.39%**. This fixes the confirmed + GPT-OSS precision defect; it does not establish full accuracy for other models. +- The reproduced config131/C256/ISL1024 predicts **17.1702 ms** TPOT versus + historical **16.7921 ms** (2.25% APE), and fresh uninstrumented serving + **16.5492 ms** (3.75% APE). These are distinct reference observations. + +## Reproducibility + +- Frozen cohort SHA256: + `c4d9e9f777216218459fb68001b2a989dacd09b9178031a8b55b6cde439be121`. +- Native runtime remains `89d2051137b772944a3e303c452108811790bef4` with + binary SHA256 `7c0ccae3937ef48224f1a35cbab04735ea13914a395bc0853435c63425ef62b8`. + Only the reviewed Python sampler patch is applied to this frozen runtime; + report hashes distinguish it from an unmodified baseline wheel. +- Resolve the checkpoint through Gym's updated mapping and its public + `build_replay_spec` lowering; retain all frozen engine controls during the + causal replay. Each phase saves specs and native per-request reports. +- Performance data remains rc20 while silicon serving is rc14; this existing + version difference is retained, not described as an exact framework match. + Collector image: `sha256:9b3b4dfb811caa9420fa99a6f958155f6a1f727ffc2b5a5c2d9d2ce51fdc323d`. +- Source rules: `ModelConfig.get_mxfp4_quant_algo`, rc14 + `93cb6518b6d6dbd6095748189e626db731f44545`, rc18 + `15d06c0923b63ac1781784d5f59e1747bb47d5f1`, rc20 + `c25c23f71786bad54d192893d696ce8043426eca` in + [TensorRT-LLM model_config.py](https://github.com/NVIDIA/TensorRT-LLM/blob/93cb6518b6d6dbd6095748189e626db731f44545/tensorrt_llm/_torch/model_config.py). + Unreviewed releases and unknown hardware fail closed. Explicit runtime + overrides are honored. TRITON's W4A8 MXFP4/FP8 fails closed because it has no + matching predictor profile; it is never substituted with MXFP8. +- Raw artifacts: `${CAMPAIGN_ARTIFACT_ROOT}/w4a8-fix/`; + Slurm artifacts under + `${B200_ARTIFACT_ROOT}/results/trt-62-20260917/w4a8-fix/`. +- [Summary and provenance](gym-w4a8-20260918.json) and + [all paired points](gym-w4a8-20260918-paired.json.gz) include signed errors, + absolute errors, original recovery metrics, and both correction phases. +- Data checks pass all seven invariants; backend facts remain 1,922 slices. + Runner tests: 109 passed. Gym focused mapping/deployment/parser tests: 95 passed. + +## Residual follow-up + +- [Qwen scheduling and GPT-OSS TTFT diagnosis](gym-residual-20260918.md) tracks + remaining source-contract gaps separately from the completed precision fix. diff --git a/python/aisimulate/src/aisimulate/runner.py b/python/aisimulate/src/aisimulate/runner.py index a203080a2..d0b7cd44d 100644 --- a/python/aisimulate/src/aisimulate/runner.py +++ b/python/aisimulate/src/aisimulate/runner.py @@ -16,6 +16,8 @@ from numbers import Real from typing import Any, Protocol, runtime_checkable +import numpy as np + from .capacity import materialize_aic_num_gpu_blocks from .config.common import ENGINE_MODEL_CONTROL_FIELDS, is_active_engine_model_control from .power import normalize_power_summary, power_metadata @@ -352,6 +354,8 @@ def run( output_requirements = output_requirements or ReplayOutputRequirements() if output_requirements.capture_telemetry: raise InvalidRunnerError("EngineReplayRunner's JSON runtime does not yet expose replay telemetry") + if spec.workload.get("source_type") is not None and "length_sampler" in spec.workload: + raise ValueError("length_sampler requires materialized direct synthetic replay without source_type") self.capabilities.require_compatible(spec) encoder = spec.backend_deployment.encoder if encoder is None and spec.workload.get("images") is not None: @@ -1054,6 +1058,8 @@ def _materialize_requests(spec: ReplaySpec, trace_block_size: int) -> tuple[list raise TypeError("trace_path must be a non-empty string") if workload.get("random_range_ratio", 1.0) != 1.0 or workload.get("random_seed", 0) != 0: raise ValueError("random_range_ratio and random_seed only apply to synthetic replay") + if workload.get("length_sampler", "python_random") != "python_random": + raise ValueError("length_sampler only applies to synthetic replay") configured_trace_block_size = workload.get("trace_block_size") requests = materialize_configured_traffic( { @@ -1113,7 +1119,15 @@ def _materialize_requests(spec: ReplaySpec, trace_block_size: int) -> tuple[list random_range_ratio = _random_range_ratio(workload.get("random_range_ratio", 1.0)) random_seed = _random_seed(workload.get("random_seed", 0)) - length_rng = random.Random(random_seed) + sampler = workload.get("length_sampler", "python_random") + if sampler == "numpy_random_state": + if random_seed > 0xFFFF_FFFF: + raise ValueError("numpy_random_state random_seed must be an unsigned 32-bit integer") + length_rng = np.random.RandomState(random_seed) + elif sampler == "python_random": + length_rng = random.Random(random_seed) + else: + raise ValueError(f"unsupported length_sampler: {sampler!r}") # Follow InferenceX's draw order: sample the complete ISL vector before OSL. input_lengths = _sample_synthetic_lengths(isl, request_count, random_range_ratio, length_rng) output_lengths = _sample_synthetic_lengths(osl, request_count, random_range_ratio, length_rng) @@ -1656,13 +1670,15 @@ def _sample_synthetic_lengths( upper: int, count: int, random_range_ratio: float, - rng: random.Random, + rng: random.Random | np.random.RandomState, ) -> list[int]: if random_range_ratio == 1.0: return [upper] * count lower = int(upper * random_range_ratio) if lower == 0: raise ValueError(f"random_range_ratio={random_range_ratio} gives a zero-token lower bound for length {upper}") + if isinstance(rng, np.random.RandomState): + return rng.randint(lower, upper + 1, size=count).tolist() return [rng.randint(lower, upper) for _ in range(count)] diff --git a/python/aisimulate/src/aisimulate_core/systems/data/b200_sxm/README.md b/python/aisimulate/src/aisimulate_core/systems/data/b200_sxm/README.md index c0c1c69a1..af11b49fd 100644 --- a/python/aisimulate/src/aisimulate_core/systems/data/b200_sxm/README.md +++ b/python/aisimulate/src/aisimulate_core/systems/data/b200_sxm/README.md @@ -3,9 +3,9 @@ SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All SPDX-License-Identifier: Apache-2.0 --> -# B200 TRT-LLM power import +# B200 TRT-LLM power import and reviewed refreshes -The 18 tables below derive from [AIConfigurator commit +The original import described below derived 18 tables from [AIConfigurator commit 915f590680d8a79fe9c39f6f3a9ff13bc267fcce](https://github.com/ai-dynamo/aiconfigurator/tree/915f590680d8a79fe9c39f6f3a9ff13bc267fcce/aic-core/src/aiconfigurator_core/systems/data/b200_sxm) ([upstream PR #1584](https://github.com/ai-dynamo/aiconfigurator/pull/1584)). Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -63,3 +63,27 @@ The retained local identity/latency pairs are independently pinned from AISimula commit `ffcb6576b3a60077ea1200788f1b784213f979cf`, immediately before import commit `717f973bea4ebc07673192475b3a0f743d82c168`. The test records their digests and the deterministic serialization used to reproduce them from that baseline. + +## September 17-18 precision/profile refresh + +The inventory and integrity counts above describe the original import. The +reviewed TRT-LLM campaign subsequently replaced context MLA with 3,080 measured +latency rows and refreshed GPT-OSS MXFP4 MoE rows, bringing MoE to 218,457 rows. +See the collection sidecars and `collector/trtllm/gym-w4a8-20260918.md` for +source revisions, image hashes, Slurm jobs and the historical replay evidence. + +The 324 refreshed W4A16/W4A8 MoE rows did not measure power. Their paired +power fields use `0.0/0.0` (unavailable), not null or an inferred measurement. +The packaging correction preserves every identity and latency and all 218,133 +other rows exactly. MoE now has 178,875 positive power pairs and 39,582 paired +zero sentinels. The context MLA refresh has no measured power columns. +The two corresponding file-integrity pins reflect these reviewed updates; +the other 16 pins and the attention preservation checks are unchanged. + +The 324 refreshed rows form 12 complete token curves (two quantizations, six +TP/EP configurations, `power_law_1.2`). Every point in each refreshed curve +has unavailable power, so these curves do not interpolate between measured +and unavailable power. Runtime zero energy is the existing missing-evidence +sentinel: it contributes no covered latency, and the public power result is +subject to the 90% coverage gate described in `docs/power-model.md`. The data +refresh does not qualify power accuracy or change power interpolation rules. diff --git a/python/aisimulate/src/aisimulate_core/systems/data/b200_sxm/mla/trtllm/1.3.0rc20/collection_meta.yaml b/python/aisimulate/src/aisimulate_core/systems/data/b200_sxm/mla/trtllm/1.3.0rc20/collection_meta.yaml index 675a6753c..3ff467c0d 100644 --- a/python/aisimulate/src/aisimulate_core/systems/data/b200_sxm/mla/trtllm/1.3.0rc20/collection_meta.yaml +++ b/python/aisimulate/src/aisimulate_core/systems/data/b200_sxm/mla/trtllm/1.3.0rc20/collection_meta.yaml @@ -2,12 +2,18 @@ # SPDX-License-Identifier: Apache-2.0 schema_version: 1 -provenance: collected +provenance: local runtime: framework: trtllm version: 1.3.0rc20 tables: context_mla_perf: + collector_ref: ce9a396642674e8008fda2b778580dc8d9e310b4 + collector_hash: sha256:9d396ec69d649ddfeb9c3450f0cd35881309628c5ca1ded9b4259c98cf074e2e + case_plan_hash: sha256:984b7c01ed49ccc195f4b9668b59865da2b367a9c270800f9850275e7ae16150 + collected_at: '2026-09-17' + rows: 3080 + classified_failures: 0 status: complete generation_mla_perf: status: complete diff --git a/python/aisimulate/src/aisimulate_core/systems/data/b200_sxm/mla/trtllm/1.3.0rc20/context_mla_perf.parquet b/python/aisimulate/src/aisimulate_core/systems/data/b200_sxm/mla/trtllm/1.3.0rc20/context_mla_perf.parquet index e37d86350..565cb8af1 100644 Binary files a/python/aisimulate/src/aisimulate_core/systems/data/b200_sxm/mla/trtllm/1.3.0rc20/context_mla_perf.parquet and b/python/aisimulate/src/aisimulate_core/systems/data/b200_sxm/mla/trtllm/1.3.0rc20/context_mla_perf.parquet differ diff --git a/python/aisimulate/src/aisimulate_core/systems/data/b200_sxm/moe/trtllm/1.3.0rc20/collection_meta.yaml b/python/aisimulate/src/aisimulate_core/systems/data/b200_sxm/moe/trtllm/1.3.0rc20/collection_meta.yaml index bf3fc0947..a0ddcceda 100644 --- a/python/aisimulate/src/aisimulate_core/systems/data/b200_sxm/moe/trtllm/1.3.0rc20/collection_meta.yaml +++ b/python/aisimulate/src/aisimulate_core/systems/data/b200_sxm/moe/trtllm/1.3.0rc20/collection_meta.yaml @@ -10,6 +10,21 @@ runtime: # model-filtered internal collection CI run, 8x B200, zero failures). # 2026-08-09: appended 5508 nvidia/MiniMax-M3-NVFP4 nvfp4 rows (model-filtered # internal CI run, zero failures). +# Historical stage, job 1993260: appended 81 W4A16 MXFP4 rows (TP=2/4/8, +# EP=1, power_law_1.2), producing 218376 rows from 218295 prior rows. +# Historical refresh, job 1994597: replaced 162 W4A16 MXFP4 rows; +# 218214 rows retained exact values, with 218376 total rows. +# Final refresh, job 1995371: measured 162 W4A8 MXFP4/MXFP8 rows, +# replaced 81 existing rows and added 81 TP=2/4/8, EP=1 rows. +# 218295 prior rows retain exact values; final total: 218457 rows. +# Measured-stage moe_perf.parquet SHA256 (before power sentinel normalization): +# db1930fde20b419284bf1f038fadd4b398d29701b12867fd4ac9da878808e781 +# Source, graph validation and replay: collector/trtllm/gym-w4a8-20260918.md. +# Packaging correction: 324 latency-only MXFP4 rows use the documented +# power=0.0, power_limit=0.0 unavailable sentinel instead of nulls. All +# identities, latencies and 218133 existing power pairs are unchanged. +# Packaged moe_perf.parquet SHA256: +# 9c88ebdd67b97ded16310f18abaa5459d4959ff7b880bbd15c6fec43753af411 tables: moe_perf: status: complete diff --git a/python/aisimulate/src/aisimulate_core/systems/data/b200_sxm/moe/trtllm/1.3.0rc20/moe_perf.parquet b/python/aisimulate/src/aisimulate_core/systems/data/b200_sxm/moe/trtllm/1.3.0rc20/moe_perf.parquet index 1f1075ddb..654dabbf2 100644 Binary files a/python/aisimulate/src/aisimulate_core/systems/data/b200_sxm/moe/trtllm/1.3.0rc20/moe_perf.parquet and b/python/aisimulate/src/aisimulate_core/systems/data/b200_sxm/moe/trtllm/1.3.0rc20/moe_perf.parquet differ diff --git a/python/aisimulate/src/aisimulate_core/systems/data/h200_sxm/mla/trtllm/1.3.0rc20/collection_meta.yaml b/python/aisimulate/src/aisimulate_core/systems/data/h200_sxm/mla/trtllm/1.3.0rc20/collection_meta.yaml index 675a6753c..002f19b49 100644 --- a/python/aisimulate/src/aisimulate_core/systems/data/h200_sxm/mla/trtllm/1.3.0rc20/collection_meta.yaml +++ b/python/aisimulate/src/aisimulate_core/systems/data/h200_sxm/mla/trtllm/1.3.0rc20/collection_meta.yaml @@ -2,12 +2,18 @@ # SPDX-License-Identifier: Apache-2.0 schema_version: 1 -provenance: collected +provenance: local runtime: framework: trtllm version: 1.3.0rc20 tables: context_mla_perf: + collector_ref: ce9a396642674e8008fda2b778580dc8d9e310b4 + collector_hash: sha256:9d396ec69d649ddfeb9c3450f0cd35881309628c5ca1ded9b4259c98cf074e2e + case_plan_hash: sha256:c0fd09dd4f806c0c23a84de0e452f6093b889ec3bc4adcdf1fdf4f30d2b66606 + collected_at: '2026-09-17' + rows: 3080 + classified_failures: 0 status: complete generation_mla_perf: status: complete diff --git a/python/aisimulate/src/aisimulate_core/systems/data/h200_sxm/mla/trtllm/1.3.0rc20/context_mla_perf.parquet b/python/aisimulate/src/aisimulate_core/systems/data/h200_sxm/mla/trtllm/1.3.0rc20/context_mla_perf.parquet index 91eaeceab..45ed957fb 100644 Binary files a/python/aisimulate/src/aisimulate_core/systems/data/h200_sxm/mla/trtllm/1.3.0rc20/context_mla_perf.parquet and b/python/aisimulate/src/aisimulate_core/systems/data/h200_sxm/mla/trtllm/1.3.0rc20/context_mla_perf.parquet differ diff --git a/python/aisimulate/tests/unit/collector/trtllm/test_mla_precision_mxfp4_padding.py b/python/aisimulate/tests/unit/collector/trtllm/test_mla_precision_mxfp4_padding.py new file mode 100644 index 000000000..01b4f6644 --- /dev/null +++ b/python/aisimulate/tests/unit/collector/trtllm/test_mla_precision_mxfp4_padding.py @@ -0,0 +1,144 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""GPU-free regression tests for TRT precision labels and native MXFP4 padding.""" + +import ast +import inspect +from contextlib import nullcontext +from pathlib import Path +from types import SimpleNamespace +from unittest.mock import MagicMock + +import pytest + +pytestmark = pytest.mark.unit +COLLECTOR = Path(__file__).resolve().parents[4] / "collector" / "trtllm" + + +@pytest.mark.parametrize("sm", [80, 89, 90, 100, 103, 120, 121]) +@pytest.mark.parametrize("kv_dtype", ["bfloat16", "fp8"]) +@pytest.mark.parametrize("context", [True, False]) +def test_mla_logged_compute_precision(sm, kv_dtype, context): + tree = ast.parse((COLLECTOR / "collect_mla.py").read_text()) + function = next( + node for node in tree.body if isinstance(node, ast.FunctionDef) and node.name == "_run_attn_for_backend" + ) + call = next( + node + for node in ast.walk(function) + if isinstance(node, ast.Call) and isinstance(node.func, ast.Name) and node.func.id == "log_perf" + ) + fields = next(item.value.elts[0] for item in call.keywords if item.arg == "item_list") + dtype_expr = next( + value + for key, value in zip(fields.keys, fields.values, strict=True) + if isinstance(key, ast.Constant) and key.value == "mla_dtype" + ) + actual = eval( + compile(ast.Expression(dtype_expr), "collect_mla.py", "eval"), + { + "dtype_str": kv_dtype, + "is_context_phase": context, + "get_sm_version": lambda: sm, + }, + ) + assert actual == (kv_dtype if context and sm in (90, 100, 103, 120) else "bfloat16") + + +class ReachedNativeBuilder(Exception): + pass + + +@pytest.mark.parametrize("sm", [90, 100, 103, 120]) +@pytest.mark.parametrize("tp", [2, 4, 8]) +@pytest.mark.parametrize("quant", ["w4a16_mxfp4", "w4a8_mxfp4_mxfp8"]) +@pytest.mark.parametrize("model_name", ["openai/gpt-oss-120b", "openai/gpt-oss-20b", "other-mxfp4-model"]) +def test_unaligned_mxfp4_honors_native_padding_window(sm, tp, quant, model_name): + tree = ast.parse((COLLECTOR / "collect_moe.py").read_text()) + function = next(node for node in tree.body if isinstance(node, ast.FunctionDef) and node.name == "run_moe_torch") + native_builder = MagicMock(side_effect=ReachedNativeBuilder) + namespace = { + "torch": MagicMock(), + "gc_collect": lambda: None, + "aic_debug": 0, + "QuantAlgo": MagicMock(), + "QuantConfig": MagicMock(), + "Mapping": SimpleNamespace, + "SimpleNamespace": SimpleNamespace, + "ModelConfig": lambda **_: SimpleNamespace(), + "NON_GATED_MOE_MODELS": [], + "ActivationType": MagicMock(), + "get_sm_version": lambda: sm, + "_MXFP4_MOE_TYPES": {"w4a16_mxfp4", "w4a8_mxfp4_mxfp8"}, + "RenormalizeMoeRoutingMethod": MagicMock(), + "create_moe": native_builder, + "inspect": SimpleNamespace(signature=lambda _: SimpleNamespace(parameters={})), + } + exec(compile(ast.Module(body=[function], type_ignores=[]), "collect_moe.py", "exec"), namespace) + if sm in (90, 120): + with pytest.raises(ValueError, match="weight-layout alignment"): + namespace["run_moe_torch"](quant, [1], 2880, 2880, 4, 128, tp, 1, False, model_name, perf_filename="unused") + native_builder.assert_not_called() + return + with pytest.raises(ReachedNativeBuilder): + namespace["run_moe_torch"](quant, [1], 2880, 2880, 4, 128, tp, 1, False, model_name, perf_filename="unused") + assert native_builder.call_args.kwargs["intermediate_size"] == 2880 + assert native_builder.call_args.kwargs["model_config"].mapping.moe_tp_size == tp + assert native_builder.call_args.kwargs["bias"] == model_name.startswith("openai/gpt-oss-") + if model_name.startswith("openai/gpt-oss-"): + assert namespace["RenormalizeMoeRoutingMethod"].call_args.kwargs["output_dtype"] is namespace["torch"].bfloat16 + else: + assert not namespace["RenormalizeMoeRoutingMethod"].call_args.kwargs + + +@pytest.mark.parametrize("outcomes", ["fail,fail", "fail,pass", "pass,fail", "pass,pass", "fatal", "cached"]) +def test_moe_autotuning_requires_success_or_loaded_cache(outcomes): + tree = ast.parse((COLLECTOR / "collect_moe.py").read_text()) + function = next(node for node in tree.body if isinstance(node, ast.FunctionDef) and node.name == "run_moe_torch") + tuning = next( + node + for node in ast.walk(function) + if isinstance(node, ast.If) + and isinstance(node.test, ast.UnaryOp) + and isinstance(node.test.op, ast.Not) + and isinstance(node.test.operand, ast.Name) + and node.test.operand.id == "cache_loaded" + ) + + class AcceleratorError(RuntimeError): + pass + + failure = RuntimeError("ordinary tuning failure") + fatal = AcceleratorError("CUDA context lost") + effects = {"fail": failure, "pass": None, "fatal": fatal, "cached": None} + forward = MagicMock(side_effect=[effects[item] for item in outcomes.split(",")]) + namespace = { + "torch": SimpleNamespace( + AcceleratorError=AcceleratorError, + cuda=SimpleNamespace(synchronize=lambda: None), + inference_mode=nullcontext, + ), + "cache_loaded": outcomes == "cached", + "num_tokens_lists": [2, 4, 8], + "max_tokens": 4, + "inspect": inspect, + "autotune": nullcontext, + "moe": SimpleNamespace(forward=forward), + "hidden_states_max_tokens": MagicMock(), + "logits_max_tokens": MagicMock(), + "min_latency_mode": False, + } + block = compile(ast.Module(body=[tuning], type_ignores=[]), "collect_moe.py", "exec") + if outcomes == "fail,fail": + with pytest.raises(RuntimeError, match="any eligible token count") as caught: + exec(block, namespace) + assert caught.value.__cause__ is failure + elif outcomes == "fatal": + with pytest.raises(AcceleratorError) as caught: + exec(block, namespace) + assert caught.value is fatal + else: + exec(block, namespace) + expected_calls = 0 if outcomes == "cached" else 1 if outcomes == "fatal" else 2 + assert forward.call_count == expected_calls diff --git a/python/aisimulate/tests/unit/sdk/task_v2/test_task_config.py b/python/aisimulate/tests/unit/sdk/task_v2/test_task_config.py index b74aed194..3dc039e4c 100644 --- a/python/aisimulate/tests/unit/sdk/task_v2/test_task_config.py +++ b/python/aisimulate/tests/unit/sdk/task_v2/test_task_config.py @@ -1407,17 +1407,20 @@ def test_large_ep_replica_size_is_bounded(): assert max(kw["num_gpu_list"]) <= t.total_gpus -def test_explicit_fmha_fp8_not_downgraded(): - """V3/Kimi context fmha fp8->bf16 downgrade fires only on HF-inferred fp8 (matches v1's - `not explicit_fmha_mode` guard). An explicit fp8 is kept, so validate can fail fast - (instead of silently modelling bf16).""" +def test_explicit_fmha_precision_preserved_with_trtllm_fp8_profiles(): + """Audited rc20 FP8 profiles retain inferred FP8 and both explicit overrides.""" from aisimulate.sdk import common - base = dict(serving_mode="agg", model_path="deepseek-ai/DeepSeek-V3", system_name="h200_sxm", backend_name="trtllm") - assert Task(**base).fmha_quant_mode == common.FMHAQuantMode.bfloat16 # HF-inferred -> downgraded - assert ( - Task(**base, fmha_quant_mode=common.FMHAQuantMode.fp8).fmha_quant_mode == common.FMHAQuantMode.fp8 - ) # explicit -> kept + base = dict( + serving_mode="agg", + model_path="deepseek-ai/DeepSeek-V3", + system_name="h200_sxm", + backend_name="trtllm", + backend_version="1.3.0rc20", + ) + assert Task(**base).fmha_quant_mode == common.FMHAQuantMode.fp8 + for precision in (common.FMHAQuantMode.fp8, common.FMHAQuantMode.bfloat16): + assert Task(**base, fmha_quant_mode=precision).fmha_quant_mode == precision def test_database_mode_is_forwarded_to_view_loader(monkeypatch): diff --git a/python/aisimulate/tests/unit/tools/test_power_data.py b/python/aisimulate/tests/unit/tools/test_power_data.py index e3e223caa..834b208fc 100644 --- a/python/aisimulate/tests/unit/tools/test_power_data.py +++ b/python/aisimulate/tests/unit/tools/test_power_data.py @@ -4,9 +4,10 @@ """Regression evidence for the B200 TRT-LLM 1.3.0rc20 import. Source: ai-dynamo/aiconfigurator at 915f590680d8a79fe9c39f6f3a9ff13bc267fcce, -under src/aisimulate_core/systems/data/b200_sxm/. +under aic-core/src/aiconfigurator_core/systems/data/b200_sxm/. See the adjacent data README for attribution and the two attention merges. -These pins describe this import; future reviewed dataset updates may update them. +The context MLA and MoE pins include the reviewed September 17-18 refresh +documented in the data README; other pins still describe the original import. Version-independent power-field checks live in test_power_data_invariants.py. """ @@ -61,7 +62,7 @@ ), ( "mla/trtllm/1.3.0rc20/context_mla_perf.parquet", - "ccfbf6075f4644bd4e39b224552160a3214920ddce7bfd7195347dc0225950f9", + "905f09a335f56c3a4c37cc7eb74113c1c026e3c57a8ff3dfc40a7c98d1fab30a", ), ( "mla/trtllm/1.3.0rc20/generation_mla_perf.parquet", @@ -81,7 +82,7 @@ ), ( "moe/trtllm/1.3.0rc20/moe_perf.parquet", - "942d7f65c9b624d1ff261f3764820d44ce385cdd55a23dbf949edbbea8792d5a", + "9c88ebdd67b97ded16310f18abaa5459d4959ff7b880bbd15c6fec43753af411", ), ( "quantize/trtllm/1.3.0rc20/computescale_perf.parquet", @@ -102,7 +103,7 @@ ], ) def test_b200_import_file(relative, sha256): - # Sixteen pins equal the upstream bytes; two pin the reviewed attention merges. + # Fourteen pins equal upstream bytes; two pin attention merges and two the MLA/MoE refresh. assert hashlib.sha256((DATA_ROOT / relative).read_bytes()).hexdigest() == sha256 diff --git a/tests/test_runner.py b/tests/test_runner.py index cda19ad04..272191f88 100644 --- a/tests/test_runner.py +++ b/tests/test_runner.py @@ -6,13 +6,17 @@ import json import math import pickle +from dataclasses import replace import pytest +from pydantic import ValidationError import aisimulate from aisimulate import capacity as aic +from aisimulate.cli_args import build_parser from aisimulate.compiler import prediction_to_replay_spec from aisimulate.config.cli import CorePredictionConfig +from aisimulate.config.traffic import SyntheticSource from aisimulate.replay.config import ReplayCliConfig, ReplayOutputConfig from aisimulate.runner import ( EngineReplayRunner, @@ -27,6 +31,7 @@ ReplaySpec, RuntimeHookSpec, ) +from aisimulate.sweeper.config import Workload from aisimulate_core.sdk.deepseek_v41 import MODEL_PATH as DEEPSEEK_V41_MODEL_PATH pytestmark = [ @@ -1485,6 +1490,123 @@ def test_runner_rejects_overflowing_ordinary_metric(): _normalize_engine_replay_report({"output_throughput_tok_s": 10**400}, include_native_report=False) +@pytest.mark.parametrize("sampler", ["numpy_random_state", "python_random"]) +def test_runner_honors_length_sampler(sampler): + # Fixed vectors from the benchmark's NumPy RandomState contract and the + # legacy Python sampler. The entire input vector is drawn first. + expected = { + "numpy_random_state": [(8, 4), (9, 4), (8, 5), (9, 4), (9, 4)], + "python_random": [(9, 5), (9, 5), (8, 5), (9, 5), (10, 5)], + } + for _ in range(2): + runtime = RecordingRuntime() + EngineReplayRunnerFactory(runtime=runtime).create(0).run( + _spec( + workload={ + "isl": 10, + "osl": 5, + "request_count": 5, + "arrival_interval_ms": 0.0, + "random_range_ratio": 0.8, + "random_seed": 0, + "length_sampler": sampler, + } + ) + ) + assert [(r["input_tokens"], r["output_tokens"]) for r in runtime.execution_spec["requests"]] == expected[ + sampler + ] + + +def test_runner_rejects_unknown_length_sampler(): + with pytest.raises(ValueError, match="length_sampler"): + EngineReplayRunnerFactory(runtime=RecordingRuntime()).create(0).run( + _spec( + workload={ + "isl": 10, + "osl": 5, + "request_count": 2, + "arrival_interval_ms": 0.0, + "length_sampler": "typo", + } + ) + ) + + +@pytest.mark.parametrize( + "sampler,seed", + [("numpy_random_state", 0xFFFF_FFFF), ("python_random", 0x1_0000_0000), ("python_random", 0xFFFF_FFFF_FFFF_FFFF)], +) +def test_runner_sampler_seed_upper_bounds(sampler, seed): + runtime = RecordingRuntime() + EngineReplayRunnerFactory(runtime=runtime).create(0).run( + _spec( + workload={ + "isl": 10, + "osl": 5, + "request_count": 2, + "arrival_interval_ms": 0.0, + "random_range_ratio": 0.8, + "random_seed": seed, + "length_sampler": sampler, + } + ) + ) + assert len(runtime.execution_spec["requests"]) == 2 + + +def test_runner_rejects_numpy_seed_above_uint32(): + with pytest.raises(ValueError, match="numpy_random_state random_seed.*32-bit"): + EngineReplayRunnerFactory(runtime=RecordingRuntime()).create(0).run( + _spec( + workload={ + "isl": 10, + "osl": 5, + "request_count": 2, + "arrival_interval_ms": 0.0, + "random_seed": 0x1_0000_0000, + "length_sampler": "numpy_random_state", + } + ) + ) + + +@pytest.mark.parametrize("sampler", ["numpy_random_state", "typo"]) +def test_runner_rejects_nondefault_sampler_for_trace(sampler): + with pytest.raises(ValueError, match="length_sampler only applies to synthetic replay"): + EngineReplayRunnerFactory(runtime=RecordingRuntime()).create(0).run( + _spec(workload={"trace_path": "unused.jsonl", "length_sampler": sampler}) + ) + + +@pytest.mark.parametrize("schema", [Workload, SyntheticSource]) +def test_length_sampler_is_not_exposed_by_public_workload_schemas(schema): + with pytest.raises(ValidationError) as error: + schema.model_validate({"length_sampler": "numpy_random_state"}) + assert any(e["loc"] == ("length_sampler",) and e["type"] == "extra_forbidden" for e in error.value.errors()) + + +def test_length_sampler_is_not_exposed_by_public_cli(capsys): + with pytest.raises(SystemExit) as error: + build_parser().parse_args(["predict", "-c", "unused.yaml", "--length-sampler", "numpy_random_state"]) + assert error.value.code == 2 + assert "unrecognized arguments: --length-sampler numpy_random_state" in capsys.readouterr().err + + +@pytest.mark.parametrize("source_type", ["synthetic", "trace"]) +@pytest.mark.parametrize("sampler", ["numpy_random_state", "python_random", "typo"]) +@pytest.mark.parametrize("deployment_mode", ["agg", "disagg", "afd", "afd+pd"]) +def test_workload_driver_rejects_length_sampler_before_native_execution(source_type, sampler, deployment_mode): + runtime = RecordingRuntime() + spec = _spec(workload={"source_type": source_type, "length_sampler": sampler}) + spec = replace(spec, backend_deployment=replace(spec.backend_deployment, deployment_mode=deployment_mode)) + with pytest.raises( + ValueError, match="length_sampler requires materialized direct synthetic replay without source_type" + ): + EngineReplayRunnerFactory(runtime=runtime).create(0).run(spec) + assert runtime.execution_spec is None + + @pytest.mark.parametrize("layout", ["flat", "null_flat", "canonical", "canonical_fallback", "nested"]) @pytest.mark.parametrize("version", ["", "next", "literal_next"]) def test_direct_replay_capacity_uses_the_resolved_timing_version(monkeypatch, tmp_path, layout, version):