[feat][nv] Add Kimi K3 B200 configs - #2618
Conversation
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| # Throughput runs use the committed K7 golden synthetic AL 3.84. Eval | ||
| # selects the paired real-verification recipe instead. | ||
| speculative-config: '{"model":"Inferact/Kimi-K3-DSpark","num_speculative_tokens":7,"method":"dspark","attention_backend":"FLASHINFER_MLA","draft_sample_method":"probabilistic","rejection_sample_method":"synthetic","synthetic_acceptance_length":3.84}' | ||
| speculative-config: '{"model":"Inferact/Kimi-K3-DSpark","num_speculative_tokens":3,"method":"dspark","attention_backend":"TOKENSPEED_MLA","draft_sample_method":"probabilistic","rejection_sample_method":"synthetic","synthetic_acceptance_length":3.0}' |
There was a problem hiding this comment.
🟡 The comment above speculative-config in agg-b200-tp8dp2-vllm-simple-offload-dspark-agentic.yaml (lines 96-97) still says "Throughput runs use the committed K7 golden synthetic AL 3.84", but this PR changed this recipe's speculative-config to num_speculative_tokens:3/synthetic_acceptance_length:3.0 (banded DSpark: level 7 at concurrency<=8, level 3 above). Please update the comment to reflect level 3 / AL 3.0 for this higher-concurrency offload arm.
Extended reasoning...
The comment block directly above the speculative-config line in agg-b200-tp8dp2-vllm-simple-offload-dspark-agentic.yaml (lines 96-97) reads:
# Throughput runs use the committed K7 golden synthetic AL 3.84. Eval
# selects the paired real-verification recipe instead.
That text was accurate before this PR, when the recipe used num_speculative_tokens:7 and synthetic_acceptance_length:3.84 (K7 = DSpark level 7, AL 3.84). This PR changes that same speculative-config line (line 98) to num_speculative_tokens:3 and synthetic_acceptance_length:3.0, but leaves the comment text untouched, so the comment now describes a configuration the file no longer uses.
The change is intentional and correct on the config side: perf-changelog.yaml's new entry for this PR describes banded DSpark — "level 7 at concurrency 8 and below and level 3 above it" — and this offload recipe runs at much higher concurrency (max-num-seqs: 96), so moving it to level 3 / AL 3.0 is the right call. The sibling agg-b200-tp8dp2-latency-dspark-agentic.yaml, which runs at low concurrency (max-num-seqs: 16) and correctly stayed on num_speculative_tokens:7/AL 3.84, kept the identical comment — and there it is still accurate, confirming the comment in the offload file is what fell out of sync.
Nothing prevents this drift structurally: the comment is a free-text annotation next to a JSON blob in YAML, so there is no lint or schema check tying the prose to the num_speculative_tokens/synthetic_acceptance_length values. It is purely on the author to keep them in sync when editing the JSON.
Concrete proof: (1) pre-PR, this file had num_speculative_tokens:7, synthetic_acceptance_length:3.84, matching "K7 golden synthetic AL 3.84". (2) The PR diff for this file changes only the JSON values to num_speculative_tokens:3, synthetic_acceptance_length:3.0, and does not touch the two comment lines above it. (3) Reading the file post-PR, the comment and the config it annotates now disagree — a maintainer skimming the comment would believe this arm still runs K7/AL 3.84 when it actually runs K3/AL 3.0.
The fix is a one-line comment edit: change "Throughput runs use the committed K7 golden synthetic AL 3.84" to something like "Throughput runs use the committed K3 golden synthetic AL 3.0 (banded DSpark for this higher-concurrency offload arm)". This has no effect on runtime behavior since YAML comments are inert, so it is a documentation-only nit rather than a blocking issue.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31891188634 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31892447465 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31893639408 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31903549275 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31904552121 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31905071348 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31909756784 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31910027316 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31910303104 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31910406129 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31913443079 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31913534317 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31920547327 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31921630464 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31923518793 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31929860188 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31930157551 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31930245322 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31930356749 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31930627976 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31930693552 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=31930784480 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32101185082 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32101982974 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32101982974 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32143081226 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32143487706 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32143982420 |
2 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32143982420 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32143982420 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32174865887 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32175631733 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32179193775 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32179193775 |
dd56804 to
8c5a8b5
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32179193775 |
8c5a8b5 to
c150eb1
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32198442075 |
c150eb1 to
4cfd94f
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32198704284 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32198975090 |
4cfd94f to
afae6b6
Compare
Two-node TP8 x DP2 x EP16 with DCP8, offloading every point to a Mooncake external tier. Concurrency 1 to 8 drafts with DSpark level 7, 10 to 16 with level 3, and 18 to 24 runs unspeculated, with the golden acceptance length injected only for throughput so eval verifies against the target model. Runs on the b200-nscale cluster, rendered by upstream srt-slurm.
afae6b6 to
ced99b4
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32206823975 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32206823975 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32206823975 |
|
closed in favor of #2672 |
1 participant
No description provided.