Repository navigation
[Doc] Kimi-K3: move the GB300 AgentX presets to the Oct 26 points with KV-balanced decode - #64665
Merged
wzhao18 merged 2 commits intoOct 30, 2026
Merged
Conversation
…h KV-balanced decode Signed-off-by: Ankur Singh <ankusingh@nvidia.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Contributor
wzhao18
approved these changes
Oct 30, 2026
Contributor
|
/ci run |
wzhao18
enabled auto-merge (squash)
October 30, 2026 06:02
|
✅ Triggered Buildkite CI #98598 for commit |
github-actions
Bot
added
the
ready
wzhao18
temporarily deployed
to
recipes-preview
October 30, 2026 06:17 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
vllm-recipes-publisher Bot
pushed a commit
to vllm-project/recipes
that referenced
this pull request
Oct 30, 2026
vllm-project/vllm@main:recipes 10579b71ebcc → d0d3005affa3 [latest.vllm] vllm-project/vllm@10579b7...d0d3005 Commits: - vllm-project/vllm@d0d3005affa3 [Doc] Kimi-K3: move the GB300 AgentX presets to the Oct 26 points with KV-balanced decode (vllm-project/vllm#64665) Recipes: - moonshotai/Kimi-K3 (modified): https://recipes.vllm.ai/moonshotai/Kimi-K3 Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
github-actions Bot
added a commit
to Yogaraj-Alamenda/recipes
that referenced
this pull request
Oct 30, 2026
vllm-project/vllm@main:recipes f81a414faed6 → a20280f30e31 [latest.vllm] vllm-project/vllm@f81a414...a20280f Commits: - vllm-project/vllm@d14f72c7fe13 [Rust Frontend] Add native GLM-5.3 / GLM-5.3-Flash chat rendering (vllm-project/vllm#62424) - vllm-project/vllm@d0d3005affa3 [Doc] Kimi-K3: move the GB300 AgentX presets to the Oct 26 points with KV-balanced decode (vllm-project/vllm#64665) Recipes: - moonshotai/Kimi-K3 (modified): https://recipes.vllm.ai/moonshotai/Kimi-K3 - zai-org/GLM-5.3-Flash (modified): https://recipes.vllm.ai/zai-org/GLM-5.3-Flash - zai-org/GLM-5.3 (modified): https://recipes.vllm.ai/zai-org/GLM-5.3 Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
github-actions Bot
added a commit
to yiminghub2024/recipes
that referenced
this pull request
Oct 30, 2026
vllm-project/vllm@main:recipes f81a414faed6 → 617135eed34b [latest.vllm] vllm-project/vllm@f81a414...617135e Commits: - vllm-project/vllm@d14f72c7fe13 [Rust Frontend] Add native GLM-5.3 / GLM-5.3-Flash chat rendering (vllm-project/vllm#62424) - vllm-project/vllm@d0d3005affa3 [Doc] Kimi-K3: move the GB300 AgentX presets to the Oct 26 points with KV-balanced decode (vllm-project/vllm#64665) Recipes: - moonshotai/Kimi-K3 (modified): https://recipes.vllm.ai/moonshotai/Kimi-K3 - zai-org/GLM-5.3-Flash (modified): https://recipes.vllm.ai/zai-org/GLM-5.3-Flash - zai-org/GLM-5.3 (modified): https://recipes.vllm.ai/zai-org/GLM-5.3 Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
github-actions Bot
added a commit
to monkeyLoveding/recipes
that referenced
this pull request
Oct 30, 2026
…mni@cda4a1ef83b2 vllm-project/vllm@main:recipes 8084f0624dba → 24420b7a341a [latest.vllm] vllm-project/vllm@8084f06...24420b7 Commits: - vllm-project/vllm@d14f72c7fe13 [Rust Frontend] Add native GLM-5.3 / GLM-5.3-Flash chat rendering (vllm-project/vllm#62424) - vllm-project/vllm@d0d3005affa3 [Doc] Kimi-K3: move the GB300 AgentX presets to the Oct 26 points with KV-balanced decode (vllm-project/vllm#64665) Recipes: - moonshotai/Kimi-K3 (modified): https://recipes.vllm.ai/moonshotai/Kimi-K3 - zai-org/GLM-5.3-Flash (modified): https://recipes.vllm.ai/zai-org/GLM-5.3-Flash - zai-org/GLM-5.3 (modified): https://recipes.vllm.ai/zai-org/GLM-5.3 vllm-project/vllm-omni@main:recipes e45e43bcc1d4 → cda4a1ef83b2 [latest.vllm-omni] vllm-project/vllm-omni@e45e43b...cda4a1e Commits: - vllm-project/vllm-omni@b5a6fd836185 [Model] MiniMax-H3: Continue a previous video from cached latents across requests (vllm-project/vllm-omni#9342) Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Update the two GB300 Kimi-K3 AgentX disaggregated presets in
recipes/models/moonshotai/Kimi-K3.yamlto the lanes qualified in SemiAnalysisAI/InferenceX#4638 (headbc4d49f5, sweep run 40830592414, attempt 1):kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg(presetgb300_agentx_pd) andkimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dep16-disagg(presetgb300_agentx_pd_dep16). Both lanes now run imagevllm/vllm-openai:nightly-3966924d566340e971a6da313e790d05492aa505(vLLM0.33.0rc1.dev229+g3966924d5in the run logs) with Dynamoa0de541ea8c861d5ac01d15953ca529bea68637b, and their throughput points run decode attention onKIMI_K3_KV_BALANCED_MLA.Each preset now has one
points.<id>per measured operating point, and the worker commands, environment block and Mooncake connector move out of the runbooks into structured fields. No open PR updates these lanes or their points; #64586 (drop dead envs from Kimi-K3) and #64617 (a B300 preset) edit the same file, so whichever merges second needs a rebase.Claims
gb300_agentx_pd(AgentX xPyD, default on GB300pd_cluster, as before):sourceis [Frontend] Dynamic RoPE scaling #4638 atbc4d49f5df307680be5a1567e01b838b83b31b62. The image is pinned by digest (@sha256:f255c4fa…), and the Dynamo commit is unchanged.TOKENSPEED_MLA,latency_specialized,--max-num-seqs 12):1p3d_c16,1p2d_c24(the default point, as the preset rendered before) and1p2d_c32.KIMI_K3_KV_BALANCED_MLA):2p4d_c112,2p4d_c128,2p3d_c128,2p3d_c152,3p4d_c240,3p3d_c240and3p3d_c264.--attention-backend KIMI_K3_KV_BALANCED_MLA, with the DSpark draft'sattention_backendon the same backend;tokenspeed_mla_min_split_kv: 8in--attention-config;--max-num-seqs 128,--max-cudagraph-capture-size 512and--use-replayssm; vLLM's default CUDA graph mode.TOKENSPEED_MLAfor target and draft.--cp-kv-cache-interleave-size 1and autotune off.MultiConnectorfrom the runbook into structured fields. It comes fromselect.kv_offload: kv_store_distributed_mooncake, withjson_argsfor the lane's connector options: recompute on failed loads,kv_buffer_device: cuda,enforce_handshake_compat: false,enable_cross_layers_blocks: falseandenable_offload: false.--data-parallel-hybrid-lband--no-enable-prefix-cachingwithremove_args, replacing the earlier undo flags.VLLM_KIMI_K3_SHARD_SP_SHARED_EXPERT=0on decode too,DYN_TCP_REQUEST_TIMEOUT=120andETCD_LEASE_TTL=600.--kv-events-config) are removed, because the lane no longer publishes them.gb300_agentx_pd_dep16(AgentX 3P1D DEP16, opt-in):sourceis [Frontend] Dynamic RoPE scaling #4638 at the same commit, and the image is the same digest-pinned image.KIMI_K3_KV_BALANCED_MLA. Prefill stays onTOKENSPEED_MLA.c256,c288andc336(default). The lane runs one configuration at all three client counts.select+json_args. The decode role also sets the store child tokv_consumerwithenable_lookup: false.Runbooks keep the evidence and processes the builder does not render:
least-loaded, session affinity).The duplicated Workers/Environment command blocks are removed.
meta.date_updatedis set to 2026-10-29.Validation
node scripts/validate-source.mjs --source vllm --dir <this checkout>/recipespasses (vllm-project/recipes @2ac6bc4): 220 recipes, JSON API built.I built the full JSON API before and after the change. Only Kimi-K3 family files change:
moonshotai/Kimi-K3.jsonand its promoted variantsRedHatAI/Kimi-K3-NVFP4,RedHatAI/Kimi-K3-FP8-BLOCKandEco-Tech/Kimi-K3-w4a8(onlypresetsandmeta.date_updatedchange);presets/gb300_agentx_pd.jsonandpresets/gb300_agentx_pd_dep16.json;hw/gb300/strategies/pd_cluster.jsonfiles.No
strategy_overrides, hardware or variant field changes, so the Baseline (non-preset) commands are unchanged on every GPU and strategy. The default GB300pd_clusterrendering (default preset, point1p2d_c24) changes only in:MultiConnectorwithMOONCAKE_CONFIG_PATHandPYTHONHASHSEED=0, which the builder's Mooncake option adds;Against the source: every one of the 13 rendered points matches the lane's effective arguments and environment on both roles at the PR head. The script merges
disagg-variants.yaml/disagg-tp8-dep16.yamlbase roles with each point's override and checks against the master config's search space: concurrency, prefill and decode worker counts, two trays per TP8 worker and four per DEP16 worker, and the image. The only flags the builder adds are restatements of the lane's effective values:--no-disable-hybrid-kv-cache-manager;--max-num-batched-tokens 16384on the xPyD pools;--data-parallel-hybrid-lbon the DEP16 decode, which the lane's per-node DP launch also adds;--disable-flashinfer-autotunenext to--no-enable-flashinfer-autotuneon the specialized prefill.Against the logs: the rendered
2p4d_c112andc256prefill and decode commands and environment match the launch commands logged in run 40830592414 (2P4D c112 and 3P1D c256 eval jobs), apart from placement, parser and InferenceX plumbing (listed below). The worker logs show:Using AttentionBackendEnum.KIMI_K3_KV_BALANCED_MLA backend(32x per job) and prefill onTOKENSPEED_MLA;max_num_batched_tokens=16384on both xPyD roles;FULL_AND_PIECEWISE.Accuracy, from the sweep's eval jobs with block verification: Kimi tool-call schema
em_strict0.985-0.998 (n=408) at every point, and GSM8Kem_strict0.966-0.972 (n=1319) where run.Details
Schema limitations:
nodesstays at two trays per TP8 worker, because a largernodeswould render one wider TP group instead of more workers.gb300_low_latencyis unchanged. The #4638 aggregateddcp8-aggc1 point is an image-only bump, and its serving flags already match that preset'stp16point, apart from the image and the synthetic acceptance. A point can pin its ownimage, butsource(href/key/revision) exists only on the preset, and that preset cites vllm-project/recipes#1134 with the rollingnightlyimage for both its points. Citing the InferenceX key would mean re-sourcing or splitting that preset, so I left it.Excluded as InferenceX-only, placement or no-op:
rejection_sample_method: synthetic,synthetic_acceptance_lengthon throughput runs; the presets keep block verification), AIPerf/trace-replay settings,dram-utilizationand Tachometer (disabled in [Frontend] Dynamic RoPE scaling #4638).--served-model-nameand the local/modelpath,--device-ids,--dump-config-to,--request-plane tcp(the builder setsDYN_REQUEST_PLANE=tcp), Slurm/container/health-check settings,VLLM_SERVER_DEV_MODE,TILELANG_CLEANUP_TEMP_FILES,WITH_NVIDIA_PEERMEM,NCCL_NET_PLUGIN, the per-host Mooncake/NIXL/OTEL addresses, andNVSHMEM_HCA_LIST(a site value; the runbook says to set it).--dyn-tool-call-parser,--dyn-reasoning-parserand--dyn-enable-structural-tag, which the runbook documents.VLLM_RPC_TIMEOUTandVLLM_PREFIX_CACHE_RETENTION_INTERVAL. The lane sets them, but vLLM at the pinned commit3966924dreads neither (git grepfinds no reference undervllm/), andprefix_cache_retention_intervalalready defaults to 0 (vllm/config/cache.py). [Doc][Recipe] Kimi-K3: drop removed retention and RPC timeout envs #64586 removes them from the other Kimi-K3 presets.Known renderer behaviour: like the other presets that tune a composed connector by name, the
json_argson--kv-transfer-configassume the Mooncake store selected by the preset. If a user turns that KV offload option off while the preset is active, the renderer raises an error for the missingMooncakeStoreConnectorchild.Generated preset JSON, before and after
Condensed: the default point in full (
docker_*/commandfields and per-host env omitted), then each other point's differences from it.public/moonshotai/Kimi-K3/presets/gb300_agentx_pd.jsonbefore (condensed){ "preset": "gb300_agentx_pd", "label": "AgentX xPyD", "hardware": "gb300", "strategy": "pd_cluster", "default": true, "source": { "label": "InferenceX #4047", "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/4047", "key": "kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg" }, "image": { "vllm": "vllm/vllm-openai:nightly-e4a5b5537344b103ed10de7764378ef09604ef3d", "dynamo": "a0de541ea8c861d5ac01d15953ca529bea68637b" }, "points": { "default": { "prefill": { "nodes": 2, "parallelism": "tep", "argv": [ "--model moonshotai/Kimi-K3", "--trust-remote-code", "--disaggregation-mode prefill", "--kv-transfer-config {\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"fail\"}", "--tensor-parallel-size 8", "--enable-expert-parallel", "--nnodes 2", "--node-rank 0", "--master-addr $PREFILL_NODE_1", "--max-num-batched-tokens 16384", "--no-disable-hybrid-kv-cache-manager", "--disable-flashinfer-autotune", "--decode-context-parallel-size 8", "--dcp-comm-backend a2a", "--attention-backend TOKENSPEED_MLA", "--kv-cache-dtype fp8", "--mamba-ssm-cache-dtype float32", "--cp-kv-cache-interleave-size 1", "--enable-prefix-caching", "--prefix-match-unit 128", "--gpu-memory-utilization 0.95", "--enable-cumem-allocator", "--no-enable-flashinfer-autotune", "--load-format fastsafetensors", "--stream-interval 10", "--language-model-only", "--speculative-config {\"model\":\"Inferact/Kimi-K3-DSpark\",\"attention_backend\":\"TOKENSPEED_MLA\",\"method\":\"dspark\",\"num_speculative_tokens\":4,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}", "--moe-backend deep_gemm_mega_moe", "--attention-config {\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true,\"trtllm_ragged_backend\":\"cute-dsl\"}", "--max-num-seqs 32", "--max-cudagraph-capture-size 8192", "--compilation-config {\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"cudagraph_capture_sizes\":[4,8,12,16,20,24,32,40,48,64,80,96,128,160,256,384,512,640,768,896,1024,1152,1280,1408,1536,1664,1792,1920,2048,2304,2560,2816,3072,3328,3584,3840,4096,4608,5120,5632,6144,6656,7168,7680,8192]}", "--reasoning-parser kimi_k3" ], "env": { "NCCL_CUMEM_ENABLE": "1", "NCCL_MNNVL_ENABLE": "1", "NCCL_NVLS_ENABLE": "1", "UCX_NET_DEVICES": "all", "DYN_REQUEST_PLANE": "tcp", "VLLM_SSM_CONV_STATE_LAYOUT": "DS", "VLLM_KIMI_K3_SHARD_SP_SHARED_EXPERT": "0", "UCX_TLS": "all", "VLLM_USE_DIRECT_DCP_A2A": "1", "VLLM_USE_DIRECT_DCP_Q_GATHER": "1", "VLLM_USE_DIRECT_DCP_KV_GATHER": "1", "VLLM_ALLREDUCE_USE_FLASHINFER": "1", "VLLM_USE_NCCL_SYMM_MEM": "0", "VLLM_USE_V2_MODEL_RUNNER": "1", "VLLM_PREFIX_CACHE_RETENTION_INTERVAL": "0", "NCCL_P2P_LEVEL": "NVL", "VLLM_MOONCAKE_LOAD_RECV_THREADS": "4", "MC_SLICE_SIZE": "1048576", "MC_MAX_MR_SIZE": "4294967296", "MC_ENABLE_DEST_DEVICE_AFFINITY": "1", "UCX_MEMTYPE_CACHE": "n", "UCX_MEMTYPE_REG_WHOLE": "n", "UCX_RCACHE_MAX_UNRELEASED": "1024", "UCX_TCP_AF_PRIO": "inet", "TOKENSPEED_MLA_FP16_PARTIALS": "0", "VLLM_ENGINE_READY_TIMEOUT_S": "3600", "VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS": "1800", "VLLM_RPC_TIMEOUT": "600000", "VLLM_ENABLE_GEMM_RS": "1", "DYN_BACKEND_ADMISSION_ENGINE_REQUEST_LIMIT": "1024", "DYN_BACKEND_ADMISSION_ENGINE_REQUEST_WAITING_LIMIT": "1024", "VLLM_USE_RUST_FRONTEND": "1" } }, "decode": { "nodes": 2, "parallelism": "tp", "argv": [ "--model moonshotai/Kimi-K3", "--trust-remote-code", "--disaggregation-mode decode", "--kv-transfer-config {\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"fail\"}", "--tensor-parallel-size 8", "--nnodes 2", "--node-rank 0", "--master-addr $DECODE_NODE_1", "--data-parallel-hybrid-lb", "--no-enable-prefix-caching", "--no-disable-hybrid-kv-cache-manager", "--decode-context-parallel-size 8", "--dcp-comm-backend a2a", "--attention-backend TOKENSPEED_MLA", "--kv-cache-dtype fp8", "--mamba-ssm-cache-dtype float32", "--cp-kv-cache-interleave-size 1", "--enable-prefix-caching", "--prefix-match-unit 128", "--gpu-memory-utilization 0.95", "--enable-cumem-allocator", "--no-enable-flashinfer-autotune", "--load-format fastsafetensors", "--stream-interval 10", "--language-model-only", "--speculative-config {\"model\":\"Inferact/Kimi-K3-DSpark\",\"attention_backend\":\"TOKENSPEED_MLA\",\"method\":\"dspark\",\"num_speculative_tokens\":4,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}", "--no-data-parallel-hybrid-lb", "--max-num-batched-tokens 16384", "--additional-config {\"decode_backend\":\"latency_specialized\"}", "--max-num-seqs 12", "--max-cudagraph-capture-size 64", "--compilation-config {\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"cudagraph_capture_sizes\":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64]}", "--attention-config {\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true}", "--reasoning-parser kimi_k3" ], "env": { "DYN_REQUEST_PLANE": "tcp", "VLLM_SSM_CONV_STATE_LAYOUT": "DS", "UCX_TLS": "all", "VLLM_USE_DIRECT_DCP_A2A": "1", "VLLM_USE_DIRECT_DCP_Q_GATHER": "1", "VLLM_USE_DIRECT_DCP_KV_GATHER": "1", "VLLM_ALLREDUCE_USE_FLASHINFER": "1", "VLLM_USE_NCCL_SYMM_MEM": "0", "VLLM_USE_V2_MODEL_RUNNER": "1", "VLLM_PREFIX_CACHE_RETENTION_INTERVAL": "0", "NCCL_CUMEM_ENABLE": "1", "NCCL_MNNVL_ENABLE": "1", "NCCL_NVLS_ENABLE": "1", "NCCL_P2P_LEVEL": "NVL", "VLLM_MOONCAKE_LOAD_RECV_THREADS": "4", "MC_SLICE_SIZE": "1048576", "MC_MAX_MR_SIZE": "4294967296", "MC_ENABLE_DEST_DEVICE_AFFINITY": "1", "UCX_MEMTYPE_CACHE": "n", "UCX_MEMTYPE_REG_WHOLE": "n", "UCX_RCACHE_MAX_UNRELEASED": "1024", "UCX_TCP_AF_PRIO": "inet", "TOKENSPEED_MLA_FP16_PARTIALS": "0", "VLLM_ENGINE_READY_TIMEOUT_S": "3600", "VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS": "1800", "VLLM_RPC_TIMEOUT": "600000", "VLLM_ENABLE_GEMM_RS": "0", "VLLM_USE_RUST_FRONTEND": "1" } } } } }public/moonshotai/Kimi-K3/presets/gb300_agentx_pd.jsonafter (condensed){ "preset": "gb300_agentx_pd", "label": "AgentX xPyD", "hardware": "gb300", "strategy": "pd_cluster", "default": true, "source": { "label": "InferenceX #4638", "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/4638", "key": "kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg", "revision": "bc4d49f5df307680be5a1567e01b838b83b31b62" }, "image": { "vllm": "vllm/vllm-openai:nightly-3966924d566340e971a6da313e790d05492aa505@sha256:f255c4fad2f30ff5ee80db14d3a03afe9ae0f10d3470339864ff04745fe44cb3", "dynamo": "a0de541ea8c861d5ac01d15953ca529bea68637b" }, "default_point": "1p2d_c24", "points": { "1p3d_c16": { "label": "16 (1P3D)", "concurrency": 16, "prefill": "same as default point", "decode": "same as default point" }, "1p2d_c24": { "label": "24 (1P2D)", "concurrency": 24, "prefill": { "nodes": 2, "parallelism": "tep", "argv": [ "--model moonshotai/Kimi-K3", "--trust-remote-code", "--disaggregation-mode prefill", "--tensor-parallel-size 8", "--enable-expert-parallel", "--nnodes 2", "--node-rank 0", "--master-addr $PREFILL_NODE_1", "--max-num-batched-tokens 16384", "--no-disable-hybrid-kv-cache-manager", "--disable-flashinfer-autotune", "--decode-context-parallel-size 8", "--dcp-comm-backend a2a", "--attention-backend TOKENSPEED_MLA", "--kv-cache-dtype fp8", "--enable-prefix-caching", "--prefix-match-unit 128", "--gpu-memory-utilization 0.95", "--enable-cumem-allocator", "--load-format fastsafetensors", "--stream-interval 10", "--language-model-only", "--speculative-config {\"model\":\"Inferact/Kimi-K3-DSpark\",\"attention_backend\":\"TOKENSPEED_MLA\",\"method\":\"dspark\",\"num_speculative_tokens\":4,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}", "--mamba-ssm-cache-dtype float32", "--cp-kv-cache-interleave-size 1", "--no-enable-flashinfer-autotune", "--moe-backend deep_gemm_mega_moe", "--attention-config {\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true,\"trtllm_ragged_backend\":\"cute-dsl\"}", "--max-num-seqs 32", "--max-cudagraph-capture-size 8192", "--compilation-config {\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"cudagraph_capture_sizes\":[4,8,12,16,20,24,32,40,48,64,80,96,128,160,256,384,512,640,768,896,1024,1152,1280,1408,1536,1664,1792,1920,2048,2304,2560,2816,3072,3328,3584,3840,4096,4608,5120,5632,6144,6656,7168,7680,8192]}", "--reasoning-parser kimi_k3", "--kv-transfer-config {\"kv_connector\":\"MultiConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"connectors\":[{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"fail\",\"kv_buffer_device\":\"cuda\",\"kv_connector_extra_config\":{\"enforce_handshake_compat\":false,\"enable_cross_layers_blocks\":false}},{\"kv_connector\":\"MooncakeStoreConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"load_async\":true,\"lookup_async\":true,\"enable_cross_layers_blocks\":false,\"enable_offload\":false},\"kv_load_failure_policy\":\"recompute\"}]},\"kv_load_failure_policy\":\"recompute\"}" ], "env": { "NCCL_CUMEM_ENABLE": "1", "NCCL_MNNVL_ENABLE": "1", "NCCL_NVLS_ENABLE": "1", "UCX_NET_DEVICES": "all", "DYN_REQUEST_PLANE": "tcp", "MOONCAKE_CONFIG_PATH": "$MOONCAKE_CONFIG_PATH", "PYTHONHASHSEED": 0, "VLLM_SSM_CONV_STATE_LAYOUT": "DS", "VLLM_KIMI_K3_SHARD_SP_SHARED_EXPERT": "0", "UCX_TLS": "all", "VLLM_USE_DIRECT_DCP_A2A": "1", "VLLM_USE_DIRECT_DCP_Q_GATHER": "1", "VLLM_USE_DIRECT_DCP_KV_GATHER": "1", "VLLM_ALLREDUCE_USE_FLASHINFER": "1", "VLLM_USE_NCCL_SYMM_MEM": "0", "VLLM_USE_V2_MODEL_RUNNER": "1", "NCCL_P2P_LEVEL": "NVL", "VLLM_MOONCAKE_LOAD_RECV_THREADS": "4", "MC_SLICE_SIZE": "1048576", "MC_MAX_MR_SIZE": "4294967296", "MC_ENABLE_DEST_DEVICE_AFFINITY": "1", "UCX_MEMTYPE_CACHE": "n", "UCX_MEMTYPE_REG_WHOLE": "n", "UCX_RCACHE_MAX_UNRELEASED": "1024", "UCX_TCP_AF_PRIO": "inet", "TOKENSPEED_MLA_FP16_PARTIALS": "0", "DYN_TCP_REQUEST_TIMEOUT": "120", "ETCD_LEASE_TTL": "600", "VLLM_ENGINE_READY_TIMEOUT_S": "3600", "VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS": "1800", "VLLM_ENABLE_GEMM_RS": "1", "DYN_BACKEND_ADMISSION_ENGINE_REQUEST_LIMIT": "1024", "DYN_BACKEND_ADMISSION_ENGINE_REQUEST_WAITING_LIMIT": "1024", "VLLM_USE_RUST_FRONTEND": "1" } }, "decode": { "nodes": 2, "parallelism": "tp", "argv": [ "--model moonshotai/Kimi-K3", "--trust-remote-code", "--disaggregation-mode decode", "--tensor-parallel-size 8", "--nnodes 2", "--node-rank 0", "--master-addr $DECODE_NODE_1", "--no-disable-hybrid-kv-cache-manager", "--decode-context-parallel-size 8", "--dcp-comm-backend a2a", "--attention-backend TOKENSPEED_MLA", "--kv-cache-dtype fp8", "--enable-prefix-caching", "--prefix-match-unit 128", "--gpu-memory-utilization 0.95", "--enable-cumem-allocator", "--load-format fastsafetensors", "--stream-interval 10", "--language-model-only", "--speculative-config {\"model\":\"Inferact/Kimi-K3-DSpark\",\"attention_backend\":\"TOKENSPEED_MLA\",\"method\":\"dspark\",\"num_speculative_tokens\":4,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}", "--mamba-ssm-cache-dtype float32", "--cp-kv-cache-interleave-size 1", "--no-enable-flashinfer-autotune", "--max-num-batched-tokens 16384", "--attention-config {\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true}", "--additional-config {\"decode_backend\":\"latency_specialized\"}", "--max-num-seqs 12", "--max-cudagraph-capture-size 64", "--compilation-config {\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"cudagraph_capture_sizes\":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64]}", "--reasoning-parser kimi_k3", "--kv-transfer-config {\"kv_connector\":\"MultiConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"connectors\":[{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"fail\",\"kv_buffer_device\":\"cuda\",\"kv_connector_extra_config\":{\"enforce_handshake_compat\":false,\"enable_cross_layers_blocks\":false}},{\"kv_connector\":\"MooncakeStoreConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"load_async\":true,\"lookup_async\":true,\"enable_cross_layers_blocks\":false,\"enable_offload\":false},\"kv_load_failure_policy\":\"recompute\"}]},\"kv_load_failure_policy\":\"recompute\"}" ], "env": { "DYN_REQUEST_PLANE": "tcp", "MOONCAKE_CONFIG_PATH": "$MOONCAKE_CONFIG_PATH", "PYTHONHASHSEED": 0, "VLLM_SSM_CONV_STATE_LAYOUT": "DS", "UCX_TLS": "all", "VLLM_USE_DIRECT_DCP_A2A": "1", "VLLM_USE_DIRECT_DCP_Q_GATHER": "1", "VLLM_USE_DIRECT_DCP_KV_GATHER": "1", "VLLM_ALLREDUCE_USE_FLASHINFER": "1", "VLLM_KIMI_K3_SHARD_SP_SHARED_EXPERT": "0", "VLLM_USE_NCCL_SYMM_MEM": "0", "VLLM_USE_V2_MODEL_RUNNER": "1", "NCCL_CUMEM_ENABLE": "1", "NCCL_MNNVL_ENABLE": "1", "NCCL_NVLS_ENABLE": "1", "NCCL_P2P_LEVEL": "NVL", "VLLM_MOONCAKE_LOAD_RECV_THREADS": "4", "MC_SLICE_SIZE": "1048576", "MC_MAX_MR_SIZE": "4294967296", "MC_ENABLE_DEST_DEVICE_AFFINITY": "1", "UCX_MEMTYPE_CACHE": "n", "UCX_MEMTYPE_REG_WHOLE": "n", "UCX_RCACHE_MAX_UNRELEASED": "1024", "UCX_TCP_AF_PRIO": "inet", "TOKENSPEED_MLA_FP16_PARTIALS": "0", "DYN_TCP_REQUEST_TIMEOUT": "120", "ETCD_LEASE_TTL": "600", "VLLM_ENGINE_READY_TIMEOUT_S": "3600", "VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS": "1800", "VLLM_ENABLE_GEMM_RS": "0", "VLLM_USE_RUST_FRONTEND": "1" } } }, "1p2d_c32": { "label": "32 (1P2D)", "concurrency": 32, "prefill": "same as default point", "decode": "same as default point" }, "2p4d_c112": { "label": "112 (2P4D)", "concurrency": 112, "prefill": { "argv (changed or added vs default point)": [ "--mamba-ssm-cache-dtype bfloat16", "--enable-flashinfer-autotune" ], "argv removed vs default point": [ "--disable-flashinfer-autotune", "--cp-kv-cache-interleave-size", "--no-enable-flashinfer-autotune" ] }, "decode": { "argv (changed or added vs default point)": [ "--speculative-config {\"model\":\"Inferact/Kimi-K3-DSpark\",\"attention_backend\":\"KIMI_K3_KV_BALANCED_MLA\",\"method\":\"dspark\",\"num_speculative_tokens\":4,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}", "--mamba-ssm-cache-dtype bfloat16", "--enable-flashinfer-autotune", "--attention-config {\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true,\"tokenspeed_mla_min_split_kv\":8}", "--attention-backend KIMI_K3_KV_BALANCED_MLA", "--max-num-seqs 128", "--max-cudagraph-capture-size 512", "--use-replayssm" ], "argv removed vs default point": [ "--cp-kv-cache-interleave-size", "--no-enable-flashinfer-autotune", "--additional-config", "--compilation-config" ] } }, "2p4d_c128": { "label": "128 (2P4D)", "concurrency": 128, "prefill": { "argv (changed or added vs default point)": [ "--mamba-ssm-cache-dtype bfloat16", "--enable-flashinfer-autotune" ], "argv removed vs default point": [ "--disable-flashinfer-autotune", "--cp-kv-cache-interleave-size", "--no-enable-flashinfer-autotune" ] }, "decode": { "argv (changed or added vs default point)": [ "--speculative-config {\"model\":\"Inferact/Kimi-K3-DSpark\",\"attention_backend\":\"KIMI_K3_KV_BALANCED_MLA\",\"method\":\"dspark\",\"num_speculative_tokens\":4,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}", "--mamba-ssm-cache-dtype bfloat16", "--enable-flashinfer-autotune", "--attention-config {\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true,\"tokenspeed_mla_min_split_kv\":8}", "--attention-backend KIMI_K3_KV_BALANCED_MLA", "--max-num-seqs 128", "--max-cudagraph-capture-size 512", "--use-replayssm" ], "argv removed vs default point": [ "--cp-kv-cache-interleave-size", "--no-enable-flashinfer-autotune", "--additional-config", "--compilation-config" ] } }, "2p3d_c128": { "label": "128 (2P3D)", "concurrency": 128, "prefill": { "argv (changed or added vs default point)": [ "--mamba-ssm-cache-dtype bfloat16", "--enable-flashinfer-autotune" ], "argv removed vs default point": [ "--disable-flashinfer-autotune", "--cp-kv-cache-interleave-size", "--no-enable-flashinfer-autotune" ] }, "decode": { "argv (changed or added vs default point)": [ "--speculative-config {\"model\":\"Inferact/Kimi-K3-DSpark\",\"attention_backend\":\"KIMI_K3_KV_BALANCED_MLA\",\"method\":\"dspark\",\"num_speculative_tokens\":4,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}", "--mamba-ssm-cache-dtype bfloat16", "--enable-flashinfer-autotune", "--attention-config {\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true,\"tokenspeed_mla_min_split_kv\":8}", "--attention-backend KIMI_K3_KV_BALANCED_MLA", "--max-num-seqs 128", "--max-cudagraph-capture-size 512", "--use-replayssm" ], "argv removed vs default point": [ "--cp-kv-cache-interleave-size", "--no-enable-flashinfer-autotune", "--additional-config", "--compilation-config" ] } }, "2p3d_c152": { "label": "152 (2P3D)", "concurrency": 152, "prefill": { "argv (changed or added vs default point)": [ "--mamba-ssm-cache-dtype bfloat16", "--enable-flashinfer-autotune" ], "argv removed vs default point": [ "--disable-flashinfer-autotune", "--cp-kv-cache-interleave-size", "--no-enable-flashinfer-autotune" ] }, "decode": { "argv (changed or added vs default point)": [ "--speculative-config {\"model\":\"Inferact/Kimi-K3-DSpark\",\"attention_backend\":\"KIMI_K3_KV_BALANCED_MLA\",\"method\":\"dspark\",\"num_speculative_tokens\":4,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}", "--mamba-ssm-cache-dtype bfloat16", "--enable-flashinfer-autotune", "--attention-config {\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true,\"tokenspeed_mla_min_split_kv\":8}", "--attention-backend KIMI_K3_KV_BALANCED_MLA", "--max-num-seqs 128", "--max-cudagraph-capture-size 512", "--use-replayssm" ], "argv removed vs default point": [ "--cp-kv-cache-interleave-size", "--no-enable-flashinfer-autotune", "--additional-config", "--compilation-config" ] } }, "3p4d_c240": { "label": "240 (3P4D)", "concurrency": 240, "prefill": { "argv (changed or added vs default point)": [ "--mamba-ssm-cache-dtype bfloat16", "--enable-flashinfer-autotune" ], "argv removed vs default point": [ "--disable-flashinfer-autotune", "--cp-kv-cache-interleave-size", "--no-enable-flashinfer-autotune" ] }, "decode": { "argv (changed or added vs default point)": [ "--speculative-config {\"model\":\"Inferact/Kimi-K3-DSpark\",\"attention_backend\":\"KIMI_K3_KV_BALANCED_MLA\",\"method\":\"dspark\",\"num_speculative_tokens\":4,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}", "--mamba-ssm-cache-dtype bfloat16", "--enable-flashinfer-autotune", "--attention-config {\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true,\"tokenspeed_mla_min_split_kv\":8}", "--attention-backend KIMI_K3_KV_BALANCED_MLA", "--max-num-seqs 128", "--max-cudagraph-capture-size 512", "--use-replayssm" ], "argv removed vs default point": [ "--cp-kv-cache-interleave-size", "--no-enable-flashinfer-autotune", "--additional-config", "--compilation-config" ] } }, "3p3d_c240": { "label": "240 (3P3D)", "concurrency": 240, "prefill": { "argv (changed or added vs default point)": [ "--mamba-ssm-cache-dtype bfloat16", "--enable-flashinfer-autotune" ], "argv removed vs default point": [ "--disable-flashinfer-autotune", "--cp-kv-cache-interleave-size", "--no-enable-flashinfer-autotune" ] }, "decode": { "argv (changed or added vs default point)": [ "--speculative-config {\"model\":\"Inferact/Kimi-K3-DSpark\",\"attention_backend\":\"KIMI_K3_KV_BALANCED_MLA\",\"method\":\"dspark\",\"num_speculative_tokens\":4,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}", "--mamba-ssm-cache-dtype bfloat16", "--enable-flashinfer-autotune", "--attention-config {\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true,\"tokenspeed_mla_min_split_kv\":8}", "--attention-backend KIMI_K3_KV_BALANCED_MLA", "--max-num-seqs 128", "--max-cudagraph-capture-size 512", "--use-replayssm" ], "argv removed vs default point": [ "--cp-kv-cache-interleave-size", "--no-enable-flashinfer-autotune", "--additional-config", "--compilation-config" ] } }, "3p3d_c264": { "label": "264 (3P3D)", "concurrency": 264, "prefill": { "argv (changed or added vs default point)": [ "--mamba-ssm-cache-dtype bfloat16", "--enable-flashinfer-autotune" ], "argv removed vs default point": [ "--disable-flashinfer-autotune", "--cp-kv-cache-interleave-size", "--no-enable-flashinfer-autotune" ] }, "decode": { "argv (changed or added vs default point)": [ "--speculative-config {\"model\":\"Inferact/Kimi-K3-DSpark\",\"attention_backend\":\"KIMI_K3_KV_BALANCED_MLA\",\"method\":\"dspark\",\"num_speculative_tokens\":4,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}", "--mamba-ssm-cache-dtype bfloat16", "--enable-flashinfer-autotune", "--attention-config {\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true,\"tokenspeed_mla_min_split_kv\":8}", "--attention-backend KIMI_K3_KV_BALANCED_MLA", "--max-num-seqs 128", "--max-cudagraph-capture-size 512", "--use-replayssm" ], "argv removed vs default point": [ "--cp-kv-cache-interleave-size", "--no-enable-flashinfer-autotune", "--additional-config", "--compilation-config" ] } } } }public/moonshotai/Kimi-K3/presets/gb300_agentx_pd_dep16.jsonbefore (condensed){ "preset": "gb300_agentx_pd_dep16", "label": "AgentX 3P1D DEP16", "hardware": "gb300", "strategy": "pd_cluster", "default": false, "source": { "label": "InferenceX #4516", "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/4516", "key": "kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dep16-disagg" }, "image": { "vllm": "vllm/vllm-openai:nightly-e4a5b5537344b103ed10de7764378ef09604ef3d", "dynamo": "a0de541ea8c861d5ac01d15953ca529bea68637b" }, "points": { "default": { "prefill": { "nodes": 2, "parallelism": "tep", "argv": [ "--model moonshotai/Kimi-K3", "--trust-remote-code", "--disaggregation-mode prefill", "--kv-transfer-config {\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"fail\"}", "--tensor-parallel-size 8", "--nnodes 2", "--node-rank 0", "--master-addr $PREFILL_NODE_1", "--no-disable-hybrid-kv-cache-manager", "--attention-backend TOKENSPEED_MLA", "--kv-cache-dtype fp8", "--mamba-ssm-cache-dtype bfloat16", "--prefix-match-unit 128", "--gpu-memory-utilization 0.95", "--enable-expert-parallel", "--enable-ep-weight-filter", "--moe-backend deep_gemm_mega_moe", "--max-num-seqs 32", "--enable-cumem-allocator", "--enable-flashinfer-autotune", "--load-format fastsafetensors", "--stream-interval 10", "--language-model-only", "--speculative-config {\"model\":\"Inferact/Kimi-K3-DSpark\",\"attention_backend\":\"TOKENSPEED_MLA\",\"method\":\"dspark\",\"num_speculative_tokens\":4,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}", "--quantization-config {\"targets\":{\"*.in_proj_qkvgfab\":\"mxfp8\",\"*.fused_qkv_a_g_proj\":\"mxfp8\",\"*.routed_expert_down_proj\":\"mxfp8\",\"*.routed_expert_up_proj\":\"mxfp8\"}}", "--enable-prefix-caching", "--max-num-batched-tokens 16384", "--max-cudagraph-capture-size 8192", "--compilation-config {\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"cudagraph_capture_sizes\":[4,8,12,16,20,24,32,40,48,64,80,96,128,160,256,384,512,640,768,896,1024,1152,1280,1408,1536,1664,1792,1920,2048,2304,2560,2816,3072,3328,3584,3840,4096,4608,5120,5632,6144,6656,7168,7680,8192]}", "--attention-config {\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true,\"trtllm_ragged_backend\":\"cute-dsl\"}", "--reasoning-parser kimi_k3" ], "env": { "NCCL_CUMEM_ENABLE": "1", "NCCL_MNNVL_ENABLE": "1", "NCCL_NVLS_ENABLE": "1", "UCX_NET_DEVICES": "all", "DYN_REQUEST_PLANE": "tcp", "VLLM_SSM_CONV_STATE_LAYOUT": "DS", "VLLM_KIMI_K3_SHARD_SP_SHARED_EXPERT": "0", "UCX_TLS": "all", "VLLM_ALLREDUCE_USE_FLASHINFER": "1", "VLLM_ENABLE_GEMM_RS": "1", "VLLM_USE_NCCL_SYMM_MEM": "0", "VLLM_USE_V2_MODEL_RUNNER": "1", "NCCL_P2P_LEVEL": "NVL", "NVSHMEM_REMOTE_TRANSPORT": "none", "NVSHMEM_DISABLE_MNNVL": "0", "NVSHMEM_IB_ENABLE_IBGDA": "0", "NVSHMEM_ENABLE_NIC_PE_MAPPING": "1", "VLLM_MOONCAKE_LOAD_RECV_THREADS": "4", "MC_SLICE_SIZE": "1048576", "MC_MAX_MR_SIZE": "4294967296", "MC_ENABLE_DEST_DEVICE_AFFINITY": "1", "UCX_MEMTYPE_CACHE": "n", "UCX_MEMTYPE_REG_WHOLE": "n", "UCX_RCACHE_MAX_UNRELEASED": "1024", "UCX_TCP_AF_PRIO": "inet", "VLLM_ENGINE_READY_TIMEOUT_S": "3600", "VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS": "1800", "VLLM_RPC_TIMEOUT": "600000", "DYN_BACKEND_ADMISSION_ENGINE_REQUEST_LIMIT": "1024", "DYN_BACKEND_ADMISSION_ENGINE_REQUEST_WAITING_LIMIT": "1024", "VLLM_USE_RUST_FRONTEND": "1" } }, "decode": { "nodes": 4, "parallelism": "dep", "argv": [ "--model moonshotai/Kimi-K3", "--trust-remote-code", "--disaggregation-mode decode", "--kv-transfer-config {\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"fail\"}", "--tensor-parallel-size 1", "--data-parallel-size 16", "--data-parallel-size-local 4", "--data-parallel-start-rank 0", "--data-parallel-address $DECODE_NODE_1", "--data-parallel-rpc-port $DECODE_DP_RPC_PORT", "--data-parallel-hybrid-lb", "--no-enable-prefix-caching", "--no-disable-hybrid-kv-cache-manager", "--attention-backend TOKENSPEED_MLA", "--kv-cache-dtype fp8", "--mamba-ssm-cache-dtype bfloat16", "--prefix-match-unit 128", "--gpu-memory-utilization 0.95", "--enable-expert-parallel", "--enable-ep-weight-filter", "--moe-backend deep_gemm_mega_moe", "--max-num-seqs 32", "--enable-cumem-allocator", "--enable-flashinfer-autotune", "--load-format fastsafetensors", "--stream-interval 10", "--language-model-only", "--speculative-config {\"model\":\"Inferact/Kimi-K3-DSpark\",\"attention_backend\":\"TOKENSPEED_MLA\",\"method\":\"dspark\",\"num_speculative_tokens\":4,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}", "--compilation-config {\"cudagraph_mode\":\"FULL_AND_PIECEWISE\"}", "--quantization-config {\"targets\":{\"*.in_proj_qkvgfab\":\"mxfp8\",\"*.fused_qkv_a_g_proj\":\"mxfp8\",\"*.routed_expert_down_proj\":\"mxfp8\",\"*.routed_expert_up_proj\":\"mxfp8\",\"*.o_proj\":\"mxfp8\",\"*.shared_experts.gate_up_proj\":\"mxfp8\",\"*.shared_experts.down_proj\":\"mxfp8\"}}", "--max-num-batched-tokens 512", "--max-cudagraph-capture-size 512", "--use-replayssm", "--attention-config {\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true,\"tokenspeed_mla_min_split_kv\":8}", "--reasoning-parser kimi_k3" ], "env": { "DYN_REQUEST_PLANE": "tcp", "VLLM_SSM_CONV_STATE_LAYOUT": "DS", "UCX_TLS": "all", "VLLM_ALLREDUCE_USE_FLASHINFER": "1", "VLLM_ENABLE_GEMM_RS": "1", "VLLM_USE_NCCL_SYMM_MEM": "0", "VLLM_USE_V2_MODEL_RUNNER": "1", "NCCL_CUMEM_ENABLE": "1", "NCCL_MNNVL_ENABLE": "1", "NCCL_NVLS_ENABLE": "1", "NCCL_P2P_LEVEL": "NVL", "NVSHMEM_REMOTE_TRANSPORT": "none", "NVSHMEM_DISABLE_MNNVL": "0", "NVSHMEM_IB_ENABLE_IBGDA": "0", "NVSHMEM_ENABLE_NIC_PE_MAPPING": "1", "VLLM_MOONCAKE_LOAD_RECV_THREADS": "4", "MC_SLICE_SIZE": "1048576", "MC_MAX_MR_SIZE": "4294967296", "MC_ENABLE_DEST_DEVICE_AFFINITY": "1", "UCX_MEMTYPE_CACHE": "n", "UCX_MEMTYPE_REG_WHOLE": "n", "UCX_RCACHE_MAX_UNRELEASED": "1024", "UCX_TCP_AF_PRIO": "inet", "VLLM_ENGINE_READY_TIMEOUT_S": "3600", "VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS": "1800", "VLLM_RPC_TIMEOUT": "600000", "VLLM_USE_RUST_FRONTEND": "1" } } } } }public/moonshotai/Kimi-K3/presets/gb300_agentx_pd_dep16.jsonafter (condensed){ "preset": "gb300_agentx_pd_dep16", "label": "AgentX 3P1D DEP16", "hardware": "gb300", "strategy": "pd_cluster", "default": false, "source": { "label": "InferenceX #4638", "href": "https://github.com/SemiAnalysisAI/InferenceX/pull/4638", "key": "kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dep16-disagg", "revision": "bc4d49f5df307680be5a1567e01b838b83b31b62" }, "image": { "vllm": "vllm/vllm-openai:nightly-3966924d566340e971a6da313e790d05492aa505@sha256:f255c4fad2f30ff5ee80db14d3a03afe9ae0f10d3470339864ff04745fe44cb3", "dynamo": "a0de541ea8c861d5ac01d15953ca529bea68637b" }, "default_point": "c336", "points": { "c256": { "label": "256 (3P1D)", "concurrency": 256, "prefill": "same as default point", "decode": "same as default point" }, "c288": { "label": "288 (3P1D)", "concurrency": 288, "prefill": "same as default point", "decode": "same as default point" }, "c336": { "label": "336 (3P1D)", "concurrency": 336, "prefill": { "nodes": 2, "parallelism": "tep", "argv": [ "--model moonshotai/Kimi-K3", "--trust-remote-code", "--disaggregation-mode prefill", "--tensor-parallel-size 8", "--nnodes 2", "--node-rank 0", "--master-addr $PREFILL_NODE_1", "--no-disable-hybrid-kv-cache-manager", "--attention-backend TOKENSPEED_MLA", "--kv-cache-dtype fp8", "--mamba-ssm-cache-dtype bfloat16", "--prefix-match-unit 128", "--gpu-memory-utilization 0.95", "--enable-expert-parallel", "--enable-ep-weight-filter", "--moe-backend deep_gemm_mega_moe", "--max-num-seqs 32", "--enable-cumem-allocator", "--enable-flashinfer-autotune", "--load-format fastsafetensors", "--stream-interval 10", "--language-model-only", "--speculative-config {\"model\":\"Inferact/Kimi-K3-DSpark\",\"attention_backend\":\"TOKENSPEED_MLA\",\"method\":\"dspark\",\"num_speculative_tokens\":4,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}", "--quantization-config {\"targets\":{\"*.in_proj_qkvgfab\":\"mxfp8\",\"*.fused_qkv_a_g_proj\":\"mxfp8\",\"*.routed_expert_down_proj\":\"mxfp8\",\"*.routed_expert_up_proj\":\"mxfp8\"}}", "--enable-prefix-caching", "--max-num-batched-tokens 16384", "--max-cudagraph-capture-size 8192", "--compilation-config {\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"cudagraph_capture_sizes\":[4,8,12,16,20,24,32,40,48,64,80,96,128,160,256,384,512,640,768,896,1024,1152,1280,1408,1536,1664,1792,1920,2048,2304,2560,2816,3072,3328,3584,3840,4096,4608,5120,5632,6144,6656,7168,7680,8192]}", "--attention-config {\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true,\"trtllm_ragged_backend\":\"cute-dsl\"}", "--reasoning-parser kimi_k3", "--kv-transfer-config {\"kv_connector\":\"MultiConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"connectors\":[{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"fail\",\"kv_buffer_device\":\"cuda\",\"kv_connector_extra_config\":{\"enforce_handshake_compat\":false,\"enable_cross_layers_blocks\":false}},{\"kv_connector\":\"MooncakeStoreConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"load_async\":true,\"lookup_async\":true,\"enable_cross_layers_blocks\":false,\"enable_offload\":false},\"kv_load_failure_policy\":\"recompute\"}]},\"kv_load_failure_policy\":\"recompute\"}" ], "env": { "NCCL_CUMEM_ENABLE": "1", "NCCL_MNNVL_ENABLE": "1", "NCCL_NVLS_ENABLE": "1", "UCX_NET_DEVICES": "all", "DYN_REQUEST_PLANE": "tcp", "MOONCAKE_CONFIG_PATH": "$MOONCAKE_CONFIG_PATH", "PYTHONHASHSEED": 0, "VLLM_SSM_CONV_STATE_LAYOUT": "DS", "VLLM_KIMI_K3_SHARD_SP_SHARED_EXPERT": "0", "UCX_TLS": "all", "VLLM_ALLREDUCE_USE_FLASHINFER": "1", "VLLM_ENABLE_GEMM_RS": "1", "VLLM_USE_NCCL_SYMM_MEM": "0", "VLLM_USE_V2_MODEL_RUNNER": "1", "NCCL_P2P_LEVEL": "NVL", "NVSHMEM_REMOTE_TRANSPORT": "none", "NVSHMEM_DISABLE_MNNVL": "0", "NVSHMEM_IB_ENABLE_IBGDA": "0", "NVSHMEM_ENABLE_NIC_PE_MAPPING": "1", "VLLM_MOONCAKE_LOAD_RECV_THREADS": "4", "MC_SLICE_SIZE": "1048576", "MC_MAX_MR_SIZE": "4294967296", "MC_ENABLE_DEST_DEVICE_AFFINITY": "1", "UCX_MEMTYPE_CACHE": "n", "UCX_MEMTYPE_REG_WHOLE": "n", "UCX_RCACHE_MAX_UNRELEASED": "1024", "UCX_TCP_AF_PRIO": "inet", "DYN_TCP_REQUEST_TIMEOUT": "120", "ETCD_LEASE_TTL": "600", "VLLM_ENGINE_READY_TIMEOUT_S": "3600", "VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS": "1800", "DYN_BACKEND_ADMISSION_ENGINE_REQUEST_LIMIT": "1024", "DYN_BACKEND_ADMISSION_ENGINE_REQUEST_WAITING_LIMIT": "1024", "VLLM_USE_RUST_FRONTEND": "1" } }, "decode": { "nodes": 4, "parallelism": "dep", "argv": [ "--model moonshotai/Kimi-K3", "--trust-remote-code", "--disaggregation-mode decode", "--tensor-parallel-size 1", "--data-parallel-size 16", "--data-parallel-size-local 4", "--data-parallel-start-rank 0", "--data-parallel-address $DECODE_NODE_1", "--data-parallel-rpc-port $DECODE_DP_RPC_PORT", "--data-parallel-hybrid-lb", "--no-enable-prefix-caching", "--no-disable-hybrid-kv-cache-manager", "--kv-cache-dtype fp8", "--mamba-ssm-cache-dtype bfloat16", "--prefix-match-unit 128", "--gpu-memory-utilization 0.95", "--enable-expert-parallel", "--enable-ep-weight-filter", "--moe-backend deep_gemm_mega_moe", "--max-num-seqs 32", "--enable-cumem-allocator", "--enable-flashinfer-autotune", "--load-format fastsafetensors", "--stream-interval 10", "--language-model-only", "--speculative-config {\"model\":\"Inferact/Kimi-K3-DSpark\",\"attention_backend\":\"KIMI_K3_KV_BALANCED_MLA\",\"method\":\"dspark\",\"num_speculative_tokens\":4,\"draft_sample_method\":\"probabilistic\",\"rejection_sample_method\":\"block\"}", "--compilation-config {\"cudagraph_mode\":\"FULL_AND_PIECEWISE\"}", "--quantization-config {\"targets\":{\"*.in_proj_qkvgfab\":\"mxfp8\",\"*.fused_qkv_a_g_proj\":\"mxfp8\",\"*.routed_expert_down_proj\":\"mxfp8\",\"*.routed_expert_up_proj\":\"mxfp8\",\"*.o_proj\":\"mxfp8\",\"*.shared_experts.gate_up_proj\":\"mxfp8\",\"*.shared_experts.down_proj\":\"mxfp8\"}}", "--max-num-batched-tokens 512", "--max-cudagraph-capture-size 512", "--use-replayssm", "--attention-backend KIMI_K3_KV_BALANCED_MLA", "--attention-config {\"mla_prefill_backend\":\"TRTLLM_RAGGED\",\"use_prefill_query_quantization\":true,\"tokenspeed_mla_min_split_kv\":8}", "--reasoning-parser kimi_k3", "--kv-transfer-config {\"kv_connector\":\"MultiConnector\",\"kv_role\":\"kv_both\",\"kv_connector_extra_config\":{\"connectors\":[{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_both\",\"kv_load_failure_policy\":\"fail\",\"kv_buffer_device\":\"cuda\",\"kv_connector_extra_config\":{\"enforce_handshake_compat\":false,\"enable_cross_layers_blocks\":false}},{\"kv_connector\":\"MooncakeStoreConnector\",\"kv_role\":\"kv_consumer\",\"kv_connector_extra_config\":{\"load_async\":true,\"lookup_async\":true,\"enable_cross_layers_blocks\":false,\"enable_offload\":false,\"enable_lookup\":false},\"kv_load_failure_policy\":\"recompute\"}]},\"kv_load_failure_policy\":\"recompute\"}" ], "env": { "DYN_REQUEST_PLANE": "tcp", "MOONCAKE_CONFIG_PATH": "$MOONCAKE_CONFIG_PATH", "PYTHONHASHSEED": 0, "VLLM_SSM_CONV_STATE_LAYOUT": "DS", "UCX_TLS": "all", "VLLM_ALLREDUCE_USE_FLASHINFER": "1", "VLLM_ENABLE_GEMM_RS": "1", "VLLM_USE_NCCL_SYMM_MEM": "0", "VLLM_USE_V2_MODEL_RUNNER": "1", "NCCL_CUMEM_ENABLE": "1", "NCCL_MNNVL_ENABLE": "1", "NCCL_NVLS_ENABLE": "1", "NCCL_P2P_LEVEL": "NVL", "NVSHMEM_REMOTE_TRANSPORT": "none", "NVSHMEM_DISABLE_MNNVL": "0", "NVSHMEM_IB_ENABLE_IBGDA": "0", "NVSHMEM_ENABLE_NIC_PE_MAPPING": "1", "VLLM_MOONCAKE_LOAD_RECV_THREADS": "4", "MC_SLICE_SIZE": "1048576", "MC_MAX_MR_SIZE": "4294967296", "MC_ENABLE_DEST_DEVICE_AFFINITY": "1", "UCX_MEMTYPE_CACHE": "n", "UCX_MEMTYPE_REG_WHOLE": "n", "UCX_RCACHE_MAX_UNRELEASED": "1024", "UCX_TCP_AF_PRIO": "inet", "DYN_TCP_REQUEST_TIMEOUT": "120", "ETCD_LEASE_TTL": "600", "VLLM_ENGINE_READY_TIMEOUT_S": "3600", "VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS": "1800", "VLLM_USE_RUST_FRONTEND": "1" } } } } }Pull Request Checklist
I used vLLM's
/pr-checklistskill. (Mandatory for agents, optional for humans).AI assistance was used during the creation of this PR.
Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.
Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.
Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.
Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.