Repository navigation
[Klaud Cold][agentic experiment][Variant J] Kimi-K3 B200 agg TP8xPP2 agentic — DSpark @ golden AL + prefix-cache retention 0 / Kimi-K3 B200 聚合式 TP8xPP2 智能体实验——DSpark 黄金 AL + 前缀缓存留存 0 #2379
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| #!/bin/bash | ||
| # Setup script for the Kimi-K3 vLLM bring-up image (vllm/vllm-openai:kimi-k3). | ||
| # srt-slurm runs this in every worker container before dynamo install and | ||
| # worker startup (recipe field: setup_script). | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| # The image's first decode step crashes in the KDA hybrid-state postprocess: | ||
| # vllm/v1/worker/gpu/model_states/mamba_hybrid.py, postprocess_state: | ||
| # IndexError: index_fill_(): Expected dtype int64 for index. | ||
| # torch's index_fill_ requires an int64 index tensor, but the runner passes | ||
| # the int32 idx_mapping (hit by moonshotai/Kimi-K3 agentic bring-up, first | ||
| # decode step, engine v0.1.dev19262+gb6bbf29dd). Coerce the index to int64. | ||
| # Idempotent: exits 0 if the patch is already applied. | ||
| python3 - <<'PY' | ||
| import pathlib | ||
| import re | ||
|
|
||
| import vllm.v1.worker.gpu.model_states.mamba_hybrid as mh | ||
|
|
||
| path = pathlib.Path(mh.__file__) | ||
| src = path.read_text() | ||
| if "idx_mapping.long()" in src: | ||
| print(f"mamba_hybrid index_fill_ patch already applied: {path}") | ||
| raise SystemExit(0) | ||
|
|
||
| new, n = re.subn( | ||
| r"index_fill_\(\s*0,\s*idx_mapping,", | ||
| "index_fill_(0, idx_mapping.long(),", | ||
| src, | ||
| ) | ||
| if n != 1: | ||
| raise SystemExit( | ||
| f"expected exactly one index_fill_(0, idx_mapping, ...) call in " | ||
| f"{path}, found {n} — image layout changed, refusing to patch" | ||
| ) | ||
| path.write_text(new) | ||
| print(f"Patched mamba_hybrid index_fill_ index dtype: {path}") | ||
| PY |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| diff --git a/src/srtctl/backends/vllm.py b/src/srtctl/backends/vllm.py | ||
| index 74f673b..377606a 100644 | ||
| --- a/src/srtctl/backends/vllm.py | ||
| +++ b/src/srtctl/backends/vllm.py | ||
| @@ -716,25 +716,31 @@ class VLLMProtocol: | ||
| if frontend_type == "vllm": | ||
| if mode != "agg": | ||
| raise ValueError("frontend.type: vllm supports aggregate vLLM jobs only") | ||
| - if is_multi_node: | ||
| - raise ValueError("frontend.type: vllm currently supports single-node aggregate jobs only") | ||
|
|
||
| config.pop("host", None) | ||
| config.pop("port", None) | ||
| config.pop("connector", None) | ||
| config.setdefault("served-model-name", served_model_name) | ||
|
|
||
| - cmd.extend( | ||
| - [ | ||
| - "vllm", | ||
| - "serve", | ||
| - model_arg, | ||
| - "--host", | ||
| - "0.0.0.0", | ||
| - "--port", | ||
| - str(runtime.frontend_port), | ||
| - ] | ||
| - ) | ||
| + node_rank = endpoint_nodes.index(process.node) | ||
| + cmd.extend(["vllm", "serve", model_arg]) | ||
| + if node_rank == 0: | ||
| + cmd.extend(["--host", "0.0.0.0", "--port", str(runtime.frontend_port)]) | ||
| + if is_multi_node: | ||
| + # vLLM-native multi-node serve (torchrun-style): the leader owns | ||
| + # the OpenAI server; other node ranks run headless engine workers. | ||
| + cmd.extend( | ||
| + [ | ||
| + "--master-addr", | ||
| + leader_ip, | ||
| + "--nnodes", | ||
| + str(len(endpoint_nodes)), | ||
| + "--node-rank", | ||
| + str(node_rank), | ||
| + ] | ||
| + ) | ||
| + if node_rank > 0: | ||
| + cmd.append("--headless") | ||
| if not self.set_cuda_visible_devices: | ||
| device_ids = ",".join(str(i) for i in sorted(process.gpu_indices)) | ||
| if device_ids: | ||
| diff --git a/src/srtctl/core/schema.py b/src/srtctl/core/schema.py | ||
| index 1263ddc..0ef7ae4 100644 | ||
| --- a/src/srtctl/core/schema.py | ||
| +++ b/src/srtctl/core/schema.py | ||
| @@ -1587,8 +1587,6 @@ class SrtConfig: | ||
| raise ValidationError("frontend.type: vllm supports aggregate jobs only, not disaggregated layouts") | ||
| if self.resources.num_agg < 1: | ||
| raise ValidationError("frontend.type: vllm requires resources.agg_workers >= 1") | ||
| - if (self.resources.agg_nodes or 1) != 1: | ||
| - raise ValidationError("frontend.type: vllm currently supports single-node aggregate jobs only") | ||
|
|
||
| def _validate_het_jobs(self): | ||
| """When ``resources.het_jobs`` is set to True, enforce supported shape. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,155 @@ | ||
| name: "kimik3-vllm-agg-b200-tp8pp2-dspark-retention0-agentic" | ||
|
|
||
| # Kimi-K3 MXFP4 B200 AGGREGATED TP8 x PP2 agentic recipe (2 nodes / 16 GPUs). | ||
| # The native MXFP4 checkpoint (2.8T total params, ~1.4TB of weights) does not | ||
| # fit one 8xB200 node, so TP8 shards attention/dense (/8) and PP2 splits the | ||
| # 93 layers (/2) across 16 GPUs. Plain TP (NOT TEP): expert parallelism is | ||
| # deliberately off, so the 896 routed experts are TP-sharded inside each | ||
| # pipeline stage. Node allocation = tp*pp/gpus_per_node = 8*2/8 = 2 nodes. | ||
| # Aggregated (single worker, decode num-worker 0) — no P/D split, no NIXL. | ||
| # VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION fuses the K3 LatentMoE tail path in | ||
| # the kimi-k3 bring-up image. | ||
| model: | ||
| path: "kimik3" | ||
| container: "vllm/vllm-openai:kimi-k3" | ||
| precision: "fp4" | ||
|
|
||
| identity: | ||
| model: | ||
| repo: "moonshotai/Kimi-K3" | ||
| container: | ||
| image: "vllm/vllm-openai:kimi-k3" | ||
|
|
||
| # Direct vLLM serving (frontend.type: vllm, srt-slurm PR #278 + the | ||
| # InferenceX multinode patch): `vllm serve` owns the OpenAI port itself, so | ||
| # no Dynamo frontend/worker is involved and no dynamo install is needed. | ||
| dynamo: | ||
| install: false | ||
|
|
||
| # Patches the image's mamba_hybrid postprocess_state: torch index_fill_ | ||
| # requires an int64 index but the runner passes the int32 idx_mapping, | ||
| # crashing the first decode step (IndexError: Expected dtype int64 for index). | ||
| setup_script: kimi-k3-container-deps.sh | ||
|
|
||
| slurm: | ||
| time_limit: "8:00:00" | ||
|
|
||
| health_check: | ||
| interval_seconds: 10 | ||
| max_attempts: 1440 | ||
|
|
||
| resources: | ||
| gpu_type: "b200" | ||
| gpus_per_node: 8 | ||
| agg_nodes: 2 | ||
| agg_workers: 1 | ||
| gpus_per_agg: 16 | ||
|
|
||
| infra: | ||
| etcd_nats_dedicated_node: false | ||
| nats_max_payload_mb: 32 | ||
|
|
||
| frontend: | ||
| # Direct vLLM OpenAI server (srt-slurm PR #278): the vllm serve leader owns | ||
| # the public port; rank-1 runs a headless engine worker (vLLM-native | ||
| # multi-node TP8xPP2 via --master-addr/--nnodes/--node-rank, enabled by | ||
| # patches/srt-slurm-pr278-direct-vllm-multinode.patch). | ||
| type: vllm | ||
| enable_multiple_frontends: false | ||
|
|
||
| backend: | ||
| type: vllm | ||
| connector: null | ||
| aggregated_environment: | ||
| VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION: "1" | ||
| VLLM_SERVER_DEV_MODE: "1" | ||
| # ~1.4TB of MXFP4 weights off shared Lustre: keep the engine-ready window | ||
| # generous, and let one long AgentX request hold a PP stage beyond vLLM's | ||
| # 300-second model-execution default. | ||
| VLLM_ENGINE_READY_TIMEOUT_S: "3600" | ||
| VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS: "1800" | ||
| # Prefix-cache retention (variant J): 0, on the otherwise-unchanged | ||
| # DSpark variant I config. Any positive value must be a multiple of | ||
| # Kimi-K3's KDA-hybrid scheduler_block_size (3145728; the GB recipes' | ||
| # 32768 is hard-rejected at engine init — verified on this PR family), so | ||
| # 0 is the only value below one 3.1M-token scheduler block. | ||
| VLLM_PREFIX_CACHE_RETENTION_INTERVAL: "0" | ||
| # No VLLM_PREFIX_CACHE_RETENTION_INTERVAL: the GB200/GB300 AgentX value | ||
| # (32768) hard-fails engine init on Kimi-K3 — the KDA hybrid gives it a | ||
| # scheduler_block_size of 3145728 and the interval must be a multiple of | ||
| # it ("VLLM_PREFIX_CACHE_RETENTION_INTERVAL (32768) must be non-negative | ||
| # and a multiple of scheduler_block_size (3145728)"). Default retention | ||
| # served fine in earlier runs. | ||
| NCCL_CUMEM_ENABLE: "1" | ||
| TILELANG_CLEANUP_TEMP_FILES: "1" | ||
| UCX_MEMTYPE_CACHE: "n" | ||
| UCX_MEMTYPE_REG_WHOLE: "n" | ||
| UCX_NET_DEVICES: "mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_10:1,mlx5_11:1" | ||
| HF_HUB_CACHE: "/hf_hub_cache" | ||
| HUGGINGFACE_HUB_CACHE: "/hf_hub_cache" | ||
| vllm_config: | ||
| aggregated: | ||
| served-model-name: "moonshotai/Kimi-K3" | ||
| tensor-parallel-size: 8 | ||
| pipeline-parallel-size: 2 | ||
| trust-remote-code: true | ||
| load-format: fastsafetensors | ||
| moe-backend: auto | ||
| # 0.90, not 0.95: the flashinfer trtllm MXFP4 MoE kernel allocates a | ||
| # ~1.6 GiB runtime workspace OUTSIDE vLLM's memory pool on the first | ||
| # forward; at 0.95 a 178 GiB B200 has only ~1.35 GiB free and the first | ||
| # warmup request OOMs (seen on the dynamo-frontend variants). 0.90 | ||
| # matches the GB200/GB300 agentic recipes. | ||
| gpu-memory-utilization: 0.90 | ||
| no-enable-flashinfer-autotune: true | ||
| # kimi_k3 parsers via the native vllm serve OpenAI-frontend flags — | ||
| # legitimate here because this recipe serves directly with vllm serve | ||
| # (frontend.type: vllm), not through the dynamo worker entrypoint that | ||
| # rejects them. | ||
| enable-auto-tool-choice: true | ||
| tool-call-parser: kimi_k3 | ||
| reasoning-parser: kimi_k3 | ||
| # DSpark (DeepSeek's confidence-scheduled semi-autoregressive | ||
| # speculative decoding, arXiv 2607.05147): parallel backbone drafting | ||
| # plus a lightweight sequential head — reported ~27-31% higher accepted | ||
| # length than EAGLE3 and 60-85% faster per-user generation than MTP-1 | ||
| # at matched throughput. Variant I uses the Inferact draft head, | ||
| # probabilistic draft sampling + block rejection sampling (the #2366 | ||
| # acceptance-rate optimization), k=7, and synthetic_acceptance_length | ||
| # 3.84 = the committed golden AL for exactly this sampling config at | ||
| # k=7 (golden_al_distribution/kimik3_dspark_probabilistic_sample_method_ | ||
| # block_rejection_sample_method.yaml, measured in run 30316471205). | ||
| speculative-config: '{"model":"Inferact/Kimi-K3-DSpark", "num_speculative_tokens":7, "method": "dspark", "attention_backend": "FLASHINFER_MLA", "draft_sample_method": "probabilistic", "rejection_sample_method": "block", "synthetic_acceptance_length": 3.84}' | ||
| # No explicit max-model-len: let vLLM derive the native 1M window from | ||
| # the model config (agentic trajectories blow past any small cap, and | ||
| # K3's KDA layers keep per-token KV small — only the 24 gated-MLA | ||
| # layers hold cache). Prefix caching stays on (default) for trajectory | ||
| # reuse. Cap prefill chunks so a single long request cannot OOM a | ||
| # pipeline stage; let vLLM pick max-num-seqs. | ||
| max-num-batched-tokens: 8192 | ||
|
|
||
| sbatch_directives: | ||
| segment: "1" | ||
|
|
||
| srun_options: | ||
| container-remap-root: "" | ||
|
|
||
| benchmark: | ||
| type: custom | ||
| command: bash /infmax-workspace/benchmarks/multi_node/agentic_srt.sh | ||
| env: | ||
| INFMAX_CONTAINER_WORKSPACE: "/infmax-workspace" | ||
| RESULT_DIR: "/logs/agentic" | ||
| PORT: "8000" | ||
| # Keep the aggregate worker in the multinode result schema so ingestion | ||
| # uses the zero decode-worker count instead of duplicating TP into P and D. | ||
| IS_MULTINODE: "true" | ||
| # aiperf's conv-aware routing emits nvext.session_control, a removed POC | ||
| # field this dynamo build 400-rejects at warmup (schema moved to | ||
| # router/routing_constraints/agent_hints). Same opt-out as the GB300 | ||
| # aggregate AgentX recipes — and with a single aggregate worker there is | ||
| # no P/D routing to bind anyway. | ||
| AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING: "0" | ||
| AIPERF_DATASET_MMAP_CACHE_DIR: "/aiperf_mmap_cache" | ||
| HF_HUB_CACHE: "/hf_hub_cache" | ||
| WEKA_LOADER_OVERRIDE: "semianalysis_cc_traces_weka_062126" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -5138,3 +5138,19 @@ | |
| - "Add a one-node 4P1D TP1-prefill/TEP4-decode concurrency-4096 point on the refreshed image while preserving the legacy 4P2D DEP2-prefill/TEP4-decode point unchanged" | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2310 | ||
|
|
||
|
|
||
| - config-keys: | ||
| - kimik3-fp4-b200-dynamo-vllm-agentic | ||
| description: | ||
| - "Add Kimi-K3 MXFP4 B200 aggregated multinode Dynamo-vLLM agentic-coding bring-up (new model on B200; first kimik3 benchmark config)" | ||
| - "Aggregated TP8 x PP2 across 2 B200 nodes (16 GPUs), plain TP (NOT TEP: ep 1, no enable-expert-parallel) — the native MXFP4 checkpoint (2.8T total params, ~1.4TB weights) does not fit one 8xB200 node, so TP8 shards attention/dense and PP2 splits the 93 layers. Aggregated mode (prefill num-worker 1 + decode num-worker 0, RECIPES.md section 5): one worker serves prefill and decode, no P/D KV transfer" | ||
| - "Dedicated bring-up image vllm/vllm-openai:kimi-k3 with VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1 (fuses the K3 LatentMoE tail path), --load-format fastsafetensors, --moe-backend auto, --gpu-memory-utilization 0.90 (0.95 OOMs: the flashinfer trtllm MXFP4 MoE kernel allocates a ~1.6 GiB runtime workspace outside vLLM's pool on the first forward), --no-enable-flashinfer-autotune, --trust-remote-code, --enable-auto-tool-choice, --tool-call-parser kimi_k3, --reasoning-parser kimi_k3 (agentic experiment Variant D: the OpenAI-frontend parser flags are legitimate here because serving is DIRECT vllm serve, not the dynamo worker entrypoint that rejects them)" | ||
| - "No explicit max-model-len (vLLM derives the native 1M window from the model config; K3's KDA layers keep per-token KV small — only the 24 gated-MLA layers hold cache), prefix caching on for trajectory reuse, max-num-batched-tokens 8192 so a single long prefill cannot OOM a pipeline stage; conc 1/8/16/32" | ||
| - "DIRECT vLLM serving via srt-slurm PR #278 (kylliang/direct-aggregate-vllm, frontend.type: vllm): vllm serve owns the OpenAI port itself, removing the dynamo frontend/worker/router entirely (dynamo install: false) and with it the kimi_k3 tiktoken tokenizer gap that 404'd every request on dynamo <=1.2.1. PR #278 validates single-node only, so patches/srt-slurm-pr278-direct-vllm-multinode.patch extends it to vLLM-native multi-node serve (--master-addr/--nnodes/--node-rank, headless non-leader ranks) for the 2-node TP8xPP2 topology" | ||
| - "Model pre-staged at /lustre/fsw/models/Kimi-K3 (moonshotai/Kimi-K3); launcher launch_b200-dgxc.sh gains the kimik3/fp4 model-path mapping, pins the agentic srt-slurm base to upstream NVIDIA/srt-slurm v1.0.36 (validated in #2302/#2341; replaces the cquil11/srt-slurm-nv fork branch whose older srtctl schema rejects newer recipe fields such as benchmark.aiperf_server_metrics), overlays the kimi-k3 agentic recipes onto the clone, and adds the agentic default_mounts (/aiperf_mmap_cache, /hf_hub_cache) already used by the GB200/GB300 agentic paths" | ||
| - "aiperf conv-aware routing disabled (AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING=0, same opt-out as the GB300 aggregate AgentX recipes): aiperf's nvext.session_control is a removed POC field this dynamo build 400-rejects at warmup (fifth sweep attempt: tokenizer registered, engine served, all 5 warmup requests 400'd); with a single aggregate worker there is no P/D routing to bind" | ||
| - "In-container vLLM patch via setup_script kimi-k3-container-deps.sh: the kimi-k3 image's first decode step crashes in the KDA hybrid-state postprocess (vllm/v1/worker/gpu/model_states/mamba_hybrid.py postprocess_state, IndexError: index_fill_(): Expected dtype int64 for index — torch requires an int64 index but the runner passes the int32 idx_mapping; sixth sweep attempt, first warmup request 500s then the model 503s). The patch coerces the index with .long(), is idempotent, and refuses to run if the image layout changed" | ||
| - "Agentic experiment Variant I (of the #2359 direct-vllm Variant D): DSpark speculative decoding with the Inferact/Kimi-K3-DSpark draft head — num_speculative_tokens 7, FLASHINFER_MLA attention backend, probabilistic draft sampling + block rejection sampling (the #2366 acceptance-rate optimization), synthetic_acceptance_length 3.84 = the committed golden AL for this exact sampling config at k=7 (#2368's golden_al_distribution curve, measured in run 30316471205). Conc curve widened to 1/2/4/8/16/32; spec-decoding labeled dspark (new schema value alongside mtp/draft_model)" | ||
| - "Agentic experiment Variant J (of the #2376 DSpark Variant I): identical DSpark-at-golden-AL config plus VLLM_PREFIX_CACHE_RETENTION_INTERVAL=0 (the only setting below one 3.1M-token KDA-hybrid scheduler block; positive values must be multiples of scheduler_block_size 3145728)" | ||
| - "Recipe: benchmarks/multi_node/srt-slurm-recipes/vllm/kimi-k3/agentic/agg-b200-tp8pp2-dspark-retention0-agentic.yaml on the cluster:b200-dgxc pool" | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/2379 | ||
|
Contributor
There was a problem hiding this comment. 🟡 The changelog entry for this PR's model (kimik3-fp4-b200-dynamo-vllm-agentic) ends with a literal unfilled placeholder Extended reasoning...What's wrong
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXX
Why validation doesn't catch it
ImpactEvery other Proof
FixReplace |
||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟡 Line 76 sets
VLLM_PREFIX_CACHE_RETENTION_INTERVAL: "0", but the comment block immediately below (lines 77-82) begins "No VLLM_PREFIX_CACHE_RETENTION_INTERVAL: ..." and explains why the var is deliberately omitted — this is leftover copy-paste from Variant I's recipe (which didn't set the var at all) and directly contradicts the setting one line above. It's comment-only (no runtime effect), but the stale block should be removed so a future editor doesn't mistakenly delete line 76 thinking the var should stay unset.Extended reasoning...
What the bug is: In
agg-b200-tp8pp2-dspark-retention0-agentic.yaml, thebackend.aggregated_environmentblock setsVLLM_PREFIX_CACHE_RETENTION_INTERVAL: "0"on line 76. That line is correctly preceded (lines 71-75) by a comment explaining the Variant J rationale: 0 is the only value below one 3.1M-token KDA-hybrid scheduler block, since any positive value must be a multiple ofscheduler_block_size(3,145,728). Immediately after that setting, lines 77-82 contain a second comment block that opens with "No VLLM_PREFIX_CACHE_RETENTION_INTERVAL: the GB200/GB300 AgentX value (32768) hard-fails engine init..." and closes with "Default retention served fine in earlier runs."How it manifests / code path: This second comment is leftover text carried over verbatim from Variant I's recipe (the predecessor PR #2376), which did not set this env var at all — hence the original "No VLLM_PREFIX_CACHE_RETENTION_INTERVAL" phrasing made sense there. When Variant J (this PR) added the explicit
"0"setting on line 76, the old comment explaining the absence of the setting was never removed or updated, so it now sits directly beneath the line it contradicts.Why existing code doesn't prevent it: This is a pure YAML-comment issue — comments have no runtime effect, are not parsed or validated by
utils/matrix_logic/validation.pyor any CI check, and the engine still receives the correct explicit"0"value regardless of what the comment says. Nothing in the pipeline flags contradictory adjacent comments.Impact: No functional impact on this sweep — the YAML value is what's used at runtime, not the comment. The risk is purely to future maintainers: someone skimming lines 77-82 could reasonably conclude the variable is intentionally left unset (matching Variant I's behavior) and delete or "fix" line 76 accordingly, which would change runtime behavior (0 vs. vLLM's default retention interval). It's a maintenance/readability hazard, not a merge-blocking defect.
Proof (step-by-step):
VLLM_PREFIX_CACHE_RETENTION_INTERVAL: "0"→ confirms the var IS set to 0.How to fix: Delete the stale lines 77-82 comment block entirely (its information is now redundant with the correct explanation on lines 71-75), or fold the "32768 hard-fails engine init" rationale into a single unified comment above line 76 explaining both why 0 was chosen and why the GB-style 32768 value can't be used.