fix(attention): qualify compact NVFP4 MTP verifier route

[yatesdr]

Summary

Keep compact nvfp4_ds_mla MTP verification on SparkInfer's established
extend path in auto mode. Continue auto-routing the already-qualified
fp8_ds_mla verifier through decode, and preserve explicit
VLLM_B12X_MLA_SPEC_EXTEND_AS_DECODE=1 as an opt-in for further qualification.

This corrects the scope of 3e731bc0. Its new decode-verifier correctness test
uses fp8_ds_mla, while GLM-5.2 production uses the distinct compact
nvfp4_ds_mla record and BF16-QK kernel arm.

Root cause

The v20 regression is not consistent with persistent KV-byte corruption:

• the compact-NVFP4 cache writer, dequantization primitives, established extend
  front door, and paged-indexer orchestration are executable-AST identical
  between the passing v19 stack and v20 after normalizing the B12X → SparkInfer
  namespace move;
• DCP4 global/local position mapping is bijective through the tested 475k
  range; and
• the four-supertile paged top-k fold at 118,750 local rows is mathematically
  equivalent to direct top-k.

The relevant execution delta is that 3e731bc0 changed the default from
extend to decode for genuine MTP verification without qualifying the compact
NVFP4 kernel arm.

Change

auto is now format-qualified:

Mode	fp8_ds_mla	nvfp4_ds_mla
0	extend	extend
auto	decode	extend
1	decode	decode

The policy also controls scratch sizing. At MNS16/MTP3, compact NVFP4 no longer
reserves 64 decode rows for verifier batches; it reserves the 16 rows needed by
ordinary decode. For 64 gathered heads, 32 split slots, and value width 512,
this removes at least 99.38 MiB/GPU of verifier-only scratch:

tmp_output: 96.00 MiB
tmp_lse:     0.38 MiB
output:      3.00 MiB

Ordinary one-token decode, MNS16, max model length, and KV format are unchanged.

Internal field evidence

The production comparison that motivated this patch:

• v19 compact NVFP4 + i8_ring: 5/5 deep-needle pass at
  50k/200k/300k/350k/475k; decode baseline 165.1 aggregate tok/s at C16.
• v20 compact NVFP4 + confirmed i8_ring + default auto route:
  pass at 50k/150k, genuine miss by 300k.
• v20 forced-one-split decode experiment: moved the pass boundary through 300k
  but still genuinely missed at 350k/475k and reduced the measured KV pool
  from 557,824 to 525,568.

The last result is important: it falsifies the earlier hypothesis that BF16
split partials were the complete cause. Reducing splits changes the margin but
does not qualify the compact-NVFP4 decode verifier.

Needle classification captured content, reasoning_content,
finish_reason, and token usage. Values found in reasoning at 250k/300k count
as passes; the 350k/475k cases ended normally with coherent answers saying the
ticket was absent, so they are genuine retrieval misses rather than parser or
budget artifacts.

Validation

Completed without a GPU boot:

• Python syntax compilation;
• git diff --check;
• unit tests added for every mode/format combination, invalid input, and
  FP8/compact-NVFP4 scratch sizing;
• executable source proof:
  • matching compact-KV writer/reader AST hashes;
  • exhaustive DCP4 inverse mapping for [0, 475000);
  • adversarial 12,000-candidate exact-radix overflow equality;
  • production-geometry long-context top-k fold equality;
  • isolated execution of the patched route policy.

The source proof reports:

PASS: compact-NVFP4 KV writer/reader primitives are unchanged
PASS: patched route preserves fp8 auto and restores NVFP4 extend
PASS: DCP4 mapping is bijective through 475,000 positions
PASS: >4,096-candidate radix overflow fallback returns exact top-k
PASS: four-chunk 118,750-row top-k fold equals direct top-k
scratch: rows 64 -> 16 minimum recovered=99.38 MiB/GPU

The local Mac Python environment does not include pytest, so the full vLLM test
module was not collected here. The dependency-free proof executes the same
route helper directly and passed; CUDA test collection remains part of the GPU
gate.

GPU validation is still required before this should leave draft:

1. cold needles at 50k/150k/250k/300k/350k/475k, accepting either response
   field;
2. decode C1/C4/C8/C16;
3. clean KV-pool measurement with diagnostics disabled; and
4. confirmation that the route INFO line selects extend for compact NVFP4.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: f95a9af1-8ece-4ba3-bde1-eef0ae948cae

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

PRs do not trigger a full CI run by default. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

Agent Guidelines

IMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban.

🚀

[malaiwah]

Reproduction Attempt on AIBoss (RTX 5090)

We attempted to reproduce the nvfp4_ds_mla MTP verifier route regression described in this PR using the CI fixture model malaiwah/GLM-5.2-SIQ-Fruit-Instruct (5.04B params, 2.885 GiB) on a single RTX 5090 (32 GB, SM120) with the exact r28 release image (voipmonitor/vllm:gilded-gnosis-v20-vllme1e9426-si200c1db-fi801d57a-cu132-20260804-r28).

What was confirmed

The route selection root cause described in the PR is present in r28. In b12x_mla_sparse.py (line 1425–1430):

spec_decode_mode = os.getenv("VLLM_B12X_MLA_SPEC_EXTEND_AS_DECODE", "auto").strip().lower()
disabled_modes = {"0", "false", "off", "no"}
forced_modes = {"1", "true", "on", "yes"}
self.spec_extend_as_decode = spec_decode_mode not in disabled_modes

With auto (the documented default), spec_extend_as_decode = True for all KV formats including nvfp4_ds_mla — this is the bug. With =0, it is False (extend path — safe). Both routes boot and serve successfully with nvfp4_ds_mla + MTP1 on the RTX 5090. MTP acceptance with =0 (extend): 372/460 = 80.9%.

Why the retrieval bug could not be reproduced

The CI fixture model has max_position_embeddings = 65536 (64k), while the regression described in this PR manifests at 300k+ context on the full GLM-5.2 model. Needle-in-haystack tests showed identical results for both routes:

Context	auto (decode path)	=0 (extend path)
1k	PASS	PASS
4k	PASS	PASS
8k	FAIL	FAIL
16k	FAIL	FAIL
32k	FAIL	FAIL
48k	FAIL	FAIL

The failures at 8k+ are from the 5B CI fixture's limited retrieval capability, not the route bug — both routes degrade identically because the model cannot do needle retrieval beyond ~4k tokens regardless of the verifier path.

Conclusion

The code-level root cause is confirmed in r28, but the retrieval failure itself requires the full GLM-5.2 model at 300k+ context — not reproducible with the Fruit CI fixture at 64k max context on a single RTX 5090. The fix remains relevant for production deployments running nvfp4_ds_mla + MTP at long context.

[malaiwah]

Relationship Between PR #171 and Issue #182

Summary

PR #171 does not fix issue #182. Issue #182's investigation explicitly tested and refuted the PR #171 hypothesis. They address different root causes in the same symptom space (nvfp4_ds_mla + long-context retrieval).

Issue #182 root cause (not PR #171)

Issue #182 tracks a deep-context retrieval regression where a frozen 343,727-token prompt retrieves correctly on v19 but misses on v20. The investigation (23 comments, ~36 hours of work) localized the root cause to the sparse-indexer top-k selection policy during prefill, not the MTP verifier route:

• v20 commits 1012199e (exact overflow rescan) and 83a58444 (widened coarse radix from 8 to 10 bits, candidate buffer 4096 to 8192) changed the sparse-indexer selector.
• The new exact selector is mathematically correct for the E4M3-query/FP8-key proxy, but it systematically moves ~7.4-8.4% of the 2,048-entry sparse-attention budget from older positions into the newest quarter of context.
• The needle at position ~137,496 gets dropped from the top-2,048 at multiple early sparse layers (starting at layer 34), because the exact proxy scores favor recent tokens.
• Even a full BF16/FP32 oracle excludes the needle from top-2,048 — the issue is the selection distribution, not quantization precision.

The actual fix is the bounded_compat selector policy (SparkInfer PR #82), which preserves the historical v19 selection distribution. It passed all frozen gates: 250k control, 3/3 frozen 350k failures, and a randomized 50k-475k cold ladder.

PR #171 was explicitly refuted for issue #182

Issue #182's experiments table contains this row:

PR #171 NVFP4 MTP verifier route | No-#171 image still missed the first cold causal row; deep physical-index decode/extend oracle prepared | Refuted end-to-end

They tested an image without PR #171's route change and the 350k retrieval still missed — proving the MTP verifier route is not the cause of the retrieval regression.

What PR #171 actually fixes (and why it's still valuable)

PR #171 addresses a different, narrower issue: commit 3e731bc0 changed the default VLLM_B12X_MLA_SPEC_EXTEND_AS_DECODE from extend to decode for MTP verification without qualifying by KV format. For nvfp4_ds_mla (compact NVFP4 records), the decode kernel arm is the wrong verifier path. This is a correctness issue in MTP verification, not in prefill retrieval.

The PR's own field evidence confirms this distinction:

• v19 compact NVFP4 + i8_ring: 5/5 deep-needle pass at 50k-475k
• v20 with auto route: pass at 50k/150k, genuine miss by 300k
• v20 with forced decode (reducing splits): moved the pass boundary through 300k but still missed at 350k/475k

The last result is important — it shows that the MTP route affects the margin of the failure but is not the root cause. Issue #182's investigation then identified the actual root cause as the selector policy.

Where they overlap and where they don't

Aspect	PR #171	Issue #182
Layer	MTP verifier route (decode vs extend)	Sparse-indexer top-k selection (prefill)
Root cause	3e731bc0 unqualified format switch	1012199e + 83a58444 exact overflow rescan
KV format	nvfp4_ds_mla	nvfp4_ds_mla
Fix	Format-qualify auto mode	bounded_compat selector policy (SparkInfer PR #82)
Scratch benefit	Reclaims ~99 MiB/GPU verifier scratch	N/A
Fixes issue #182?	No (explicitly refuted)	N/A
Still needed?	Yes — correct MTP verifier routing for nvfp4_ds_mla is independently valuable	Yes — selector policy is the actual fix

Relevance to the production stack

The production container runs VLLM_B12X_MLA_SPEC_EXTEND_AS_DECODE=0 (forced extend), which avoids PR #171's bug. Issue #182's regression is present in the r28 image's default exact selector — the bounded_compat fix is not in r28 (the build docs explicitly state: "the later bounded_compat commits on build PR #5; that selector policy was not part of this candidate"). Long-context retrieval quality at 300k+ on nvfp4_ds_mla may be affected in production if prompts exceed ~250k tokens.

[yatesdr]

Closing under the repository’s no-draft-PR policy. The branch and discussion remain preserved; any successor will be submitted as ready only after local correctness, E2E, and regression validation.