[weireweire]

Motivation

DeepSeek-V4.1-Flash low-latency decode with DSpark and the NVIDIA megakernel plugin. SGLANG_OPT_DSV41_TARGET_METADATA_EARLY (from #45450) builds the target verify metadata on a side stream beside the draft graph, but only for the candidate_unfiltered verify variant. Verify contexts past the candidate span (> 16K tokens) select candidate_filtered, so they still rebuild the full metadata inside the verify graph: about 39 µs of head kernels per step at bs=1. Long agentic contexts take this variant on every decode step.

Modifications

• Add candidate_filtered to DSV41_EARLY_VERIFY_VARIANTS, so both DSpark verify variants stage their metadata beside the draft.
  • The metadata build (make_forward_metadata_from_raw_verify) does not depend on the variant.
  • The candidate_all index fill that only the in-graph upgrade does is not used by either DSpark verify variant, so the existing assert still holds.
  • The candidate filtering itself runs in the layer forward, which is recorded in the verify graph either way.
  • take() still rejects a staged entry whose key or tensors do not match the replayed batch, and batch sizes with no variant keep the in-graph path.
• Update the runner staging unit test, which asserted that the filtered variant is not staged, and two stale comments.

Accuracy Tests

GB300, TP4, DeepSeek-V4.1-Flash, megakernel plugin, SGLANG_ENABLE_NV_MEGA_RUNTIME_OPTIMIZATION=1, real acceptance:

check	without	with
Long-context retrieval, 20K-160K tokens (exercises candidate_filtered)	48 / 48	48 / 48

GSM8K prompts are shorter than 16K tokens, so they never take the changed path.

test/registered/unit/model_executor/runner/test_early_verify_metadata.py passes on CPU.

Speed Tests and Profiling

GB300, TP4, DeepSeek-V4.1-Flash, megakernel plugin, AgentX (InferenceX agentic traces), concurrency 1, 3,600 s, simulated acceptance 3.51 with the stock accept path. Both arms run the same SGLang nightly (2026-10-30) with two other unmerged step-time changes applied (the accept-buffer writes of #47318, and one copy instead of three for the deferred commit KV), so only this change differs. TPS/User = 1 / ITL.

	without	with
verify head kernels per step (nsys)	~56 µs	~19.5 µs
decode step p50		-20.8 µs
TPS/User P90	934.1	953.0 (+2.0%)
accept length	3.51	3.52

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

---

CI States

Latest PR Test (Base): ❌ Run #41184332020
Latest PR Test (Extra): ❌ Run #41184331973
Latest PR Test (AMD ROCm 10): ❌ Run #41184331921

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[nvpohanh]

[by Claude Code] This review is done by agents. Please feel free to disagree if they do not make sense.

[nvpohanh]

[by Claude Code] Severity: style | Confidence: Medium

Issue

With both DSpark verify variants in this set, the variant no longer changes the verify prep: both run _install_early_verify_metadata into the same (bs, "early") object. But DecodeCudaGraphRunner._glue_key still adds the variant label. So each bs captures two identical metadata glue graphs at runtime (two warmups, two captures, two private pools). Also, take() rejects the staged work when the stage-time and replay-time variants differ, even though the staged metadata is the same. The _glue_key comment "(early install or Raw copy)" is now stale.

Fix

Key on attention_variant in DSV41_EARLY_VERIFY_VARIANTS instead of the label, or at least update the _glue_key comment.