[Test] Deterministic e2e regression tests (xfail) for hybrid-Mamba prefix cache corruption (#43559)

[puririshi98]

[Test] Deterministic e2e regression tests (xfail) for hybrid-Mamba prefix cache corruption (#43559)

Status / supersession

Open as the red/xfail reproducer; superseded by #48361 on its merge,
after which this PR will be closed. The earlier caveat here — that this
branch's copy of the test file had diverged ahead of the copy
shipped in #48361 — is resolved: #48361's branch now carries the same
hardened file (fork-runner-safe GeometryUnsupported /
ControlQualityFailure hard-fail guards replacing in-body
pytest.skip / pytest.fail, the engine-boot AssertionError →
GeometryUnsupported mapping, the cache_config.block_size ↔
MambaSpec geometry cross-invariant, and the Nemotron-Super
large_gpu_mark bump to min_gb=140). The two copies now differ only
in posture: #48361 drops the two xfail markers, the XFAIL_REASON
constant, and the CI step's --runxfail, and its docstrings describe
the fixed-tree status instead of the live bug (executable content
verified identical by AST compare modulo docstrings/decorators).

Purpose

#43559 reports silent, hard-to-reproduce output corruption on hybrid-Mamba
models when prefix caching and MTP speculative decoding are combined. The
maintainer ask in that thread was a reliable reproducer. This PR delivers
exactly that: two deterministic, output-level e2e regression tests that fail
on main (validated red at e08a9151 and re-confirmed at 88e5e2c5)
for mechanistically understood reasons, and will guard
against regressions once the open fixes land.

This PR intentionally contains no source fix. Both corruption mechanisms
already have open fix PRs — #45477 (chunked-prefill fragmentation caching a
mid-block Mamba state as a block-boundary snapshot) and #47861 / #45614 /
#46281 (Mamba hit length overrunning the attention-verified hit under the
spec-decode lookahead). Adding a fifth fix would duplicate them; nobody had
yet contributed e2e output-level coverage, which is a different layer from
those PRs' unit/scheduler tests.

The tests

tests/v1/e2e/test_hybrid_mamba_prefix_cache_correctness.py, parametrized
over Qwen/Qwen3.6-27B-FP8 (TP1) and
nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 (TP4), plus an optional
non-blocking Buildkite step:

• Cold-race: 16 identical prompts, sized between one and two Mamba
  blocks, prefilled cold and concurrently under a small token budget.
  Asserts output-distinctness and needle recall relative to an APC-off
  control, margin-graded rather than zero-tolerance: benign batch-shape
  nondeterminism flips symmetrically across arms, while [Bug]: Accuracy drops ~20% when --enable-prefix-caching is used together with MTP speculative decoding (Qwen3.6 35B-A3B) #43559
  corruption is one-sided. Runs Nemotron-Super only: Qwen corrupts under this
  geometry even with APC off, so the control-quality gate cannot grade
  the cache there — hence three result rows below, not four.
• Multi-turn: turn-2 prompts resume from Mamba blocks written during
  speculative decode. Asserts recall parity vs. an APC-off control.

Both are xfail(raises=CorruptionDetected, strict=False): only genuine
corruption xfails. Engagement failures (zero cache hits / zero drafts),
weak-control results, geometry drift, and boot errors hard-fail, so the
tests cannot pass vacuously. Geometry and control-quality guards raise
plain Exception subclasses rather than pytest.skip/pytest.fail:
the fork-based per-test runner reports in-body skips as PASS and loses
Failed (a BaseException) diagnostics, so both would otherwise be
silent-green vectors on the --runxfail step. Remove the markers
together with --runxfail in the CI step once a fix merges.

Results on main at validation time (e08a9151, GB200)

Run with --runxfail so the raw failures are visible:

Arm	Result
Cold-race, Nemotron-Super TP4	CorruptionDetected: 16 identical prompts → 3 distinct outputs (needle 824568 corrupted to 82423567 / 82447757); control: 1
Multi-turn, Nemotron-Super TP4	CorruptionDetected: 17/32 recall flips vs. control, 0 in the benign direction
Multi-turn, Qwen3.6-27B-FP8 TP1	CorruptionDetected: 14/32 one-sided flips — first reproduction on the issue's own model family (batch-1 harnesses in the thread could not)

Prefix-cache and spec-decode engagement verified via llm.get_metrics()
counters in every run. The corruption is unchanged at e12b91b03
(which contains #48132), ruling that fix out for this issue.

pytest tests/v1/e2e/test_hybrid_mamba_prefix_cache_correctness.py \
    --runxfail -v -s -rA -k "nemotron-super"   # 4x >=140GB GPUs (H200/GB200 class)
pytest tests/v1/e2e/test_hybrid_mamba_prefix_cache_correctness.py \
    --runxfail -v -s -rA -k "qwen"             # 1x >=80GB GPU

Run note: in the executed copies the two HF model ids were substituted with
pre-downloaded local paths (HF_HUB_OFFLINE=1 breaks model-redirect
resolution); everything else was byte-identical to the diff as of the
validation commit. The diff has since received post-validation amendments:
comment/docstring rewording, ruff-format whitespace, review-round hardening
of the geometry/control-quality guard paths (now fork-runner-safe hard
failures instead of skips), a hardware-capacity gate bump on the
Nemotron-Super param (large_gpu_mark min_gb 80 → 140, matching the CI
step's H200-class targeting), and a comment-only calibration-data sync
with #48361's copy of the file. None of these touches the validated trigger
geometry or grading logic, which are unchanged from the validation commit.
The branch has also since merged main @ 1ef1c7ebb (2026-07-11); that
merge touched only surrounding steps in engine.yaml, not this PR's step
or the test file.

Cross-verification: these tests discriminate between the open fix PRs

We ran the two Nemotron-Super arms against each open fix PR, hand-rebased
onto main as of the cross-verification runs (88e5e2c5, at which the
clean-main baseline is RED on both arms). GB200, one run per arm; "zero-hit" = the test's anti-vacuous
liveness/engagement gate fired because prefix-cache hits dropped to
exactly 0 while queries stayed >150k.

Condition	Cold-race (chunk-fragmentation lane)	Multi-turn (eagle hit-length lane)
clean main	🔴 corrupts (cache engaged, 103k hits)	🔴 corrupts (59k hits)
#47861	⚠️ zero-hit: caching effectively disabled	✅ cured, engaged (wave-2 hit ratio 0.477)
#45614	🔴 still corrupts (3 distinct outputs / 16 identical prompts)	✅ cured, engaged
#46281	⚠️ zero-hit: caching effectively disabled	✅ cured, engaged
#45477 (hand-port)	⚠️ zero-hit both arms — port inconclusive, needs author rebase	⚠️ zero-hit
#45477 + #47861	⚠️ zero-hit	✅ cured, engaged

Takeaways: the multi-turn lane is cured by three independent PRs; no
open PR fixes the cold-race lane — it either still corrupts (#45614) or
goes quiet only because hybrid prefix caching stops producing hits under
this pattern (a caching regression, not a cure — which is exactly what the
liveness gate in these tests exists to catch). Caveats: hand-rebased
ports, not the PRs' own branches; single run per arm.

AI assistance disclosure

Developed with AI assistance (Claude Code), including adversarial
multi-agent review; every line reviewed by the human submitter, who ran the
tests and can defend the change end-to-end.

[puririshi98]

Update: the fix PR #48361 now supersedes this draft — it includes this test file (xfail markers removed; the tests pass with the fix, red→green validated on GB200 for both models). Keeping this draft open only until #48361 resolves; see the verification matrix there and on #43559.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.