[Bugfix][Tests] Stabilize B12X linear kernel checks

[lukealonso]

Summary

• Allow the expected normalized error from B12X four-way split-K while tightening the cosine-similarity requirement.
• Make the B12X W4A16 fallback test independent of the host GPU and unrelated NVFP4 backend registrations.

Rationale

B12X block-FP8 split-K CTAs atomically accumulate into a BF16 output. Atomic ordering is nondeterministic, so individual outputs can alternate between adjacent BF16 values one ULP apart. On the 48-SM four-slice route, the normalized mean absolute difference was 0.00240-0.00252, while cosine similarity remained 0.9999935-0.9999940 across ten seeds. This changes the normalized-error limit from 0.002 to 0.003 and tightens the cosine floor from 0.9999 to 0.99999.

The W4A16 fallback test previously depended on the complete CUDA NVFP4 registry. #53014 registered FlashInferCuteDslNvFp4W4A16LinearKernel ahead of Marlin, so SM12x CI selected FlashInfer even though the test had explicitly made Marlin available. The test now installs a minimal B12X/Marlin registry with monkeypatch, directly exercising its intended contract. Future backend registrations therefore cannot change its result; adding W4A16 support to the B12X linear kernel will still invalidate the fallback test as intended.

Duplicate-work check

I searched open PRs for B12X block-FP8 tolerance and W4A16 fallback changes. #43929 concerns NVFP4 MoE W4A16 routing and does not modify these tests or the dense linear selector. No open PR addresses these two CI failures.

Validation

uvx --from pre-commit pre-commit run --files \
  tests/kernels/quantization/test_block_fp8.py \
  tests/model_executor/kernels/test_b12x_linear.py
# all hooks passed

CUDA_VISIBLE_DEVICES=11 .venv/bin/python -m pytest -v -s \
  tests/model_executor/kernels/test_b12x_linear.py::test_b12x_backend_preserves_w4a16_fallback \
  tests/kernels/quantization/test_block_fp8.py::test_w8a8_block_fp8_b12x_matmul
# 5 passed

The tests used b12x==1.3.0 and source at 605c3ddcba. The precompiled extension carrier was from 3b45d053b4 because the wheel for the newly published main SHA was not yet available; the two intervening commits only add documentation annotations and update the CUTLASS source revision. These test paths use Python selection logic and the separately installed B12X kernel.

Model evaluation is not applicable because this changes test acceptance and isolation only; runtime code is unchanged.

AI assistance

AI assistance was used for failure investigation, implementation, validation, and PR drafting. The submitter reviewed every changed line and the validation results.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[mgoin]

/ci run

[github-actions]

✅ Triggered Buildkite CI #86913 for commit 532cd88264a0.