[Perf][SM70] Narrow Qwen3.8 AWQ active grouped decode

[Leonccaa]

Purpose

Reuse the existing SM70 AWQ active-stage operator for Qwen3.8 TP4 small-batch
decode, and document the separately validated existing autotune configuration.

• Narrow gate: 512 experts/top-k 10, effective g32, hidden 2560, local intermediate
  160, and 2–8 input tokens. Native-g32 checkpoint used for GPU validation.
• No new public operator, Python wrapper or GEMM kernel. Preserve multi-row
  active-expert segments and offsets-based scheduling; align runtime/warmup
  admission and annotate the existing scratch mutations.
• C1 and unmatched contracts retain existing dispatch. The model-specific
  grouped route defaults on; its rollback is
  VLLM_SM70_AWQ_QWEN38_MOE_COMPACT_GROUPED_DECODE=0 before startup.
• Existing VLLM_SM70_AWQ_TUNE_SMALL_SHAPES=1 is documented as optional and
  stays globally default-off. No precision, scheduler, KV or deployment change.

Scope, duplicate check, and base alignment

This is the reduced-surface successor to closed
1CatAI/1Cat-vLLM#491, not a
reopening of its new-operator implementation. Its closure was a maintenance
cost/benefit decision. Open-PR searches in both repositories for AWQ, grouped
decode and 1CatAI#491 found no duplicate. Pending
QSA ordering #494 is a separate
correctness fix used by both arms of the historical model validation.

At Leon's request, fork main was fast-forwarded from b410e59eb9 to upstream
755baae1d075ee04fa9096b23fc0225b23589a86 before opening this PR. The patch was
ported to that base, preserving its indexed-prefill and compact-metadata
initialization. The original validated 5cceeaad89 branch is retained.
This PR contains only five production files, one focused test file and one
design/evidence document; unrelated upstream commits are in the base.

Draft review gate: the new base changes HC, NVFP4 and scratch lifetimes.
The native build, actual Flash binding checks and bounded AWQ model acceptance
now pass. New-base integration results are listed separately below; the older
OFF/ON tables remain frozen causal comparisons. Final human review is still
required. Commit bc4686eadd only adds acceptance documentation; the five
production files are unchanged from tested native source b50bb1f5037d.
Follow-up fbaa195f47 only corrects the documented engine-reported KV capacity;
it includes no C1 prepare candidate or production-code change.

AI assistance: OpenAI Codex assisted with implementation, analysis, tests and
documentation. Leon directed the scope and requested fork submission. This is
a fork review draft; final human line-by-line review is not claimed complete.
No upstream submission is made here.

Test Plan

Current aligned branch, CPU:

.venv/bin/python -m pytest -q \
  tests/quantization/test_sm70_awq_active_grouped_decode.py \
  tests/quantization/test_sm70_awq_indexed_prefill.py \
  tests/quantization/test_sm70_awq_compact_metadata.py \
  tests/quantization/test_awq_sm70_persistent_cap.py

Run changed-file pre-commit hooks and git diff main --check. The new boundary
cases cover 1/2/8/9/127/128 tokens and disjoint indexed-prefill/grouped-decode
admission. The source-matched native build and bounded new-base graph/route,
natural-EOS and model acceptance are complete; see their exact scope below.

Test Result

• Aligned-tree CPU regression: 70 passed. Changed-file hooks passed.
• Update 2026-09-05: native targets _C, _moe_C, and
  _C_stable_libtorch built successfully from PR head b50bb1f5037d
  for SM70, with exit 0 and no OOM. In the intended Torch 2.10.0/CUDA 12.8
  environment, a no-GPU-device container imported all three plus the fixed
  Flash-V100 binary, verified their SHA-256 values and key operator registration,
  and confirmed CUDA remained uninitialized. This is an import/hash gate,
  not a GPU correctness or performance pass. Both model-validation arms
  will still use the same separately tracked QSA [Bugfix][SM70] Stabilize QSA page4 order under KV relocation 1CatAI/1Cat-vLLM#494 fix.
• Integration follow-up: the first new-base NVFP4 model attempt exited during
  the C4 quality check, before long-context scoring. The QSA interface loaded
  the image's package-local Flash-V100 binary (18 XQA arguments), while the
  provenance check inspected a different top-level binary (19 arguments).
  The new launcher had omitted the full Flash Python package path. This is
  a validation-runtime import-shadowing bug, not an AWQ performance or precision
  result. The launcher now mounts the pinned Python package alongside its
  native extension and checks actual/top-level binding identity and a real
  19-argument call before model loading and in all TP workers. A clean Flash
  rebuild and the CPU binding check passed; the r2 model retry subsequently
  completed. No route or quality checks were disabled, and the historical
  tables remain unchanged.
• An additional unchanged fallback test module had five successful test bodies
  but five teardown errors: CPU-only host torch.accelerator.empty_cache()
  raises Cannot access accelerator device when none is available. Its test,
  global cleanup fixture and cleanup implementation are unchanged from main.
  This expanded invocation was 75 passed, 5 errors, not an all-green run;
  no unrelated cleanup fix is included.
• Frozen source validation previously completed an extension build, 51
  GPU-directed tests and 32 dynamic-route operator comparisons. These are
  retained historical checks, not a rebuild of the aligned tree.

New-base bounded integration acceptance (r2)

On base 755baae1d075, using the same frozen prompt IDs and runtime contract,
AWQ grouped decode and optional autotune enabled, shared overlap disabled:

Cell	Pure aggregate tok/s	Mean pure ITL ms	Prefill+mixed s	Pure window s	E2E s
C1×64K	51.2897	19.4971	14.5356	6.2196	20.7552
C4×64K	131.7116	30.3694	79.8826	8.9286	89.4635
C8×16K	247.5878	32.3118	31.3695	9.8228	41.6304

Each cell scored once; 319/294/304 pure-window tokens per request, no
multi-token deliveries. All 18 short quality requests stopped naturally at EOS,
passed basic answer checks and had finite recorded logprobs. C4 self-repeats
matched complete token IDs and top-5 logprobs in 4/4 cases; post-score C1 profile
output matched the scored output. All four workers reported 48 admitted grouped
layers and tuning enabled. Exit 0, no OOM; resource-tracker semaphore/shared
memory cleanup warnings at shutdown remain in the raw evidence.

Startup: 438.4613 seconds. Engine-reported KV capacity: 427,385 tokens;
configured cache blocks: 563; graph memory: 0.37 GiB. Correction: the earlier
563×784 extrapolation was not the engine-reported token capacity.
This is integration acceptance, not a new OFF/ON comparison;
upstream changes also contribute. No cross-version bitwise or broad quality
claim, no change to the precision deferral, and no production deployment.
The separate QSA 1CatAI#494 port remains part of the validation stack, not this PR.
Evidence ID: qwen38-c1-main-20260905-r2/awq.

Implementation-only benefit (frozen r3)

Both AWQ arms have autotune off. Pure aggregate tok/s:

Cell	Grouped OFF	Grouped ON	Gain	NVFP4
C1×64K	48.65	48.60	−0.09%	54.65
C4×64K	110.02	116.20	+5.62%	130.91
C8×16K	207.29	211.94	+2.24%	237.58

Additional existing-autotune benefit (fresh-process pair)

Both AWQ arms have grouped decode on. Independent processes, private empty
caches and no imported LUT; only the tuning flag changes:

Cell	Tuning OFF	Tuning ON	Gain	Historical r3 NVFP4
C1×64K	48.7874	48.5898	−0.40%	54.6544
C4×64K	116.1290	129.2741	+11.32%	130.9051
C8×16K	212.0154	242.7240	+14.48%	237.5780

Four V100 PCIe 32 GB; TP4/MTP0/FP16 KV, language-only MRv2 graphs, chunked
prefill with 8192 batched tokens, prefix cache off, identical corrected QSA
and frozen input IDs. Each cell/arm scored once, 320 output tokens,
ignore_eos=false / min_tokens=0. Common all-active windows have
319/294/304 tokens per request, with no multi-token deliveries. NVFP4 is not a
new third arm in the tuning experiment. Do not add the two experiments'
percentages, claim statistical superiority, or extrapolate C8×64K.

Both formats already had the existing FP16 GEMV, fused HC, fused GDN input,
and cuBLAS QSA-indexer opt-ins enabled; runtime-lossy QPN8 was off. Actual
collectives were PyNCCL on PCIe V100. Later fully-connected custom-AR/SXM2 HC
sharding evidence does not prove the sharded route can run on this topology.

The new OFF baseline is within 0.4% of historical grouped ON. A separate
post-score C4 profile confirms actual W13 tile M64×128×32 → M8×256×64 on all
four ranks, 5.531 → 2.109 GPU ms/step; other major categories are nearly
unchanged. Profile durations are neither new unprofiled ITLs nor rank sums.

Precision findings and deliberate deferral

• Grouped-only AWQ OFF/ON: 17/18 complete short-output token sequences match.
• Tuning pair: all 36 short requests stop naturally at EOS, pass basic answer
  checks and have finite recorded logprobs. Same-arm C4 repeats match complete
  token IDs and recorded top-5 logprobs in 4/4 cases per arm.
• Across tuning arms, 16/18 complete token sequences match. Two C8 differences
  are arithmetic wording (and / +, answer 156 unchanged) and an open Chinese
  response (71/70 tokens). Both fresh arms also differ from historical grouped
  ON in 2/18 cases: baseline process drift means not every difference can be
  assigned to tuning.
• W13 tile/reduction choices have small real-weight rounding differences;
  the separate tuning probe reached max absolute difference 0.0009765625.
  These bounded tests found no new functional failure, but do not establish
  bitwise equality or broad model quality.
• We explicitly record and defer the precision-sensitive cross-batch/
  cross-process issue. It is separate from the QSA ordering bug. This PR does
  not attempt a general determinism fix or alter HC/NCCL precision defaults.

Capacity, startup, and remaining limit

Tuning OFF/ON both report 386,392 KV tokens and 0.37 GiB graph memory;
initialization-to-ready is 431.090/427.330 seconds. No extra total startup or
KV-capacity cost was observed in this single pair; full memory-peak equality
and startup acceleration are not claimed. C4/C8 E2E seconds are
94.6550→93.2887 and 44.8195→43.3021, only −1.44%/−3.39%; long-prefill blocking
remains. The design document includes phase times, provenance and limitations.

Next work is a separate dual-format C1 investigation, including shared
projection/HC costs and each format's MoE preparation. NVFP4 is a reference,
not a performance ceiling. New upstream C1/HC changes must be assessed before
reusing old bottleneck attribution.

Review checklist

• Narrow scope, rollback, duplicate check and AI assistance documented.
• Implementation-only and opt-in configuration gains separated.
• CPU tests, historical GPU evidence and precision limitations recorded.
• Fork main aligned without force-push; unrelated branches retained.
• Source-matched native build and CPU-only import/hash acceptance.
• Bounded GPU model acceptance of the new aligned tree (with separate QSA fix).
• Final human review before merge or upstream submission.