evidence(W_O K-parallel): validation harness + 8.39x sweep + NCU memory-bound

[Natfii]

Summary

• W_O K-parallel validation passed. Bit-exact correctness (max_abs=0.0) for all four split-order variants against per-variant FP32 reference; sweep reaches 8.39× speedup at 32 W_O CTAs (vs 4 baseline).
• NCU classification: scaled endpoint becomes memory-bound (55.95% peak DRAM bw, 6.51% SM busy) while the baseline is latency-limited (8.06% / 1.17%). The W_O lever converts a latency-limited kernel into a memory-bound one — PR evidence(β-coop region breakdown): 36% K-reducible, W_O is the bottleneck #6's matrix gate now passes at the scaled endpoint.
• Production parity gap recorded as audit follow-up, not a conclusion — the slope is valid evidence; absolute-timing mismatch needs a separate denominator + launch-shape audit.

Curated evidence: benchmarks/nvllm/traces/cute_paged_attn/2026-05-03-w-o-k-parallel-harness/summary.md.

Scope

• Updated PR evidence(β-coop region breakdown): 36% K-reducible, W_O is the bottleneck #6 breakdown summaries (mirror pair) — adds the "NCU pivot" subsection recording the structural failures of --replay-mode kernel (deadlocks cooperative grid barrier) and --replay-mode application (long-running vllm serve blocks application-replay), with decision-record line that R2-R4 coupling carries W_O to validation despite the matrix gate not yet being met at that point.
• NCU attempt logs under docs/research/2026-05-02-beta-region-breakdown/ncu-attempt{1,2}/ — provenance for the failed in-process attempts that drove the pivot.
• New harness under docs/research/2026-05-03-w-o-k-parallel-harness/:
  • microkernel.py — W_O kernel with wo_split Constexpr + per-K-group block-scale dequant (production-equivalent FP4 + FP8 path)
  • torch_reference.py — three FP32 references; reference_split_order(wo_split) is AUTHORITATIVE per variant (mirrors each kernel's reduction tree)
  • run_harness.py — configurable sweep + NCU re-exec, fail-fast correctness gate
  • run_sweep.sh — docker-wrapped 4-variant sweep (CTAs ∈ {4, 8, 16, 32})
  • README.md — design contract + ratification record
• Evidence artifacts under benchmarks/nvllm/traces/cute_paged_attn/2026-05-03-w-o-k-parallel-harness/:
  • 4 variant scratchpads (50 launches each, timing.csv + 4 correctness JSONs)
  • ncu/wo_split_1/ and ncu/wo_split_8/ — bounded NCU runs (filtered by --kernel-name regex:wo_kernel_body)
  • ncu/ncu_unfiltered_aborted/wo_split_1/ — first-attempt metadata only (the 2.7 GB partial .ncu-rep was discarded; first attempt aborted at 5h31m due to unfiltered profiling of non-target kernels)
  • summary.md — curated headline numbers + classification verdicts

Total new evidence: ~1.1 MB (NCU reports 324 KB + 525 KB; everything else is text).

Headline numbers

wo_split	W_O CTAs	Mean elapsed (μs)	Speedup	NCU classification
1	4	13754	1.00×	latency-limited
2	8	5176	2.66×	(unprofiled)
4	16	2693	5.11×	(unprofiled)
8	32	1639	8.39×	memory-bound

NCU at the two endpoints (same kernel symbol, same grid, only the bx < wo_split gate differs):

Metric	wo_split=1	wo_split=8
Max Bandwidth (% peak DRAM)	8.06%	55.95%
SM Busy	1.17%	6.51%
No-Eligible Warps (% cycles)	98.33%	91.86%
Eligible Warps / Scheduler	0.02	0.08

Test plan

• Sweep correctness gate at all 4 splits (max_abs=0.0 against reference_split_order(wo_split))
• NCU on baseline endpoint (wo_split=1)
• NCU on scaled endpoint (wo_split=8)
• Verify aborted-attempt metadata preserved (no 2.7 GB .ncu-rep in tree)
• DRAM headroom corrected to ~44% (matches 55.95% peak)
• No __pycache__ in harness tree
• Reviewer reproduce: bash docs/research/2026-05-03-w-o-k-parallel-harness/run_sweep.sh
• Reviewer reproduce: NCU command in summary.md § How to reproduce

Notes

• AI-assisted: harness design + implementation + NCU run iteration done with Claude Opus 4.7. Curated commit message + summary.md authored under direct user direction with explicit framing constraints (slope vs absolute parity, classification wording).
• No production code changed. This is harness + evidence only. Production integration (β-coop W_O K-parallel) is the next ticket; design notes in docs/research/2026-05-03-w-o-k-parallel-harness/README.md §1b.

🤖 Generated with Claude Code