perf(gfx1250): pin the sweeps that fault, put mori_ep back, fix three…

[JiaoliangYu]

… notes

--dsv4 came back 8 failed / 10 ops on the 20260827 image. Every failure was a shape a kernel cannot survive, not a bench bug, so pin past them and record what was measured. No operator and no tuned row is touched here.

mega_moe drops tokens/rank=65536 (40 -> 36 cases). It dies in pipe.setup() asking 7.5 GB for cco's VMM arena against a 4 GiB default. Note that the earlier explanation named the wrong allocator: that 4 GiB is Communicator.DEFAULT_PER_RANK_VMM, a ccoCommCreate argument with no environment variable behind it, and test_mega_moe_gfx1250.py:512 calls Communicator.init() without passing it. MORI_SHMEM_HEAP_SIZE is read only in mori/src/shmem/init.cpp and feeds a different allocator, so exporting it for that one case does nothing -- and exporting it sweep-wide took a machine down, because that heap is preallocated per rank for every case (reverted here). The error's own "Hint: Increase via MORI_SHMEM_HEAP_SIZE" is what points the wrong way.

mla_v4_prefill pins n=1024, the only tier that never faulted. Measured with --no-verify, so no reference is involved: 1024 3/3 pass, 2048 2/3 fault, 4096 1/1, 8192 1/1, 16384 3/3, 65536 1/1. Only 16384 is reproducibly dead; 2048 passed on the third try, so the middle tiers are not shown to be deterministic. 16384 is the DSv4 prefill chunk and is what this op exists to measure, so this is coverage lost to a kernel bug. AITER_BENCH_TOKENS still overrides.

mla_v4_prefill_fp8 drops nnz_prefix=8192, the only value that faults, at (n=2048, total_pages=16384, mode=fixed). The kernel is not what breaks: that point passes with --no-verify at 2040 TFLOPS, so the reference or the comparison is what dies. Dropped anyway because this op runs the UT bare, where verify is on, and the UT prints its table only at the end, so one fault costs every shape that already ran.

mori_ep goes back into the suite. The two all2all legs are what either MoE path pays around the GEMMs, so a DSv4 sweep without them measures half the layer. It was held out for a reason that no longer holds -- run_mori_ep stopped rebuilding mori, so the dev ROCm toolchain the old note cited is not needed. It keeps its own TOKENS sweep (64..16384), matching mori_dsv4_bench.sh.

The note on a8w8_blockscale claimed gfx1250 has no working path for it. Wrong: at -m 512 the six DSv4 (n,k) give err=0 at 1210-3564 TFLOPS over ck/asm/flydsl. What kills it is the tuning table -- a libtype=triton row routes into gemm_afp8wfp8_preshuffle with backend="gluon", and that kernel does not compile here (PassManager::run failed in triton's make_llir). gfx1250 has 11 such rows, all at M=16 and M=64, added by #4773, which replaced 11 faster flydsl rows and carried their tflops/bw columns over unchanged so the rows disagree with their own us. All 11 were run: 11/11 fail across K 768..7168 and N 1536..16384; three shapes whose M misses those rows pass. So it is the gluon kernel, not eleven unlucky configs, and re-tuning cannot fix it either. It stays out of the defaults: pinning -m would make it run, but the reported kernel would no longer be the tuned one.

Verified per-op on 20260827 / b45-1: the eleven --dsv4 ops are rc=0, 56 cases, 0 failed.

Motivation

Technical Details

Test Plan

Test Result

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.