[gfx1250] Sweep the fp4 dispatch wire in mori_ep - #5147
Merged
Conversation
mori_ep measured only the bf16 wire, which is not the leg DSv4 serves on.
The receiver hands the dispatched payload straight to the expert GEMM as its
A operand, and that GEMM is a4w4 -- ATOM's serve script pins MEGA_WIRE=fp4
with AITER_FORCE_A8W4=0 -- so bf16 alone measures a path production does not
take, and misses the wire whose payload is a quarter as wide.
Sweeps bf16 and fp4 by default, one child each: bench_ep.py reads $DISP once
at import and builds the transport for that dtype, so the tiers cannot share
a process. $DISP still overrides, now comma-separated, and is forwarded
unvalidated -- mori owns the value set, this file does not restate it.
fp4 rows are labelled UNCHECKED. mori forces its own comparison off on fp4
("fp4 combine is too lossy to compare"), so CHECK=1 is silently dropped
there; without the label a passing fp4 row reads as verified next to a bf16
row that actually was.
JiaoliangYu
added a commit
to JiaoliangYu/aiter
that referenced
this pull request
Sep 3, 2026
mori_ep measured only the bf16 wire, which is not the leg DSv4 serves on.
The receiver hands the dispatched payload straight to the expert GEMM as its
A operand, and that GEMM is a4w4 -- ATOM's serve script pins MEGA_WIRE=fp4
with AITER_FORCE_A8W4=0 -- so bf16 alone measures a path production does not
take, and misses the wire whose payload is a quarter as wide.
Sweeps bf16 and fp4 by default, one child each: bench_ep.py reads $DISP once
at import and builds the transport for that dtype, so the tiers cannot share
a process. $DISP still overrides, now comma-separated, and is forwarded
unvalidated -- mori owns the value set, this file does not restate it.
fp4 rows are labelled UNCHECKED. mori forces its own comparison off on fp4
("fp4 combine is too lossy to compare"), so CHECK=1 is silently dropped
there; without the label a passing fp4 row reads as verified next to a bf16
row that actually was.
Co-authored-by: Yu <jiaolyu@amd.com>
JiaoliangYu
added a commit
that referenced
this pull request
Sep 3, 2026
* perf(gfx1250): add the combined gfx1250 microbench (#5076) Two files, both new, on top of main: op_tests/bench_gfx1250_combo.py combined bench op_tests/triton_tests/attention/test_mla_v4_triton.py MLA v4 triton reference The bench imports the top-level @benchmark sweep fns from the op_tests siblings and runs each over its own shape axes, printing only the per-op summary tables -- all the JIT/ROCTracer/import noise is silenced at the fd level. Two suites: --perf for hardware-oriented single-op numbers, --dsv4 for the DeepSeek-V4 shapes. Token sweeps come from AITER_BENCH_TOKENS; ops whose axis means something else, or whose usable range is fixed, pin their sweep in the source and say why. Everything else the branch used to carry has reached main on its own since: test_pa_sparse_prefill.py (#4926), test_mega_moe_gfx1250.py (#5052), test_flydsl_qk_norm_rope_quant.py and the mla_v4 hsa artifacts. Only these two files were ever unique to it, so this is what is left to upstream. Notes worth carrying, all measured on gfx1250 / 20260827-28: a16w16 does not pre-check the >4 GiB operand limit. That guard belongs to one fallback path -- opus_dispatch_a16w16_gfx1250 searches the tuned table first and only reaches check_shape_4g on a miss, en route to the split-K kid whose launcher builds the 32-bit gmem descriptors. A tuned 4wave_wl_co winner never gets there. Predicting it in Python skipped shapes that tuning had already made runnable, so the kernel is left to raise and the exception is recorded as a row. It also checks the error ratio the UT returns against _A16W16_MAX_ERR: all four M=65536 shapes come back err=0.96-0.99 while every other row is 0 or ~1e-5, and nothing in the UT raises or warns, so those used to print as data. a8w8_blockscale sweeps M from 1024. Below that, get_CKGEMM_config's M -> get_padded_m -> nextPow2 retry lands on #4773's M=16/M=64 gluon rows, and the UT's extra "ck strided x_scale" check (line 120, added by #4406 and gated on ck_preshuffle alone) hands triton a stride != 1 specialization that fails to compile in make_llir. The mxfp8_128 path declares its layout with is_x_scale_transposed=True and never reads the stride, so that check tests nothing there. Fixing the gate is upstream's call; starting at 1024 keeps this bench clear of it. Verified 36/36, err=0, 2207-7003 TFLOPS. mla_v4_prefill is pinned to n=1024 and mla_v4_prefill_fp8 drops nnz_prefix=8192; both are kernel/verify faults, measured and documented at the pins. Co-authored-by: Yu <jiaolyu@amd.com> * perf(gfx1250): let AITER_BENCH_TOKENS reach every op (#5082) Three token constants were hardcoded and never consulted the variable -- _INVERSE_ROPE_TOKENS, _MLA_DECODE_TOKENS, _MEGA_MOE_TOKENS -- with comments saying so ("Pinned, not env-driven", "deliberately not consulted"). The reasoning was that a global token count means the wrong thing for those ops, which is true of the default but not of an explicit request: if a caller sets the variable, that is their decision to make, including asking for a shape the op is known to fail on. All nine token constants now go through _tokens(): unset, each op runs its own default and says at its constant why that default is not the shared list; set, the variable wins everywhere and the file does not argue with it. Behaviour with the variable unset is unchanged. The module docstring said "Two ops ignore it and pin their sweep in the source" -- no longer true, and it only listed two of the five ops whose default differs. It now lists all five with the reason for each. Should have been part of #5076; amended in after that PR was pushed. Co-authored-by: Yu <jiaolyu@amd.com> * perf(gfx1250): sweep the fp4 dispatch wire beside bf16 (#5147) mori_ep measured only the bf16 wire, which is not the leg DSv4 serves on. The receiver hands the dispatched payload straight to the expert GEMM as its A operand, and that GEMM is a4w4 -- ATOM's serve script pins MEGA_WIRE=fp4 with AITER_FORCE_A8W4=0 -- so bf16 alone measures a path production does not take, and misses the wire whose payload is a quarter as wide. Sweeps bf16 and fp4 by default, one child each: bench_ep.py reads $DISP once at import and builds the transport for that dtype, so the tiers cannot share a process. $DISP still overrides, now comma-separated, and is forwarded unvalidated -- mori owns the value set, this file does not restate it. fp4 rows are labelled UNCHECKED. mori forces its own comparison off on fp4 ("fp4 combine is too lossy to compare"), so CHECK=1 is silently dropped there; without the label a passing fp4 row reads as verified next to a bf16 row that actually was. Co-authored-by: Yu <jiaolyu@amd.com> * perf(gfx1250): follow the sparse-prefill UT through #5084 (#5170) #5084 rewrote test_pa_sparse_prefill.py and mla_v4_prefill stopped running against it: --nnz-prefix is gone, so the child aborted with "unrecognized arguments" before reaching a kernel. - drop --nnz-prefix. The nnz axis no longer exists: the CSR is generated from --mode under --seed (sparse draws a random nnz per row, dense fills every row), so nnz is an outcome, not an input - delete the mla_v4_prefill_fp8 op. Its entire content was that nnz sweep, and it has no equivalent under the new UT. mla_v4_prefill already covers fp8 -- and now covers the backends too, since #5084 compares opus/asm/triton - _space_table: decide a data row by counting numeric fields instead of testing the first one. The new table leads with prec/mode (bf16, dense), so the first-field test dropped the whole table and the op reported "no result rows" while the UT itself had exited 0 Raise _MLA_PREFILL_TOKENS to 1024..16384. Re-measured on the new UT (b45-2, one process per tier): 1024 through 16384 all clean, where 4096/8192/16384 faulted on the old one. 16384 is the DSv4 prefill chunk and the reason this op exists, so this is the coverage the pin was costing us. 65536 stays out. It faults with a memory access at 0x7f2ddbec0000 and writes an 89 GB coredump doing it -- a third of the free disk on a shared host -- and it is past the chunk size the model prefills anyway. Both the tier sweep and this op run --no-verify, hardcoded at the call site, and that is now load-bearing rather than a speed choice: with verification on, even n=1024 dies at the first case (fp8/dense, fault at 0x43000), so the reference or the comparison is what breaks, not the kernel under test. These are therefore timings from an unverified kernel. The comment says so, because a16w16's M=65536 rows looked exactly this healthy until _A16W16_MAX_ERR was added and caught err=0.99. Co-authored-by: Yu <jiaolyu@amd.com> * perf(gfx1250): stop every child from shelling out to rocminfo (#5180) chip_info runs rocminfo twice per process -- once for the arch, once for the CU count -- and rocminfo takes a per-device rocm_smi mutex on its way in. One process is fine, and the nine single-GPU ops never noticed. A torchrun op starts four ranks at once, and they contend for that mutex. Both multi-GPU ops have now lost a run to it. On b45-1 a rank lost the mutex and aborted: init_mutex /rocm_smi_renderD128: unlock timed lock, ret: 1 terminate called after throwing an instance of 'std::runtime_error' what(): Allgather operation failed The allgather is the symptom, not the cause: the rank died first and took the collective with it. On b45-2 the other shape of the same problem -- four rocminfo processes sat in the mutex for minutes, one wedged in D state, the op never produced a line, and even docker stop hung waiting for the driver to let go. Detect once here, where the call is serial, and hand the answer down: GPU_ARCHS -> get_gfx_list() skips _detect_native() CU_NUM -> get_cu_num_custom_op() skips its own rocminfo Both are read from the environment before either shells out, so a child that inherits them runs no rocminfo at all. Set on os.environ in main() for the children that inherit our environment, and setdefault() in _run_child for the ones handed an explicit env -- setdefault throughout, so an exported value from the caller still wins. The value is what this process detected on this machine, not a hardcoded gfx1250, so it is the same answer the child would have computed. Verified in the child's /proc/<pid>/environ: GPU_ARCHS=gfx1250, AITER_GPU_ARCHS=gfx1250, CU_NUM=256, matching get_gfx()/get_cu_num() here. mega_moe on b45-1 after the change: rc=0, all 36 configs, zero mutex warnings, and zero rocminfo processes observed while it ran. Before it, the same op on the same host was rc=1 with 6 mutex warnings. Co-authored-by: Yu <jiaolyu@amd.com> * test_common：data generate --------- Co-authored-by: Yu <jiaolyu@amd.com>
JiaoliangYu
added a commit
that referenced
this pull request
Sep 13, 2026
* Gfx1250/microbench (#5241) * perf(gfx1250): add the combined gfx1250 microbench (#5076) Two files, both new, on top of main: op_tests/bench_gfx1250_combo.py combined bench op_tests/triton_tests/attention/test_mla_v4_triton.py MLA v4 triton reference The bench imports the top-level @benchmark sweep fns from the op_tests siblings and runs each over its own shape axes, printing only the per-op summary tables -- all the JIT/ROCTracer/import noise is silenced at the fd level. Two suites: --perf for hardware-oriented single-op numbers, --dsv4 for the DeepSeek-V4 shapes. Token sweeps come from AITER_BENCH_TOKENS; ops whose axis means something else, or whose usable range is fixed, pin their sweep in the source and say why. Everything else the branch used to carry has reached main on its own since: test_pa_sparse_prefill.py (#4926), test_mega_moe_gfx1250.py (#5052), test_flydsl_qk_norm_rope_quant.py and the mla_v4 hsa artifacts. Only these two files were ever unique to it, so this is what is left to upstream. Notes worth carrying, all measured on gfx1250 / 20260827-28: a16w16 does not pre-check the >4 GiB operand limit. That guard belongs to one fallback path -- opus_dispatch_a16w16_gfx1250 searches the tuned table first and only reaches check_shape_4g on a miss, en route to the split-K kid whose launcher builds the 32-bit gmem descriptors. A tuned 4wave_wl_co winner never gets there. Predicting it in Python skipped shapes that tuning had already made runnable, so the kernel is left to raise and the exception is recorded as a row. It also checks the error ratio the UT returns against _A16W16_MAX_ERR: all four M=65536 shapes come back err=0.96-0.99 while every other row is 0 or ~1e-5, and nothing in the UT raises or warns, so those used to print as data. a8w8_blockscale sweeps M from 1024. Below that, get_CKGEMM_config's M -> get_padded_m -> nextPow2 retry lands on #4773's M=16/M=64 gluon rows, and the UT's extra "ck strided x_scale" check (line 120, added by #4406 and gated on ck_preshuffle alone) hands triton a stride != 1 specialization that fails to compile in make_llir. The mxfp8_128 path declares its layout with is_x_scale_transposed=True and never reads the stride, so that check tests nothing there. Fixing the gate is upstream's call; starting at 1024 keeps this bench clear of it. Verified 36/36, err=0, 2207-7003 TFLOPS. mla_v4_prefill is pinned to n=1024 and mla_v4_prefill_fp8 drops nnz_prefix=8192; both are kernel/verify faults, measured and documented at the pins. Co-authored-by: Yu <jiaolyu@amd.com> * perf(gfx1250): let AITER_BENCH_TOKENS reach every op (#5082) Three token constants were hardcoded and never consulted the variable -- _INVERSE_ROPE_TOKENS, _MLA_DECODE_TOKENS, _MEGA_MOE_TOKENS -- with comments saying so ("Pinned, not env-driven", "deliberately not consulted"). The reasoning was that a global token count means the wrong thing for those ops, which is true of the default but not of an explicit request: if a caller sets the variable, that is their decision to make, including asking for a shape the op is known to fail on. All nine token constants now go through _tokens(): unset, each op runs its own default and says at its constant why that default is not the shared list; set, the variable wins everywhere and the file does not argue with it. Behaviour with the variable unset is unchanged. The module docstring said "Two ops ignore it and pin their sweep in the source" -- no longer true, and it only listed two of the five ops whose default differs. It now lists all five with the reason for each. Should have been part of #5076; amended in after that PR was pushed. Co-authored-by: Yu <jiaolyu@amd.com> * perf(gfx1250): sweep the fp4 dispatch wire beside bf16 (#5147) mori_ep measured only the bf16 wire, which is not the leg DSv4 serves on. The receiver hands the dispatched payload straight to the expert GEMM as its A operand, and that GEMM is a4w4 -- ATOM's serve script pins MEGA_WIRE=fp4 with AITER_FORCE_A8W4=0 -- so bf16 alone measures a path production does not take, and misses the wire whose payload is a quarter as wide. Sweeps bf16 and fp4 by default, one child each: bench_ep.py reads $DISP once at import and builds the transport for that dtype, so the tiers cannot share a process. $DISP still overrides, now comma-separated, and is forwarded unvalidated -- mori owns the value set, this file does not restate it. fp4 rows are labelled UNCHECKED. mori forces its own comparison off on fp4 ("fp4 combine is too lossy to compare"), so CHECK=1 is silently dropped there; without the label a passing fp4 row reads as verified next to a bf16 row that actually was. Co-authored-by: Yu <jiaolyu@amd.com> * perf(gfx1250): follow the sparse-prefill UT through #5084 (#5170) #5084 rewrote test_pa_sparse_prefill.py and mla_v4_prefill stopped running against it: --nnz-prefix is gone, so the child aborted with "unrecognized arguments" before reaching a kernel. - drop --nnz-prefix. The nnz axis no longer exists: the CSR is generated from --mode under --seed (sparse draws a random nnz per row, dense fills every row), so nnz is an outcome, not an input - delete the mla_v4_prefill_fp8 op. Its entire content was that nnz sweep, and it has no equivalent under the new UT. mla_v4_prefill already covers fp8 -- and now covers the backends too, since #5084 compares opus/asm/triton - _space_table: decide a data row by counting numeric fields instead of testing the first one. The new table leads with prec/mode (bf16, dense), so the first-field test dropped the whole table and the op reported "no result rows" while the UT itself had exited 0 Raise _MLA_PREFILL_TOKENS to 1024..16384. Re-measured on the new UT (b45-2, one process per tier): 1024 through 16384 all clean, where 4096/8192/16384 faulted on the old one. 16384 is the DSv4 prefill chunk and the reason this op exists, so this is the coverage the pin was costing us. 65536 stays out. It faults with a memory access at 0x7f2ddbec0000 and writes an 89 GB coredump doing it -- a third of the free disk on a shared host -- and it is past the chunk size the model prefills anyway. Both the tier sweep and this op run --no-verify, hardcoded at the call site, and that is now load-bearing rather than a speed choice: with verification on, even n=1024 dies at the first case (fp8/dense, fault at 0x43000), so the reference or the comparison is what breaks, not the kernel under test. These are therefore timings from an unverified kernel. The comment says so, because a16w16's M=65536 rows looked exactly this healthy until _A16W16_MAX_ERR was added and caught err=0.99. Co-authored-by: Yu <jiaolyu@amd.com> * perf(gfx1250): stop every child from shelling out to rocminfo (#5180) chip_info runs rocminfo twice per process -- once for the arch, once for the CU count -- and rocminfo takes a per-device rocm_smi mutex on its way in. One process is fine, and the nine single-GPU ops never noticed. A torchrun op starts four ranks at once, and they contend for that mutex. Both multi-GPU ops have now lost a run to it. On b45-1 a rank lost the mutex and aborted: init_mutex /rocm_smi_renderD128: unlock timed lock, ret: 1 terminate called after throwing an instance of 'std::runtime_error' what(): Allgather operation failed The allgather is the symptom, not the cause: the rank died first and took the collective with it. On b45-2 the other shape of the same problem -- four rocminfo processes sat in the mutex for minutes, one wedged in D state, the op never produced a line, and even docker stop hung waiting for the driver to let go. Detect once here, where the call is serial, and hand the answer down: GPU_ARCHS -> get_gfx_list() skips _detect_native() CU_NUM -> get_cu_num_custom_op() skips its own rocminfo Both are read from the environment before either shells out, so a child that inherits them runs no rocminfo at all. Set on os.environ in main() for the children that inherit our environment, and setdefault() in _run_child for the ones handed an explicit env -- setdefault throughout, so an exported value from the caller still wins. The value is what this process detected on this machine, not a hardcoded gfx1250, so it is the same answer the child would have computed. Verified in the child's /proc/<pid>/environ: GPU_ARCHS=gfx1250, AITER_GPU_ARCHS=gfx1250, CU_NUM=256, matching get_gfx()/get_cu_num() here. mega_moe on b45-1 after the change: rc=0, all 36 configs, zero mutex warnings, and zero rocminfo processes observed while it ran. Before it, the same op on the same host was rc=1 with 6 mutex warnings. Co-authored-by: Yu <jiaolyu@amd.com> * test_common：data generate --------- Co-authored-by: Yu <jiaolyu@amd.com> * Blyu/opus gemm benchmark update (#5250) * [test] Add bench controls to test_opus_a16w16_gemm Add configurable warmup/iteration counts, data-init modes (zero/constant/uniform/norm) with a seed, and rotating-tensor control to the opus a16w16 GEMM test, and report both TFLOPs and bandwidth (TB/s) explicitly across all three timing paths. The opus graph sweep table now also surfaces the tuned CSV's splitK and kernelName (kid) per shape, so the algorithmic winner is visible and tracks CSV updates. test_a16w16 keeps a backward-compatible signature so the gfx1250 combo bench is unaffected. Co-authored-by: Cursor <cursoragent@cursor.com> * [test] Fix ruff C408 in test_opus_a16w16_gemm Rewrite init_kwargs dict() call as a dict literal to satisfy `ruff check` (C408); black is unchanged. Co-authored-by: Cursor <cursoragent@cursor.com> * test_common：data generate * [test] Adopt shared test_common data-init in a16w16 gemm Use the cherry-picked aiter.test_common data-init API (make_generator / add_data_init_args / fill / DATA_DISTS) in test_opus_a16w16_gemm.py instead of the local DATA_INITS/_make_generator/_make_tensor helpers, so operand init and --data-init/--seed CLI match every other op test. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: binglue.lyu <blyu@heliosp-1b114-d07-2.mnb.dcgpu> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: JiaoliangYu <jiaolyu@amd.com> Co-authored-by: blyu <blyu@amd.com> * [ASM][gfx1250] enhance ASM GEMM microbench (#5251) * test(gemm): use common data initialization options for gfx1250 asm gemm Signed-off-by: Double Young <yang.yang2@amd.com> * test(gemm): expose warmup/iters/graph/rotate CLIs for gfx1250 asm gemm Signed-off-by: Double Young <yang.yang2@amd.com> * test(gemm): show algorithmic details for gfx1250 asm gemm Signed-off-by: Double Young <yang.yang2@amd.com> --------- Signed-off-by: Double Young <yang.yang2@amd.com> * test mega moe data init and warmup (#5248) * Dev/ubench gemm (#5252) * update * update * support a8w8 ubench * revert ck module * apply aiter common interface --------- Co-authored-by: yadaish <yadai@amd.com> * [Bench] Add data-init options to qk_norm_rope_quant test (#5253) Add --seed, --init {normal,uniform,zero,constant}, --init-scale, and --init-val CLI flags so callers can control how input tensors are initialised, enabling reproducible perf sweeps across data distributions. Co-authored-by: jli-melchior <Jun.Li@amd.com> Co-authored-by: Claude <noreply@anthropic.com> * [HIP] Add data-init to test_mhc (#5245) Wire test_common.fill, --data-init, and --seed so MHC inputs are reproducible. Co-authored-by: Cursor <cursoragent@cursor.com> * perf(gfx1250): add ubench init controls and SMI monitor (#5257) Co-authored-by: JiaoliangYu <jiaolyu@amd.com> * [Perf][gfx1250] Complete DSv4 attention data-init support (#5258) * perf(gfx1250): complete DSv4 data init controls * perf(gfx1250): report per-case SMI telemetry * perf(gfx1250): monitor Mega MoE per rank --------- Co-authored-by: JiaoliangYu <jiaolyu@amd.com> * perf(gfx1250): add M512 A8W8 DSv4 case (#5284) Co-authored-by: JiaoliangYu <jiaolyu@amd.com> * [Perf][gfx1250] Emit ubench summaries as JSON (#5288) * docs(gfx1250): document bundled amdsmi binding * perf(gfx1250): emit benchmark summaries as JSON --------- Co-authored-by: JiaoliangYu <jiaolyu@amd.com> * fix(gfx1250): scope bundled amdsmi import (#5291) Co-authored-by: JiaoliangYu <jiaolyu@amd.com> * perf(gfx1250): reuse process-wide amdsmi state (#5302) Co-authored-by: JiaoliangYu <jiaolyu@amd.com> * [Perf][gfx1250] Label SMI replays per benchmark call (#5304) * perf(gfx1250): label SMI replays per benchmark call * perf(gfx1250): cover saturated DSv4 MLA decode KV * perf(gfx1250): sweep DSv4 score-QK decode batches * perf(gfx1250): cover TP4 inverse RoPE shape * perf(gfx1250): add M256 blockscale coverage * perf(gfx1250): add 64K Mega MoE coverage --------- Co-authored-by: JiaoliangYu <jiaolyu@amd.com> * [ASM][gfx1250] mla v4 prefill: rebuild the sparse_pfl kernel with the head schedule and the PTL prefetch hazard fixed Replaces mla_a8w8_qh128_1tg_32mx4_32nx1_sparse_pfl.co (462a5ee0, as shipped by #4926) with a build carrying the 09-07/09-08 head work plus one correctness fix. The correctness fix. The page-table prefetch built its row offset from a VGPR written by v_writelane_b32 at the end of KV_page_prefetch_build; the consumers are the two v_lshlrev at the top of KV_page_prefetch_issue, on the far side of the _s_p_tiles branch with zero VALU in between: v_writelane_b32 v4, s60, 0 s_cmp_eq_u32 s103, 0 s_cbranch_scc0 label_1028 v_lshlrev_b32 v2, 9, v4 <- 2 SALU after the write SCHED_MODE 2 has no VALU->VALU interlock, so the shift could read the stale register (Q addr-gen leftovers). `stale << 9` off a bounds-free 64-bit base is a wild address, observed under batched prefill as Memory access fault by GPU node-3 ... Reason: Page not present No numeric test can catch this: a prefetch has no destination register, so the bad address never reaches a result. Fixed with one s_wait_alu placed before the branch, covering both the fall-through (extend) and taken (prefix) arms. +4 bytes; the core loop is byte-identical. Also in this build, relative to 462a5ee0: * head schedule: HEAD_INDPTR_EARLY, HEAD_INDEX_HOIST, HEAD_TDM_SLOT_AWARE * HEAD_KV_PTL_* page-table prefetch over tile 0's KV rows * the packed-row scale tail mask (448:511 pad), which 462a5ee0 lacks -- that pad reads as NaN once the host stops zero-filling it Source: junxiaguo/my_agent_skills @ 495b486 (0805_base/32mx4_32nx1.sp3). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * support print overlap ratio (#5311) * [ASM][gfx1250] mla v4 prefill: update sparse_pfl kernel binary Replaces the mla_a8w8_qh128_1tg_32mx4_32nx1_sparse_pfl.co binary added in e566d68 with a newer build. Validated end-to-end on DeepSeek-V4-Pro (gfx1250, tp4, expert parallel, dp attention, kv_cache_dtype=fp8, index_cache_dtype=fp8): lm_eval gsm8k, full 1319 questions, 3-shot flexible-extract exact_match 0.9507 +/- 0.006 strict-match exact_match 0.9500 +/- 0.006 No memory access faults, page-not-present or HSA_STATUS errors in the server log across the full run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Revert the two sparse_pfl kernel binary updates This reverts 12f16de ("[ASM][gfx1250] mla v4 prefill: update sparse_pfl kernel binary") and e566d68 ("[ASM][gfx1250] mla v4 prefill: rebuild the sparse_pfl kernel with the head schedule and the PTL prefetch hazard fixed"), restoring hsa/gfx1250/mla_v4/mla_a8w8_qh128_1tg_32mx4_32nx1_sparse_pfl.co to 462a5ee0 -- the build shipped by #4926. No other file is touched; both reverted commits changed only that binary. The kernel work is being resubmitted against main directly rather than landing here first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * [Perf][gfx1250] Use tuned paths in combo benchmarks (#5323) * perf(gfx1250): compare MegaMoE modes in one run * perf(gfx1250): honor tuned backend in a16w16 bench * perf(gfx1250): add Mori EP inputs and telemetry * perf(gfx1250): add 16K Mega MoE coverage --------- Co-authored-by: JiaoliangYu <jiaolyu@amd.com> * style: satisfy black and ruff checks * test: keep MLA benchmark entrypoints out of pytest * test: use architecture fp8 dtype for blockscale gemm * test: unify FlyDSL grouped GEMM coverage * test: unify architecture-specific Mega MoE runners * test: preserve grouped GEMM suites behind arch gates * bench: default combo output to readable tables * test: place unified FlyDSL MoE coverage under flydsl tests * style: remove stale shebangs from merged MoE test * test: isolate FlyDSL MoE suites by runtime arch * test: split FlyDSL MoE and grouped GEMM suites * ci: leave test sharding config unchanged * test: move grouped GEMM suite under FlyDSL tests * refactor: move SMI monitor into aiter package * test: restore readable benchmark summaries * style: satisfy Black after JSON helper removal * revert: restore JSON benchmark summaries * perf: preserve profiler kernel names * perf: avoid clearing allocator cache per iteration * fix: address microbenchmark review feedback * refactor: split benchmark support helpers Signed-off-by: JiaoliangYu <jiaolyu@amd.com> * fix: serialize benchmark dtypes as strings * style: format benchmark reporting helper Signed-off-by: JiaoliangYu <jiaolyu@amd.com> --------- Signed-off-by: Double Young <yang.yang2@amd.com> Signed-off-by: JiaoliangYu <jiaolyu@amd.com> Co-authored-by: Yu <jiaolyu@amd.com> Co-authored-by: demonsan <39889339+demonsan@users.noreply.github.com> Co-authored-by: binglue.lyu <blyu@heliosp-1b114-d07-2.mnb.dcgpu> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: blyu <blyu@amd.com> Co-authored-by: Double Young <yang.yang2@amd.com> Co-authored-by: yanboshao <yashao@amd.com> Co-authored-by: yadaish <yadai@amd.com> Co-authored-by: jli-melchior <Jun.Li@amd.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: la <junchen2@amd.com> Co-authored-by: junxiaguo <JunXia.Guo@amd.com> Co-authored-by: Xin Huang <Xin.Huang@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
mori_epmeasured only the bf16 dispatch wire, which is not the leg DSv4serves on: the receiver hands the payload straight to the expert GEMM as its A
operand, and that GEMM is a4w4 (
MEGA_WIRE=fp4,AITER_FORCE_A8W4=0in theEP4 serve script). So the tier that production runs was not being measured.
bf16,fp4by default, one child per wire —bench_ep.pyreads$DISPonce at import and builds the transport for that dtype, so thetiers cannot share a process
$DISPstill overrides, now comma-separated (DISP=bf16,fp8,fp4), and isforwarded unvalidated: mori owns the value set
UNCHECKED— mori forces its own comparison off onfp4 ("fp4 combine is too lossy to compare"), so
CHECK=1is silentlydropped there and a passing fp4 row is unchecked, not verified
Measured on b45-1, 4x gfx1250, hip/eager (us):
combine is identical on both wires (489 vs 491 us at 16384), as it must be —
it stays bf16, which is also what makes this a clean single-variable A/B.
Two things the numbers show: combine is now the larger leg (180 vs 489 us at
16384), and fp4 does not saturate — 1618 GB/s on bf16 vs 1096 GB/s on fp4, so
a quarter the bytes buys only 2.65x. Small tiers gain least (1.37x at ct=64).
Full 12-op sweep re-run after this change: 11 rc=0,
a16w16rc=1 on the fourknown M=65536 rows. No
us=0/inf.