Skip to content

[Perf][gfx1250] Reuse process-wide amdsmi state - #5302

Merged
JiaoliangYu merged 1 commit into
ROCm:gfx1250/microbenchfrom
JiaoliangYu:perf/gfx1250-cache-amdsmi-state
Sep 7, 2026
Merged

[Perf][gfx1250] Reuse process-wide amdsmi state#5302
JiaoliangYu merged 1 commit into
ROCm:gfx1250/microbenchfrom
JiaoliangYu:perf/gfx1250-cache-amdsmi-state

Conversation

@JiaoliangYu

@JiaoliangYu JiaoliangYu commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Summary

  • initialize amdsmi lazily once per Python process and shut it down once via atexit
  • enumerate and cache the BDF-to-handle map once per process
  • cache HIP-device-to-BDF and HIP-device-to-handle lookups
  • keep GpuMonitor.start() / stop() scoped to starting and joining the sampling thread
  • preserve the existing sampling window and metric semantics

This removes repeated amdsmi_init, processor enumeration, PCI BDF lookup, and amdsmi_shut_down work from every benchmark case.

Validation

  • python3 -m py_compile op_tests/smi_monitor.py
  • git diff --check
  • ruff check op_tests/smi_monitor.py
  • fake-amdsmi smoke: two sequential monitors produced exactly one init, one processor enumeration, one BDF mapping, and one process shutdown

Runtime GPU validation is left to the submitter.

@JiaoliangYu
JiaoliangYu merged commit 4c225d6 into ROCm:gfx1250/microbench Sep 7, 2026
3 checks passed
JiaoliangYu added a commit that referenced this pull request Sep 13, 2026
@JiaoliangYu Yu
@demonsan binglue.lyu @cursoragent blyu @dbyoung18 @yanboshao @yadaish @jli-melchior @claude @junhaha666 @junxiaguo @gyohuangxin
[Triton/Gluon] [OPUS] [FlyDSL] Gfx1250/microbench (#5391)
6566fb4 
* Gfx1250/microbench (#5241)

* perf(gfx1250): add the combined gfx1250 microbench (#5076)

Two files, both new, on top of main:

  op_tests/bench_gfx1250_combo.py                        combined bench
  op_tests/triton_tests/attention/test_mla_v4_triton.py  MLA v4 triton reference

The bench imports the top-level @benchmark sweep fns from the op_tests
siblings and runs each over its own shape axes, printing only the per-op
summary tables -- all the JIT/ROCTracer/import noise is silenced at the fd
level. Two suites: --perf for hardware-oriented single-op numbers, --dsv4
for the DeepSeek-V4 shapes. Token sweeps come from AITER_BENCH_TOKENS; ops
whose axis means something else, or whose usable range is fixed, pin their
sweep in the source and say why.

Everything else the branch used to carry has reached main on its own since:
test_pa_sparse_prefill.py (#4926), test_mega_moe_gfx1250.py (#5052),
test_flydsl_qk_norm_rope_quant.py and the mla_v4 hsa artifacts. Only these
two files were ever unique to it, so this is what is left to upstream.

Notes worth carrying, all measured on gfx1250 / 20260827-28:

a16w16 does not pre-check the >4 GiB operand limit. That guard belongs to
one fallback path -- opus_dispatch_a16w16_gfx1250 searches the tuned table
first and only reaches check_shape_4g on a miss, en route to the split-K
kid whose launcher builds the 32-bit gmem descriptors. A tuned 4wave_wl_co
winner never gets there. Predicting it in Python skipped shapes that tuning
had already made runnable, so the kernel is left to raise and the exception
is recorded as a row. It also checks the error ratio the UT returns against
_A16W16_MAX_ERR: all four M=65536 shapes come back err=0.96-0.99 while
every other row is 0 or ~1e-5, and nothing in the UT raises or warns, so
those used to print as data.

a8w8_blockscale sweeps M from 1024. Below that, get_CKGEMM_config's
M -> get_padded_m -> nextPow2 retry lands on #4773's M=16/M=64 gluon rows,
and the UT's extra "ck strided x_scale" check (line 120, added by #4406 and
gated on ck_preshuffle alone) hands triton a stride != 1 specialization
that fails to compile in make_llir. The mxfp8_128 path declares its layout
with is_x_scale_transposed=True and never reads the stride, so that check
tests nothing there. Fixing the gate is upstream's call; starting at 1024
keeps this bench clear of it. Verified 36/36, err=0, 2207-7003 TFLOPS.

mla_v4_prefill is pinned to n=1024 and mla_v4_prefill_fp8 drops
nnz_prefix=8192; both are kernel/verify faults, measured and documented at
the pins.

Co-authored-by: Yu <jiaolyu@amd.com>

* perf(gfx1250): let AITER_BENCH_TOKENS reach every op (#5082)

Three token constants were hardcoded and never consulted the variable --
_INVERSE_ROPE_TOKENS, _MLA_DECODE_TOKENS, _MEGA_MOE_TOKENS -- with comments
saying so ("Pinned, not env-driven", "deliberately not consulted"). The
reasoning was that a global token count means the wrong thing for those
ops, which is true of the default but not of an explicit request: if a
caller sets the variable, that is their decision to make, including asking
for a shape the op is known to fail on.

All nine token constants now go through _tokens(): unset, each op runs its
own default and says at its constant why that default is not the shared
list; set, the variable wins everywhere and the file does not argue with
it. Behaviour with the variable unset is unchanged.

The module docstring said "Two ops ignore it and pin their sweep in the
source" -- no longer true, and it only listed two of the five ops whose
default differs. It now lists all five with the reason for each.

Should have been part of #5076; amended in after that PR was pushed.

Co-authored-by: Yu <jiaolyu@amd.com>

* perf(gfx1250): sweep the fp4 dispatch wire beside bf16 (#5147)

mori_ep measured only the bf16 wire, which is not the leg DSv4 serves on.
The receiver hands the dispatched payload straight to the expert GEMM as its
A operand, and that GEMM is a4w4 -- ATOM's serve script pins MEGA_WIRE=fp4
with AITER_FORCE_A8W4=0 -- so bf16 alone measures a path production does not
take, and misses the wire whose payload is a quarter as wide.

Sweeps bf16 and fp4 by default, one child each: bench_ep.py reads $DISP once
at import and builds the transport for that dtype, so the tiers cannot share
a process. $DISP still overrides, now comma-separated, and is forwarded
unvalidated -- mori owns the value set, this file does not restate it.

fp4 rows are labelled UNCHECKED. mori forces its own comparison off on fp4
("fp4 combine is too lossy to compare"), so CHECK=1 is silently dropped
there; without the label a passing fp4 row reads as verified next to a bf16
row that actually was.

Co-authored-by: Yu <jiaolyu@amd.com>

* perf(gfx1250): follow the sparse-prefill UT through #5084 (#5170)

#5084 rewrote test_pa_sparse_prefill.py and mla_v4_prefill stopped running
against it: --nnz-prefix is gone, so the child aborted with "unrecognized
arguments" before reaching a kernel.

- drop --nnz-prefix. The nnz axis no longer exists: the CSR is generated from
  --mode under --seed (sparse draws a random nnz per row, dense fills every
  row), so nnz is an outcome, not an input
- delete the mla_v4_prefill_fp8 op. Its entire content was that nnz sweep, and
  it has no equivalent under the new UT. mla_v4_prefill already covers fp8 --
  and now covers the backends too, since #5084 compares opus/asm/triton
- _space_table: decide a data row by counting numeric fields instead of
  testing the first one. The new table leads with prec/mode (bf16, dense), so
  the first-field test dropped the whole table and the op reported "no result
  rows" while the UT itself had exited 0

Raise _MLA_PREFILL_TOKENS to 1024..16384. Re-measured on the new UT (b45-2,
one process per tier): 1024 through 16384 all clean, where 4096/8192/16384
faulted on the old one. 16384 is the DSv4 prefill chunk and the reason this
op exists, so this is the coverage the pin was costing us.

65536 stays out. It faults with a memory access at 0x7f2ddbec0000 and writes
an 89 GB coredump doing it -- a third of the free disk on a shared host -- and
it is past the chunk size the model prefills anyway.

Both the tier sweep and this op run --no-verify, hardcoded at the call site,
and that is now load-bearing rather than a speed choice: with verification on,
even n=1024 dies at the first case (fp8/dense, fault at 0x43000), so the
reference or the comparison is what breaks, not the kernel under test. These
are therefore timings from an unverified kernel. The comment says so, because
a16w16's M=65536 rows looked exactly this healthy until _A16W16_MAX_ERR was
added and caught err=0.99.

Co-authored-by: Yu <jiaolyu@amd.com>

* perf(gfx1250): stop every child from shelling out to rocminfo (#5180)

chip_info runs rocminfo twice per process -- once for the arch, once for the
CU count -- and rocminfo takes a per-device rocm_smi mutex on its way in. One
process is fine, and the nine single-GPU ops never noticed. A torchrun op
starts four ranks at once, and they contend for that mutex.

Both multi-GPU ops have now lost a run to it. On b45-1 a rank lost the mutex
and aborted:

    init_mutex /rocm_smi_renderD128: unlock timed lock, ret: 1
    terminate called after throwing an instance of 'std::runtime_error'
      what():  Allgather operation failed

The allgather is the symptom, not the cause: the rank died first and took the
collective with it. On b45-2 the other shape of the same problem -- four
rocminfo processes sat in the mutex for minutes, one wedged in D state, the op
never produced a line, and even docker stop hung waiting for the driver to
let go.

Detect once here, where the call is serial, and hand the answer down:

    GPU_ARCHS -> get_gfx_list()          skips _detect_native()
    CU_NUM    -> get_cu_num_custom_op()  skips its own rocminfo

Both are read from the environment before either shells out, so a child that
inherits them runs no rocminfo at all. Set on os.environ in main() for the
children that inherit our environment, and setdefault() in _run_child for the
ones handed an explicit env -- setdefault throughout, so an exported value
from the caller still wins.

The value is what this process detected on this machine, not a hardcoded
gfx1250, so it is the same answer the child would have computed. Verified in
the child's /proc/<pid>/environ: GPU_ARCHS=gfx1250, AITER_GPU_ARCHS=gfx1250,
CU_NUM=256, matching get_gfx()/get_cu_num() here.

mega_moe on b45-1 after the change: rc=0, all 36 configs, zero mutex warnings,
and zero rocminfo processes observed while it ran. Before it, the same op on
the same host was rc=1 with 6 mutex warnings.

Co-authored-by: Yu <jiaolyu@amd.com>

* test_common：data generate

---------

Co-authored-by: Yu <jiaolyu@amd.com>

* Blyu/opus gemm benchmark update (#5250)

* [test] Add bench controls to test_opus_a16w16_gemm

Add configurable warmup/iteration counts, data-init modes
(zero/constant/uniform/norm) with a seed, and rotating-tensor control
to the opus a16w16 GEMM test, and report both TFLOPs and bandwidth
(TB/s) explicitly across all three timing paths.

The opus graph sweep table now also surfaces the tuned CSV's splitK and
kernelName (kid) per shape, so the algorithmic winner is visible and
tracks CSV updates. test_a16w16 keeps a backward-compatible signature so
the gfx1250 combo bench is unaffected.

Co-authored-by: Cursor <cursoragent@cursor.com>

* [test] Fix ruff C408 in test_opus_a16w16_gemm

Rewrite init_kwargs dict() call as a dict literal to satisfy
`ruff check` (C408); black is unchanged.

Co-authored-by: Cursor <cursoragent@cursor.com>

* test_common：data generate

* [test] Adopt shared test_common data-init in a16w16 gemm

Use the cherry-picked aiter.test_common data-init API (make_generator /
add_data_init_args / fill / DATA_DISTS) in test_opus_a16w16_gemm.py instead of
the local DATA_INITS/_make_generator/_make_tensor helpers, so operand init and
--data-init/--seed CLI match every other op test.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: binglue.lyu <blyu@heliosp-1b114-d07-2.mnb.dcgpu>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: JiaoliangYu <jiaolyu@amd.com>
Co-authored-by: blyu <blyu@amd.com>

* [ASM][gfx1250] enhance ASM GEMM microbench (#5251)

* test(gemm): use common data initialization options for gfx1250 asm gemm

Signed-off-by: Double Young <yang.yang2@amd.com>

* test(gemm): expose warmup/iters/graph/rotate CLIs for gfx1250 asm gemm

Signed-off-by: Double Young <yang.yang2@amd.com>

* test(gemm): show algorithmic details for gfx1250 asm gemm

Signed-off-by: Double Young <yang.yang2@amd.com>

---------

Signed-off-by: Double Young <yang.yang2@amd.com>

* test mega moe data init and warmup (#5248)

* Dev/ubench gemm (#5252)

* update

* update

* support a8w8 ubench

* revert ck module

* apply aiter common interface

---------

Co-authored-by: yadaish <yadai@amd.com>

* [Bench] Add data-init options to qk_norm_rope_quant test (#5253)

Add --seed, --init {normal,uniform,zero,constant}, --init-scale,
and --init-val CLI flags so callers can control how input tensors
are initialised, enabling reproducible perf sweeps across data
distributions.

Co-authored-by: jli-melchior <Jun.Li@amd.com>
Co-authored-by: Claude <noreply@anthropic.com>

* [HIP] Add data-init to test_mhc (#5245)

Wire test_common.fill, --data-init, and --seed so MHC inputs are reproducible.

Co-authored-by: Cursor <cursoragent@cursor.com>

* perf(gfx1250): add ubench init controls and SMI monitor (#5257)

Co-authored-by: JiaoliangYu <jiaolyu@amd.com>

* [Perf][gfx1250] Complete DSv4 attention data-init support (#5258)

* perf(gfx1250): complete DSv4 data init controls

* perf(gfx1250): report per-case SMI telemetry

* perf(gfx1250): monitor Mega MoE per rank

---------

Co-authored-by: JiaoliangYu <jiaolyu@amd.com>

* perf(gfx1250): add M512 A8W8 DSv4 case (#5284)

Co-authored-by: JiaoliangYu <jiaolyu@amd.com>

* [Perf][gfx1250] Emit ubench summaries as JSON (#5288)

* docs(gfx1250): document bundled amdsmi binding

* perf(gfx1250): emit benchmark summaries as JSON

---------

Co-authored-by: JiaoliangYu <jiaolyu@amd.com>

* fix(gfx1250): scope bundled amdsmi import (#5291)

Co-authored-by: JiaoliangYu <jiaolyu@amd.com>

* perf(gfx1250): reuse process-wide amdsmi state (#5302)

Co-authored-by: JiaoliangYu <jiaolyu@amd.com>

* [Perf][gfx1250] Label SMI replays per benchmark call (#5304)

* perf(gfx1250): label SMI replays per benchmark call

* perf(gfx1250): cover saturated DSv4 MLA decode KV

* perf(gfx1250): sweep DSv4 score-QK decode batches

* perf(gfx1250): cover TP4 inverse RoPE shape

* perf(gfx1250): add M256 blockscale coverage

* perf(gfx1250): add 64K Mega MoE coverage

---------

Co-authored-by: JiaoliangYu <jiaolyu@amd.com>

* [ASM][gfx1250] mla v4 prefill: rebuild the sparse_pfl kernel with the head schedule and the PTL prefetch hazard fixed

Replaces mla_a8w8_qh128_1tg_32mx4_32nx1_sparse_pfl.co (462a5ee0, as shipped by
#4926) with a build carrying the 09-07/09-08 head work plus one correctness fix.

The correctness fix. The page-table prefetch built its row offset from a VGPR
written by v_writelane_b32 at the end of KV_page_prefetch_build; the consumers
are the two v_lshlrev at the top of KV_page_prefetch_issue, on the far side of
the _s_p_tiles branch with zero VALU in between:

    v_writelane_b32 v4, s60, 0
    s_cmp_eq_u32    s103, 0
    s_cbranch_scc0  label_1028
    v_lshlrev_b32   v2, 9, v4      <- 2 SALU after the write

SCHED_MODE 2 has no VALU->VALU interlock, so the shift could read the stale
register (Q addr-gen leftovers). `stale << 9` off a bounds-free 64-bit base is a
wild address, observed under batched prefill as

    Memory access fault by GPU node-3 ... Reason: Page not present

No numeric test can catch this: a prefetch has no destination register, so the
bad address never reaches a result. Fixed with one s_wait_alu placed before the
branch, covering both the fall-through (extend) and taken (prefix) arms.
+4 bytes; the core loop is byte-identical.

Also in this build, relative to 462a5ee0:
  * head schedule: HEAD_INDPTR_EARLY, HEAD_INDEX_HOIST, HEAD_TDM_SLOT_AWARE
  * HEAD_KV_PTL_* page-table prefetch over tile 0's KV rows
  * the packed-row scale tail mask (448:511 pad), which 462a5ee0 lacks -- that
    pad reads as NaN once the host stops zero-filling it

Source: junxiaguo/my_agent_skills @ 495b486 (0805_base/32mx4_32nx1.sp3).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* support print overlap ratio (#5311)

* [ASM][gfx1250] mla v4 prefill: update sparse_pfl kernel binary

Replaces the mla_a8w8_qh128_1tg_32mx4_32nx1_sparse_pfl.co binary added in
e566d68 with a newer build.

Validated end-to-end on DeepSeek-V4-Pro (gfx1250, tp4, expert parallel,
dp attention, kv_cache_dtype=fp8, index_cache_dtype=fp8):

  lm_eval gsm8k, full 1319 questions, 3-shot
    flexible-extract  exact_match  0.9507 +/- 0.006
    strict-match      exact_match  0.9500 +/- 0.006

  No memory access faults, page-not-present or HSA_STATUS errors in the
  server log across the full run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Revert the two sparse_pfl kernel binary updates

This reverts 12f16de ("[ASM][gfx1250] mla v4 prefill: update sparse_pfl
kernel binary") and e566d68 ("[ASM][gfx1250] mla v4 prefill: rebuild the
sparse_pfl kernel with the head schedule and the PTL prefetch hazard
fixed"), restoring
hsa/gfx1250/mla_v4/mla_a8w8_qh128_1tg_32mx4_32nx1_sparse_pfl.co to
462a5ee0 -- the build shipped by #4926.

No other file is touched; both reverted commits changed only that binary.

The kernel work is being resubmitted against main directly rather than
landing here first.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* [Perf][gfx1250] Use tuned paths in combo benchmarks (#5323)

* perf(gfx1250): compare MegaMoE modes in one run

* perf(gfx1250): honor tuned backend in a16w16 bench

* perf(gfx1250): add Mori EP inputs and telemetry

* perf(gfx1250): add 16K Mega MoE coverage

---------

Co-authored-by: JiaoliangYu <jiaolyu@amd.com>

* style: satisfy black and ruff checks

* test: keep MLA benchmark entrypoints out of pytest

* test: use architecture fp8 dtype for blockscale gemm

* test: unify FlyDSL grouped GEMM coverage

* test: unify architecture-specific Mega MoE runners

* test: preserve grouped GEMM suites behind arch gates

* bench: default combo output to readable tables

* test: place unified FlyDSL MoE coverage under flydsl tests

* style: remove stale shebangs from merged MoE test

* test: isolate FlyDSL MoE suites by runtime arch

* test: split FlyDSL MoE and grouped GEMM suites

* ci: leave test sharding config unchanged

* test: move grouped GEMM suite under FlyDSL tests

* refactor: move SMI monitor into aiter package

* test: restore readable benchmark summaries

* style: satisfy Black after JSON helper removal

* revert: restore JSON benchmark summaries

* perf: preserve profiler kernel names

* perf: avoid clearing allocator cache per iteration

* fix: address microbenchmark review feedback

* refactor: split benchmark support helpers

Signed-off-by: JiaoliangYu <jiaolyu@amd.com>

* fix: serialize benchmark dtypes as strings

* style: format benchmark reporting helper

Signed-off-by: JiaoliangYu <jiaolyu@amd.com>

---------

Signed-off-by: Double Young <yang.yang2@amd.com>
Signed-off-by: JiaoliangYu <jiaolyu@amd.com>
Co-authored-by: Yu <jiaolyu@amd.com>
Co-authored-by: demonsan <39889339+demonsan@users.noreply.github.com>
Co-authored-by: binglue.lyu <blyu@heliosp-1b114-d07-2.mnb.dcgpu>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: blyu <blyu@amd.com>
Co-authored-by: Double Young <yang.yang2@amd.com>
Co-authored-by: yanboshao <yashao@amd.com>
Co-authored-by: yadaish <yadai@amd.com>
Co-authored-by: jli-melchior <Jun.Li@amd.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: la <junchen2@amd.com>
Co-authored-by: junxiaguo <JunXia.Guo@amd.com>
Co-authored-by: Xin Huang <Xin.Huang@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@JiaoliangYu