[rpoornac]

Summary

• Power-mode tuning (--select power, from [HIP] [CK] [JIT] [Tuner] Rank accuracy-qualified kernels by package energy #6970) ranks candidates by us × package watts. The power sample launched each candidate from Python, so a kernel whose launch takes longer than its GPU time left the GPU idle between launches and read low. [HIP] [CK] [JIT] [Tuner] Rank accuracy-qualified kernels by package energy #6970's launch-gap correction recovers most of that for torch.mm, but split-K FlyDSL GEMMs still read 12-28% under their true power, enough to pick the wrong winner on close shapes, and kernels under ~5 µs were left unmeasured.
• package_power.measure_kernel now captures the candidate's call into a CUDA graph holding about 1 ms of GPU work and replays it for the sample window, so the sensor sees the kernel back to back with no host gaps. A call that cannot be captured (host sync, CPU-side work) is logged once per reason and sampled with the existing eager loop and correction. AITER_POWER_GRAPH=0 forces the eager path. Without a timed us, one warm call is timed to size the graph.
• The capture helper is shared in aiter/utility/cuda_graph.py; IQ2R's timer uses it, and IQ2R now hands its call list to the power sample like every other tuner. No tuner decides how power is measured.
• Latency mode (the default) never reaches this code.

PR Self-Review

Reviewed revision

f34fb38

Behavior and intent

Power-mode tuning only. measure_kernel (the single power-measurement function from #6970) replays a CUDA graph of the timed call instead of launching it eagerly from Python, because eager launches leave the GPU idle whenever the host launch path is longer than the kernel, which under-reads power for short and split-K kernels and can flip the energy ranking. Eager sampling with the #6970 launch-gap correction remains the fallback when capture fails, and AITER_POWER_GRAPH=0 forces it. Latency tuning and runtime dispatch are unchanged.

Validation and unverified scope

Hardware: 8× MI355X (gfx950), ROCm 7.2.4, power cap 1400 W, clocks auto (SCLK 500-2400 MHz), GPUs otherwise idle.

• Graph vs eager on MI355X: FlyDSL split-K 4 at 256×4096×4096 reads 936 W by graph replay vs 619 W eager-corrected; split-K 2 980 vs 988; torch.mm 64×7168×2048 779 vs 639. Graph readings match an independent CUDA-graph measurement (892 / 987 / 768 W).
• Unit: python -m unittest op_tests.tuning_tests.test_fast_tuning — 59 tests pass, including real-sensor tests: a launch-bound GEMM (~100 µs host work per 22 µs kernel) reads 881 W by graph replay vs 850 W back to back and 594 W eager; a failed capture leaves the device usable.
• Other tuning suites pass: test_mp_tuner_logic, test_compare_logic, test_config_shape_collision, test_csv_validation, test_fhmoe_tuner, test_a6w6_tuning, test_mixed_mxfp_tuning. test_tuner_infra has 2 failures (a_preshuffle blank-cell) that also occur on main.
• End to end, --select power tunes on GPUs 0-7 with the shipped low/mid/high shapes: a16w16 (all backends, 4 shapes), gradlib hipBLASLt (4), a8w8 (3), a8w8 blockscale (3), CK MoE ASM FP8 (3), MXFP4 staged (3) and fast scan (2), IQ2R (3). All exit 0 with no capture fallback; every row is stamped with finite watts; every pick is the minimum-energy measured candidate. a16w16 16×128×128 now measures all 82 candidates (39 of 40 FlyDSL candidates were unmeasured with eager sampling). MXFP4 32 tokens reads 797-805 W vs ~540 W eager.
• Known risk, not observed: a kernel that faults only under graph replay would kill its mp_tuner worker, and with --shape_grouped that drops every candidate of that shape (the shape comes out untuned). It did not happen on the eight tuner paths above (0 capture fallbacks, 0 crashes); AITER_POWER_GRAPH=0 avoids it, and automatic eager retry is left for a follow-up if it shows up in practice.
• Unverified: gfx942 and gfx1250 not run; grouped MoE (gfx1250 only) and K5 (no shipped shapes) covered by unit tests only; which sampling path each candidate used is logged, not recorded as a CSV column; the power score is still an isolated replay, so a winner can change in a model or at another power cap.

PR-specific question

No kernel dispatch path changes. Which kernel runs is still chosen by the tuned CSV for the arch/dtype/shape keys. This changes only how --select power measures a candidate during tuning; runtime never captures or replays graphs. Graph replay applies only inside measure_kernel, which the latency policy never calls, and an uncapturable call falls back to the previous eager measurement.

Changed op tests

op_tests/tuning_tests/test_fast_tuning.py — the only changed op test. New cases, none redundant with existing ones:

• test_power_sample_replays_a_graph_and_falls_back_to_eager: graph path used when capture succeeds (window replays the graph and frees it), eager path with the correction when capture fails, and AITER_POWER_GRAPH=0 skips capture.
• test_power_sample_times_the_call_when_no_us_is_given: without us, one timed call sizes the graph and drives the fallback correction.
• test_graph_holds_about_a_millisecond_for_a_timed_call (GPU): the captured graph holds ~1 ms of work.
• test_failed_graph_capture_returns_none (GPU): a host-syncing call is rejected, logged, and the device stays usable.
• test_graph_replay_reads_a_launch_bound_kernel_at_full_power (real sensor): host work between launches does not dilute the graph sample.

Test plan

• python -m unittest op_tests.tuning_tests.test_fast_tuning (59 tests, MI355X)
• --select power tunes on 8 tuner paths, low/mid/high shapes, MI355X (see Validation above)
• gfx942 / gfx1250 power tunes

[github-actions]

🏷️ CI Guide

Runs automatically on every PR:

• ✅ Pre-checks (submodule verification, code formatting)
• ✅ Aiter op tests (gfx942 + gfx950)
• ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label	Tests
ci:gfx1250-ffm-triton	Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x	Run an additional Triton test job on MI300X in PRs (added automatically when gfx942 configs change); main branch always runs both MI35X and MI300X
ci:triton-355	Run the full Triton test suite on MI35X, not only the tests the change affects
ci:windows	Windows JIT op smoke test on the gfx1201 ROCm runner
ci:multi-gpu	Aiter multi-GPU tests on the 8-GPU runner
ci:sglang	SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom	ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full	ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm	vLLM benchmark: DeepSeek-V4-Pro, MiniMax-M3-MXFP4
ci:all	All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 7480 --add-label <label>

One backend per PR:
A PR changes one kernel backend: [Triton/Gluon] (Triton and Gluon count as one), [HIP], [ASM], [CK], [OPUS] or [FlyDSL]. If the title ends up with two backend tags, split the PR -- as stacked pull requests when one part cannot merge without the other.

PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to stop the title rewrites; labels stay in sync either way.

[aiter-bot]

READY FOR HUMAN REVIEW

PR Self-Review

Revision f34fb38b7367 · mode shadow

Author self-review accepted. Readiness report:

Author answers

• PASS — current-SHA answers are present and structurally specific

Show the author's self-review answers

Behavior and intent

Power-mode tuning only. `measure_kernel` (the single power-measurement function from #6970) replays a CUDA graph of the timed call instead of launching it eagerly from Python, because eager launches leave the GPU idle whenever the host launch path is longer than the kernel, which under-reads power for short and split-K kernels and can flip the energy ranking. Eager sampling with the #6970 launch-gap correction remains the fallback when capture fails, and `AITER_POWER_GRAPH=0` forces it. Latency tuning and runtime dispatch are unchanged.

Validation and unverified scope

Hardware: 8× MI355X (gfx950), ROCm 7.2.4, power cap 1400 W, clocks auto (SCLK 500-2400 MHz), GPUs otherwise idle.
- Graph vs eager on MI355X: FlyDSL split-K 4 at 256×4096×4096 reads 936 W by graph replay vs 619 W eager-corrected; split-K 2 980 vs 988; `torch.mm` 64×7168×2048 779 vs 639. Graph readings match an independent CUDA-graph measurement (892 / 987 / 768 W).
- Unit: `python -m unittest op_tests.tuning_tests.test_fast_tuning` — 59 tests pass, including real-sensor tests: a launch-bound GEMM (~100 µs host work per 22 µs kernel) reads 881 W by graph replay vs 850 W back to back and 594 W eager; a failed capture leaves the device usable.
- Other tuning suites pass: `test_mp_tuner_logic`, `test_compare_logic`, `test_config_shape_collision`, `test_csv_validation`, `test_fhmoe_tuner`, `test_a6w6_tuning`, `test_mixed_mxfp_tuning`. `test_tuner_infra` has 2 failures (`a_preshuffle` blank-cell) that also occur on main.
- End to end, `--select power` tunes on GPUs 0-7 with the shipped low/mid/high shapes: a16w16 (all backends, 4 shapes), gradlib hipBLASLt (4), a8w8 (3), a8w8 blockscale (3), CK MoE ASM FP8 (3), MXFP4 staged (3) and fast scan (2), IQ2R (3). All exit 0 with no capture fallback; every row is stamped with finite watts; every pick is the minimum-energy measured candidate. a16w16 16×128×128 now measures all 82 candidates (39 of 40 FlyDSL candidates were unmeasured with eager sampling). MXFP4 32 tokens reads 797-805 W vs ~540 W eager.
- Known risk, not observed: a kernel that faults only under graph replay would kill its `mp_tuner` worker, and with `--shape_grouped` that drops every candidate of that shape (the shape comes out untuned). It did not happen on the eight tuner paths above (0 capture fallbacks, 0 crashes); `AITER_POWER_GRAPH=0` avoids it, and automatic eager retry is left for a follow-up if it shows up in practice.
- Unverified: gfx942 and gfx1250 not run; grouped MoE (gfx1250 only) and K5 (no shipped shapes) covered by unit tests only; which sampling path each candidate used is logged, not recorded as a CSV column; the power score is still an isolated replay, so a winner can change in a model or at another power cap.

PR-specific question

No kernel dispatch path changes. Which kernel runs is still chosen by the tuned CSV for the arch/dtype/shape keys. This changes only how `--select power` measures a candidate during tuning; runtime never captures or replays graphs. Graph replay applies only inside `measure_kernel`, which the latency policy never calls, and an uncapturable call falls back to the previous eager measurement.

Changed op tests

`op_tests/tuning_tests/test_fast_tuning.py` — the only changed op test. New cases, none redundant with existing ones:
- `test_power_sample_replays_a_graph_and_falls_back_to_eager`: graph path used when capture succeeds (window replays the graph and frees it), eager path with the correction when capture fails, and `AITER_POWER_GRAPH=0` skips capture.
- `test_power_sample_times_the_call_when_no_us_is_given`: without `us`, one timed call sizes the graph and drives the fallback correction.
- `test_graph_holds_about_a_millisecond_for_a_timed_call` (GPU): the captured graph holds ~1 ms of work.
- `test_failed_graph_capture_returns_none` (GPU): a host-syncing call is rejected, logged, and the device stays usable.
- `test_graph_replay_reads_a_launch_bound_kernel_at_full_power` (real sensor): host work between launches does not dilute the graph sample.

PR state

• PASS — pull request is marked ready for review

Copilot findings

• PASS — no active Copilot finding lacks a substantive author reply
• ADVISORY — Copilot did not review the current head SHA

Code and op-test hygiene

• PASS — no committed artifact or conflict marker detected

Required CI

• PASS — Black: passed
• PASS — Ruff: passed
• PASS — Repository dependency: passed
• PASS — Aiter Test Gate: passed

_{Shadow mode reports what would block but keeps the status successful. Set AITER_PR_SELF_REVIEW_MODE=enforce only after reviewing real false positives.}