[rpoornac]

Summary

• Add --select power (experimental, opt-in) so tuners that support it keep the kernel with the lowest package energy (us × PPT watts) among kernels that pass errRatio. Latency remains the default, and a latency tune writes the same CSV schema as before.
• Every latency-or-power decision goes through one SelectionPolicy (aiter/utility/selection_policy.py), built from --select. Every power read goes through one function, package_power.measure_kernel. Tuners do not branch on the select mode and do not read or write AITER_SELECT. AITER_SELECT=power only makes runtime dispatch overlay tables from aiter/configs/power_configs/.
• Power is the mean power1_input (PPT) of the HIP device under test while the timed kernel launches back to back for AITER_POWER_WINDOW_S (default 0.4 s). When host launch gaps leave the GPU partly idle, the reading is scaled to active power from the launch count and the timed us. Windows under 20% busy stay unmeasured.
• Power rows record select=power, perf_level, power_cap_w, and sclk_max_mhz, and a power tune of a shipped table is written to aiter/configs/power_configs/. Only rows whose watts were measured are written; unmeasured shapes stay untuned for the next power tune, and the AITER_SELECT=power overlay uses only rows stamped select=power.
• Latency tunes keep main's output schemas, including the MoE -o2 profile and profile_fmoe.csv.

PR Self-Review

Reviewed revision

bbc0ae5

Behavior and intent

Tuners rank candidates by latency only, so the tuned tables always pick the fastest kernel even when a slightly slower one uses much less package energy. This PR adds an opt-in --select power mode that ranks accuracy-qualified candidates by us × package watts and stores those rows separately (aiter/configs/power_configs/), loaded only with AITER_SELECT=power. Default behavior is unchanged: --select latency keeps the lowest us and writes no power columns. Supported tuners: the shared GEMM tuners (GemmCommonTuner: a16w16 incl. hipBLASLt/asm/FlyDSL/Opus/Triton/torch, a8w8, blockscale, gradlib), the fused-MoE CK two-stage, grouped, and FlyDSL MXFP4 paths, the GDN K5 BV sweep, and IQ2R. Conv3d, a6w6, and the mixed-MXFP tuners (a6w4/a4w6) reject --select power, because their gain guard keeps a kernel only for speed.

Validation and unverified scope

Hardware: 8× MI355X (gfx950), ROCm 7.2.4, power cap 1400 W, clocks auto (SCLK 500-2400 MHz), GPUs otherwise idle.

• Unit: python -m unittest op_tests.tuning_tests.test_fast_tuning — 54 tests pass, including a real-sensor test (8192³ bf16 GEMM must read > 1.5× idle and ≥ 0.85× a back-to-back loop; skips without power1_input).
• Other tuning suites pass: test_mp_tuner_logic, test_compare_logic, test_config_shape_collision, test_fhmoe_tuner, test_a6w6_tuning, test_mixed_mxfp_tuning. test_tuner_infra has 2 failures (a_preshuffle blank-cell) that also occur on main ([FlyDSL] Add gfx1250 grouped MoE A-preshuffle and tuned CSV validation #6709).
• End to end, latency and power tunes on GPUs 0-7, low/mid/high shapes from the shipped untuned CSVs: a16w16 (all backends + hipBLASLt, 4 shapes), gradlib (4), a8w8 (3), a8w8 blockscale (3), CK MoE ASM FP8 (3), MXFP4 staged (3) and fast scan (2), IQ2R (3), and a16w16 with --mp 2. On every shape the latency CSV has no power columns, every power row has finite watts and the conditions, and the power pick is the minimum-energy measured candidate. Examples: a8w8 16384×8192×1024 is 23% slower and 9% lower energy; blockscale 1024×1536×7168 is 24% slower and 16% lower energy; MXFP4 32 tokens is 4% slower and 18% lower energy.
• MoE profile schema (FP8 per_1x128, ASM, 3 shapes): the latency -o2 profile has main's 29 columns with no power columns, and profile_fmoe.csv has none; the power run keeps the energy-ranked profile and stamps all rows.
• Power sample vs CUDA-graph replay (no launch gaps): torch.mm within ~10%; split-K FlyDSL still reads 12-28% low after the launch-gap correction, though the energy ranking on that shape matches the graph-replay ranking.
• Unverified: grouped MoE (gfx1250 only) is covered by unit tests only; K5 has no shipped untuned shapes; gfx942 not run; kernels under ~5 µs are left unmeasured by design; the power score is an isolated replay, so a winner can change in a model, at another power cap, or next to other kernels; end-to-end model perf/W (MLPerf) is not part of this PR. Graph-replay power sampling is planned as a follow-up.

PR-specific question

No kernel dispatch path changes. Which kernel runs is still chosen by the tuned CSV for the arch/dtype/shape keys, as before. A power row can reach dispatch only through AITER_SELECT=power (overlay of aiter/configs/power_configs/) or an explicit AITER_CONFIG_* path. Without AITER_SELECT=power, power_configs/ is ignored. A power row is written only for a candidate that passed the same errRatio accuracy gate as latency mode and has finite measured watts; a batch with no finite watts stays a latency result. Merging a power row and a latency row for one shape, or power rows tuned under different perf level, cap, or SCLK, raises instead of auto-resolving. Tuners that cannot rank by energy reject --select power (SUPPORTS_POWER_SELECT = False).

Changed op tests

op_tests/tuning_tests/test_fast_tuning.py — the only changed op test. Its new cases cover power selection, and none duplicate an existing case:

• Ranking and policy: energy vs latency order, unmeasured rows ranked last, MoE stage-energy pairs, candidate_beats, SelectionPolicy decisions, mp_tuner using the activated policy, a16w16 passing its policy to hipBLASLt.
• Measurement: the timed kernel is replayed without run_perftest options; bounded in-flight launches; launch-gap correction; real-sensor large-GEMM check.
• Output and loading: only measured rows are kept and stamped, power output redirected to power_configs/, latency frames and the latency MoE profile without power columns, power/latency merge conflicts, the power_configs/ overlay (unstamped rows and files without select ignored).
• Build safety: jit.core loaded as in setup.py (with and without duplicate shapes) does not import the aiter package.
• Guard: no tuner file decides the select mode, reads power, or uses AITER_SELECT itself.

Test plan

• python -m unittest op_tests.tuning_tests.test_fast_tuning (54 tests, MI355X)
• Latency and power tunes on 8 tuner paths, low/mid/high shapes, MI355X (see Validation above)
• MLPerf and other end-to-end perf/W runs (out of scope for this PR)

Made with Cursor

[github-actions]

🏷️ CI Guide

Runs automatically on every PR:

• ✅ Pre-checks (submodule verification, code formatting)
• ✅ Aiter op tests (gfx942 + gfx950)
• ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label	Tests
ci:gfx1250-ffm-triton	Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x	Run an additional Triton test job on MI300X in PRs (added automatically when gfx942 configs change); main branch always runs both MI35X and MI300X
ci:triton-355	Run the full Triton test suite on MI35X, not only the tests the change affects
ci:windows	Windows JIT op smoke test on the gfx1201 ROCm runner
ci:multi-gpu	Aiter multi-GPU tests on the 8-GPU runner
ci:sglang	SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom	ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full	ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm	vLLM benchmark: DeepSeek-V4-Pro, MiniMax-M3-MXFP4
ci:all	All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 6970 --add-label <label>

One backend per PR:
A PR changes one kernel backend: [Triton/Gluon] (Triton and Gluon count as one), [HIP], [ASM], [CK], [OPUS] or [FlyDSL]. If the title ends up with two backend tags, split the PR -- as stacked pull requests when one part cannot merge without the other.

PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to stop the title rewrites; labels stay in sync either way.

[yzhou103]

Thanks for putting this together. Power-aware kernel selection is an interesting and potentially valuable direction, especially for sustained workloads where energy efficiency is an important objective.
But I have a design concern about whether the current measurement can provide a stable, workload-independent power score.
The direction is valuable, but the current implementation may not produce a stable, workload-independent power score.

The tuner measures package PPT while replaying each kernel in isolation for approximately 0.4 seconds, under whatever clock, thermal, and power-management policy the firmware currently applies. This may not represent the kernel’s behavior inside a real model:

Small shapes may be dominated by telemetry granularity, launch overhead, or background power.
The replay itself may push the GPU into a sustained power cap.
Candidates are measured sequentially without cooldown or randomized ordering, so one candidate may affect the thermal/frequency state of the next one.
The same shape may behave differently across models. A kernel selected by this tuner could improve perf/W in model A but be slower or less efficient in model B, because neighboring kernels, concurrency, communication, and the GPU’s thermal/power state change the operating point. Since the tuned CSV is primarily keyed by shape and reused across workloads, an isolated replay may select a winner that is not optimal for every model.
There is also no explicit latency or throughput constraint in the current objective:

score = runtime_us × package_watts
A substantially slower kernel can win as long as its measured power decreases proportionally. That may be valid for raw energy minimization, but it may violate production latency or throughput requirements.

Another concern is ownership of the problem. If tuning is performed at maximum frequency/power but the selected kernel performs poorly in production because the GPU reaches a power cap, are we observing a firmware or power-management issue? In that case, selecting a slower, lower-power kernel may hide the underlying problem instead of fixing it. The power cap and frequency policy may need to be addressed by firmware or the platform power manager, and the tuner should expose the issue rather than encode a workaround into the kernel configuration.

Before treating this as a production tuning strategy, could we clarify:

Is the goal to minimize energy subject to a latency/throughput bound, or to minimize raw runtime × watts?
How do we distinguish a genuinely more energy-efficient kernel from one that is simply being throttled differently by firmware?
Should this be validated under fixed clocks and explicit power-cap settings?
Do we have representative end-to-end results showing that the power-selected kernel improves perf/W without violating latency targets?
How do we handle cases where the best kernel differs between model A and model B?
I suggest keeping this feature explicitly experimental unless it adds a latency guard or Pareto-based selection, records the relevant clock/power-cap state, and is validated on representative model workloads. Otherwise, the tuner may mask a firmware power-management problem by selecting a power-friendly but slower operator.

[rpoornac]

Thanks for the design notes. This PR keeps power selection opt-in. Latency is still the default, and the README now says so.

Answers to the questions, as the code stands:

• The objective is raw package energy, runtime × PPT watts, among kernels that already pass errRatio. There is no latency or throughput bound on top of that. A slower kernel can win when its joules are lower.
• The sample is an isolated replay of about 0.4 s on power1_input. It is not a model-level score. Sequential candidates can see different clocks, and the same shape can prefer a different kernel in another model because the CSV key is the shape.
• We have not shown an end-to-end perf/watt win. At an open cap the package watts of different tiles sat together; the watt gap showed up with the clock held fixed. That is why this should not be the production default yet.
• A missing sensor no longer changes the latency order, and a single AITER_POWER_PATH is rejected for a multi-process tune (see the two threads above).

A latency guard or a Pareto pick would be a follow-up. I would rather not add one in this PR until there is a fixed-clock or capped end-to-end run to set the bound against.

[yzhou103]

Thanks, that clarifies that the feature is opt-in, is intended to use per-model tuned CSVs, and does not directly modify PM firmware behavior.

Before treating this as a product-facing feature, I think we still need to clarify the actual motivation and target scenario. What concrete workload or deployment problem is this intended to solve? For example, is there a real model running under a power cap where this selection provides a measurable benefit?

So far, the evidence appears to be based on isolated kernel replay. We do not yet have end-to-end model results showing improved perf/W, lower energy, or acceptable latency impact. Without that validation, it is difficult to determine whether this is ready for product release or should remain an experimental tuning option.

Also, AITER currently merges tuned CSVs by shape-based keys by default. How is model separation guaranteed in the actual config loading and merge path? If configs from different models are merged, model-specific power winners can still collapse into a single row and be applied across workloads.

Although this feature does not directly change PM firmware, it still selects kernels based on behavior observed under the current PM policy. That behavior may depend on power caps, clocks, thermal state, and the surrounding model workload.

[rpoornac]

Thanks. Following up on my earlier note: we have end-to-end workload runs showing perf/W gains at a fixed power cap and at a fixed clock cap. I'd prefer to walk through that data with you offline rather than post it here. Until it is validated more widely, the feature stays experimental and opt-in.

On model separation, you're right that the default load merges tables by shape key. d9cd30c keeps power rows in aiter/configs/power_configs/. They load only with AITER_SELECT=power and replace only the shapes they cover. An explicit AITER_CONFIG_* path loads one model's table with no merge. A merge that finds a power row and a latency row for one shape, or power rows tuned under different conditions, now raises instead of auto-picking.

On power-management dependence: each power row now records the perf level, power cap, and SCLK range it was tuned under, so a winner from one state is not silently compared with one from another.

[yzhou103]

I ran the latest head (94d257f) on an MI355X. The unit tests pass, and the raw PPT read works (about 310–326 W for a tiny GEMM, about 1372–1398 W for an 8k bf16 GEMM at the 1400 W cap). But an a16w16 tune with --select power --shape_grouped --mp 1 (torch and triton backends) writes watts and tflops_per_w as Null for every shape, while the rows are still stamped select=power.

Cause: _replay in mp_tuner.worker calls func(*args, **kwargs), but those kwargs are the run_perftest kwargs (num_warmup, num_iters, ...), so every replay raises TypeError: run_torch_gemm_a16w16() got an unexpected keyword argument 'num_warmup'. record_package_watts swallows the error and returns NaN, and the missing-watts fallback then keeps latency order. As a result, power mode silently produces latency picks on what looks like every mp_tuner-based tuner that passes perftest kwargs.

[yzhou103]

Power measurement on the mp_tuner path

Thanks for fixing the num_warmup TypeError; watts are now recorded. However, the new replay mostly measures idle time, so the watts can't be used for ranking yet.

Problem. replay_timed_call runs the full run_perftest(...) repeatedly inside the 0.4 s power window. Each call includes profiler start/stop, warmup, synchronization and trace parsing, so the GPU is idle most of the time. Same kernel on an MI355X (gfx950):

shape (bf16 mm)    us      back-to-back W   GPU busy   run_perftest replay W   GPU busy
64x7168x2048       13.8    541-638          ~100%      343-388                 ~3%
256x4096x4096      22.5    1067-1242        ~100%      367-553                 ~7%
4096^3             107.6   1340-1397        ~100%      576-873                 ~21%

A real gemm_a16w16_tune.py --select power --mp 1 run wrote 348 / 515 / 335 W for these shapes. These values are idle-dominated, like the replay column, and far below the back-to-back numbers. Shorter kernels are diluted more, so us * watts ends up close to latency order plus noise. The unit tests mock the measurement and don't catch this.

Scope. This affects mp_tuner.worker, i.e. every GemmCommonTuner subclass and gradlib. MoE, gdn_k5 and iq2r measure closures that launch the kernel directly, so they look fine. I checked those by reading the code, not by measuring.

Clarification. By "measure in the timing call" I meant sampling inside run_perftest's replay loop, not calling run_perftest again inside the window. Sorry for the ambiguity. Either fix is fine:

1. Sample in the existing perftest replay hook (the AITER_SMI_MONITOR path). It replays func back to back with the same rotated args or graph as the timer. Note that it reads amdsmi current_socket_power rather than hwmon power1_input; please confirm the two agree, or use one sampler for both.

2. Replay the kernel directly, with the run_perftest options stripped. This is the original _replay without the kwargs bug. I checked on gfx950 that it works and keeps the GPU fully busy:

   _PERFTEST_KEYS = set(inspect.signature(run_perftest).parameters) - {"func", "args", "kwargs"}
   
   def kernel_replay(func, args, kwargs):
       kernel_kwargs = {k: v for k, v in kwargs.items() if k not in _PERFTEST_KEYS}
       return lambda: func(*args, **kernel_kwargs)
   
   watts = record_package_watts(kernel_replay(func, args, kwargs), device_index=gpu_id)

Please also add one non-mocked test, e.g. watts for a large GEMM should be well above idle and close to a back-to-back loop. It would have caught both the NaN bug and this dilution.

[yzhou103]

• Minor: the prebuild guard doesn't cover the duplicate-shape branch. test_prebuild_config_load_does_not_import_aiter_package only exercises tables without duplicates; _raise_on_duplicate_shapes returns early there. With an overlapping shape (e.g. two AITER_CONFIG_BF16_BATCHED_GEMM files sharing one row), loading jit.core the way setup.py does fails with KeyError: 'module_aiter_core' in both latency and power mode, because the auto-dedup branch now imports sort_frame_for_selection / power_selection_enabled from aiter.utility. Main auto-resolves the same case. The shipped tables don't hit this, so it's an edge case, but a duplicate-shape case in that test would make the guard complete.

[rpoornac]

Thanks @yzhou103. 5355cd8 covers both comments from this morning and the three maintainability points you sent separately.

Power sample measured idle time. I went with your option 2. package_power.kernel_call(func, args, kwargs) removes the run_perftest options, and measure_kernel launches the kernel back to back for the window. That is now the only path to the sensor. MI355X (gfx950), bf16 torch.mm, GPU 0 otherwise idle:

shape	us	old run_perftest replay W	new sample W	back-to-back loop W
64x7168x2048	16.1	356	697	806
256x4096x4096	21.8	478	983	1042
4096^3	116.2	601	1247	1279

The old column reproduces your table. New test test_power_sample_of_a_large_gemm_tracks_a_busy_gpu reads the real sensor and does not mock it. It runs an 8192^3 bf16 GEMM through the tuner's sample and requires more than 1.5x idle and at least 0.85x a back-to-back loop. Locally it read idle 238 W, loop 1313 W, tuner 1339 W. It skips when power1_input is not readable.

Prebuild guard missed the duplicate-shape branch. The dedup in _raise_on_duplicate_shapes now imports from aiter.utility only when a running import aiter asked for power tables. Otherwise it sorts by lowest us, as main does. test_prebuild_duplicate_shapes_resolve_without_aiter_package loads jit.core the way setup.py does, with two AITER_CONFIG_BF16_BATCHED_GEMM files sharing a row, in both latency and power mode. It fails on the previous commit with KeyError: 'module_aiter_core'.

Maintainability.

1. Ranking in one place. New aiter/utility/selection_policy.py: make_policy(args.select) returns LATENCY or POWER, and tuners get it as self.policy / self.policy_for(args). It wraps the existing helpers: rank, beats (with an optional latency noise margin for the MoE flat check), best(candidates, latency_choice=...) for tuners with their own latency rule (the MXFP4 finalist gate), sort_frame, pair_energy, accurate_count, plus output helpers (attach_columns, result_metrics, watt_text, profile_file). rank_moe_pairs, the grouped, MXFP4, and staged loops, K5, IQ2R, a16w16, gradlib, and base_tuner all call the policy. The staged MXFP4 search now ranks and measures through it too; before this it compared us only.
2. Measurement in one place. Tuners hand policy.measure(call) / policy.record(target, key, call) the kernel launch. mp_tuner calls policy.measure_timed(func, args, kwargs). The latency policy never touches the sensor and adds no watts column.
3. AITER_SELECT is runtime-only. The tuner no longer writes it, and --select no longer defaults from it. mp_tuner takes policy= or reads the one TunerCommon.run activates (use_policy), so the ~17 existing mp_tuner callers needed no change. jit/core and power_configs are the only readers. This also stops AITER_SELECT=power from leaking into the --compare / --run_config benchmarks during a tune.

test_tuners_do_not_decide_select_mode_themselves enforces this across base_tuner, mp_tuner, gradlib, and every csrc/**/*_tune.py: none of them may call power_selection_enabled, select_power, the sensor functions, the raw rank helpers, or AITER_SELECT. A few tuners still read policy.measures_power only to decide whether to print watts columns in their log output.

Tuner unit tests pass locally (50). test_tune_pipeline and test_run_config need Ninja for the JIT build, which this host lacks, so I'm relying on CI for those.

[aiter-bot]

READY FOR HUMAN REVIEW

PR Self-Review

Revision bbc0ae5a8954 · mode shadow

Author self-review accepted. Readiness report:

Author answers

• PASS — current-SHA answers are present and structurally specific

Show the author's self-review answers

Behavior and intent

Tuners rank candidates by latency only, so the tuned tables always pick the fastest kernel even when a slightly slower one uses much less package energy. This PR adds an opt-in `--select power` mode that ranks accuracy-qualified candidates by `us × package watts` and stores those rows separately (`aiter/configs/power_configs/`), loaded only with `AITER_SELECT=power`. Default behavior is unchanged: `--select latency` keeps the lowest `us` and writes no power columns. Supported tuners: the shared GEMM tuners (`GemmCommonTuner`: a16w16 incl. hipBLASLt/asm/FlyDSL/Opus/Triton/torch, a8w8, blockscale, gradlib), the fused-MoE CK two-stage, grouped, and FlyDSL MXFP4 paths, the GDN K5 BV sweep, and IQ2R. Conv3d, a6w6, and the mixed-MXFP tuners (a6w4/a4w6) reject `--select power`, because their gain guard keeps a kernel only for speed.

Validation and unverified scope

Hardware: 8× MI355X (gfx950), ROCm 7.2.4, power cap 1400 W, clocks auto (SCLK 500-2400 MHz), GPUs otherwise idle.
- Unit: `python -m unittest op_tests.tuning_tests.test_fast_tuning` — 54 tests pass, including a real-sensor test (8192³ bf16 GEMM must read > 1.5× idle and ≥ 0.85× a back-to-back loop; skips without `power1_input`).
- Other tuning suites pass: `test_mp_tuner_logic`, `test_compare_logic`, `test_config_shape_collision`, `test_fhmoe_tuner`, `test_a6w6_tuning`, `test_mixed_mxfp_tuning`. `test_tuner_infra` has 2 failures (`a_preshuffle` blank-cell) that also occur on main (#6709).
- End to end, latency and power tunes on GPUs 0-7, low/mid/high shapes from the shipped untuned CSVs: a16w16 (all backends + hipBLASLt, 4 shapes), gradlib (4), a8w8 (3), a8w8 blockscale (3), CK MoE ASM FP8 (3), MXFP4 staged (3) and fast scan (2), IQ2R (3), and a16w16 with `--mp 2`. On every shape the latency CSV has no power columns, every power row has finite watts and the conditions, and the power pick is the minimum-energy measured candidate. Examples: a8w8 16384×8192×1024 is 23% slower and 9% lower energy; blockscale 1024×1536×7168 is 24% slower and 16% lower energy; MXFP4 32 tokens is 4% slower and 18% lower energy.
- MoE profile schema (FP8 per_1x128, ASM, 3 shapes): the latency `-o2` profile has main's 29 columns with no power columns, and `profile_fmoe.csv` has none; the power run keeps the energy-ranked profile and stamps all rows.
- Power sample vs CUDA-graph replay (no launch gaps): `torch.mm` within ~10%; split-K FlyDSL still reads 12-28% low after the launch-gap correction, though the energy ranking on that shape matches the graph-replay ranking.
- Unverified: grouped MoE (gfx1250 only) is covered by unit tests only; K5 has no shipped untuned shapes; gfx942 not run; kernels under ~5 µs are left unmeasured by design; the power score is an isolated replay, so a winner can change in a model, at another power cap, or next to other kernels; end-to-end model perf/W (MLPerf) is not part of this PR. Graph-replay power sampling is planned as a follow-up.

PR-specific question

No kernel dispatch path changes. Which kernel runs is still chosen by the tuned CSV for the arch/dtype/shape keys, as before. A power row can reach dispatch only through `AITER_SELECT=power` (overlay of `aiter/configs/power_configs/`) or an explicit `AITER_CONFIG_*` path. Without `AITER_SELECT=power`, `power_configs/` is ignored. A power row is written only for a candidate that passed the same `errRatio` accuracy gate as latency mode and has finite measured watts; a batch with no finite watts stays a latency result. Merging a power row and a latency row for one shape, or power rows tuned under different perf level, cap, or SCLK, raises instead of auto-resolving. Tuners that cannot rank by energy reject `--select power` (`SUPPORTS_POWER_SELECT = False`).

Changed op tests

`op_tests/tuning_tests/test_fast_tuning.py` — the only changed op test. Its new cases cover power selection, and none duplicate an existing case:
- Ranking and policy: energy vs latency order, unmeasured rows ranked last, MoE stage-energy pairs, `candidate_beats`, `SelectionPolicy` decisions, `mp_tuner` using the activated policy, a16w16 passing its policy to hipBLASLt.
- Measurement: the timed kernel is replayed without `run_perftest` options; bounded in-flight launches; launch-gap correction; real-sensor large-GEMM check.
- Output and loading: only measured rows are kept and stamped, power output redirected to `power_configs/`, latency frames and the latency MoE profile without power columns, power/latency merge conflicts, the `power_configs/` overlay (unstamped rows and files without `select` ignored).
- Build safety: `jit.core` loaded as in `setup.py` (with and without duplicate shapes) does not import the `aiter` package.
- Guard: no tuner file decides the select mode, reads power, or uses `AITER_SELECT` itself.

PR state

• PASS — pull request is marked ready for review

Copilot findings

• PASS — no active Copilot finding lacks a substantive author reply
• ADVISORY — Copilot did not review the current head SHA

Code and op-test hygiene

• PASS — no committed artifact or conflict marker detected

Required CI

• PASS — Black: passed
• PASS — Ruff: passed
• PASS — Repository dependency: passed
• PASS — Aiter Test Gate: passed

_{Shadow mode reports what would block but keeps the status successful. Set AITER_PR_SELF_REVIEW_MODE=enforce only after reviewing real false positives.}

[yzhou103]

Thanks for the update. I checked that latency mode is unchanged versus main (849f711), with AITER_SELECT unset:

• All 27 runtime config files are byte-identical.
• post_process and the merge give identical output on fixed synthetic results, including ties and invalid times.
• Real a16w16 and fused-MoE latency tunes pick the same kernels, with the same tuned-CSV schema.
  I also ran gemm_a8w8_mxfp8_bpreshuffle with --select power. Its own result_to_df override still gets watts, tflops_per_w, select=power and the power-state columns, so centralizing these in post_process and run() works.

Please fix in this PR:

1. MoE profile schema changes in latency mode. The -o2 profile file now gets rank, watts1, watts2, energy, watts and tflops_per_w, and profile_fmoe.csv gets watts. All of them are NaN in latency mode, and the -o2 column and row order also differ from main. The tuned CSV is unchanged, but anything that reads the profile files sees a different schema. Please add these columns only when select=power, and keep the latency output the same as main.

2. Unmeasured rows are written to power_configs. In a power tune, shapes whose watts could not be measured fall back to a latency rank. _stamp_power_rows leaves select and the power-state columns blank on those rows, but still returns them, so they are written to aiter/configs/power_configs/. When no row in a batch is measured, the whole batch is written ("leaving this batch as a latency result"). At runtime, _overlay_power_tables replaces the shipped latency row with any power_configs row regardless of select, so with AITER_SELECT=power these latency-ranked rows override the shipped (or model_configs) rows.

   Suggested fix:

   • In _stamp_power_rows, keep only measured rows and log a warning with the skipped shape keys. If nothing is measured, return an empty frame. Skipped shapes then show up as untuned and are retried on the next power tune.
   • In _overlay_power_tables, keep only rows where row_select_mode(select) == "power" before deduplicating, and ignore a power file that has no select column. This protects against older or hand-copied tables.

Everything else looks good to me.

[github-actions]

🏷️ CI Guide

Runs automatically on every PR:

• ✅ Pre-checks (submodule verification, code formatting)
• ✅ Aiter op tests (gfx942 + gfx950)
• ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label	Tests
ci:gfx1250-ffm-triton	Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x	Run an additional Triton test job on MI300X in PRs (added automatically when gfx942 configs change); main branch always runs both MI35X and MI300X
ci:triton-355	Run the full Triton test suite on MI35X, not only the tests the change affects
ci:windows	Windows JIT op smoke test on the gfx1201 ROCm runner
ci:multi-gpu	Aiter multi-GPU tests on the 8-GPU runner
ci:sglang	SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom	ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full	ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm	vLLM benchmark: DeepSeek-V4-Pro, MiniMax-M3-MXFP4
ci:all	All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 6970 --add-label <label>

One backend per PR:
A PR changes one kernel backend: [Triton/Gluon] (Triton and Gluon count as one), [HIP], [ASM], [CK], [OPUS] or [FlyDSL]. If the title ends up with two backend tags, split the PR -- as stacked pull requests when one part cannot merge without the other.

PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to stop the title rewrites; labels stay in sync either way.

[rpoornac]

Thanks @yzhou103 for checking latency mode against main. Both are fixed in 6fae460.

1. MoE profile schema in latency mode. Stage rows carry watts only with --select power, and rank_moe_pairs returns the frame unchanged in latency mode. The -o2 file starts from self.columns as on main, and the pairs are written in main's order. The latency winner is still the lowest-us row (same as main's idxmin). FP8 per_1x128 MoE tune on MI355X (ASM, 1/64/4096 tokens): the latency -o2 profile went from 35 to main's 29 columns, with no rank, watts1, watts2, energy, watts, or tflops_per_w, and profile_fmoe.csv has no watts. The power run still writes the energy-ranked profile and stamps every row.

2. Unmeasured rows in power_configs. I took your suggestion:

• _stamp_power_rows keeps only measured rows and logs the skipped shape keys. If nothing is measured, it returns zero rows (columns kept, because K5's result_to_csv needs them). Those shapes are reported as untuned and retried by the next power tune.
• _overlay_power_tables uses only rows where row_select_mode(select) == "power", so a file without a select column contributes nothing, and it logs how many rows it ignored. The filter runs after the duplicate-shape check, because that check can rewrite source files row by row, and filtering first would delete the ignored rows from the user's file.

New tests: latency rank_moe_pairs keeps its columns and order; mixed and all-unmeasured batches drop the unmeasured rows; the overlay ignores unstamped rows and files without select. test_fast_tuning (54), test_config_shape_collision, test_fhmoe_tuner, test_mp_tuner_logic, and test_compare_logic pass locally.

[yzhou103]

LGTM