[rpoornac]

Follow-up from #6970, raised in #5846.

package_power.measure_kernel returned NaN watts for every exception, and _sync_cuda, which the sampling loop calls, swallowed a failed synchronize. A GPU fault during a --select power sampling pass therefore never reached the mp_tuner worker. The candidate that faulted was not blamed, and the lost context could surface on the next candidate in the same worker.

Change (aiter/utility/package_power.py only)

• measure_kernel re-raises an AcceleratorError as is. For any other error it checks whether a synchronize now fails; if so it raises that error, with the original as its cause. Any other error, such as a sensor that cannot be read, still gives NaN watts with a warning.
• _sync_cuda raises a failed synchronize. It skips only when torch or an initialized CUDA context is missing.
• The sampling thread is stopped and joined even when the final synchronize raises.

How the fault is handled

• With [CK] [FlyDSL] [CI] [Tuning] Per-candidate faults, selection proof and auto-select in --compare #5846, the worker call is outside any catch-all and the parent tells a fault apart by type, so the candidate that faulted is the one that fails and the pool restarts.
• On main today, work_group's catch-all marks that candidate's group failed, as it does for any other fault in a group.

Latency tunes do not sample power, so they are unchanged.

PR Self-Review

Reviewed revision

e29d98a

Behavior and intent

A GPU fault during the power-sampling pass of a --select power tune is now raised to the tuner instead of read as NaN watts, so it is charged to the candidate that caused it. A sensor that cannot be read still gives NaN. Latency selection is unchanged.

Validation and unverified scope

CPU only (HIP_VISIBLE_DEVICES=-1), on main 29625f7:

• python -m unittest op_tests.tuning_tests.test_fast_tuning: 58 passed, 1 skipped, 1 error. The error is test_k5_post_process_keeps_lower_energy_duplicate, where Triton finds no GPU driver; it errors the same way on main without a GPU.
• test_mp_tuner_logic, test_tuner_infra and test_compare_logic pass except the two test_tuner_infra grouped-MoE tests that fail on main because of [FlyDSL] Add gfx1250 grouped MoE A-preshuffle and tuned CSV validation #6709's a_preshuffle column (fixed in [CK] [FlyDSL] [CI] [Tuning] Per-candidate faults, selection proof and auto-select in --compare #5846).
• Black 26.10.1 and Ruff 0.16.0 are clean on the changed files.

Unverified: not run on a GPU, and not run together with #5846's fault handling. A real faulting kernel during the power pass is covered only by the mocked tests below.

PR-specific question

Does raising the fault change anything for a tune on main before #5846 lands? Yes, and only under --select power. On main, work_group's catch-all now marks the faulting candidate's group as failed instead of recording NaN watts for it and running the next candidate on the broken context. With #5846, the parent tells the fault apart by type and fails only that candidate. Latency tunes never call measure_kernel, so they are unchanged. No .github/ changes.

Changed op tests

• test_fast_tuning.py (extended): five CPU tests for package_power. An AcceleratorError from the sampled call is raised unchanged; an error that breaks synchronize is raised as that synchronize error, with the original as its cause; an error that leaves synchronize working still gives NaN; the synchronize check itself; and the sampling thread ending when the final synchronize raises. No existing test covered the exception path of measure_kernel.

Made with Cursor

[github-actions]

🏷️ CI Guide

Runs automatically on every PR:

• ✅ Pre-checks (submodule verification, code formatting)
• ✅ Aiter op tests (gfx942 + gfx950)
• ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label	Tests
ci:gfx1250-ffm-triton	Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x	Run an additional Triton test job on MI300X in PRs (added automatically when gfx942 configs change); main branch always runs both MI35X and MI300X
ci:triton-355	Run the full Triton test suite on MI35X, not only the tests the change affects
ci:windows	Windows JIT op smoke test on the gfx1201 ROCm runner
ci:multi-gpu	Aiter multi-GPU tests on the 8-GPU runner
ci:sglang	SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom	ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full	ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm	vLLM benchmark: DeepSeek-V4-Pro, MiniMax-M3-MXFP4
ci:all	All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 7493 --add-label <label>

One backend per PR:
A PR changes one kernel backend: [Triton/Gluon] (Triton and Gluon count as one), [HIP], [ASM], [CK], [OPUS] or [FlyDSL]. If the title ends up with two backend tags, split the PR -- as stacked pull requests when one part cannot merge without the other.

PR title tags & labels:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title and as PR labels automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf], op tags like [MLA], and human labels (ci:*) are left untouched. Add the no-auto-title label to stop the title rewrites; labels stay in sync either way.

[aiter-bot]

READY FOR HUMAN REVIEW

PR Self-Review

Revision e29d98a9516c · mode shadow

Author self-review accepted. Readiness report:

Author answers

• PASS — current-SHA answers are present and structurally specific

Show the author's self-review answers

Behavior and intent

A GPU fault during the power-sampling pass of a `--select power` tune is now raised to the tuner instead of read as NaN watts, so it is charged to the candidate that caused it. A sensor that cannot be read still gives NaN. Latency selection is unchanged.

Validation and unverified scope

CPU only (`HIP_VISIBLE_DEVICES=-1`), on main `29625f7`:

- `python -m unittest op_tests.tuning_tests.test_fast_tuning`: 58 passed, 1 skipped, 1 error. The error is `test_k5_post_process_keeps_lower_energy_duplicate`, where Triton finds no GPU driver; it errors the same way on main without a GPU.
- `test_mp_tuner_logic`, `test_tuner_infra` and `test_compare_logic` pass except the two `test_tuner_infra` grouped-MoE tests that fail on main because of #6709's `a_preshuffle` column (fixed in #5846).
- Black 26.10.1 and Ruff 0.16.0 are clean on the changed files.

Unverified: not run on a GPU, and not run together with #5846's fault handling. A real faulting kernel during the power pass is covered only by the mocked tests below.

PR-specific question

Does raising the fault change anything for a tune on main before #5846 lands? Yes, and only under `--select power`. On main, `work_group`'s catch-all now marks the faulting candidate's group as failed instead of recording NaN watts for it and running the next candidate on the broken context. With #5846, the parent tells the fault apart by type and fails only that candidate. Latency tunes never call `measure_kernel`, so they are unchanged. No `.github/` changes.

Changed op tests

- `test_fast_tuning.py` (extended): five CPU tests for `package_power`. An `AcceleratorError` from the sampled call is raised unchanged; an error that breaks synchronize is raised as that synchronize error, with the original as its cause; an error that leaves synchronize working still gives NaN; the synchronize check itself; and the sampling thread ending when the final synchronize raises. No existing test covered the exception path of `measure_kernel`.

Made with [Cursor](https://cursor.com)

PR state

• PASS — pull request is marked ready for review

Copilot findings

• PASS — no active Copilot finding lacks a substantive author reply
• ADVISORY — Copilot did not review the current head SHA

Code and op-test hygiene

• PASS — no committed artifact or conflict marker detected

Required CI

• PASS — Black: passed
• PASS — Ruff: passed
• PASS — Repository dependency: passed
• PASS — Aiter Test Gate: passed

_{Shadow mode reports what would block but keeps the status successful. Set AITER_PR_SELF_REVIEW_MODE=enforce only after reviewing real false positives.}