fix(hrr): parse the lines decode-and-triage really receives by andraghetti · Pull Request #14136 · ROCm/rocm-systems

andraghetti · 2026-10-23T09:11:39Z

Motivation

The decode-and-triage skill misreads the HIP version and GPU arch on Windows. On Linux hosts with only amd-smi, it never finds a GPU. Its analyzer also misses three log lines that CLR, the reader and hrr-playback print.

Technical Details

check_replay_compat.py:

Windows HIP version: the patch is HIP_VERSION % 100000, as hip_capture_metadata.cpp records it. Before, the two strings never matched.
Windows GPU arch: read gcnArchName from hipDeviceProp_tR0000, which the exported hipGetDeviceProperties fills. Attribute 57 is hipDeviceAttributeMaxThreadsPerMultiProcessor, not the arch.
Linux without rocm-smi: run amd-smi static --asic and read its GPU, MARKET_NAME and TARGET_GRAPHICS_VERSION fields. amd-smi rejects the rocm-smi options it got before.
amd-smi prints TARGET_GRAPHICS_VERSION: N/A when its ASIC query fails. That GPU keeps its place and skips the arch check.

analyze_replay_finding.py:

CLR prints "Memory Fault Error" and the faulting-address bracket with separate log calls. The pattern now matches the bracket on its own line.
The reader prints "Version mismatch in : file=N reader=M". The pattern now allows the file name.
When no runtime fault line names a kernel, the kernel comes from the "[HRR] GPU error after ''" line that --sync-after-launch prints.

hrr_playback.cpp: the --version comment quotes the mismatch line as the reader prints it.

Each test fixture is a line the tools print, and its comment cites the source line that prints it. The amd-smi fields come from amdsmi-cli-tool.md.

Issue Tracking

JIRA ID: AIRUNTIME-3062

Test Plan

python3 -m pytest projects/hrr/skills/decode-and-triage and python3 -m compileall -q projects/hrr/skills on head 31276d0, as the "HRR skills" workflow runs them.
Negative control: the two scripts reverted to develop, same tests.
The Windows probes never ran on Windows. The tests run them against a ctypes fake that writes at the C struct offsets. The amd-smi path never ran against a real amd-smi.
Skills, plus one comment in hrr_playback.cpp. No compiled code changes, so the PR carries ci:skip.

Test Result

Check	This PR (`31276d0`)	Negative control
decode-and-triage pytest	92 pass, 1 skip (needs bash >= 4.4)	7 fail: the 7 new or changed tests

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

check_replay_compat.py: - Windows HIP version: the patch is HIP_VERSION % 100000, as capture records it, not (v // 100) % 1000, so the two strings can match. - Windows GPU arch: read gcnArchName from hipDeviceProp_tR0000, which the exported hipGetDeviceProperties fills, instead of attribute 57 (hipDeviceAttributeMaxThreadsPerMultiProcessor) printed in hex. - Linux without rocm-smi: run `amd-smi static --asic` and parse its GPU/MARKET_NAME/TARGET_GRAPHICS_VERSION fields; amd-smi has no --showid/--showproductname, so the GPU count and arch checks were skipped. analyze_replay_finding.py: - CLR prints "Memory Fault Error" and the faulting-address bracket with separate log calls, so match the bracket on its own line. - hrr_reader prints "Version mismatch in <file>: file=N reader=M". - Take the kernel from the "[HRR] GPU error after '<kernel>'" line that --sync-after-launch prints when no runtime fault line names one. Tests use the lines the tools print, quoting each format string. Jira: AIRUNTIME-3062

therock-pr-bot · 2026-10-23T09:12:42Z

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

Copilot

🟡 Changes recommended

Unavailable amd-smi architecture values cause false compatibility mismatches and can block replay under --strict-arch.

1 open issue

Treat unavailable ASIC architecture as unknown during compatibility checks · New

What changed in this PR

This PR aligns HRR’s decode-and-triage helpers with actual platform probe output and replay logs, improving compatibility checks and fault identification.

Changes:

Correct Windows HIP version and GPU architecture probing.
Add Linux GPU discovery through amd-smi static --asic.
Recognize additional runtime fault, version mismatch, and synchronized-launch log formats, with regression tests.

File	Description
`projects/hrr/skills/decode-and-triage/tests/test_check_replay_compat.py`	Adds mocked Windows and Linux probe coverage.
`projects/hrr/skills/decode-and-triage/tests/test_analyze_replay_finding.py`	Tests actual fault and replay log formats.
`projects/hrr/skills/decode-and-triage/tests/fixtures/version_mismatch.log`	Includes the filename emitted by the reader.
`projects/hrr/skills/decode-and-triage/scripts/check_replay_compat.py`	Fixes Windows probes and adds Linux `amd-smi` parsing.
`projects/hrr/skills/decode-and-triage/scripts/analyze_replay_finding.py`	Updates log matching and kernel-name fallback.

_{Review effort: Balanced}

amd-smi static --asic prints TARGET_GRAPHICS_VERSION: N/A when the ASIC query fails. The parser now keeps that GPU's slot as an empty string, so evaluate_compat() skips the comparison instead of reporting a mismatch, and the report shows it as n/a.

danielbolsson

Checked a3e776b: the Windows HIP patch parse, gcnArchName at offset 396, the amd-smi static --asic parser with N/A as an empty arch, and the three analyzer patterns. Nothing blocking.

Nit: check_replay_compat.py:711 - _parse_amd_smi_static can return fewer archs than count if a GPU block lacks TARGET_GRAPHICS_VERSION; pad with "" to count.

- test_check_replay_compat: drop the fake hipDeviceGetAttribute; the Windows probe no longer calls it, and the test still fails on develop. Keep only the amd-smi fields the probe reads in its sample output. - test_analyze_replay_finding: drop the fault_class and failing_call_index asserts and the Fatal line from the sync-after-launch test; they pass on develop and test_fatal_api_abort covers them. - Cite the CLR and HRR sources by file:line instead of pasting their format strings. - hrr_playback.cpp: quote the reader's mismatch line as hrr_reader.cpp:168 prints it. AIRUNTIME-3062: tests trimmed, the fix and its negative controls unchanged.

systems-assistant · 2026-10-27T04:30:32Z

TheRock Submodule Bump Activity

Newest first

2026-10-27 — Included in TheRock via ROCm/TheRock#9493.

andraghetti requested a review from a team as a code owner October 23, 2026 09:11

Copilot AI balanced review requested due to automatic review settings October 23, 2026 09:11

andraghetti added the ci:skip Skip all CI builds/tests for this PR label Oct 23, 2026

github-actions Bot added the project: hrr label Oct 23, 2026

Copilot started reviewing on behalf of andraghetti October 23, 2026 09:12

View session

Copilot AI reviewed Oct 23, 2026

View reviewed changes

Comment thread projects/hrr/skills/decode-and-triage/scripts/check_replay_compat.py Resolved

systems-assistant Bot added the organization: ROCm label Oct 23, 2026

danielbolsson approved these changes Oct 26, 2026

View reviewed changes

andraghetti merged commit c8a1bc3 into develop Oct 26, 2026
33 of 35 checks passed

andraghetti deleted the users/andraghetti/hrr-triage-log-parsing branch October 26, 2026 13:46

Navigation Menu

Repository navigation

fix(hrr): parse the lines decode-and-triage really receives - #14136

fix(hrr): parse the lines decode-and-triage really receives#14136
andraghetti merged 3 commits into
developfrom
users/andraghetti/hrr-triage-log-parsing

andraghetti commented Oct 23, 2026 •

edited

Loading

Uh oh!

therock-pr-bot Bot commented Oct 23, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

danielbolsson left a comment

Uh oh!

Uh oh!

systems-assistant Bot commented Oct 27, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

andraghetti commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

Issue Tracking

Test Plan

Test Result

Submission Checklist

Uh oh!

therock-pr-bot Bot commented Oct 23, 2026

✅ All Policy Checks Passed

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

danielbolsson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

systems-assistant Bot commented Oct 27, 2026

TheRock Submodule Bump Activity

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

andraghetti commented Oct 23, 2026 •

edited

Loading