[danielbolsson]

Summary

The decode-and-triage skill was run end to end against real archives on a gfx950 host, and reviewed against the HRR runbooks. It did not survive either exercise: on a crashing archive it produced no finding at all, and on the archives where it did produce one, the verdict was often wrong.

This is a skill-only change. No capture or playback source is touched.

It is based on #11157, which moves the skill to projects/hrr/skills/, so the diff here is the fixes alone. The base returns to develop once that one lands.

This replaces #9758. That PR was stacked on #10535, which moves the same skill files to projects/clr/.claude/skills/ instead. #10535 and #11157 are a rename/rename conflict over the same sources and cannot both land; #11157's destination is the one that puts the skill under @ROCm/hrr-reviewers, so #10535 is closed and the fixes are replayed here.

Two files needed real merging rather than a path rewrite — ensure_playback.sh and triage_archive.sh; see the commit message for what was kept from each side. All 48 unit tests pass.

What was wrong

#	Symptom on a real archive	Cause
1	Native replay impossible even with hrr-playback supplied	export_playback_env ended in a trailing &&, which under set -e became the function's exit status and killed the script before it printed the path
2	A replay that faulted produced a log and no finding	run_native_replay re-armed set -e inside itself, cancelling the set +e the caller wraps it in, so any non-zero replay exit ended the script before the analyzer ran
3	A hung replay produced nothing and never returned	no bound on the replay, so the analyzer was never reached
4	Metadata-only triage reported no events, kernels or completeness	--no-replay never resolved a playback binary, so --info never ran
5	A packaged playback failed to load its own libamdhip64	the library path handled only a playback inside a CLR build tree, not bin/ and lib/ siblings
6	A queue abort reported as a hang with an unknown kernel	hang was reachable from any HSA_STATUS_ERROR_*, and the abort line carries a kernel but no faulting address, so the memory-fault regex could not see it
7	A read-only page fault reported as replay_fatal_api	the generic abort branch was tested before the memory-fault branch
8	A clean replay reported a hipBLASLt GEMM as its kernel	the name was matched out of the archive's own kernel listing
9	A fault with no attribution reported no kernel at all	the --info kernel table was not parsed, so a single-kernel archive named nothing
10	The kernel name dropped on some ROCm builds	RE_MEM_FAULT_ERR required a host: field that not every build emits
11	Replay picked the busiest GPU and reported its full VRAM as free	the awk matched Used Memory and Total Memory as separate patterns, but the used line reads VRAM Total Used Memory and matches both
12	Replay could land on a device other than the one picked	the native path masked with ROCR_VISIBLE_DEVICES, which re-indexes underneath a HIP mask; the Windows and container paths already used HIP_VISIBLE_DEVICES
13	Complete:, Events:, Kernels: and both format versions unreported	uppercase-only matcher, unparsed labelled lines, versions recorded only in prose
14	Last progress kernel always empty	assigned unconditionally, so the --info pass erased what the replay log had recorded
15	A replay killed by a signal reported no class and still named a kernel	the log ends where the process died, so the truncation read as insufficient signal and the kernel was then inferred from a single-kernel archive

Two of these deserve a note.

Nothing beats a wrong verdict for cost. Defects 6 and 8 are the expensive ones: one reports a hang that never happened and drops the kernel the runtime had just named, the other prints a GEMM next to a PASS, where it reads as a culprit. Both send a kernel developer after a defect the run never observed.

A hang is now bounded, not detected. The replay is stopped after HRR_REPLAY_TIMEOUT seconds, default 1800, and the runner records that in the log so the finding says hang rather than nothing. The bound goes around the playback process, not around a container that may wrap it: a timeout on the container leaves the replay running inside it.

What changed in behaviour

• Replay runs with --sync-after-launch by default; --no-sync opts out. Measured cost on a real vLLM capture of 13233 kernels: 41.7 s against 39.6 s.
• A clean replay implicates no kernel. A fault with no attribution in the log takes the kernel from an archive holding exactly one, marked in the finding as inferred; more than one stays unknown, and names truncated by the --info column width are never used.
• New outcome HANG. HSA_STATUS_ERROR_MEMORY_FAULT classifies as illegal_memory_access, the other HSA statuses as replay_aborted, and the raw status goes in the notes.
• A replay process that dies on a signal is its own finding: outcome ABORT, fault class replay_crashed, no kernel implicated, because nothing about the workload was established. The runner records the signal in the log the way it already records a timeout, and the classifier ranks that marker last, so a log which explains its own fault is still believed first: a memory fault also dies on a signal.
• A fault on a <<<>>>-launched ATen kernel keeps its class and gains a caveat. Capture records the pointers these kernels pass inside by-value structs and replay translates them, so such a fault can be genuine and reclassifying it would suppress a real finding; but the translation is a heuristic and older archives carry no offsets, so the skill asks for the original failure signature instead of resolving the ambiguity.

Testing

Fixtures are unedited tool output recorded on a gfx950 host, so a change in what hrr-playback prints fails a test rather than silently degrading a finding. They sit under the skill's evals/fixtures/, with the recorded evidence rather than beside the scripts that read it. Device selection, playback discovery and the library layout are covered without a GPU by lifting the shell functions out of the script and running them against recorded input.

Unit suite: 24 tests on develop, 48 here. On this head 46 pass, 1 skips below bash 4.4, and 1 fails: test_probe_host_reads_comgr, which fails identically on the unmodified develop copies of the same two files and is untouched here.

Each fix was also removed in turn to confirm the test guarding it fails without it, since a test that passes either way proves nothing.

The exit-code plumbing behind the new class was checked against a real signal death through the same pipe-to-tee shape the runner uses: PIPESTATUS[0] carries 139 for SIGSEGV and 134 for SIGABRT.

End to end with triage_archive.sh on a gfx950 host:

Archive	Result
vLLM inference capture, format v4	13233 kernels replayed, 32 device-to-host checks exact, PASS / replay_pass, no kernel implicated, Complete: yes, 185469 events
Capture whose workload faults	fault reproduced, run survives it, MAF / illegal_memory_access with fault address, kernel crash_oob and the dispatch grid. Previously: no finding at all
PyTorch capture indexing past its embedding table	replay reproduces the capture-time failure exactly, same HSA_STATUS_ERROR_EXCEPTION on the same kernel. ABORT / replay_aborted, full indexSelectSmallIndex signature, ATen caveat. Previously: UNKNOWN / hang, kernel unknown
Hanging capture, 45 s bound	HANG / hang with a finding. Previously: ran until killed, no finding
Archive the playback build cannot read	version_mismatch with archive version 3 and reader version 4, no kernel invented
Metadata-only, no GPU	5 events, 1 kernel, crash_oob, Complete: no. Previously: every field n/a

Known gaps

• Multi-process archives are untested: none of the 13 archives available here has more than one pid-* directory, and load_capture_metadata reads only <archive>/manifest.json, so an archive root loses the preflight and the capture fields.
• A hang is inferred from the bound firing, not from any signal the replay emits; a replay legitimately slower than the timeout will be reported as a hang.
• No live hrr-playback was made to die on a signal, so that path rests on the runner and analyser tests plus the exit-code check above rather than on a recorded run.
• Windows was not exercised. triage_archive.ps1 gains no changes here beyond already using the HIP mask, and the PowerShell runner records no equivalent marker yet.

Issue Tracking

JIRA ID: AIRUNTIME-2695

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

[Copilot]

🟡 Changes recommended

triage_archive.sh currently uses tee -a in record_replay_stop in a way that can print replay-stop markers to stdout and contaminate the script’s intended stdout output (notably --format json).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR improves the HRR decode-and-triage skill so it produces accurate, actionable findings across real-world archives (including crashes, queue aborts, and hangs), by making replay execution more robust and enhancing log parsing/kernel attribution.

Changes:

• Harden replay execution in triage_archive.sh (timeout bounding, correct device masking, improved GPU selection parsing, metadata-only --info support, optional --sync-after-launch).
• Improve replay-log analysis in analyze_replay_finding.py (new hang/signal handling, better classification ordering, queue-abort kernel extraction, --info totals + kernel table parsing, post-parse attribution finalize step).
• Add unit tests and recorded fixtures to lock behavior against real tool output.

File summaries

File	Description
projects/hrr/skills/decode-and-triage/SKILL.md	Updates runbook guidance for new outcomes/attribution behavior (ABORT/HANG, kernel attribution rules, ATen caveat).
projects/hrr/skills/decode-and-triage/scripts/triage_archive.sh	Adds replay timeout bounding, sync control, improved GPU picking, HIP device masking, metadata-only playback discovery, and replay-stop markers.
projects/hrr/skills/decode-and-triage/scripts/analyze_replay_finding.py	Enhances parsing/classification (hang vs abort, signal death, queue abort kernel, --info fields/table parsing) and adds finalize attribution pass.
projects/hrr/skills/decode-and-triage/scripts/ensure_playback.sh	Ensures ROCR_LIB can be prioritized in LD_LIBRARY_PATH for standalone layouts.
projects/hrr/skills/decode-and-triage/scripts/test_triage_archive.py	Adds unit tests for shell helpers (GPU selection, HIP masking, replay stop markers, library path layout, ensure_playback behavior).
projects/hrr/skills/decode-and-triage/scripts/test_analyze_replay_finding.py	Adds regression tests using recorded logs/--info dumps to validate classification and attribution behavior.
projects/hrr/skills/decode-and-triage/evals/fixtures/*	Adds/updates fixtures representing real replay/log outputs used by the new unit tests.

Review details

• Files reviewed: 9/15 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[therock-pr-bot]

✅ All Policy Checks Passed

Check	Status	Details
📝 PR Description	✅ Pass	—
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

🎉 All policy checks passed!

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

[systems-assistant]

TheRock Submodule Bump Activity

Newest first

• 2026-09-13 — Included in TheRock via ROCm/TheRock#8175.