danielbolsson · Sep 11, 2026 · Aug 26, 2026 · Aug 26, 2026 · Aug 26, 2026 · Aug 31, 2026
@@ -172,12 +172,67 @@ native. Use `--no-replay` for metadata-only (no GPU).
 | `PASS` | Replay completed; D2H checks passed |
 | `MAF` | GPU memory access fault during replay |
 | `FAIL` | Replay finished but validation failed (e.g. D2H mismatch) |
-| `ABORT` | Replay stopped early (fatal API, version mismatch, user abort) |
+| `ABORT` | Replay stopped early (fatal API, version mismatch, queue abort, user abort, replay process killed by a signal) |
+| `HANG` | Replay made no progress and the runner had to stop it (see `HRR_REPLAY_TIMEOUT`) |
 | `UNKNOWN` | Insufficient signal to classify (e.g. metadata-only triage, missing log) |
 
 When outcome is `UNKNOWN` or fault class is `unknown`, say so explicitly in the
 summary — do not invent a fault type.
 
+A fault on the GPU and a crash of the replay process are different findings.
+The runtime's `Memory access fault by GPU node-N ... on address 0x...` line is
+the GPU one: outcome `MAF`, fault class `illegal_memory_access`, or
+`read_only_page_fault` when the reason names a read-only page. The replay
+process dying on a signal without printing any of that is the host one: outcome
+`ABORT`, fault class `replay_crashed`, and it implicates no kernel, because
+nothing about the workload was established. Report that as a failed run rather
+than as the user's defect.
+
+An HSA queue abort (`HSA_STATUS_ERROR_EXCEPTION`, `ABORTED`, `MEMORY_FAULT`) is
+not a hang: `MEMORY_FAULT` is a fault and the other two are hardware exceptions,
+which an out-of-bounds access raises. The abort line carries the kernel but no
+faulting address, so read the kernel from there. A genuine hang shows up as no
+progress against the clock, which is what `HRR_REPLAY_TIMEOUT` reports; do not
+report one on the strength of an HSA status alone.
+
+### A faulting ATen kernel needs the original failure signature
+
+PyTorch and vLLM reach the GPU through `<<<>>>` (`hipLaunchByPtr`), and those
+kernels pass device pointers inside by-value structs: `vectorized_elementwise_kernel`
+takes a `std::array<char*,N>`, `reduce_kernel` a config struct with pointers at
+arbitrary offsets. Capture records those offsets and replay translates them, with
+a defensive rescan for any the capture-time heuristic missed, so on a current
+build these kernels replay faithfully and a fault on one is a real finding.
+
+Two things still make such a fault ambiguous: the detector is a value-based
+heuristic, and an archive recorded before that support landed carries no offsets
+at all, in which case replay launches with a capture-time address and faults on a
+workload that is fine. Both look identical to a workload defect.
+
+So the fault class stays what the evidence says (`illegal_memory_access`), the
+finding carries a note, and you **ask the user for their original failure
+signature** before calling it their bug. If their run never faulted here, suspect
+the recording. Faults on `hipModuleLaunchKernel`-launched kernels (hipBLASLt
+GEMMs, custom HIP kernels) do not carry this ambiguity.
+
+### Kernel attribution
+
+Replay runs with `--sync-after-launch`, which `triage_archive.sh` adds for you.
+Without it the GPU is serialized only once at the end, so a fault is reported but
+not attributed and the finding has no failing event and no kernel. Pass
+`--no-sync` only when throughput matters more than attribution.
+
+A clean replay implicates no kernel, so the finding reports none even though the
+archive lists the kernels it ran. Never present a kernel next to a `PASS`.
+
+A memory fault can tear the process down before the failing dispatch is
+attributed, leaving a log with no kernel at all. When the archive holds exactly
+one kernel, the finding names it and says the name was inferred; confirm it with
+`--sync-after-launch`, which stops replay on the faulting dispatch. With more
+than one kernel it stays unknown rather than guessing. Names that `--info`
+truncated to its column width are never used: a cut-off symbol cannot be looked
+up or handed to a kernel developer.
+
 ### Finding summary template (Step 5 / W4)
 
 Print a structured summary directly in the chat reply:

@@ -0,0 +1,29 @@
+HRR Archive: /captures/crash_v4.hrr/pid-79
+========================================
+Complete:     NO (no shutdown trailer; capture likely crashed)
+Recovered:    5 events
+Events:       5
+Kernels:      1
+Blobs:        1
+Code Objects: 2
+Threads:      1
+
+Event Type Breakdown:
+  Type                   Count
+  ----                   -----
+  hipDeviceSynchronize   1
+  hipMalloc              1
+  hipMemcpy              1
+  hipModuleLaunchKernel  1
+  hipModuleLoad          1
+
+Kernel Summary (first 20):
+  ID   Kernel                                             Grid            Block           SharedMem
+  --   ------                                             ----            -----           ---------
+  0    crash_oob                                          [64,1,1]        [64,1,1]        0
+
+Kernel Call Counts:
+  Kernel                                                       Calls
+  ------                                                       -----
+  crash_oob                                                    1
+
@@ -0,0 +1,135 @@
+HRR Archive: /captures/qwen06b_offline_v4.hrr/pid-1
+========================================
+Complete:     yes (clean shutdown)
+Recovered:    185469 events
+Events:       185469
+Kernels:      13233
+Blobs:        4458
+Code Objects: 33
+Threads:      2
+
+Event Type Breakdown:
+  Type                   Count
+  ----                   -----
+  __hipPopCallConfiguration 12945
+  __hipPushCallConfiguration 12945
+  __hipRegisterFatBinary 4083
+  hipDeviceGetAttribute  2827
+  hipDeviceGetStreamPriorityRange 1
+  hipDevicePrimaryCtxGetState 44
+  hipDeviceSynchronize   1
+  hipDriverGetVersion    1
+  hipEventCreateWithFlags 66
+  hipEventDestroy        64
+  hipEventQuery          31
+  hipEventRecord         131
+  hipEventSynchronize    66
+  hipFree                11
+  hipFuncGetAttribute    12
+  hipGetDevice           81383
+  hipGetDeviceCount      4
+  hipGetDevicePropertiesR0600 3295
+  hipGetLastError        1241
+  hipHostMalloc          29
+  hipMalloc              107
+  hipMemGetInfo          5
+  hipMemcpyAsync         354
+  hipMemcpyWithStream    509
+  hipMemset              1
+  hipMemsetAsync         32
+  hipModuleLaunchKernel  13233
+  hipModuleLoad          2
+  hipModuleLoadDataEx    4
+  hipModuleUnload        2
+  hipPointerGetAttribute 448
+  hipRuntimeGetVersion   2827
+  hipSetDevice           48075
+  hipStreamCreateWithPriority 1
+  hipStreamGetCaptureInfo 509
+  hipStreamIsCapturing   118
+  hipStreamWaitEvent     32
+  hipGetProcAddress      30
+
+Kernel Summary (first 20):
+  ID   Kernel                                             Grid            Block           SharedMem
+  --   ------                                             ----            -----           ---------
+  0    _ZN2at6native29vectorized_elementwise_kernelILi... [1,1,1]         [256,1,1]       0
+  1    _ZN2at6native29vectorized_elementwise_kernelILi... [1,1,1]         [256,1,1]       0
+  2    _ZN2at6native29vectorized_elementwise_kernelILi... [128,1,1]       [256,1,1]       0
+  3    _ZN2at6native29vectorized_elementwise_kernelILi... [16,1,1]        [256,1,1]       0
+  4    _ZN2at6native29vectorized_elementwise_kernelILi... [16,1,1]        [256,1,1]       0
+  5    _ZN2at6native29vectorized_elementwise_kernelILi... [16,1,1]        [256,1,1]       0
+  6    _ZN2at6native29vectorized_elementwise_kernelILi... [2,1,1]         [256,1,1]       0
+  7    _ZN2at6native29vectorized_elementwise_kernelILi... [1,1,1]         [256,1,1]       0
+  8    _ZN2at6native29vectorized_elementwise_kernelILi... [1,1,1]         [256,1,1]       0
+  9    _ZN2at6native29vectorized_elementwise_kernelILi... [1,1,1]         [256,1,1]       0
+  10   _ZN2at6native29vectorized_elementwise_kernelILi... [16,1,1]        [256,1,1]       0
+  11   _ZN2at6native29vectorized_elementwise_kernelILi... [1,1,1]         [256,1,1]       0
+  12   _ZN2at6native29vectorized_elementwise_kernelILi... [1,1,1]         [256,1,1]       0
+  13   _ZN2at6native29vectorized_elementwise_kernelILi... [1,1,1]         [256,1,1]       0
+  14   _ZN2at6native29vectorized_elementwise_kernelILi... [8192,1,1]      [256,1,1]       0
+  15   _ZN2at6native29vectorized_elementwise_kernelILi... [4,1,1]         [256,1,1]       0
+  16   _ZN2at6native29vectorized_elementwise_kernelILi... [1,1,1]         [256,1,1]       0
+  17   _ZN2at6native29vectorized_elementwise_kernelILi... [1,1,1]         [256,1,1]       0
+  18   _ZN2at6native29vectorized_elementwise_kernelILi... [1,1,1]         [256,1,1]       0
+  19   _ZN2at6native29vectorized_elementwise_kernelILi... [16,1,1]        [256,1,1]       0
+  ... and 13213 more
+
+Kernel Call Counts:
+  Kernel                                                       Calls
+  ------                                                       -----
+  Cijk_Alik_Bljk_BBS_BH_Bias_HA_S_SAV_UserArgs_MT128x128x12... 28
+  Cijk_Alik_Bljk_BBS_BH_Bias_HA_S_SAV_UserArgs_MT192x128x12... 28
+  Cijk_Alik_Bljk_BBS_BH_Bias_HA_S_SAV_UserArgs_MT256x256x64... 114
+  Cijk_Alik_Bljk_BBS_BH_Bias_HA_S_SAV_UserArgs_MT64x64x256_... 56
+  _Z11wvSplitKrc_I14__hip_bfloat16Li64ELi16ELi4ELi8ELi1ELi1... 112
+  _Z16wvSplitK_hf_sml_I14__hip_bfloat16Li64ELi1ELi16ELi8ELi... 1736
+  _Z16wvSplitK_hf_sml_I14__hip_bfloat16Li64ELi2ELi16ELi8ELi... 1768
+  _Z35paged_attention_ll4mi_reduce_kernelI14__hip_bfloat16S... 896
+  _Z38paged_attention_ll4mi_QKV_mfma4_kernelI14__hip_bfloat... 896
+  _ZN12_GLOBAL__N_129elementwise_kernel_with_indexIiZZZN2at... 2
+  _ZN2at6native12_GLOBAL__N_121indexSelectSmallIndexIN3c108... 32
+  _ZN2at6native12_GLOBAL__N_123cunn_SoftMaxForwardGmemILi4E... 2
+  _ZN2at6native12_GLOBAL__N_126CatArrayBatchedCopy_contigIN... 1
+  _ZN2at6native12_GLOBAL__N_143distribution_elementwise_gri... 2
+  _ZN2at6native12_GLOBAL__N_143distribution_elementwise_gri... 2
+  _ZN2at6native13reduce_kernelILi512ELi1ENS0_8ReduceOpIfNS0... 36
+  _ZN2at6native24index_elementwise_kernelILi128ELi4EZNS0_16... 32
+  _ZN2at6native24index_elementwise_kernelILi128ELi4EZNS0_16... 32
+  _ZN2at6native24vectorized_gather_kernelILi16ElEEvPcS2_PT0... 4
+  _ZN2at6native27unrolled_elementwise_kernelINS0_11FillFunc... 32
+  _ZN2at6native27unrolled_elementwise_kernelINS0_21CUDAFunc... 32
+  _ZN2at6native29vectorized_elementwise_kernelILi16ENS0_11F... 56
+  _ZN2at6native29vectorized_elementwise_kernelILi16ENS0_11F... 3
+  _ZN2at6native29vectorized_elementwise_kernelILi4ENS0_11Fi... 125
+  _ZN2at6native29vectorized_elementwise_kernelILi4ENS0_11Fi... 43
+  _ZN2at6native29vectorized_elementwise_kernelILi4ENS0_11Fi... 9
+  _ZN2at6native29vectorized_elementwise_kernelILi4ENS0_13AU... 1
+  _ZN2at6native29vectorized_elementwise_kernelILi4ENS0_13BU... 1
+  _ZN2at6native29vectorized_elementwise_kernelILi4ENS0_13Bi... 2
+  _ZN2at6native29vectorized_elementwise_kernelILi4ENS0_15CU... 32
+  _ZN2at6native29vectorized_elementwise_kernelILi4ENS0_15CU... 32
+  _ZN2at6native29vectorized_elementwise_kernelILi4EZNS0_21c... 4
+  _ZN2at6native29vectorized_elementwise_kernelILi4EZNS0_25b... 1
+  _ZN2at6native29vectorized_elementwise_kernelILi4EZNS0_34b... 34
+  _ZN2at6native29vectorized_elementwise_kernelILi4EZZZNS0_1... 2
+  _ZN2at6native29vectorized_elementwise_kernelILi4EZZZNS0_1... 1
+  _ZN2at6native29vectorized_elementwise_kernelILi4EZZZNS0_1... 1
+  _ZN2at6native29vectorized_elementwise_kernelILi4EZZZNS0_2... 1
+  _ZN2at6native29vectorized_elementwise_kernelILi8ENS0_11Fi... 170
+  _ZN2at6native32elementwise_kernel_manual_unrollILi128ELi4... 1
+  _ZN2at6native32elementwise_kernel_manual_unrollILi128ELi4... 36
+  _ZN2at6native32elementwise_kernel_manual_unrollILi128ELi4... 98
+  _ZN2at6native32elementwise_kernel_manual_unrollILi128ELi4... 2
+  _ZN2at6native32elementwise_kernel_manual_unrollILi128ELi4... 1
+  _ZN2at6native32elementwise_kernel_manual_unrollILi128ELi4... 2
+  _ZN4vllm15rms_norm_kernelIN3c108BFloat16ELi8ELi2EEEvPT_PK... 34
+  _ZN4vllm15rms_norm_kernelIN3c108BFloat16ELi8ELi3EEEvPT_PK... 1904
+  _ZN4vllm18act_and_mul_kernelIN3c108BFloat16E15__hip_bfloa... 952
+  _ZN4vllm23rotary_embedding_kernelIN3c108BFloat16ELb1EEEvP... 952
+  _ZN4vllm24reshape_and_cache_kernelI14__hip_bfloat16S1_LNS... 924
+  _ZN4vllm25fused_add_rms_norm_kernelIN3c108BFloat16ELi8EEE... 1904
+  _compute_slot_mapping_kernel                                 32
+  _fwd_kernel                                                  28
+  _topk_topp_kernel                                            2
+
@@ -0,0 +1,13 @@
+[triage] archive=/data/llama_run.hrr/pid-680 replay=native
+[triage] native replay playback=/opt/rocm/bin/hrr-playback GPU=3 --sync-after-launch
+[HRR] Archive : 177203 events, 8214 kernels, 4733 blobs, 41 code objects
+[HRR] Complete:     yes (clean shutdown)
+[HRR] [replay] Device state OK — beginning replay
+[HRR] Event 41188: hipMallocAsync
+[HRR] Event 41202: hipModuleLaunchKernel   -> Kernel '_ZN2at6native29vectorized_elementwise_kernelILi4EZZZNS0_18add_kernel_cudaERNS_18TensorIteratorBaseEENKUlvE_clEvENKUlvE2_clEvEUlffE_NS_6detail5ArrayIPcLi3EEEEEviT0_T1_' OK
+[HRR] Event 41266: hipMemcpyAsync
+[HRR] Event 41310: hipLaunchKernel
+:0:rocdevice.cpp :3608: Memory Fault Error [GPU index: 0, faulting addr: 0x7f8c11a00000, kernel: _ZN2at6native29vectorized_elementwise_kernelILi4EZZZNS0_18mul_kernel_cudaERNS_18TensorIteratorBaseEENKUlvE_clEvENKUlvE2_clEvEUlffE_NS_6detail5ArrayIPcLi3EEEEEviT0_T1_]
+Memory access fault by GPU node-4 (Agent handle: 0x55d1a2f00c40) on address 0x7f8c11a00000. Reason: Unknown.
+Aborted (core dumped)
+[triage] native replay exit=134
@@ -0,0 +1,8 @@
+[HRR] Archive : 5 events, 1 kernels, 1 blobs, 2 code objects
+[HRR] Threads : 1 captured
+[HRR] Device  : AMD Radeon Graphics (gfx950:sramecc+:xnack-)
+[HRR] Mode    : single-threaded
+[HRR] [replay] Device state OK — beginning replay
+[HRR] Pre-loading 2 code objects...
+[HRR] All code objects pre-loaded OK.
+[triage] replay timed out after 1800s
@@ -0,0 +1,3 @@
+[HRR] Archive has no clean-shutdown trailer (capture likely crashed); recovered 5 events
+Memory access fault by GPU node-2 (Agent handle: 0x55a197628600) on address 0x7f91da000000. Reason: Unknown.
+GPU coredump: execvp failed: No such file or directory
@@ -0,0 +1,20 @@
+[HRR] NOOP playback handler called for hipDevicePrimaryCtxGetState — this API is not replayed; results may differ from capture.
+[HRR] NOOP playback handler called for hipGetProcAddress — this API is not replayed; results may differ from capture.
+[HRR] NOOP playback handler called for hipFuncGetAttribute — this API is not replayed; results may differ from capture.
+[HRR] NOOP playback handler called for hipPointerGetAttribute — this API is not replayed; results may differ from capture.
+[HRR] Archive : 185469 events, 13233 kernels, 4458 blobs, 33 code objects
+[HRR] Threads : 2 captured
+[HRR] Device  : AMD Radeon Graphics (gfx950:sramecc+:xnack-)
+[HRR] Mode    : single-threaded
+[HRR] [replay] Device memory: 257418 MB free / 258032 MB total
+[HRR] [replay] Device state OK — beginning replay
+[HRR] Pre-loading 33 code objects...
+[HRR] All code objects pre-loaded OK.
+
+[HRR] -- Replay summary ------------------------------
+[HRR]   Wall time      : 39145.5 ms
+[HRR]   Threads used   : 1
+[HRR]   Kernels launched: 13233
+[HRR]   Graphs launched : 0
+[HRR]   D2H checks     : 32 pass (32 exact, 0 within tol), 0 fail, 0 skipped
+[HRR] PASS
@@ -0,0 +1,14 @@
+[HRR] Archive has no clean-shutdown trailer (capture likely crashed); recovered 2727 events
+[HRR] NOOP playback handler called for hipDevicePrimaryCtxGetState — this API is not replayed; results may differ from capture.
+[HRR progress] elapsed_s=0.0 seq=1301 kernels=1 d2h_pass=0 d2h_fail=0 d2h_attempted=0 last="_ZN2at6native29vectorized_elementwise_kernelILi4ENS0_11FillFunctorIfEESt5arrayIPcLm1EEEEviT0_T1_"
+Queue error: HSA_STATUS_ERROR_EXCEPTION: An HSAIL operation resulted in a hardware exception.
+GPU coredump: Pipe handler '/usr/share/apport/apport' not found or not executable.
+Falling back to file-based dump. Set HSA_COREDUMP_PATTERN to override.
+VGPU=0x5617c727ec30 SWq=0x7f69e8266000, HWq=0x7f6109e00000, id=1
+	Dispatch Header =0xb02 (type=2, barrier=1, acquire=1, release=1), setup=3
+	grid=[256, 1, 1], workgroup=[256, 1, 1]
+	private_seg_size=0, group_seg_size=0
+	kernel_obj=0x7f69de8aaa40, kernarg_address=0x0x7f60fea00100
+	completion_signal=0x0
+	rptr=18, wptr=20
+:0:rocdevice.cpp            :4312: 3445930657021 us:  Callback: Queue 0x7f6109e00000 aborting with error : HSA_STATUS_ERROR_EXCEPTION: An HSAIL operation resulted in a hardware exception. code: 0x1016 [host: 3d890059944d, GPU index: 0, kernel: void at::native::(anonymous namespace)::indexSelectSmallIndex<float, long, unsigned int, 2, 2, -2>(at::cuda::detail::TensorInfo<float, unsigned int>, at::cuda::detail::TensorInfo<float const, unsigned int>, at::cuda::detail::TensorInfo<long const, unsigned int>, int, int, unsigned int, long)]
@@ -0,0 +1,7 @@
+====================== ROCm System Management Interface ======================
+================================ Memory Usage ================================
+GPU[0]		: VRAM Total Memory (B): 274877906944
+GPU[0]		: VRAM Total Used Memory (B): 264877906944
+GPU[1]		: VRAM Total Memory (B): 274877906944
+GPU[1]		: VRAM Total Used Memory (B): 1073741824
+============================== End of ROCm SMI Log ===========================
@@ -0,0 +1 @@
+[HRR] Version mismatch: file=3 reader=4