Skip to content
Merged
Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
80ccdb0
hrr: snapshot pinned host memory kernels read, and restore it at replay
andraghetti Oct 9, 2026
4e60f35
hrr: test and document pinned host snapshots
andraghetti Oct 9, 2026
b6583ca
hrr: mark a pinned snapshot as last only after its event is written; …
andraghetti Oct 9, 2026
c32db55
Merge branch 'develop' into users/andraghetti/hrr-pinned-host-min
andraghetti Oct 12, 2026
b3bd5da
fix(hrr): hold pinned tracking across fork and failed frees, translat…
andraghetti Oct 12, 2026
7d8bac0
Merge branch 'develop' into users/andraghetti/hrr-pinned-host-min
andraghetti Oct 12, 2026
fa1ffc5
fix(hrr): keep the pinned last record out of graph captures, drop for…
andraghetti Oct 13, 2026
f43cf80
Merge branch 'develop' into users/andraghetti/hrr-pinned-host-min
andraghetti Oct 14, 2026
aa91d3f
docs(hrr): drop the lock-order note for the removed pinned fork handler
andraghetti Oct 14, 2026
9e23895
fix(hrr): copy pinned host memory before hashing it, hold its lock ac…
andraghetti Oct 15, 2026
499c82b
fix(hrr): restore pinned snapshots by the launch stream's capture state
andraghetti Oct 15, 2026
89cc5f4
Merge branch 'develop' of github.com:ROCm/rocm-systems into users/and…
andraghetti Oct 15, 2026
69d5514
fix(hrr): sync a pinned restore's stream without invalidating a capture
andraghetti Oct 15, 2026
9ddccdd
refactor(hrr): read the pinned snapshot settings in the HRR capture code
andraghetti Oct 15, 2026
5643bd7
Merge branch 'develop' into users/andraghetti/hrr-pinned-host-min
andraghetti Oct 15, 2026
274fb02
Merge branch 'develop' into users/andraghetti/hrr-pinned-host-min
andraghetti Oct 16, 2026
d6b99b1
fix(hrr): queue pinned restores on the launch stream instead of waiti…
andraghetti Oct 19, 2026
a7b0018
fix(hrr): track hipMallocHost for pinned snapshots, test the other fr…
andraghetti Oct 19, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
273 changes: 269 additions & 4 deletions projects/clr/hipamd/src/hrr/hip_capture.cpp

Large diffs are not rendered by default.

6 changes: 4 additions & 2 deletions projects/clr/hipamd/src/hrr/hip_capture_writer.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -153,8 +153,10 @@ static constexpr size_t kPathMax = 4096;
static constexpr size_t kMetadataJsonMax = 128u * 1024u;
static constexpr size_t kEmergencyManifestMax = kMetadataJsonMax + 1024u;

// Lock order: g_reopen_mu, then g_file_mu, then g_blob_mu, then
// g_unreplayable_mu. atfork_prepare is the only path that holds them all.
// Lock order: g_pinned_mu (hip_capture.cpp, held while a launch copies pinned
// host memory or writes its event), then g_reopen_mu, then g_file_mu, then
// g_blob_mu, then g_unreplayable_mu. Only fork() holds them all: the pinned
// prepare handler takes g_pinned_mu, then atfork_prepare takes the rest.
static std::mutex g_file_mu;
static int g_events_fd = -1;
// g_base_dir is the archive path requested via HIP_HRR_CAPTURE_OUTPUT.
Expand Down
105 changes: 59 additions & 46 deletions projects/hrr/DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,7 @@ the parent that exited cleanly needs no repair. A sub-archive is skipped when
payloads, finds it complete in the sense `load_archive` uses. `--info` on a
root applies the same check before it shows a process as complete.

### Archive Format (v9)
### Archive Format (v10)
```
capture.hrr/
manifest.json { version, capture_mode, owner_pid, processes[] }
Expand Down Expand Up @@ -286,7 +286,7 @@ projects/hrr/ — standalone HRR project (portable layer)
Not used by capture — it is the format an
out-of-tree producer writes and playback reads
playback/
hrr_reader.h/.cpp — archive loader, v7 format; record framing
hrr_reader.h/.cpp — archive loader, v10 format; record framing
(read_raw_record / open_record_stream) shared
with the region sidecars
hrr_region_map.h/.cpp — region timeline: merge, cursor, live block set,
Expand Down Expand Up @@ -371,13 +371,13 @@ The generator classifies each API:
Generated capture shims for manual APIs are pass-throughs (no `write_event()`).
When adding HIP API support, update this script to classify the API in the appropriate capture and playback policy sets. APIs requiring non-trivial serialization or replay belong in `MANUAL_CAPTURE_APIS` and/or `MANUAL_PLAYBACK_APIS`; intentionally unsupported replay APIs belong in `NOOP_PLAYBACK_APIS`.

## Archive Format (v9)
## Archive Format (v10)

Single-authority definition in `hrr_api_args.h` (auto-generated):

```
HRR_MAGIC = 0x52524845 ("HRRE")
HRR_VERSION = 9
HRR_VERSION = 10
```

Version history, so an archive written by an older runtime can be placed:
Expand Down Expand Up @@ -414,12 +414,15 @@ Version history, so an archive written by an older runtime can be placed:
- **v9** records `hipKernelSetAttributeForDevice`, appended to
`HipDispatchTable`, which shifts the compiler API IDs up by one again. A v8
reader would decode those events as the wrong API.
- **v10** puts pinned host snapshot records after a kernel launch's arguments
(`num_snapshots` was always 0). A v9 reader would not restore them. See
Pinned Host Snapshots below.

```
<output_dir>/
manifest.json { version, capture_mode, owner_pid, processes[] }
(version here is the manifest schema = 1, distinct from the
events.bin HRR_VERSION = 9)
events.bin HRR_VERSION = 10)
pid-<pid>/
manifest.json { pid, parent_pid, complete, event_count, blob_count }
active one line naming the process instance (boot id and
Expand Down Expand Up @@ -537,41 +540,51 @@ At playback:
- `playback_hipHostUnregister` translates the pointer, calls `hipHostUnregister`, frees the
backing buffer, and removes both map entries.

### Sysmem Update Tracking (planned — not yet implemented)

**Problem:** `hipHostRegister` captures the buffer contents at registration time only.
After that, the CPU may modify the buffer without going through any HIP API:

```
hipHostRegister(ptr, sz) ← blob captured (initial state)
// CPU writes to ptr[] ← invisible to capture layer
hipMemcpyAsync(d, ptr, sz, H2D, stream) ← H2D blob captures current contents ✓
// CPU writes to ptr[] again
hipModuleLaunchKernel(...) ← kernel reads ptr directly via mapped flag ✗ stale
```

H2D memcpy is already handled — the src buffer is re-snapshotted at each call regardless
of registration. The gap is direct GPU reads from registered host memory (mapped flag)
after a CPU write that was not routed through a memcpy.

**Planned design:**

1. New synthetic event `HRR_SYSMEM_UPDATE` (not a real HIP API):
fields: `hostPtr u64`, `sizeBytes u64`, `blob_hash_lo u64`, `blob_hash_hi u64`.

2. Per-region `last_hash` stored alongside `g_pinned_reg_map`. Before each H2D memcpy src
check and before each kernel launch for pointer args in registered ranges: hash the
current contents, compare with `last_hash`. Emit `HRR_SYSMEM_UPDATE` + write blob only
if hash changed. Content-addressed storage deduplicates unchanged regions automatically.

3. Optional sync-gated dirty flag: set `dirty=true` for all registered regions after any
`hipStreamSynchronize` / `hipDeviceSynchronize` / `hipEventSynchronize`. Only hash
(step 2) if `dirty==true`. Avoids hashing in pure-GPU loops where the CPU never
touches the buffer between launches. Falls back to always-hash if sync events are
not captured.

4. At replay: `HRR_SYSMEM_UPDATE` handler `memcpy`s the blob into the live registered
buffer, ordered by sequence ID like all other events.
### Pinned Host Snapshots

A kernel can read pinned host memory that the host filled with plain stores, which no HIP
call records; replay would run it on whatever its fresh buffer holds. Capture tracks the
ranges of `hipHostMalloc` and `hipHostRegister` (dropped on `hipHostFree`, `hipFreeHost`,
`hipFree` and `hipHostUnregister`). Before each launch it reads the 8-byte-aligned words of
the arguments, by-value structs included; for each tracked allocation a word points into,
it copies the whole allocation under the tracking lock, writes the copy as a blob and adds a
record (base, offset 0, length, blob hash, direction 0) to the launch event, unless the hash
and the launch stream equal those of the allocation's last record. Struct words pointing into such an allocation
are flagged for translation like
embedded device pointers, and replay translates them without the device-address shape
check. Replay checks each record names the base of a pinned host
allocation it made, lies inside it and matches its blob's length, refusing and counting it
otherwise; then it queues a host function that copies the blobs in on the launch stream,
ahead of the kernel, without waiting for the stream. `hipHostFree` and `hipHostUnregister`
wait for every stream, so the allocation outlives a queued copy. The summary prints
restored, refused and skipped counts.

Kept deliberately simple:

- Capture does not wait for the stream: the bytes are those at enqueue time, which is what
the kernel reads unless earlier queued work still changes them. If that work is writing
the allocation during the copy, replay restores the partial bytes over its output.
- `HIP_HRR_HOST_SNAPSHOTS=0` turns snapshots off; an allocation larger than
`HIP_HRR_HOST_SNAPSHOT_MAX_MB` (64) is not snapshotted and is listed under
`unreplayable_apis`.
- Replay restores nothing for a launch replayed into a capturing stream, and capture does
not snapshot at graph launch. A record for a launch into a capturing stream does not
become the allocation's last, so the next launch outside the capture records again.
- `hipHostAlloc`, `hipMallocHost` and `hipMemAllocHost` are not tracked, since replay makes
no buffer for them, nor is a `hipHostRegister` device alias that differs from the host
address.
- An unchanged allocation gets no record, so device writes into it between two launches
stay in replay's buffer as they did at capture; a host store that puts back exactly the
last recorded bytes after such a device write is missed.
- A restore is ordered on the launch stream only; work on other streams reading the buffer
is not waited for, which is why a launch on another stream records again.

**Threat model.** The snapshots put into the archive host memory the application never
handed to a HIP call: for a serving stack, token IDs, sampling state and request metadata,
including stale bytes from earlier requests in a reused buffer. The archive must be
handled as at least as sensitive as the workload's data. Replay treats every record as
untrusted input and writes only inside a pinned host allocation it made itself, never
through a recorded address.

## Kernel Argument Capture

Expand Down Expand Up @@ -659,10 +672,9 @@ This detector is a value-based heuristic with three deliberate properties:
previously caused occasional replay faults on ATen elementwise kernels.
`HIP_HRR_PTR_RELAX=1` disables the replay-side guard for debugging.

**Scope:** only `hipMemoryTypeDevice`/`Unified` words are flagged. A pinned/host
(`hipMemoryTypeHost`) pointer embedded by value keeps its capture-time host VA at
replay (invalid in the replay process); translating embedded host pointers is
intentionally out of scope.
**Scope:** `hipMemoryTypeDevice`/`Unified` words are flagged, and so are words
pointing into a pinned allocation the launch snapshots (see Pinned Host Snapshots).
Any other embedded host pointer keeps its capture-time VA at replay.

A per-launch `co_hash` (the FNV-1a-128 hash of the owning code object) **is** recorded
in kernel launch events. Playback resolves kernels by `(co_hash, name)`: it first looks
Expand Down Expand Up @@ -1273,7 +1285,8 @@ Sysmem capture is a fundamentally hard problem. Two approaches exist:
and emit `HRR_SYSMEM_UPDATE` synthetic events only when content changes. This is
significantly more complex to implement and may not be fully feasible on all platforms.

Neither approach is currently implemented.
Pinned host memory a kernel reads is now snapshotted before each launch; see Pinned Host
Snapshots above.

### Partial-Replay APIs — H2D and D2H Work, Kernel Launches Do Not

Expand Down Expand Up @@ -1491,7 +1504,7 @@ The event wire format (finding H5):
shrinking `reserved` to 2 bytes), so kernel launches with large serialized payloads
(many args / long mangled names / large by-value structs) up to ~4 GiB are recorded
normally instead of being dropped at 65535 bytes. This is the change that bumped
`HRR_VERSION` to 4; the current version is 9, see Archive Format above. The
`HRR_VERSION` to 4; the current version is 10, see Archive Format above. The
writer's single-record buffer path now writes any oversized record straight through.
- **Per-argument size limit (64 KiB) now fails loudly.** Each kernel arg's size is still
a `uint16_t`. A by-value struct argument ≥ 64 KiB cannot be represented, so the launch
Expand Down
1 change: 1 addition & 0 deletions projects/hrr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ User-facing capture, replay, and validation knobs. Implementation details can be
|----------|---------|---------|
| `HIP_HRR_CAPTURE_OUTPUT` | *(unset)* | Enable capture; path to the `.hrr` archive directory. An empty or blank value leaves capture off. Ignored, with one line on stderr, when Linux starts the program in secure-execution mode (set-user-ID, set-group-ID, file capabilities or an LSM transition) |
| `HIP_HRR_DEBUG_ARGS` | off | Dump every captured kernel arg to the log (debug / provenance). Only a Debug build of the HIP runtime (no `NDEBUG`) reads it; a release build ignores it. Either way capture says on stderr whether it reads it, whatever the log level |
| `HIP_HRR_HOST_SNAPSHOTS` | on | Record the pinned host memory each kernel reads before the launch, so replay can write it back. `HIP_HRR_HOST_SNAPSHOT_MAX_MB` (default `64`) caps the allocation size. The archive then holds that host memory (see DESIGN.md, Pinned Host Snapshots) |

On Linux the archive is readable only by the user who captured it (directories 0700, files 0600). If the archive cannot be set up, capture is disabled with a `[HRR capture] Capture disabled` message on stderr and the application runs normally.

Expand Down
6 changes: 4 additions & 2 deletions projects/hrr/include/hrr/hrr_api_args.h
Original file line number Diff line number Diff line change
Expand Up @@ -83,8 +83,10 @@
* reader decodes those events as the wrong API.
* v9: hipKernelSetAttributeForDevice was appended to HipDispatchTable,
* taking the first compiler ID and shifting the compiler API IDs up by one.
* A v8 reader would decode those events as the wrong API. */
#define HRR_VERSION ((uint16_t)9u)
* A v8 reader would decode those events as the wrong API.
* v10: a kernel launch event carries pinned host snapshot records after its
* arguments (num_snapshots was always 0). A v9 reader would not restore them. */
#define HRR_VERSION ((uint16_t)10u)

/* Written once at byte 0 of events.bin. */
#pragma pack(push, 1)
Expand Down
85 changes: 79 additions & 6 deletions projects/hrr/playback/hip_playback.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -467,11 +467,64 @@ hipFunction_t PlaybackContext::resolve_replacement(const std::string& kernel_nam
// [+12..23] block[3] (uint32_t[3])
// [+24..27] shared_mem (uint32_t)
// [+28..29] num_args (uint16_t)
// [+30..31] num_snapshots (uint16_t, always 0)
// [+30..31] num_snapshots (uint16_t)
// per arg: u8 value_kind, u16 size, <size> bytes data
// value_kind: 0=scalar, 1=gpu-pointer, 2=hidden,
// 3=scalar/struct with embedded gpu pointer(s);
// kind 3 appends u16 n_ptrs then n_ptrs * u16 byte offsets.
// per snapshot (v10): u64 ptr_handle, u64 offset, u64 length, u64 hash_lo,
// u64 hash_hi, u8 direction (0)
// u32 num_attrs, u32 attr stride, num_attrs * stride bytes

// (destination, blob, length) of each copy; blob_cache_ is never evicted.
using HostSnapCopies = std::vector<std::tuple<void*, const void*, size_t>>;
static void host_snapshot_copy(void* data) {
auto* copies = static_cast<HostSnapCopies*>(data);
for (auto& [dst, src, len] : *copies) memcpy(dst, src, len);
delete copies;
}

// Write a launch's pinned host snapshots back when its stream reaches it, by a
// host function queued ahead of the kernel. The replay thread does not wait
// for the stream, which may wait on work it has yet to submit. hipHostFree and
// hipHostUnregister wait for every stream, so the allocation outlives the copy.
// Nothing is restored when the stream captures into a graph, where the kernel
// runs only when the graph is launched.
static hipError_t restore_host_snapshots(PlaybackContext& ctx, const uint8_t* rec,
uint16_t n, hipStream_t stream,
const std::string& kernel_name) {
hipStreamCaptureStatus cs = hipStreamCaptureStatusNone;
if (hipStreamIsCapturing(stream, &cs) != hipSuccess || cs != hipStreamCaptureStatusNone) {
ctx.host_snapshots_skipped += n;
return hipSuccess;
}
auto copies = std::make_unique<HostSnapCopies>();
for (uint16_t i = 0; i < n; i++, rec += kHostSnapRecordSize) {
uint64_t hash[2];
memcpy(hash, rec + 24, 16);
size_t blob_len = 0;
const void* blob = ctx.load_blob(hash[0], hash[1], &blob_len);
// Refused for the host memory cap or a read error, not merely absent.
if (!blob && ctx.fatal_error.load(std::memory_order_acquire))
return hipErrorUnknown;
const char* why = nullptr;
void* dst = hrr_host_snapshot_target(ctx, rec, blob, blob_len, &why);
if (!dst) {
ctx.host_snapshots_refused++;
fprintf(stderr, "[HRR] '%s': pinned host snapshot %u %s — refused\n",
compact_kernel_name(kernel_name).c_str(), i, why);
continue;
}
copies->emplace_back(dst, blob, blob_len);
}
if (copies->empty()) return hipSuccess;
const size_t k = copies->size();
hipError_t r = HRR_HIP_CHECK(hipLaunchHostFunc(stream, host_snapshot_copy, copies.get()));
if (r != hipSuccess) return r;
copies.release();
ctx.host_snapshots_restored += k;
return hipSuccess;
}

// ext_global_worksize: the captured grid[] holds *global work-item counts*
// (HSA/OpenCL semantics, as passed to hipExtModuleLaunchKernel), NOT workgroup
Expand Down Expand Up @@ -843,11 +896,15 @@ static void decode_kernel_args(
// may be a genuine scalar (a large count, a double, a packed value)
// that was mis-flagged — overwriting it with null would silently
// corrupt it. Only rewrite the word when it actually resolves.
auto try_translate_word = [&](size_t off, const char* src) -> bool {
auto try_translate_word = [&](size_t off, const char* src, bool flagged) -> bool {
if (off + 8 > arg_size) return false;
uint64_t rec_ptr; memcpy(&rec_ptr, data + off, 8);
// null, small, or a packed-integer false positive
if (!va_shaped(rec_ptr)) return false;
// null, small, or a packed-integer false positive. A word capture
// flagged inside a pinned host allocation need not look like a
// device VA: a host address can have small low 32 bits.
if (!va_shaped(rec_ptr) &&
!(flagged && ctx.in_pinned_host(rec_ptr)))
return false;
void* live = ctx.translate_ptr(rec_ptr);
// Same bypassed-segment materialisation as a whole-pointer
// argument: a struct field pointing into memory HIP never saw
Expand All @@ -872,7 +929,7 @@ static void decode_kernel_args(
// First honor the capture-recorded offsets (these may be unaligned).
std::vector<char> handled(arg_size, 0);
for (uint16_t off : ptr_offsets) {
if (try_translate_word(off, "captured"))
if (try_translate_word(off, "captured", true))
for (int b = 0; b < 8 && static_cast<size_t>(off) + b < arg_size; b++)
handled[off + b] = 1;
else if (ctx.verbose)
Expand All @@ -895,7 +952,7 @@ static void decode_kernel_args(
(std::getenv("HIP_HRR_REPLAY_NO_RESCAN") != nullptr);
for (size_t off = 0; !no_rescan && off + 8 <= arg_size; ) {
if (handled[off]) { off += 1; continue; }
if (try_translate_word(off, "rescan")) off += 8;
if (try_translate_word(off, "rescan", false)) off += 8;
else off += 1;
}
} else {
Expand Down Expand Up @@ -1046,6 +1103,15 @@ static hipError_t replay_kernel_launch(PlaybackContext& ctx, const uint8_t* pl,
arg_storage, &rls, dbg_dump_ptrs ? &dbg_ptrs : nullptr,
dbg_dump_ptrs ? &dbg_args : nullptr);

const uint8_t* snapshots = p;
if (!hrr_host_snapshots_fit(p, end, num_snapshots)) {
fprintf(stderr, "[HRR] '%s': %u pinned host snapshot records run past the "
"event — refused\n", kernel_name.c_str(), num_snapshots);
ctx.host_snapshots_refused += num_snapshots;
num_snapshots = 0;
}
p += static_cast<size_t>(num_snapshots) * kHostSnapRecordSize;

// Launch-attribute tail: u32 count, u32 per-entry stride, then the
// entries. Every launch payload carries it (count 0 for a plain launch).
std::vector<hipLaunchAttribute> launch_attrs;
Expand Down Expand Up @@ -1093,6 +1159,13 @@ static hipError_t replay_kernel_launch(PlaybackContext& ctx, const uint8_t* pl,
hipStream_t stream = ctx.translate_stream(stream_rec);
const size_t kernel_ordinal =
ctx.kernels_launched.load(std::memory_order_relaxed) + 1;
if (hipError_t sr = num_snapshots ? restore_host_snapshots(ctx, snapshots, num_snapshots,
stream, kernel_name)
: hipSuccess;
sr != hipSuccess) {
for (const auto& g : rls.guards) hrr_block_guard_teardown(g);
return sr;
}

if (ctx.audit_host_args) {
for (auto& [idx, rec, live] : dbg_ptrs) {
Expand Down
Loading
Loading