[amd-vkale]

PR 5 of 5 in the kernel replay host-side perf stack, stacked on #12609. Review only its own two commits, 80fc304b4a (code) and 381cb13b24 (docs); the merge commits after them only bring in fixes from the PRs below it; full diff against develop. Code permalinks pin to 80fc304b4a, docs to 381cb13b24.

Motivation

Underlying Problem

The snapshot is a local of the replay window. Before this change, every replayed dispatch built it in fresh std::vector<char> buffers, one per tracked region, and freed them all when the window closed. For a gigabyte-scale footprint that is expensive before a single byte is copied:

• resize() zero-fills the whole footprint, even though snap() overwrites every byte straight away.
• glibc serves blocks above its mmap threshold (32 MiB at most) with a fresh mapping, so each dispatch first-touch faults the whole footprint in, page by page.
• Freeing returns that mapping to the OS, so the next dispatch faults it all in again.

Measured on the host alone, that cycle costs about 0.35 ms per MiB of footprint per dispatch: roughly 20 ms at 64 MiB, 94 ms at 256 MiB and 375 ms at 1 GiB, all before any copy. ROCr's hsa_memory_copy then locks (pins and GPU-maps) that pageable memory for every copy, which on fresh pages also means faulting them in.

The performance assessment on develop lists "The staging buffer is allocated and freed per dispatch" as an open problem, and proposes exactly this fix: a per-agent staging buffer retained across dispatches.

Notes on Bigger Picture

This is the host-side counterpart of the device backing cache in #10916, without moving the snapshot into device memory. The device-side work, with snapshots in GPU memory and blit-based restores, stays in #10916 with Mythreya Kuricheti. The two compose: on that design, host staging remains the fallback tier when GPU memory is short. With #12608, the saving shows in the "snapshot" phase of the INFO line.

Technical Details

Snapshot storage comes from a per-agent pool and goes back to it, and the pool never holds more than the last snapshot used.

flowchart LR
  S["snap(agent)"] --> A{"retained buffer with capacity in [size, 2*size]?"}
  A -- yes --> T["take it (best fit)"]
  A -- no --> N["allocate size bytes, no zero-fill"]
  T --> C["copy device to host"]
  N --> C
  C --> P["passes and restores use the snapshot"]
  P --> D["window closes, snapshot destroyed: storage returns to the agent's pool"]
  S -. "when capture ends" .-> TR["free retained storage this snap did not take"]

Loading

Code

• staging_buffer_t (memory_snapshot.hpp L44-L83) is a move-only RAII owner of one region's host storage. It exposes acquire(agent, size), data() and size(). mem_block_t::host_copy changes from std::vector<char> to it. It keeps data() and size(), so restore(), feat(rocprofiler-sdk): report kernel replay phase timings at INFO #12608's byte count and the existing snapshot tests are unchanged.
• The pool is a Synchronized map from agent handle to a capacity-ordered std::multimap of free buffers. It is deliberately leaked, so a buffer destroyed during static destruction still has a pool to return to.
• acquire() takes the best-fitting retained buffer, capped at twice the request. A small region cannot take a large buffer that a later, larger region would then have to allocate again. When nothing fits it allocates with new (std::nothrow) char[size], which does not zero-fill. A failed allocation still reports memory pressure, so snap() declines replay exactly as before.
• release() returns the storage to the owning agent's free list on destruction or move-assignment. If the free list cannot grow (bad_alloc), the storage is simply freed.
• The trim in snap() is a scope guard. When capture ends, successful or not, it frees whatever retained storage this snapshot did not take: buffers for regions that were freed or shrank. Between dispatches the pool therefore holds at most the last snapshot's footprint.
• retained_staging_bytes(agent) reports what the pool holds for an agent, for the tests.

Behaviour changes

• Host memory is now held between replayed dispatches. It is bounded by the most recent snapshot's footprint; before, it was freed when each window closed. The docs' host-RAM limitation is updated to say so.
• Per-agent isolation is kept. Storage retained for one GPU is never handed to another, matching the agent-scoped snapshots and locks.
• The memory-pressure behaviour is unchanged. When an allocation fails, the snapshot is declined and the dispatch runs once instead of aborting the application.

Docs and CHANGELOG (second commit)

• The performance assessment moves "staging allocated and freed per dispatch" from open problem to fixed, with the measured costs.
• It corrects how hsa_memory_copy treats pageable memory. ROCr locks and unlocks the range for every copy, rather than staging through driver buffers. That is what makes pinning the now-retained buffers (hsa_amd_memory_lock) the next experiment, and a policy decision rather than a code change.
• It adds a "Measuring the phases" section for feat(rocprofiler-sdk): report kernel replay phase timings at INFO #12608's INFO timings.
• using-kernel-replay.rst and kernel_replay_memory_snapshot.md update the host-RAM limitation.
• The CHANGELOG entries for the whole stack land here: retained staging, the pooled drain signal (perf(rocprofiler-sdk): take the kernel replay drain signal from the signal pool #12607), the __hip_cuid_ skip (perf(rocprofiler-sdk): skip HIP's compilation-unit markers in the kernel replay snapshot #12609) and the INFO phase timings (feat(rocprofiler-sdk): report kernel replay phase timings at INFO #12608).

Reviewer Guide

1. acquire() never hands out a buffer more than twice the request, and allocates without zero-filling. snap() copies over every byte before anything reads it.
2. Ownership: a buffer returns its storage exactly once (destructor or move-assignment), moved-from buffers own nothing, and the pool is per agent.
3. The snap() trim runs on every exit (scope guard), including declined snapshots, so retained memory cannot grow past the last footprint.
4. Allocation failure still declines replay instead of aborting.
5. Static-destruction safety: the pool is leaked so late releases have somewhere to go.

Out of scope: pinning retained buffers (documented as the next experiment), device-memory backing (#10916), and copying only what a kernel writes (the assessment's largest open lever).

Issue Tracking

JIRA ID: AIPROFSDK-68

Test Plan

• New, GPU-free: kernel-replay-staging-test, registered in tests/CMakeLists.txt L119-L135. Each test uses its own agent handle, so pool state does not leak between tests.
  • acquire_sizes_storage_to_the_request: a fresh buffer is sized to the request and writable, and nothing is retained.
  • destroyed_storage_is_reused_by_the_next_acquire: the next snapshot of the same footprint gets the very same storage back.
  • reuse_is_best_fit_within_twice_the_request: a 40 KiB request does not take a 100 KiB buffer, but a 60 KiB request does.
  • pools_are_per_agent: storage retained for one agent is not handed to another.
  • moves_return_storage_exactly_once: move construction and assignment transfer ownership, and only the final owner returns the storage.
  • snap_frees_retained_storage_it_does_not_reuse: snap() trims retained storage even when it declines the snapshot, here because no HSA runtime is live.
• Existing: unit.kernel_replay_snapshot.*, including the 8, 32 and 128 MiB snapshot bandwidth tests and the restore tests, which now run on retained storage. Also the kernel replay integration tests and the rocprofv3 kernel replay tests, in CI.
• Manual: with feat(rocprofiler-sdk): report kernel replay phase timings at INFO #12608 applied, ROCPROFILER_LOG_LEVEL=info on a multi-dispatch replay run shows the "snapshot" phase drop after the first dispatch, because later dispatches reuse retained storage.

Test Result

• Applies cleanly on perf(rocprofiler-sdk): skip HIP's compilation-unit markers in the kernel replay snapshot #12609, and on top of the four remove-callbacks PRs.
• clang-format 11 and cmake-format 0.6.13 clean.
• Pending CI; treat the Checks tab as the source of truth.

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/rocm-systems/blob/develop/CONTRIBUTING.md.

Related PRs

• docs(rocprofiler-sdk): note why kernel replay keeps restoring constant symbols #12886 ← perf(rocprofiler-sdk): take the kernel replay drain signal from the signal pool #12607 ← feat(rocprofiler-sdk): report kernel replay phase timings at INFO #12608 ← perf(rocprofiler-sdk): skip HIP's compilation-unit markers in the kernel replay snapshot #12609 ← this PR (top of the stack). test(rocprofiler-sdk): add a kernel replay benchmark harness and benchmarking guide #12885 is the measurement harness.
• perf(rocprofiler-sdk): kernel replay phase 2: performance optimizations #10916: device-side snapshot and restore optimizations (Mythreya Kuricheti). This PR is the host-side counterpart of its device backing cache.
• feat(rocprofiler-sdk): kernel replay service (callback tracing) #7960: the kernel replay service whose snapshot this changes.

  

[github-actions]

This pull request has been inactive for 25 days and will be marked as stale.

If you would like to keep this PR open, please:

• Add new commits
• Add a comment explaining why it should remain open

This PR will be automatically closed in 5 days if no further activity occurs.

[github-actions]

This pull request has been automatically closed due to inactivity (30 days with no updates).

If you'd like to continue working on this, feel free to reopen the PR or create a new one.