Repository navigation
perf(rocprofiler-sdk): GPU-backed snapshots and faster divergence-check hashing for range replay - #14808
Draft
minseobshin11 wants to merge 11 commits into
Draft
perf(rocprofiler-sdk): GPU-backed snapshots and faster divergence-check hashing for range replay#14808minseobshin11 wants to merge 11 commits into
minseobshin11 wants to merge 11 commits into
Conversation
Hold the allocation inventory lock until the restore blit and dependent replay dispatch complete, preventing concurrent frees without adding host-side synchronization. Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Flatten discontiguous snapshot regions into one compact 16-byte work stream so restore avoids per-tile descriptor searches and skewed grids. Add deterministic coverage for cross-region assignments and partial items. Co-authored-by: Cursor <cursoragent@cursor.com>
- compose replay snapshot storage tiers Plan whole regions against bounded GPU and pinned-host budgets before capture, using a variadic fallback chain ending in a null allocator. Materialize one arena per tier and verify mixed restore ordering, allocation rollback, and memory-pressure behavior. Improvements to come in subsequent commits. This is MVP for host-fallback. Co-authored-by: Cursor <cursoragent@cursor.com>
…king The host-fallback MVP described each storage tier twice: a planning-only allocator chain chose placement, then separate GPU and pinned-host allocation functions re-applied their own admission checks, and materialization was hard-coded to two tiers. Make each tier a single storage::arena<Policy, BackingAllocator> that owns its capacity policy and its physical backing: - planning::bounded_policy handles alignment, capacity, and offsets. - One hsa_pool_allocator template replaces the separate GPU and pinned allocation paths. Arena constructor arguments are forwarded to the backing allocator unchanged. - storage::fallback_allocator chains arenas and must end in the terminal null_allocator. Placements record the tier's storage index instead of a hard-coded storage kind. - Commit allocates only tiers that received reservations, and on failure rolls back already committed tiers in reverse order. GPU headroom and host-memory limits are now applied once, when planning budgets are computed. Allocation success remains authoritative, and a failed GPU commit still replans everything onto pinned host. Placement is unchanged: whole regions go to GPU first, then pinned host. Remove the unused global logical-offset mapper, the planning-only allocator aliases, and snapshot agent fields that nothing read. Rename the capture context's pinned_agent to cpu_agent. Move planner tests out of blit_copy.cpp into a GPU-independent kernel-replay-snapshot-storage-test covering fallback, alignment, overflow, rollback, three-tier chains, and move-only backing arguments. Co-authored-by: Cursor <cursoragent@cursor.com>
bounded_policy both sized requests and read the arena's cursor, so a new layout meant rewriting every policy. Give each tier three parts: - Admission: how much of a request the tier takes, given only what is available (whole_region; prefix_chunking splits on a granule). - Suballocator: where it goes. bump_suballocator commits one backing of exactly the used size. - Backing allocator: where memory comes from. hsa_pool_allocator now takes its storage kind and retain mode at runtime. arena<Admission, Suballocator> clamps admission to what is left, and fallback re-asks a tier until it declines, so a region can span tiers. Requests carry region_offset and priority; production still sends one request per whole region, so placement is unchanged. plan_and_commit disables a tier whose commit fails and replans, replacing the hard-coded GPU-to-pinned retry. Replace the cacheable flag with retain_mode and keep both GPU and pinned arenas across replays: one high-water arena per pool, reused when big enough, otherwise freed before the larger allocation. Retained pinned memory stays pinned until replaced or the process exits. Budgets count the retained arena as available: - GPU: min(alloc max, half of VRAM, retained + 95% of free) - Pinned: min(alloc max, retained + half of MemAvailable) BabelStream, 3 x 6 GiB behind a 221 GiB VRAM ballast (6 GiB spills to pinned): only the first replay pays the ~2 s pinned allocation, so later replays drop from ~3.3 s to ~0.63 s. All-GPU runs are unchanged. Rewrite the 11 CPU-only kernel-replay-snapshot-storage-test cases: chunking, priority, fragmented tiers, property-selected admission, replan and rollback, move-only backings, near-SIZE_MAX capacities. Co-authored-by: Cursor <cursoragent@cursor.com>
Every wave of the restore blit reads its arguments and copy descriptors. They lived in the host kernarg pool, so each restore depended on PCIe read latency: a 384 MiB restore took 240-510 us depending on how long the host had been idle. Allocate them from the GPU's pool when the CPU can map it, fence and read back the writes before the packet is published, and fall back to the host kernarg pool otherwise. The same restore now takes ~180 us, cold or warm. This impacts just the blit copy kernel, not the replayed kernel's own kernarg. Co-authored-by: Minseob Shin <minseob.shin11@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…ce check and kernarg retention
github-actions
Bot
added
documentation
Contributor
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Range replay re-executes a recorded range of kernel dispatches so counters can be re-collected without re-running the application. Two cost problems in the current implementation:
ROCPROF_RANGE_REPLAY_VERIFY=1) re-hashes two full snapshots per replay window. At a 256 MB footprint that check costs 755 ms of a 939 ms window (80%), making verify unusable at realistic footprints.Technical Details
Where the window goes
Stage breakdown of one replay window on this branch (verify off), MI300X (gfx942), medians of 12 windows. This table describes the resulting path; each optimization below has its own before/after measurement.
Reading it: the snapshot/restore stages are cheap because of optimization 1 (the device-resident snapshot path below). The two largest rows are not tool-optimizable: drain is the application's own kernels - the work the replay exists to re-measure - and packet submit (~1.5 us per replayed dispatch) is the SDK's instrumented dispatch path, shared with every application dispatch; optimizing it is an SDK-wide tracing project, deliberately out of scope. The one stage where the tool itself dominated was verify - 755 of 939 ms at 256 MB before this PR - which is what optimization 2 fixes.
1. GPU-backed snapshots (first four commits)
What changed. The old path captured the snapshot into host memory (a device-to-host readback per range) and restored it with host-to-device copies per pass - both scale with the snapshot footprint. The new path keeps the snapshot in device memory: captured on-device, restored by batched blit kernels on the GPU (submitted asynchronously), with automatic gpu/cpu placement and a host fallback when device staging spills. The range-replay-specific
replay_barrier_and_waitinhsa/replay_window.cppis removed;range_replay/executor.cppnow captures and restores throughkernel_replay/memory_snapshot. Kernarg staging is unchanged on both sides of this comparison (the agent's kernarg pool, retained across ranges); the device-staged variant with an explicit store fence is tracked separately in #13770.Before/after (same client and shapes on both builds, SDK phase lines, per-range medians of 3 runs x 9 ranges; both builds stage kernargs in host memory, so the snapshot path is the only difference):
The base path's capture and restore grow with the footprint (~2 GB/s and ~7 GB/s effective); this PR's capture stays under 0.5 ms and its restore is flat (~4-5 ms) for the same captured byte count. Window ratios: 1.7x / 11.7x / 35x at 4 / 64 / 256 MB. Caveat, stated rather than implied: this is a build-vs-build comparison, so the ratios measure the branch as a whole - but both sides stage kernargs identically (host memory), so the snapshot path is the dominant difference, and the restore's fixed ~4 ms blit cost is visible even at 4 MB where the base path was cheaper.
2. Divergence check (
ROCPROF_RANGE_REPLAY_VERIFY=1)The verify path digests the range-exit snapshot and the post-replay snapshot and counts divergent regions. The cost of one 256 MB digest decomposes as (standalone microbenchmark):
Two independent changes in
range_replay/digest.cpp/executor.cpp:static thread_localgrow-only scratch, removing the per-call zero-fill and page faults. Safe without clearing because every region is covered by exactly one extent, so every byte is overwritten by the readback before it is hashed.crc32lanes (runtime-probed, FNV-1a fallback kept). Safe because digests are compared only within one process by one binary and never persisted; the hash function may change freely between releases.Verify-path cost, medians over 12 windows:
Window with verify on at 256 MB: 939 to 47 ms (20x). The result is readback-bound (18.5 GB/s, the DMA floor) for a host-side hash. The verify-off path is unchanged within noise.
Future work: hashing on the GPU itself - the snapshot is already device-resident, so a blit-launched reduction could avoid the 18 GB/s readback entirely, an estimated ~10x beyond T1+T2. It requires new device code and a new correctness surface for an opt-in path that already costs ~29 ms per range, so it is deliberately not attempted here.
Docs: the soundness page's divergence-check section is updated for the new hash and the never-persisted rationale.
Issue Tracking
Related: #10864 (base branch), #10916 (kernel-replay snapshot machinery).
Test Plan
ctest -L range-replay(unit, integration, and the two nightly perf tests)Test Result
divergence=0on all verify-enabled runs.Submission Checklist