Repository navigation
fix(hrr): keep a hipHostRegister replay buffer alive while a graph node points into it - #15100
Conversation
…de points into it A graph memcpy node keeps the host pointer it was built with. Replay freed the buffer behind a hipHostRegister on hipHostUnregister, and when the same address was registered again, so the next launch of that graph read or wrote freed heap. The register buffers now live in host_reg_blocks, kept with the per-address block mechanism the graph landing buffers use (host_block): registering the same address again reuses its block, an outgrown block is kept, unregister drops only the mapping, and every block stays charged to the host budget until the context is destroyed. A block whose unregister fails is not reused. AIRUNTIME-3081
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
There was a problem hiding this comment.
🟡 Changes recommended
Critical buffer-lifetime and alignment defects can free still-registered or GPU-in-use memory and cause corruption between otherwise independent cross-GPU buffers.
4 open issues
What changed in this PR
This PR extends the stable host-buffer lifetime introduced in #14156 to hipHostRegister replay buffers, preventing graph nodes from referencing freed storage after re-registration or unregister.
Changes:
- Shares context-owned block storage between registration and graph landing buffers, retaining outgrown blocks and their budget charges.
- Updates unregister and teardown handling to separate registration cleanup from buffer ownership.
- Adds GPU-dependent lifetime and failed-replacement tests, and updates documentation.
| File | Description |
|---|---|
projects/hrr/tests/unit/hrr_unregister_fail_test.cc |
Adds buffer-lifetime and failed-replacement regression tests. |
projects/hrr/tests/unit/hrr_host_budget_test.cc |
Updates comments describing retained registration charges. |
projects/hrr/playback/hrr_playback.cpp |
Removes direct registration-buffer frees during teardown. |
projects/hrr/playback/hip_playback.h |
Introduces shared host-block storage and retained budget accounting. |
projects/hrr/playback/hip_playback.cpp |
Reuses registration blocks and retains storage after unregister. |
projects/hrr/DESIGN.md |
Documents registration-buffer ownership and lifetime. |
Review effort: Balanced
danielbolsson
left a comment
There was a problem hiding this comment.
LGTM: host_block reuse keeps the registered buffer alive past hipHostUnregister, charged once, and the new GraphNodeKeepsBuffer test covers it. Nits: projects/hrr/playback/hrr_playback.cpp:2035 — the fatal-error return skips cleanup(), so ~PlaybackContext frees blocks still registered (unregister them first); projects/hrr/playback/hip_playback.h:925 — a throwing make_unique after charge_host leaves the charge taken, catch and release.
Host blocks are 64-byte aligned, an allocation failure returns hipErrorMemoryAllocation instead of throwing, and the fatal path exits without destroying the context while queued work may still use a block. AIRUNTIME-3081
TheRock Submodule Bump ActivityNewest first
|


Motivation
A graph memcpy node keeps the host pointer it was built with. Replay freed the buffer behind a
hipHostRegisteronhipHostUnregister, and again when the same address was registered a second time, so the next launch of that graph read or wrote freed heap. The copy size and direction come from the archive.Technical Details
host_reg_blocks, kept with the per-address block mechanism fix(hrr): keep graph landing buffers stable until teardown #14156 added for graph landing buffers. Its body moves intoPlaybackContext::host_block, shared by both maps (LandingBufferis renamedHostBlock).hipHostUnregisterdrops the mapping only. Every block stays charged to the host budget until the context is destroyed, so replay memory at an address grows only when a registration outgrows its block.remove_allocno longer gives back ahipHostRegistercharge. The blocks arenew[]arrays owned by the context, so theposix_memalign/_aligned_mallocallocation and its sixfree/_aligned_freesites are gone.Issue Tracking
JIRA ID: AIRUNTIME-3081
Test Plan
hrr_unregister_fail_test.cc(hrr-handler-tests, AddressSanitizer build):GraphNodeKeepsBuffer(register, take the pointer a memcpy node is given, register again, unregister, write through it) andFailedReplaceTakesNewBlock. Both need a GPU and skip without one.pAIRUNTIME-3081-0c8cd41on head 0c8cd41, base develop b09c3ab.Test Result
GraphNodeKeepsBufferaborts with heap-use-after-free andFailedReplaceTakesNewBlockfails; every other case passes.Submission Checklist