Repository navigation
fix(clr): give graph signal sets back to a device pool instead of destroying them - #14878
Merged
Merged
Conversation
✅ All Policy Checks Passed
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
mangupta
reviewed
Nov 3, 2026
…troying them
Each graph executable takes its signal sets from GraphSignalManager,
which pre-creates 16 at instantiation and creates another on the launch
path whenever more launches are in flight. Each set was its own signal
batch, an allocation in device memory, and destroying the executable
destroyed every set. The last destroy of a batch's signals frees the
allocation with an unmap from the GPU, about 230 us per set in PyFR's
graphs.
With DEBUG_CLR_DEV_SIGNALS=1 those destroys landed in later graph
launches. RetiringSignalTracker keeps a reference to each signal its
GPU waits depend on, until the waits have passed, so it often held the
last reference to a destroyed graph's signals. It releases them from
Reclaim, 64 at a time, inside whichever hipGraphLaunch runs next:
PyFR's Couette graphs stalled for 4 to 15 ms every 16 launches after
another graph was destroyed. Without device signals the same destroys
ran inside hipGraphExecDestroy.
Devices now keep a pool of armed graph signals. AcquireHwEvents takes a
set from it, creating signals in batches of at least 256 when it runs
short, and ReturnHwEvents gives a set back. GraphSignalManager takes its
pre-created sets and the sets for extra launches in flight from the
pool, and returns them when the executable is destroyed or rebuilt for
a different set size. The pool holds a reference to each signal, so the
tracker's release never destroys one, and the pool's signals are only
destroyed with the device.
PyFR Couette graph launches, DEBUG_HIP_GRAPH_MIN_OVERLAP=0, MI300X,
DEBUG_CLR_DEV_SIGNALS=1:
graph avg launch us launches over 500 us
before after before after
graph_3 30.9 31.1 0 0
graph_5 27.7 8.9 4 0
graph_7 44.6 15.7 3 0
Medians, device time and the step simulation are unchanged. Without
device signals, the run creates 11 signal batches instead of 456, and
instantiation is faster (graph_7: 13.2 to 5.2 ms).
saleelk
force-pushed
the
users/saleelk/clr-graph-signal-pool
branch
from
November 3, 2026 02:35
a3b383a to
c93560a
Compare
Contributor
|
🤖 AI-assisted pre-review (for @mangupta) Re-reviewed the updated head ( |
gandryey
approved these changes
Nov 3, 2026
Contributor
TheRock Submodule Bump ActivityNewest first
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Each graph executable takes its signal sets from GraphSignalManager,
which pre-creates 16 at instantiation and creates another on the launch
path whenever more launches are in flight. Each set was its own signal
batch, an allocation in device memory, and destroying the executable
destroyed every set. The last destroy of a batch's signals frees the
allocation with an unmap from the GPU, about 230 us per set in PyFR's
graphs.
With DEBUG_CLR_DEV_SIGNALS=1 those destroys landed in later graph
launches. RetiringSignalTracker keeps a reference to each signal its
GPU waits depend on, until the waits have passed, so it often held the
last reference to a destroyed graph's signals. It releases them from
Reclaim, 64 at a time, inside whichever hipGraphLaunch runs next:
PyFR's Couette graphs stalled for 4 to 15 ms every 16 launches after
another graph was destroyed. Without device signals the same destroys
ran inside hipGraphExecDestroy.
Devices now keep a pool of armed graph signals. AcquireHwEvents takes a
set from it, creating signals in batches of at least 256 when it runs
short, and ReturnHwEvents gives a set back. GraphSignalManager takes its
pre-created sets and the sets for extra launches in flight from the
pool, and returns them when the executable is destroyed or rebuilt for
a different set size. The pool holds a reference to each signal, so the
tracker's release never destroys one, and the pool's signals are only
destroyed with the device.
PyFR Couette graph launches, DEBUG_HIP_GRAPH_MIN_OVERLAP=0, MI300X,
DEBUG_CLR_DEV_SIGNALS=1:
graph avg launch us launches over 500 us
before after before after
graph_3 30.9 31.1 0 0
graph_5 27.7 8.9 4 0
graph_7 44.6 15.7 3 0
Medians, device time and the step simulation are unchanged. Without
device signals, the run creates 11 signal batches instead of 456, and
instantiation is faster (graph_7: 13.2 to 5.2 ms).
Testing
DEBUG_HIP_GRAPH_MIN_OVERLAP=0 and DEBUG_CLR_DEV_SIGNALS=0 and 1: the
numbers above. An LD_PRELOAD probe that times every hipGraphLaunch
finds no launch over 500 us with this change; on develop the slow
launches spent their time in hsa_signal_destroy, called from
RetiringSignalTracker::ReleasePins.
JIRA ID : AIRUNTIME-2943