Repository navigation
test(rocprofiler-sdk): range replay week of 2026-10-12: C8 (stacked) - #13272
Draft
amd-vkale wants to merge 2 commits into
Conversation
…nge replay window (C8) Add tests/range-replay-concurrency. The replay window restores the agent's whole tracked inventory before every pass and at its end, so a second thread's work on the same agent has to wait for the window rather than run inside it, or its writes are restored away. A graph launch from another thread used to take the queue fast path and could do exactly that. The application holds 64 MiB of ballast so each restore takes time, records a three-kernel range on its main thread, and replays it six times. A second thread warms up a one-kernel graph before the range opens and waits. The tool releases it at the first replayed pass, after the range has left its thread, so the launches cannot decline it as a concurrent dispatch. At the last pass the tool waits until the thread has begun launching, so the launches are issued while the window still holds the agent. The test passes only if the range was replayed with divergence checking reporting no changed region, the range's result is intact, and the graph thread's counter holds all 64 increments plus the warm-up. A launch that ran inside the window would have its increment restored away and show up as a divergent region. A run without a GPU skips. Co-authored-by: Vivek Kale <vivek.kale@amd.com>
The coverage table gains the concurrency test, and every bug fixed in #10864 that had no regression test (copies unseen without copy tracing, hsa_amd_memory_async_batch_copy bypassing the decline, and another thread's graph launch inside a replay window) now names the test that guards it. Co-authored-by: Vivek Kale <vivek.kale@amd.com>
1 task
Contributor
|
This pull request has been inactive for 25 days and will be marked as stale. If you would like to keep this PR open, please:
This PR will be automatically closed in 5 days if no further activity occurs. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Week two of the range replay test roadmap (
source/docs/conceptual/range_replay/range_replay_test_plan.md): C8. A replay window restores the agent's whole tracked inventory before every pass and at its end, so another thread's work on the same agent must wait for the window instead of running inside it. Otherwise its writes are restored away. #10864 fixed a case where a graph launched from another thread took the queue fast path and could run inside the window; this is its regression test. With it, all three bugs fixed in #10864 without a test (C1, C2 and C8) have one.Stacked on #12815 (week one: C1, C2).
Technical Details
New
tests/range-replay-concurrency/, in the style oftests/range-replay-local-context. Its handshake followstests/kernel-replay-concurrency, a block of atomics the tool exports and the application finds withdlsym:main.cpp). The main thread holds 64 MiB of ballast, so each restore takes time. It records a range of three dependent kernels and replays it inrocprofiler_range_replay_end(). A second thread captures a one-kernel graph that increments its own counter, launches it once before the range opens, then waits. Every allocation and the graph setup happen before the range opens, because an allocation would decline the range, and so would a dispatch from the second thread while the range is recording.client.cpp). It asks for six passes and drives the handshake:REPLAYEDwith the expected pass and dispatch counts anddivergence_count == 0(ROCPROF_RANGE_REPLAY_VERIFY=1). The handshake overlapped the window. The application's result is intact, and the counter holds all 64 increments plus the warm-up. A launch that ran inside the window would have its increment restored away, and would also appear as a divergent region.The roadmap's coverage table gains the test, and its list of untested #10864 bugs now names the test that guards each one.
Issue Tracking
Related: #10864 (range replay, whose replay window this test exercises against another thread).
Test Plan
test-range-replay-concurrent-graph-launchon the mi325 CI jobs.-W -Wall -Wextra -Wshadow -Werroragainst the in-tree headers;Test Result
CI for this PR is pending. Nothing could be run on a GPU locally, and the device code was not compiled here, because the in-tree HIP headers need a newer clang than the one on the build machine.
Week one (#12815,
1417207891): C1 and C2 passed on every mi325 job (rhel-8.8, rhel-9.5, sles-15.6, ubuntu-22.04 twice, and code coverage). Its red jobs came from other tests:rocprofv3-test-kernel-replay-validate(ubuntu-22.04): pass 0 counted 81911 SQ_WAVES against 16384 in each replayed pass, well beyond the 8-wave tolerance. That is not the few-wave spread develop's runs showed; develop's own CI has not reached tests since 2026-10-09 (see below), so newer develop runs could not be compared.rocprofv3-test-conversion-script-validatepmc1, both prefixes (sles-15.6): a counter value of 0.0 where every value must be positive.#10864 is unchanged at
0844152fe6; its CI was summarized in #12815.Separately, develop's own SDK build has failed since 2026-10-09: ROCr #13129 added
HSA_ACCESS_PERMISSION_EX, whichhsa/utils.hppdoes not handle under-Werror=switch. This stack has not merged develop since then, so its CI is unaffected.Submission Checklist