Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -12,20 +12,23 @@ sitting and can be merged or dropped on its own. The table below lists what has
| Unit, no GPU | `range_replay/tests/` (92 tests) | eligibility and decline bookkeeping, the code-object watermark, kernarg layout, pass packets, digests, kernarg block retention, the public ABI, the single-subscriber claim, CONFIG → PASS → CLOSE ordering, the queue hook's early returns |
| Samples, GPU | `samples/range_replay/` | a replayed range with `divergence_count == 0`, a tool that opts out, a multi-queue decline |
| Integration, GPU | `tests/range-replay-local-context/` | every per-dispatch service sees the replayed dispatches and honors per-pass toggles |
| Declines, GPU | `tests/range-replay-declines/` | a device-writing copy between a range's kernels, through `hipMemcpyAsync` (C1) or `hsa_amd_memory_async_batch_copy` (C2), declines with `MEMORY_COPY_IN_RANGE` while nothing traces memory copies; no pass runs and the application's result is intact |
| Declines, GPU | `tests/range-replay-declines/` | a device-writing copy between a range's kernels, through `hipMemcpyAsync` (C1) or `hsa_amd_memory_async_batch_copy` (C2), declines with `MEMORY_COPY_IN_RANGE` while nothing traces memory copies; `hipMalloc` / `hipFree` inside a bound range decline with `ALLOCATION_CHANGED_IN_RANGE` (C3); an empty range closes with `NO_DISPATCH` (C4); no pass runs and the application's result is intact |
| Concurrency, GPU | `tests/range-replay-concurrency/` | another thread's HIP graph launches on the agent, timed into the replay window, wait for it to close: the range replays with `divergence_count == 0`, its result is intact, and every launch's increment survives (C8) |
| Performance, GPU, nightly | `tests/range-replay-perf/` | pass-count scaling, and amortization of the window cost over dispatches |

## What is missing

Most decline reasons are asserted only as bookkeeping: nothing on hardware drives an application
into them and checks that the range is declined and the application is unaffected. The three bugs
fixed in #10864 now each have a regression test:
fixed in #10864 now each have a regression test, and C3 / C4 cover allocation-set changes and an
empty range on hardware:

- device-writing copies went unseen when nothing traced memory copies (C1);
- `hsa_amd_memory_async_batch_copy` bypassed the decline (C2);
- a graph launched from another thread during a replay window could corrupt application data
(C8).
(C8);
- `hipMalloc` / `hipFree` inside a bound range must decline with `ALLOCATION_CHANGED_IN_RANGE`
(C3); an empty range must close with `NO_DISPATCH` (C4).

A small `hipMemcpyAsync` is not declined at all. HIP performs a copy of up to
`GPU_FORCE_BLIT_COPY_SIZE` (16 KiB by default) with a copy kernel on the application's own queue,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@
# reason and checks the two things a decline promises: CLOSE names the reason before any pass
# runs, and the application's own execution of the range is unchanged.
#
# A test is one RR_DECLINE_MODE of the application paired with the status the tool expects.
# A test is one RR_DECLINE_MODE of the application paired with the status and dispatch count the
# tool expects.
#
cmake_minimum_required(VERSION 3.21.0 FATAL_ERROR)

Expand Down Expand Up @@ -77,7 +78,7 @@ add_dependencies(range-replay-declines-testapp range-replay-declines-client)
# The tool's verdict alone would let a run pass while the application reported a wrong
# result, so the application's own failure line fails the test too.
function(add_rr_decline_test NAME)
cmake_parse_arguments(arg "" "MODE;EXPECT" "" ${ARGN})
cmake_parse_arguments(arg "" "MODE;EXPECT;DISPATCHES" "" ${ARGN})
rocprofiler_add_integration_execute_test(
${NAME}
COMMAND $<TARGET_FILE:range-replay-declines-testapp>
Expand All @@ -86,19 +87,35 @@ function(add_rr_decline_test NAME)
LABELS "integration-tests;range-replay"
PRELOAD "$<TARGET_FILE:range-replay-declines-client>"
ENVIRONMENT "RR_DECLINE_MODE=${arg_MODE}" "RR_DECLINE_EXPECT=${arg_EXPECT}"
"RR_DECLINE_EXPECT_DISPATCHES=${arg_DISPATCHES}"
PASS_REGULAR_EXPRESSION "\\[rr-decline\\] PASS"
FAIL_REGULAR_EXPRESSION
"\\[rr-decline\\] FAIL|\\[rr-decline\\] error|\\[app\\] FAIL|${ROCPROFILER_DEFAULT_FAIL_REGEX}"
SKIP_REGULAR_EXPRESSION "\\[app\\] SKIP")
endfunction()

# C1: a host-to-device hipMemcpyAsync between the range's two kernels, with no memory-copy
# tracing configured anywhere in the process.
# tracing configured anywhere in the process. CLOSE reports 1 dispatch: only the first;
# the copy folds the decline in before the second is counted.
add_rr_decline_test(test-range-replay-decline-hip-memcpy-async MODE hip-memcpy-async
EXPECT MEMORY_COPY_IN_RANGE)
EXPECT MEMORY_COPY_IN_RANGE DISPATCHES 1)

# C2: the same write through hsa_amd_memory_async_batch_copy, an HSA entry point
# intercepted separately from the other copy functions. Skipped where the runtime does not
# provide it.
add_rr_decline_test(test-range-replay-decline-hsa-batch-copy MODE hsa-batch-copy EXPECT
MEMORY_COPY_IN_RANGE)
MEMORY_COPY_IN_RANGE DISPATCHES 1)

# C3: hipMalloc after the binding kernel changes the tracked allocation set versus the
# entry snapshot, so execute_range declines with ALLOCATION_CHANGED_IN_RANGE. One observed
# dispatch.
add_rr_decline_test(test-range-replay-decline-hip-malloc MODE hip-malloc EXPECT
ALLOCATION_CHANGED_IN_RANGE DISPATCHES 1)

# C3: hipFree of a pre-range allocation after the binding kernel, same decline.
add_rr_decline_test(test-range-replay-decline-hip-free MODE hip-free EXPECT
ALLOCATION_CHANGED_IN_RANGE DISPATCHES 1)

# C4: begin then end with nothing recorded closes with NO_DISPATCH and zero dispatches.
add_rr_decline_test(test-range-replay-decline-empty MODE empty EXPECT NO_DISPATCH
DISPATCHES 0)
45 changes: 34 additions & 11 deletions projects/rocprofiler-sdk/tests/range-replay-declines/client.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,8 @@
* while no memory-copy tracing is configured in the process.
*
* Environment:
* RR_DECLINE_EXPECT the status CLOSE must report, by name (for example MEMORY_COPY_IN_RANGE)
* RR_DECLINE_EXPECT the status CLOSE must report, by name
* RR_DECLINE_EXPECT_DISPATCHES the dispatch_count CLOSE must report (per mode)
*/

#include "range.hpp"
Expand Down Expand Up @@ -147,17 +148,20 @@ tool_fini(void*)
return;
}

const char* expect = std::getenv("RR_DECLINE_EXPECT");
const char* status = status_name(g_status.load());
const char* expect = std::getenv("RR_DECLINE_EXPECT");
const char* expect_dispatches = std::getenv("RR_DECLINE_EXPECT_DISPATCHES");
const char* status = status_name(g_status.load());

fprintf(stderr,
"[rr-decline] configs=%d passes=%d closes=%d status=%s dispatches=%lu expected=%s\n",
"[rr-decline] configs=%d passes=%d closes=%d status=%s dispatches=%lu expected=%s "
"expected_dispatches=%s\n",
g_configs.load(),
g_passes.load(),
g_closes.load(),
status,
static_cast<unsigned long>(g_dispatches.load()),
expect != nullptr ? expect : "<unset>");
expect != nullptr ? expect : "<unset>",
expect_dispatches != nullptr ? expect_dispatches : "<unset>");

auto ok = true;
if(expect == nullptr || std::string_view{status} != std::string_view{expect})
Expand All @@ -170,20 +174,39 @@ tool_fini(void*)
fprintf(stderr, "[rr-decline] FAIL: expected exactly one CONFIG and one CLOSE\n");
ok = false;
}
// The decline is known while the range is still recording, so no pass may have run.
// The decline is known while the range is still recording, or at the start of execute_range
// before any pass runs, so no pass may have run.
if(g_passes.load() != 0)
{
fprintf(stderr, "[rr-decline] FAIL: a declined range ran replayed passes\n");
ok = false;
}
if(g_dispatches.load() != kObservedDispatches)

if(expect_dispatches == nullptr)
{
fprintf(stderr,
"[rr-decline] FAIL: CLOSE reported %lu dispatches, expected %lu\n",
static_cast<unsigned long>(g_dispatches.load()),
static_cast<unsigned long>(kObservedDispatches));
fprintf(stderr, "[rr-decline] FAIL: RR_DECLINE_EXPECT_DISPATCHES is unset\n");
ok = false;
}
else
{
char* end = nullptr;
const auto expected = std::strtoull(expect_dispatches, &end, 10);
if(end == expect_dispatches || *end != '\0')
{
fprintf(stderr,
"[rr-decline] FAIL: RR_DECLINE_EXPECT_DISPATCHES '%s' is not an integer\n",
expect_dispatches);
ok = false;
}
else if(g_dispatches.load() != expected)
{
fprintf(stderr,
"[rr-decline] FAIL: CLOSE reported %lu dispatches, expected %lu\n",
static_cast<unsigned long>(g_dispatches.load()),
static_cast<unsigned long>(expected));
ok = false;
}
}

fprintf(stderr, ok ? "[rr-decline] PASS\n" : "[rr-decline] FAIL\n");
}
Expand Down
Loading
Loading