Skip to content
25 changes: 21 additions & 4 deletions projects/clr/hipamd/src/hrr/hip_capture.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -97,11 +97,21 @@ static thread_local hipStream_t g_pushed_stream{};
// Helpers
// ---------------------------------------------------------------------------

bool hip_capture_enabled() {
return !flagIsDefault(HIP_HRR_CAPTURE_OUTPUT) &&
HIP_HRR_CAPTURE_OUTPUT[0] != '\0';
// The kernel sets AT_SECURE for a set-user-ID, set-group-ID or file-capability
// exec, and for an LSM transition. HIP_HRR_CAPTURE_OUTPUT comes from whoever
// launched the process, so such a process must not write where it says, nor
// record its memory for them.
static bool hrr_secure_exec() {
static const bool secure = hrr_cap::metadata::secure_exec();
return secure;
}

static bool hrr_capture_requested() {
return !flagIsDefault(HIP_HRR_CAPTURE_OUTPUT) && HIP_HRR_CAPTURE_OUTPUT[0] != '\0';
}

bool hip_capture_enabled() { return hrr_capture_requested() && !hrr_secure_exec(); }

const char* hip_capture_output_dir() {
return HIP_HRR_CAPTURE_OUTPUT;
}
Expand Down Expand Up @@ -1593,7 +1603,14 @@ void hrr_install_clr_exception_handler() {

void hip_capture_init() {
#if defined(HIP_HRR_CAPTURE_ENABLED)
if (!hip_capture_enabled()) return;
if (!hip_capture_enabled()) {
if (hrr_secure_exec() && hrr_capture_requested())
fprintf(stderr,
"[HRR capture] HIP_HRR_CAPTURE_OUTPUT ignored: the program was started in "
"secure-execution mode (set-user-ID, set-group-ID, file capabilities or an "
"LSM transition).\n");
return;
}

// HIP_HRR_DEBUG_ARGS traces are emitted via LogPrintfInfo (amd::LOG_INFO).
// ClPrint filters anything above AMD_LOG_LEVEL, so a user who set the trace
Expand Down
3 changes: 2 additions & 1 deletion projects/clr/hipamd/src/hrr/hip_capture.h
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,8 @@ struct Hash128 {
// Public API — called from hip_context.cpp and hip_capture.cpp
// ---------------------------------------------------------------------------

// Check if capture is enabled (HIP_HRR_CAPTURE_OUTPUT env var set and non-empty)
// Check if capture is enabled (HIP_HRR_CAPTURE_OUTPUT env var set and non-empty, and the
// process not started in secure-execution mode)
bool hip_capture_enabled();

// Return the output directory from the env var
Expand Down
18 changes: 18 additions & 0 deletions projects/clr/hipamd/src/hrr/hip_capture_metadata.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,16 @@
#include <string>
#include <vector>

// Kept out of hip_capture.cpp: glibc's <sys/auxv.h> includes <elf.h>, whose
// Elf64_* typedefs collide with the ones CLR's ELF headers put in the global
// namespace.
#if defined(__linux__)
#include <sys/auxv.h>
#ifndef AT_SECURE
#define AT_SECURE 23 // Linux ABI value; same fallback as ROCr's runtime.cpp
#endif
#endif

namespace hip {
class Device;
extern std::vector<hip::Device*> g_devices;
Expand Down Expand Up @@ -190,5 +200,13 @@ std::string collect_json() {
return os.str();
}

bool secure_exec() {
#if defined(__linux__)
return getauxval(AT_SECURE) != 0;
#else
return false;
#endif
Comment thread
Copilot marked this conversation as resolved.
Resolved
}

} // namespace metadata
} // namespace hrr_cap
4 changes: 4 additions & 0 deletions projects/clr/hipamd/src/hrr/hip_capture_metadata.h
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@
namespace hrr_cap {
namespace metadata {

// True when the kernel started this process in secure-execution mode (set-user-ID,
// set-group-ID, file capabilities or an LSM transition). Always false off Linux.
bool secure_exec();

// Collects best-effort capture environment metadata for the HRR manifest.
// The collector is safe to call from hip_capture_init(): it reads HIP runtime
// constants and initialized internal device state rather than calling public HIP
Expand Down
2 changes: 2 additions & 0 deletions projects/hip/docs/how-to/hip_record_replay.rst
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,8 @@ short run of a big model produces gigabytes.
in host memory is readable in it. Treat an archive as you would treat the input
data of the run it came from.

On Linux, a program that the kernel starts in secure-execution mode, such as a set-user-ID, set-group-ID or file-capability program, ignores ``HIP_HRR_CAPTURE_OUTPUT``: it records nothing, and says why in one line on standard error.

Your first replay
=======================================================

Expand Down
15 changes: 12 additions & 3 deletions projects/hrr/DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -660,7 +660,8 @@ with `hipEventRecord` to accumulate elapsed time into `total_graph_ms`.
## Init / Shutdown

`hip_capture_init()` is called from `hip_context.cpp` at HIP init (after `amd::Runtime`
and the live `HipDispatchTable` are ready). If `HIP_HRR_CAPTURE_OUTPUT` is set it
and the live `HipDispatchTable` are ready). If capture is enabled (see [Enable
Flag](#enable-flag)) it
snapshots the runtime dispatch table, installs runtime capture shims, opens the writer,
recovers pre-init fat binaries (compiler-table shims + retroactive sweep), and
registers `hip_capture_shutdown` via `atexit`. Runtime shims are **not** installed at
Expand All @@ -671,10 +672,18 @@ Shutdown uninstalls shims and flushes `events.bin` + `manifest.json`.

## Enable Flag

Capture is enabled when `HIP_HRR_CAPTURE_OUTPUT` is set to a non-empty directory
(see [README.md](README.md#capture-environment)). Defined as a `cstring` release flag in
Capture is enabled when `HIP_HRR_CAPTURE_OUTPUT` is set to a non-empty directory and the
process was not started in secure-execution mode (see
[README.md](README.md#capture-environment)). Defined as a `cstring` release flag in
`rocclr/utils/flags.hpp`.

On Linux the kernel sets `AT_SECURE` in the auxiliary vector for a set-user-ID,
set-group-ID or file-capability exec, and for an LSM transition. Such a process can hold
privileges that whoever set its environment does not, so it ignores the variable, as
`secure_getenv()` would, and `hip_capture_init()` prints one line on stderr saying so.
The check is `hrr_cap::metadata::secure_exec()` in `hip_capture_metadata.cpp`, and it is
always false off Linux.

## Playback Tools

| Tool | Source | Purpose |
Expand Down
2 changes: 1 addition & 1 deletion projects/hrr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ User-facing capture, replay, and validation knobs. Implementation details can be

| Variable | Default | Purpose |
|----------|---------|---------|
| `HIP_HRR_CAPTURE_OUTPUT` | *(unset)* | Enable capture; path to the `.hrr` archive directory |
| `HIP_HRR_CAPTURE_OUTPUT` | *(unset)* | Enable capture; path to the `.hrr` archive directory. Ignored, with one line on stderr, when Linux starts the program in secure-execution mode (set-user-ID, set-group-ID, file capabilities or an LSM transition) |
| `HIP_HRR_DEBUG_ARGS` | off | Dump every captured kernel arg to the log (debug / provenance) |

### `hrr-playback` CLI options
Expand Down
Loading