Skip to content

feat(rocjitsu): route AMDKFD_IOC_DBG_TRAP into a debug-session dispatcher #8393

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
amd-arosa merged 32 commits into from
Jul 15, 2026
Merged

feat(rocjitsu): route AMDKFD_IOC_DBG_TRAP into a debug-session dispatcher #8393

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
df92276
feat(rocjitsu): route AMDKFD_IOC_DBG_TRAP into a debug-session dispat…
amd-arosa Jul 9, 2026
134b02b
docs
amd-arosa Jul 10, 2026
6b4b327
make mutable std::mutex debug_mutex_;
amd-arosa Jul 10, 2026
f5b72b0
compare to kernel
amd-arosa Jul 10, 2026
44701c2
Kernel copies min(user_size, sizeof(runtime_info)) bytes (a truncatin…
amd-arosa Jul 10, 2026
05605a4
Kernel kfd_dbg_trap_enable() returns -EALREADY when already debug ena…
amd-arosa Jul 10, 2026
78290ab
Merge branch 'develop' of https://github.com/ROCm/rocm-systems into u…
amd-arosa Jul 10, 2026
833d32b
fix: We should validate dbg_fd before setting sess.enabled and return…
amd-arosa Jul 10, 2026
801c017
fix: marshal the DBG_TRAP runtime-info buffer through the RPC layer
amd-arosa Jul 10, 2026
9aede93
fix: bound the DBG_TRAP ENABLE inline buffer and response copy to the…
amd-arosa Jul 10, 2026
91ca5fb
feat: transfer the eventfd with `SCM_RIGHTS`
amd-arosa Jul 10, 2026
e6ca9df
feat: implement UniqueFd for RAII management of file descriptors in d…
amd-arosa Jul 10, 2026
d2d99a8
fix: handle short reads in rpc_recv_msg to prevent acting on partial …
amd-arosa Jul 10, 2026
fe2b0ac
fix: https://github.com/ROCm/rocm-systems/pull/8393/changes/BASE..9ae…
amd-arosa Jul 10, 2026
27d2449
add a new logger group for the driver
amd-arosa Jul 10, 2026
94e6e2d
use the driver logger
amd-arosa Jul 10, 2026
1370782
fix(rocjitsu): include <sys/types.h> for pid_t in kfd_process.h
amd-arosa Jul 10, 2026
46d5942
docs(rocjitsu): clarify DBG_TRAP by-pid gate closes cross-process attach
amd-arosa Jul 10, 2026
fb2d127
fix(rocjitsu): snapshot runtime state under one lock in DBG_TRAP ENABLE
amd-arosa Jul 10, 2026
b4ef5e7
docs(rocjitsu): note DBG_TRAP requires a post-fork re-open
amd-arosa Jul 10, 2026
69a9bf9
test(rocjitsu): cover bare DBG_TRAP DISABLE and concurrent enable/dis…
amd-arosa Jul 10, 2026
204a006
fix(rocjitsu): return -ENOSYS for recognized-but-unimplemented DBG_TR…
amd-arosa Jul 10, 2026
442c1ef
feat(rocjitsu): add support for out-of-band fd transfer in vm_execute…
amd-arosa Jul 13, 2026
da37034
EBADF fix
amd-arosa Jul 13, 2026
2502248
fix(tests): remove redundant force flag from session stop commands
amd-arosa Jul 13, 2026
545d806
fix(rocjitsu): authorize DBG_TRAP DISABLE like every other non-self op
amd-arosa Jul 14, 2026
b5e8caf
fix(rocjitsu): never trust the client dbg_fd integer in daemon mode
amd-arosa Jul 14, 2026
a5676b5
fix(rocjitsu): require the DBG_TRAP notifier fd to be writable
amd-arosa Jul 14, 2026
c3fcc3a
Revert "fix(rocjitsu): authorize DBG_TRAP DISABLE like every other no…
amd-arosa Jul 14, 2026
9c5d55b
refactor(rocjitsu): carry the DBG_TRAP notifier fd in rj_vm_cmd_t
amd-arosa Jul 14, 2026
bbdba2b
fix: A daemon-path ENABLE that the daemon fails (result != 0) must no…
amd-arosa Jul 14, 2026
dbbc2a8
Merge branch 'develop' into users/arosa/rj/rocgdb/03
amd-arosa Jul 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions emulation/rocjitsu/lib/rocjitsu/src/rocjitsu/kmd/linux/kfd_process.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,45 @@ class KfdProcess {
uint64_t r_debug = 0;
};

/// @brief Per-process debugger session state.
///
/// @details Mirrors the debug-related fields the kernel maintains on
/// @c struct @c kfd_process in
/// @c drivers/gpu/drm/amd/amdkfd/kfd_priv.h.
/// Managed by the @c AMDKFD_IOC_DBG_TRAP ioctl handler
/// (@c kfd_ioctl_set_debug_trap in the real driver).
///
/// Field mapping to @c kfd_priv.h:
/// | DebugSession field | kfd_process field |
/// |------------------------|-------------------------------------------------|
/// | @ref enabled | @c bool @c debug_trap_enabled |
/// | @ref runtime_state | @c kfd_runtime_info @c runtime_info.runtime_state (kfd_ioctl.h) |
/// | @ref exception_enable_mask | @c uint64_t @c exception_enable_mask |
/// | @ref dbg_fd | @c struct @c file* @c dbg_ev_file (flattened to fd) |
/// | @ref debugger_pid | @c struct @c kfd_process* @c debugger_process (stored as pid) |
struct DebugSession {
/// @brief Mirrors @c kfd_process::debug_trap_enabled.
/// Set when the device process is debug-attached with a reserved VMID.
bool enabled = false;

/// @brief Mirrors @c kfd_process::runtime_info.runtime_state.
/// Holds a @c kfd_dbg_runtime_state value (see @c kfd_ioctl.h).
uint32_t runtime_state = 0;

/// @brief Mirrors @c kfd_process::exception_enable_mask.
/// Bitmask of exception classes that are forwarded to the debugger.
uint64_t exception_enable_mask = 0;

/// @brief Mirrors @c kfd_process::dbg_ev_file (flattened from @c struct @c file* to fd).
/// File descriptor used as the debugger notification / poll target.
/// -1 when no debugger is attached.
int dbg_fd = -1;

/// @brief Mirrors @c kfd_process::debugger_process (stored as pid instead of pointer).
/// Linux PID of the attached debugger (ptrace parent). 0 when not attached.
pid_t debugger_pid = 0;
};

/// @brief Per-page translation entry, mirroring HW PTE fields.
/// @details Stores the host pointer for VA→PA translation and the PTE MTYPE
/// that the GPU MMU uses to override instruction-level caching. On real
Expand Down Expand Up @@ -191,6 +230,9 @@ class KfdProcess {
std::mutex runtime_mutex_;
RuntimeState runtime_state_;

mutable std::mutex debug_mutex_;
DebugSession debug_session_;

private:
};

Expand Down
2 changes: 2 additions & 0 deletions emulation/rocjitsu/lib/rocjitsu/src/rocjitsu/kmd/linux/linux_kfd.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,8 @@ const char *LinuxKfd::ioctl_name(unsigned long request) {
return "SET_SCRATCH_VA";
case AMDKFD_IOC_SET_TRAP_HANDLER:
return "SET_TRAP_HANDLER";
case AMDKFD_IOC_DBG_TRAP:
return "DBG_TRAP";
case AMDKFD_IOC_SET_XNACK_MODE:
return "SET_XNACK";
case AMDKFD_IOC_SET_MEMORY_POLICY:
Expand Down
73 changes: 73 additions & 0 deletions emulation/rocjitsu/lib/rocjitsu/src/rocjitsu/kmd/linux/remote_driver.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ constexpr bool has_embedded_pointers(unsigned long request) {
case AMDKFD_IOC_MAP_MEMORY_TO_GPU:
case AMDKFD_IOC_UNMAP_MEMORY_FROM_GPU:
case AMDKFD_IOC_GET_PROCESS_APERTURES_NEW:
case AMDKFD_IOC_DBG_TRAP:
return true;
case AMDKFD_IOC_SVM:
// SVM's variable-length attribute array is part of the ioctl payload, not a
Expand Down Expand Up @@ -334,6 +335,9 @@ int RemoteDriver::send_ioctl(unsigned long request, void *arg) {
uint64_t saved_events_ptr = 0;
uint64_t saved_apertures_ptr = 0;
uint64_t saved_device_ids_ptr = 0;
uint64_t saved_dbg_rinfo_ptr = 0;
uint32_t saved_dbg_rinfo_size = 0;
uint64_t saved_dbg_snapshot_ptr = 0;
if (has_embedded_pointers(request)) {
switch (request) {
case AMDKFD_IOC_WAIT_EVENTS:
Expand All @@ -348,6 +352,21 @@ int RemoteDriver::send_ioctl(unsigned long request, void *arg) {
saved_device_ids_ptr =
static_cast<kfd_ioctl_map_memory_to_gpu_args *>(arg)->device_ids_array_ptr;
break;
case AMDKFD_IOC_DBG_TRAP: {
auto *dbg = static_cast<kfd_ioctl_dbg_trap_args *>(arg);
switch (dbg->op) {
case KFD_IOC_DBG_TRAP_ENABLE:
saved_dbg_rinfo_ptr = dbg->enable.rinfo_ptr;
saved_dbg_rinfo_size = dbg->enable.rinfo_size;
break;
case KFD_IOC_DBG_TRAP_GET_DEVICE_SNAPSHOT:
saved_dbg_snapshot_ptr = dbg->device_snapshot.snapshot_buf_ptr;
break;
default:
break;
}
break;
}
default:
break;
}
Expand Down Expand Up @@ -385,6 +404,25 @@ int RemoteDriver::send_ioctl(unsigned long request, void *arg) {
buf.resize(buf.size() + aperture_args->num_of_nodes * sizeof(kfd_process_device_apertures));
break;
}
case AMDKFD_IOC_DBG_TRAP: {
auto *dbg = reinterpret_cast<kfd_ioctl_dbg_trap_args *>(args_base);
size_t inline_size = 0;
switch (dbg->op) {
case KFD_IOC_DBG_TRAP_ENABLE:
inline_size =
std::min(static_cast<size_t>(dbg->enable.rinfo_size), sizeof(kfd_runtime_info));
break;
case KFD_IOC_DBG_TRAP_GET_DEVICE_SNAPSHOT:
inline_size =
static_cast<size_t>(dbg->device_snapshot.num_devices) * dbg->device_snapshot.entry_size;
break;
default:
break;
}
if (inline_size > 0)
buf.resize(buf.size() + inline_size);
break;
}
default:
break;
}
Expand Down Expand Up @@ -438,6 +476,20 @@ int RemoteDriver::send_ioctl(unsigned long request, void *arg) {
static_cast<kfd_ioctl_map_memory_to_gpu_args *>(arg)->device_ids_array_ptr =
saved_device_ids_ptr;
break;
case AMDKFD_IOC_DBG_TRAP: {
auto *dbg = static_cast<kfd_ioctl_dbg_trap_args *>(arg);
switch (dbg->op) {
case KFD_IOC_DBG_TRAP_ENABLE:
dbg->enable.rinfo_ptr = saved_dbg_rinfo_ptr;
break;
case KFD_IOC_DBG_TRAP_GET_DEVICE_SNAPSHOT:
dbg->device_snapshot.snapshot_buf_ptr = saved_dbg_snapshot_ptr;
break;
default:
break;
}
break;
}
default:
break;
}
Expand All @@ -460,6 +512,27 @@ int RemoteDriver::send_ioctl(unsigned long request, void *arg) {
std::min(aperture_args->num_of_nodes * sizeof(kfd_process_device_apertures), extra));
break;
}
case AMDKFD_IOC_DBG_TRAP: {
auto *dbg = static_cast<kfd_ioctl_dbg_trap_args *>(arg);
void *dst = nullptr;
size_t copy_len = 0;
switch (dbg->op) {
case KFD_IOC_DBG_TRAP_ENABLE:
dst = reinterpret_cast<void *>(saved_dbg_rinfo_ptr);
copy_len = std::min(static_cast<size_t>(saved_dbg_rinfo_size),
std::min(static_cast<size_t>(dbg->enable.rinfo_size), extra));
break;
case KFD_IOC_DBG_TRAP_GET_DEVICE_SNAPSHOT:
dst = reinterpret_cast<void *>(saved_dbg_snapshot_ptr);
copy_len = extra;
Comment thread
amd-arosa marked this conversation as resolved.
Resolved
break;
default:
break;
}
if (dst != nullptr && copy_len > 0)
std::memcpy(dst, payload.data() + arg_size, copy_len);
break;
}
default:
break;
}
Expand Down
117 changes: 117 additions & 0 deletions emulation/rocjitsu/lib/rocjitsu/src/rocjitsu/kmd/linux/simulated_kfd.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -213,6 +213,7 @@ int SimulatedKfd::open() {
}
uint32_t pid = next_process_id_++;
auto proc = std::make_shared<KfdProcess>(pid, static_cast<uint32_t>(gpus_.size()));
proc->set_client_pid(static_cast<pid_t>(getpid()));
proc->event_state_.reset();
for (auto &g : gpus_) {
if (auto *mem = g.soc ? g.soc->memory() : nullptr) {
Expand Down Expand Up @@ -557,6 +558,8 @@ int SimulatedKfd::dispatch_ioctl(KfdProcess &proc, unsigned long request, void *
return get_available_memory_ioctl(proc, arg);
case AMDKFD_IOC_RUNTIME_ENABLE:
return runtime_enable_ioctl(proc, arg);
case AMDKFD_IOC_DBG_TRAP:
return debug_trap_ioctl(proc, arg);
case AMDKFD_IOC_SET_SCRATCH_BACKING_VA: {
auto *a = static_cast<kfd_ioctl_set_scratch_backing_va_args *>(arg);
uint32_t ord = gpu_ordinal(a->gpu_id);
Expand Down Expand Up @@ -1707,6 +1710,120 @@ int SimulatedKfd::runtime_enable_ioctl(KfdProcess &proc, void *arg) {
return 0;
}

std::shared_ptr<KfdProcess> SimulatedKfd::find_process_by_client_pid(pid_t pid) const {
if (pid == 0)
return nullptr;
std::lock_guard<std::mutex> lk(process_mutex_);
for (auto &[id, proc] : processes_)
if (proc->client_pid() == pid)
return proc;
return nullptr;
}

// in real kernel, amd/amdkfd/kfd_chardev.c kfd_ioctl_set_debug_trap
int SimulatedKfd::debug_trap_ioctl(KfdProcess &caller, void *arg) {
auto *args = static_cast<kfd_ioctl_dbg_trap_args *>(arg);
util::Logger::cp("DBG_TRAP pid=", args->pid, " op=", args->op);
Comment thread
amd-arosa marked this conversation as resolved.
Resolved

KfdProcess *target = nullptr;
std::shared_ptr<KfdProcess> target_ref;
if (caller.client_pid() != 0 && static_cast<pid_t>(args->pid) == caller.client_pid()) {
target = &caller; // self-debug (local mode)
} else {
target_ref = find_process_by_client_pid(static_cast<pid_t>(args->pid));
target = target_ref.get();
}
if (target == nullptr)
return -ESRCH;

const bool self_debug = (target == &caller);

std::lock_guard<std::mutex> lk(target->debug_mutex_);
auto &sess = target->debug_session_;

if (!self_debug && args->op != KFD_IOC_DBG_TRAP_DISABLE &&
sess.debugger_pid != caller.client_pid())
return -EPERM;

// Non-ENABLE ops require an active debug session (kernel: EINVAL).
if (args->op != KFD_IOC_DBG_TRAP_ENABLE && !sess.enabled)
return -EINVAL;

// Live runtime-enable state, set by ROCr's AMDKFD_IOC_RUNTIME_ENABLE.
const bool runtime_enabled = [&] {
std::lock_guard<std::mutex> rlk(target->runtime_mutex_);
return target->runtime_state_.enabled;
}();

// https://github.com/torvalds/linux/blob/a635d6748234582ea287c5ffeae28b9b23f91c7e/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c#L3132-L3142
switch (args->op) {
case KFD_IOC_DBG_TRAP_SET_WAVE_LAUNCH_OVERRIDE:
case KFD_IOC_DBG_TRAP_SET_WAVE_LAUNCH_MODE:
case KFD_IOC_DBG_TRAP_SUSPEND_QUEUES:
case KFD_IOC_DBG_TRAP_RESUME_QUEUES:
case KFD_IOC_DBG_TRAP_SET_NODE_ADDRESS_WATCH:
case KFD_IOC_DBG_TRAP_CLEAR_NODE_ADDRESS_WATCH:
case KFD_IOC_DBG_TRAP_SET_FLAGS:
if (!runtime_enabled)
return -EPERM;
break;
default:
break;
}

// https://github.com/torvalds/linux/blob/a635d6748234582ea287c5ffeae28b9b23f91c7e/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c#L3144
if (args->op == KFD_IOC_DBG_TRAP_SET_NODE_ADDRESS_WATCH ||
args->op == KFD_IOC_DBG_TRAP_CLEAR_NODE_ADDRESS_WATCH) {
const uint32_t gpu_id = args->op == KFD_IOC_DBG_TRAP_SET_NODE_ADDRESS_WATCH
? args->set_node_address_watch.gpu_id
: args->clear_node_address_watch.gpu_id;
if (find_gpu(gpu_id) == nullptr)
return -ENODEV;
}

// https://github.com/torvalds/linux/blob/a635d6748234582ea287c5ffeae28b9b23f91c7e/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c#L3158
switch (args->op) {
// https://github.com/torvalds/linux/blob/a635d6748234582ea287c5ffeae28b9b23f91c7e/drivers/gpu/drm/amd/amdkfd/kfd_debug.c#L788-L847
case KFD_IOC_DBG_TRAP_ENABLE: {
if (sess.enabled)
return -EALREADY; // target process is already debug enabled

const int dbg_fd = static_cast<int>(args->enable.dbg_fd);
if (!daemon_mode_ && fcntl(dbg_fd, F_GETFD) == -1)
return -EBADF;

sess.enabled = true;
Comment thread
amd-arosa marked this conversation as resolved.
Resolved
sess.debugger_pid = caller.client_pid();
sess.dbg_fd = dbg_fd;
sess.exception_enable_mask = args->enable.exception_mask;
sess.runtime_state =
runtime_enabled ? DEBUG_RUNTIME_STATE_ENABLED : DEBUG_RUNTIME_STATE_DISABLED;

// Copy kfd_runtime_info back to the debugger (kernel: kfd_dbg_trap_enable
// copies the saved runtime info and returns its size).
kfd_runtime_info info{};
{
std::lock_guard<std::mutex> rlk(target->runtime_mutex_);
info.r_debug = target->runtime_state_.r_debug;
info.ttmp_setup =
(target->runtime_state_.mode_mask & KFD_RUNTIME_ENABLE_MODE_TTMP_SAVE_MASK) ? 1u : 0u;
}
info.runtime_state = sess.runtime_state;
size_t copy_size = std::min(static_cast<size_t>(args->enable.rinfo_size), sizeof(info));
if (args->enable.rinfo_ptr != 0 && copy_size > 0)
std::memcpy(reinterpret_cast<void *>(static_cast<uintptr_t>(args->enable.rinfo_ptr)), &info,
Comment thread
amd-arosa marked this conversation as resolved.
Resolved
copy_size);
args->enable.rinfo_size = sizeof(info);
return 0;
}
case KFD_IOC_DBG_TRAP_DISABLE:
sess = KfdProcess::DebugSession{};
return 0;
default:
return -ENOSYS;
Comment thread
amd-arosa marked this conversation as resolved.
Resolved
}
}

int SimulatedKfd::set_xnack_mode_ioctl(void *arg) {
auto *args = static_cast<kfd_ioctl_set_xnack_mode_args *>(arg);
args->xnack_enabled = 0;
Expand Down
8 changes: 8 additions & 0 deletions emulation/rocjitsu/lib/rocjitsu/src/rocjitsu/kmd/linux/simulated_kfd.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,13 @@ class SimulatedKfd : public LinuxKfd {
/// @brief Look up the local-mode process.
std::shared_ptr<KfdProcess> find_local_process() const;

/// @brief Look up a KfdProcess by its client (Linux) pid. Used by
/// AMDKFD_IOC_DBG_TRAP to resolve the debug target, mirroring the kernel's
/// kfd_lookup_process_by_pid().
/// @param pid Client (Linux) pid of the target process.
/// @return The matching KfdProcess, or nullptr if none matches.
std::shared_ptr<KfdProcess> find_process_by_client_pid(pid_t pid) const;

/// @brief Look up a GpuDevice by gpu_id. Returns nullptr if not found.
GpuDevice *find_gpu(uint32_t gpu_id);
const GpuDevice *find_gpu(uint32_t gpu_id) const;
Expand Down Expand Up @@ -198,6 +205,7 @@ class SimulatedKfd : public LinuxKfd {
int ipc_import_handle_ioctl(KfdProcess &proc, void *arg);
int svm_ioctl(KfdProcess &proc, void *arg);
int runtime_enable_ioctl(KfdProcess &proc, void *arg);
int debug_trap_ioctl(KfdProcess &caller, void *arg);
int set_xnack_mode_ioctl(void *arg);
int get_tile_config_ioctl(void *arg);
bool allocate_scratch_backing(uint32_t process_id, uint64_t gpu_va, size_t size);
Expand Down
14 changes: 14 additions & 0 deletions emulation/rocjitsu/lib/rocjitsu/src/rocjitsu/vm/rj_vm.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,20 @@ void reconstruct_embedded_pointers(uint32_t cmd, void *arg, size_t arg_size, siz
args->kfd_process_device_apertures_ptr = reinterpret_cast<uint64_t>(extra);
break;
}
case AMDKFD_IOC_DBG_TRAP: {
auto *args = static_cast<kfd_ioctl_dbg_trap_args *>(arg);
switch (args->op) {
case KFD_IOC_DBG_TRAP_ENABLE:
args->enable.rinfo_ptr = reinterpret_cast<uint64_t>(extra);
break;
case KFD_IOC_DBG_TRAP_GET_DEVICE_SNAPSHOT:
args->device_snapshot.snapshot_buf_ptr = reinterpret_cast<uint64_t>(extra);
break;
default:
break;
}
break;
}
default:
break;
}
Expand Down
Loading
Loading