[rocjitsu] Reduce memory keeper startup and topology IPC

emulation/rocjitsu/docs/memory-backing.md

@@ -126,15 +126,44 @@ removes it. The helper waits for the lock through an independently opened file
 description. Retirement unmaps the lease without closing a long-lived application
 descriptor, so concurrent `close_range` or descriptor replacement cannot make it
 close an unrelated file. A helper thread waits for release of the lock and
-notifies the event loop. The helper retires
-on owner exit or exec, preserving any exported tokens until they close, without
-retaining application mappings. The interposer starts it before application
-code can lower file limits. A forked child starts its own helper under the
-child's then-current hard limits. Generated topology files and proc-maps
-snapshots are sized there and populated through mappings; these small metadata
-files must fit that helper's hard file limit. Topology population failures emit
-a warning identifying the file. Application writes, including runtime compiler
-temporary files, continue to obey the application's current limit.
+notifies the event loop. On owner exit or exec, the helper closes its listener
+and private request connections, including idle connections inherited by forked
+children. It continues serving exported tokens until they close. During
+initialization the interposer calls
+`prepare_memory_keeper` and waits for readiness before application code can
+lower file limits. A forked child starts its own helper under the child's
+then-current hard limits.
+Generated topology files and proc-maps snapshots are sized there. Topology
+files with small payloads travel with the sizing request and are written by the helper
+with positional I/O. Larger topology files and proc-maps snapshots are
+populated through shared mappings.
+Both paths preserve the caller's file offset. Metadata files must fit the
+helper's hard file limit. Topology population
+failures emit a warning identifying the file. Application writes, including
+runtime compiler temporary files, continue to obey the application's current
+limit.
+
+The owner records the helper's kernel-supplied PID from the inherited startup
+channel. Before sending a backing descriptor, it checks the listener's PID
+and credentials against the process identity reported at startup. On kernels
+with 64-bit pidfs, the helper's pidfd inode identifies that process across PID
+reuse. Otherwise the owner requires a private cookie from the startup channel
+before transferring a backing descriptor. If the helper cannot answer before
+the timeout and its lifetime is ambiguous, preparation returns `EAGAIN` and
+preserves its lease. Preparation can be retried when the helper responds.
+
+Allocation and metadata requests use temporary private connections for both
+requests and responses. Consecutive nonempty topology files share a connection. The helper
+acknowledges groups of small writes before the caller publishes the tree. Empty
+files are truncated locally, in order, without holding a connection. The caller closes each inline file descriptor after passing it to
+the helper and keeps only the current file for mapped writes. Batch size does
+not increase the application's descriptor requirement. If the kernel limits
+descriptors in flight, the caller drains the batch before retrying. At the end
+of the batch it shuts down the connection so a forked child cannot keep the
+helper endpoint open.
+
+Token reopen requests retain separate reply socket pairs because a token may
+be shared by multiple callers, including callers in different processes.
 
 Changing effective user or group credentials retires the previous helper's
 lease before using a new helper; previously exported tokens retain their keeper
@@ -159,7 +188,8 @@ sizing and application limits unchanged.
 
 Installed objects locate the helper in their installation. Only objects inside
 the configured build tree may fall back to its helper executable; an incomplete
-installation fails rather than using another build's private protocol.
+installation fails rather than using another build's private protocol. The
+startup reply identifies the private protocol before any allocation is sent.
 
 File-limit handling does not require `HSA_ENABLE_IPC_MODE_LEGACY=1`. Ordinary HIP
 allocation, copies, dispatch, and local VMM work with the runtime's default IPC

emulation/rocjitsu/lib/rocjitsu/src/rocjitsu/kmd/linux/CMakeLists.txt

@@ -9,6 +9,7 @@ rj_add_object_library(rocjitsu_kmd
     libc_passthrough.cpp
     linux_kfd.cpp
     memory_backing.cpp
+    memory_keeper_protocol.cpp
     remote_driver.cpp
     rj_daemon.cpp
     simulated_kfd.cpp

emulation/rocjitsu/lib/rocjitsu/src/rocjitsu/kmd/linux/device_backing.h

@@ -0,0 +1,28 @@
+// Copyright (c) 2026 Advanced Micro Devices, Inc.
+// SPDX-License-Identifier: MIT
+
+#pragma once
+
+/// @file device_backing.h
+/// @brief Identifiers for GPU device memory in native and shared-RAM backing.
+
+#include <cstdint>
+#include <string_view>
+
+namespace rocjitsu::kmd {
+
+inline constexpr std::string_view kDeviceBackingNamePrefix = "rocjitsu_alloc_renderD";
+
+/// @brief Persistent GPU classification in System V VMA names after IPC_RMID.
+/// @details The low slot bits select a transient creation key, released after
+/// the keeper attaches and marks the segment for removal. Exhausting these keys
+/// falls back to IPC_PRIVATE: token metadata retains device classification, but
+/// the VMA name can no longer identify the segment for ROCr GPU core dumps.
+inline constexpr uint32_t kDeviceTokenKeyPrefix = 0x524a4700;
+inline constexpr unsigned kDeviceTokenSlotBits = 8;
+inline constexpr uint32_t kDeviceTokenSlotCount = uint32_t{1} << kDeviceTokenSlotBits;
+inline constexpr unsigned kDeviceTokenSlotHexDigits = kDeviceTokenSlotBits / 4;
+static_assert(kDeviceTokenSlotBits % 4 == 0);
+static_assert((kDeviceTokenKeyPrefix & (kDeviceTokenSlotCount - 1)) == 0);
+
+} // namespace rocjitsu::kmd

emulation/rocjitsu/lib/rocjitsu/src/rocjitsu/kmd/linux/interposer.cpp

@@ -544,9 +544,7 @@ class InterposerContext {
     // Keep generated topology and proc snapshots independent of limits lowered
     // by application code before its first GPU call. Inherited hard limits on
     // GPU backing are handled by the keeper's shared-RAM fallback.
-    const int warmup = rocjitsu::create_memory_backing(1);
-    if (warmup >= 0)
-      real().close(warmup);
+    rocjitsu::prepare_memory_keeper();
   }
 
   /// @brief Conservatively detect real GPU descriptors inherited outside this backend.

emulation/rocjitsu/lib/rocjitsu/src/rocjitsu/kmd/linux/libc_passthrough.cpp

@@ -81,6 +81,7 @@ void LibcPassthrough::resolve() {
   epoll_wait = util::lookup_symbol<decltype(epoll_wait)>(handle, "epoll_wait");
   accept4 = util::lookup_symbol<decltype(accept4)>(handle, "accept4");
   getsockopt = util::lookup_symbol<decltype(getsockopt)>(handle, "getsockopt");
+  setsockopt = util::lookup_symbol<decltype(setsockopt)>(handle, "setsockopt");
   socket = util::lookup_symbol<decltype(socket)>(handle, "socket");
   connect = util::lookup_symbol<decltype(connect)>(handle, "connect");
   listen = util::lookup_symbol<decltype(listen)>(handle, "listen");
@@ -98,14 +99,14 @@ void LibcPassthrough::resolve() {
   // resolved. In release builds the asserts disappear, so the boolean must not
   // claim readiness while any later call would dereference a null function
   // pointer.
-  initialized_ = openat && close && read && write && ioctl && syscall && mmap && munmap &&
-                 mprotect && madvise && memfd_create && dup && dup2 && dup3 && fcntl && fopen &&
-                 freopen && opendir && readdir && closedir && stat && lstat && access && fstat_fn &&
-                 readlink_fn && realpath_fn && fork && getsockname && sendmsg && recvmsg &&
-                 shmget && shmat && shmdt && shmctl && getrandom && prlimit && ftruncate &&
-                 fallocate && socketpair && bind && epoll_ctl && prctl && epoll_create1 &&
-                 eventfd && epoll_wait && accept4 && getsockopt && socket && connect && listen &&
-                 execve && setsid && waitpid && poll && clone && exit_immediately;
+  initialized_ =
+      openat && close && read && write && ioctl && syscall && mmap && munmap && mprotect &&
+      madvise && memfd_create && dup && dup2 && dup3 && fcntl && fopen && freopen && opendir &&
+      readdir && closedir && stat && lstat && access && fstat_fn && readlink_fn && realpath_fn &&
+      fork && getsockname && sendmsg && recvmsg && shmget && shmat && shmdt && shmctl &&
+      getrandom && prlimit && ftruncate && fallocate && socketpair && bind && epoll_ctl && prctl &&
+      epoll_create1 && eventfd && epoll_wait && accept4 && getsockopt && setsockopt && socket &&
+      connect && listen && execve && setsid && waitpid && poll && clone && exit_immediately;
   assert(initialized_);
 }
 

emulation/rocjitsu/lib/rocjitsu/src/rocjitsu/kmd/linux/libc_passthrough.h

@@ -106,6 +106,7 @@ class LibcPassthrough {
   decltype(&::epoll_wait) epoll_wait = nullptr;
   decltype(&::accept4) accept4 = nullptr;
   decltype(&::getsockopt) getsockopt = nullptr;
+  decltype(&::setsockopt) setsockopt = nullptr;
   decltype(&::socket) socket = nullptr;
   decltype(&::connect) connect = nullptr;
   decltype(&::listen) listen = nullptr;
@@ -132,6 +133,18 @@ class LibcPassthrough {
 /// @brief Return the process-wide libc pass-through table.
 LibcPassthrough &libc_passthrough();
 
+/// @brief Call the resolved host function, falling back to libc before interposition is active.
+template <typename Function, typename Fallback, typename... Args>
+decltype(auto) host_call(Function resolved, Fallback fallback, Args... args) {
+  return (resolved ? resolved : fallback)(args...);
+}
+
+/// @brief Close an optional host descriptor, including before pass-through is resolved.
+inline void host_close_fd(int fd) {
+  if (fd >= 0)
+    host_call(libc_passthrough().close, ::close, fd);
+}
+
 /// @brief Compatibility entry points for operations absent from older libc releases.
 /// @details Prefer the resolved libc wrapper. If unavailable, use the syscall
 /// entry point; callers retain typed arguments and errno. Standalone callers
@@ -155,14 +168,7 @@ struct PassthroughFdTraits {
 
   static handle_type invalid() noexcept { return -1; }
   static bool is_valid(handle_type fd) noexcept { return fd >= 0; }
-  static void close(handle_type fd) noexcept {
-    // The table is unresolved until resolve() runs. These traits are reachable from
-    // any driver TU, so fall back to ::close rather than calling a null pointer;
-    // that path can only be taken before the interposer is live, so it cannot
-    // re-enter the hook this type exists to avoid.
-    auto *real_close = libc_passthrough().close;
-    static_cast<void>(real_close ? real_close(fd) : ::close(fd));
-  }
+  static void close(handle_type fd) noexcept { host_close_fd(fd); }
 };
 
 using UniqueDriverFd = util::BasicUniqueHandle<PassthroughFdTraits>;