UCT/CUDA_COPY: Skip dmabuf probe for async managed memory by pentschev · Pull Request #11922 · openucx/ucx

pentschev · 2026-09-08T12:39:10Z

What?

Fix CUDA async managed mempool memory detection in cuda_copy and add a regression test.

CUDA async managed allocations can report CU_POINTER_ATTRIBUTE_IS_MANAGED with no owning CUDA context. UCX now marks that case as async-managed, so mem_flags detection treats the memory as non-registerable and does not probe dmabuf export for it.

Why?

With UCX_CUDA_COPY_DMABUF=try, posting a UCP receive into CUDA 13 async managed mempool memory can crash inside cuMemGetHandleForAddressRange(). Setting UCX_CUDA_COPY_DMABUF=no avoids the crash because UCX skips the dmabuf export probe.

The crash happens during UCP memory detection when cuda_copy queries MEM_FLAGS: the allocation is managed memory, but because it has no CUDA context, it must be handled like other async managed allocations and kept off the dmabuf/registration path.

How?

The fix is local to cuda_copy memory attribute detection. When cuPointerGetAttributes() reports managed memory and the pointer has no CUDA context, mark it as is_async_managed. The existing mem_flags logic then returns non-registerable flags without calling cuMemGetHandleForAddressRange().

A CUDA 13 regression test allocates async managed memory from the default managed mempool with cudaMallocFromPoolAsync(), queries uct_md_mem_query_v2() with MEM_FLAGS while CUDA_COPY_DMABUF=try, and verifies the memory is classified as cuda-managed and not registrable.

CUDA async managed allocations can report CU_POINTER_ATTRIBUTE_IS_MANAGED with no owning CUDA context. Mark this case as async-managed so mem_flags detection keeps it non-registerable and does not call cuMemGetHandleForAddressRange(). This avoids a CUDA driver crash during UCP memory detection when UCX_CUDA_COPY_DMABUF is enabled. Add a CUDA 13 regression test that queries mem_flags for async managed mempool memory with dmabuf enabled.

svc-nvidia-pr-review · 2026-09-08T12:39:42Z

🤖 Starting review — findings will be posted here when done.

…mabuf-probe

svc-nvidia-pr-review · 2026-09-08T12:41:38Z

             * provided address and length as base address and alloc length
             * respectively */
            mem_info->type = UCS_MEMORY_TYPE_CUDA_MANAGED;
+            if (cuda_mem_ctx == NULL) {


why not gate on md->config.cuda_async_managed here as well, like the sibling branch below? otherwise pool-managed memory is marked non-registrable even when the user sets CUDA_COPY_ASYNC_MEM_TYPE=cuda.

Also worth confirming that ordinary cuMemAllocManaged memory never reports a NULL context (which would otherwise become a real registration regression), and that the config-mismatch above is intentional.

Done, added the same md->config.cuda_async_managed gate so CUDA_COPY_ASYNC_MEM_TYPE=cuda keeps this memory registrable.

svc-nvidia-pr-review · 2026-09-08T12:41:38Z

+                                       cudaMemAllocationTypeManaged));
+    ASSERT_EQ(cudaSuccess, cudaStreamCreate(&stream));
+
+    cudaError_t cuda_status = cudaMallocFromPoolAsync(&buffer, size, pool,


maybe skip instead of asserting here? a device/driver may not support managed mem pools even on CUDART>=13000, and this would turn into a hard test failure.

Done, the test now skips if the default managed mempool cannot be queried or async managed mempool allocation is unsupported.

svc-nvidia-pr-review · 2026-09-08T12:41:38Z

+
+    cuda_status = cudaStreamSynchronize(stream);
+    if (cuda_status != cudaSuccess) {
+        EXPECT_EQ(cudaSuccess, cudaFreeAsync(buffer, stream));


minor: pls cudaStreamSynchronize before cudaStreamDestroy here, to match the error path above and drain the async free.

svc-nvidia-pr-review · 2026-09-08T12:42:25Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-09-08T12:44:26Z

             * provided address and length as base address and alloc length
             * respectively */
            mem_info->type = UCS_MEMORY_TYPE_CUDA_MANAGED;
+            if (cuda_mem_ctx == NULL) {


minor: pls add a short comment here explaining why a NULL context indicates an async/stream-ordered (pool) managed allocation; the sibling else-if branch has this reasoning but it's non-obvious in the is_managed path.

Done, added a short comment explaining that managed pool allocations are stream-ordered and have no owning CUDA context.

svc-nvidia-pr-review · 2026-09-08T12:44:26Z

             * provided address and length as base address and alloc length
             * respectively */
            mem_info->type = UCS_MEMORY_TYPE_CUDA_MANAGED;
+            if (cuda_mem_ctx == NULL) {


why does a NULL context on genuine managed memory imply it's not registrable? regular cudaMallocManaged/cuMemAllocManaged memory is registrable, and with default PREF_LOC=cpu (sys_dev UNKNOWN) it used to report REGISTRABLE. can we confirm the driver always returns a non-NULL context for ordinary managed allocations, including when allocated under the pushed primary context in uct_cuda_copy_mem_alloc?

Added positive coverage for ordinary cudaMallocManaged memory and uct_mem_alloc(UCS_MEMORY_TYPE_CUDA_MANAGED). Both assert a non-NULL CUDA context and verify the allocation remains registrable.

svc-nvidia-pr-review · 2026-09-08T12:44:27Z

Test coverage gap: can we also add a positive test that ordinary cudaMallocManaged memory still reports UCS_MEM_FLAG_REGISTRABLE? The new test only covers the async-managed-pool negative case, so a genuine-managed regression from this heuristic would go uncaught. The existing detect_mem_type_cuda_managed tests only check mem_type, not mem_flags.

svc-nvidia-pr-review · 2026-09-08T14:32:32Z

🤖 Starting review — findings will be posted here when done.

Respect CUDA_COPY_ASYNC_MEM_TYPE when marking managed allocations without a CUDA context as async-managed. Add coverage for ordinary cudaMallocManaged memory remaining registrable and make the async managed mempool test skip when the device or driver does not support that allocation path.

pentschev · 2026-09-08T14:33:47Z

Test coverage gap: can we also add a positive test that ordinary cudaMallocManaged memory still reports UCS_MEM_FLAG_REGISTRABLE? The new test only covers the async-managed-pool negative case, so a genuine-managed regression from this heuristic would go uncaught. The existing detect_mem_type_cuda_managed tests only check mem_type, not mem_flags.

Done, added managed_mem_registrable to verify ordinary cudaMallocManaged memory still reports UCS_MEM_FLAG_REGISTRABLE.

svc-nvidia-pr-review · 2026-09-08T14:37:21Z

             * provided address and length as base address and alloc length
             * respectively */
            mem_info->type = UCS_MEMORY_TYPE_CUDA_MANAGED;
+            if ((cuda_mem_ctx == NULL) && md->config.cuda_async_managed) {


why is this gated on md->config.cuda_async_managed? managed pool memory is typed as cuda-managed regardless of ASYNC_MEM_TYPE, so with UCX_CUDA_COPY_ASYNC_MEM_TYPE=cuda we would still report it as registrable.

Removed the configuration dependency. The regression test now also runs with CUDA_COPY_ASYNC_MEM_TYPE=cuda, and managed-pool memory remains CUDA_MANAGED and non-registrable.

svc-nvidia-pr-review · 2026-09-08T14:37:21Z

+            if ((cuda_mem_ctx == NULL) && md->config.cuda_async_managed) {
+                /* Managed pool allocations are stream-ordered and do not have
+                 * an owning CUDA context. */
+                *is_async_managed = 1;


with the default PREF_LOC=cpu this caches sys_dev=UNKNOWN together with mem_flags=0, and ucp_memory_detect_internal() treats that combination as "no info" and falls back to ucp_memory_detect_slowpath() on every operation for these buffers. is the repeated re-detection acceptable?

This is a valid cache-performance concern. The current cache format cannot distinguish an incomplete UCM allocation event from a completed MD query when both have unknown system device and zero flags. I kept that broader UCS/UCP cache representation change out of this focused bug-fix PR.

svc-nvidia-pr-review · 2026-09-08T14:37:21Z

+    void *buffer                  = nullptr;
+    cudaError_t cuda_status;
+
+    cuda_status = cudaMallocManaged(&buffer, size);


maybe use mem_buffer buffer(size, UCS_MEMORY_TYPE_CUDA_MANAGED) here (test/AGENTS.md: reuse mem_buffer for generic allocations); it also releases the buffer if an assertion fails. Currently raw cudaMallocManaged/cudaFree leaks if an assertion aborts the test.

Replaced the raw cudaMallocManaged/cudaFree pair with mem_buffer, so the allocation is released if an assertion aborts the test.

svc-nvidia-pr-review · 2026-09-08T14:37:21Z

+        UCS_TEST_SKIP_R("failed to allocate async managed mempool memory");
+    }
+
+    cuda_status = cudaMemsetAsync(buffer, 0, size, stream);


minor: the three copies of the free/sync/destroy cleanup hide what the test checks; can we wrap the stream and the pool buffer in RAII helpers and keep only the query here?

Wrapped the stream-ordered managed-pool allocation and stream teardown in cuda_async_managed_mem_buffer. The test body now only performs setup, query, and assertions.

svc-nvidia-pr-review · 2026-09-08T14:38:04Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-09-08T14:43:36Z

             * provided address and length as base address and alloc length
             * respectively */
            mem_info->type = UCS_MEMORY_TYPE_CUDA_MANAGED;
+            if ((cuda_mem_ctx == NULL) && md->config.cuda_async_managed) {


Registrability of managed-pool memory is gated on an unrelated config knob. In this branch is_managed is already true, so mem_info->type is UCS_MEMORY_TYPE_CUDA_MANAGED regardless of UCX_CUDA_COPY_ASYNC_MEM_TYPE (md->config.cuda_async_managed only decides the type of non-managed stream-ordered allocations, see the else if at line 758). With UCX_CUDA_COPY_ASYNC_MEM_TYPE=cuda — a configuration already used by existing tests in this file (no_current_context_cuda_registrable) — the same managed-pool buffer is reported as registrable again, i.e. the bug this PR fixes reappears. Registrability is a property of the allocation, not of the reported memory type.

Why gate on md->config.cuda_async_managed here? Suggested alternative:

Suggested change

if ((cuda_mem_ctx == NULL) && md->config.cuda_async_managed) {

if (cuda_mem_ctx == NULL) {

/* Managed pool allocations are stream-ordered and do not have

* an owning CUDA context. */

*is_async_managed = 1;

}

Alternatively, CU_POINTER_ATTRIBUTE_MEMPOOL_HANDLE (already queried in cuda_ipc_md.c) detects pool allocations directly, without the ctx heuristic.

svc-nvidia-pr-review · 2026-09-08T14:43:36Z

+            if ((cuda_mem_ctx == NULL) && md->config.cuda_async_managed) {
+                /* Managed pool allocations are stream-ordered and do not have
+                 * an owning CUDA context. */
+                *is_async_managed = 1;


Cached entry becomes indistinguishable from "not detected", forcing a per-operation slow path. When the preferred location resolves to the CPU (PREF_LOC=cpu is the default, and the new positive test uses it), the managed branch sets sys_dev = UCS_SYS_DEVICE_ID_UNKNOWN; with is_async_managed = 1 the cached mem_flags is now 0. ucp_memory_detect_internal() (src/ucp/core/ucp_context.h:746) treats the (sys_dev == UNKNOWN) && (mem_flags == 0) pair as "attributes not detected" and calls ucp_memory_detect_slowpath() on every operation with such a buffer. Before this change these entries were cached with UCS_MEM_FLAG_REGISTRABLE and hit the fast path. Functionally correct, but a hot-path cost for exactly the memory this PR targets — is the per-operation slow path acceptable here?

svc-nvidia-pr-review · 2026-09-08T14:43:37Z

Residual coverage gap: async_managed_mem_pool_not_registrable is compiled only with CUDART >= 13000 and skips unless the device supports concurrent managed access and managed default mem pools; worth confirming a CI job actually runs it rather than skipping.

Residual coverage gap: Neither new test covers PREF_LOC=gpu, so the managed path where sys_dev is a real device (and mem_flags then depends on dmabuf export) stays untested.

tvegas1 · 2026-09-08T15:08:39Z

             * provided address and length as base address and alloc length
             * respectively */
            mem_info->type = UCS_MEMORY_TYPE_CUDA_MANAGED;
+            if ((cuda_mem_ctx == NULL) && md->config.cuda_async_managed) {


CU_POINTER_ATTRIBUTE_MEMPOOL_HANDLE output might be usable for detection and probably md->config.cuda_async_managed is not exactly related as it should apply to memory that is not is_managed where user can choose memory type to use, so it should be removed.

Implemented this. The code now queries CU_POINTER_ATTRIBUTE_MEMPOOL_HANDLE and uses a non-NULL pool handle to identify managed-pool allocations. I also removed the unrelated md->config.cuda_async_managed gate from the managed-memory branch. The managed-pool regression test now covers CUDA_COPY_ASYNC_MEM_TYPE=cuda.

svc-nvidia-pr-review · 2026-09-08T16:21:52Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-09-08T16:27:50Z

             * provided address and length as base address and alloc length
             * respectively */
            mem_info->type = UCS_MEMORY_TYPE_CUDA_MANAGED;
+            if (cuda_mempool != NULL) {


Hot-path concern: with the default PREF_LOC=cpu this path caches sys_dev=UNKNOWN together with mem_flags=0, which is exactly the condition that makes ucp_memory_detect_internal() call ucp_memory_detect_slowpath() (uct_md_mem_query_v2 + cuPointerGetAttributes + cuMemRangeGetAttribute + memtype-cache update under a spinlock) on every operation for these buffers. The pre-existing is_async_managed branch avoids this because it gets a real sys_dev; this new branch is the first to produce the UNKNOWN+0 combination. Can we avoid the per-op re-detection, e.g. by reporting the buffer's GPU sys_dev for managed-pool allocations?

Fixed. Managed-pool allocations now use their pointer-reported GPU sys_dev, rather than the CPU preferred-location result. This prevents caching the UNKNOWN plus zero-flags combination and avoids repeated UCP slow-path detection. The regression test now asserts a known system device.

svc-nvidia-pr-review · 2026-09-08T16:27:51Z

        attr_data[2] = &cuda_device;
        attr_type[3] = CU_POINTER_ATTRIBUTE_CONTEXT;
        attr_data[3] = &cuda_mem_ctx;
+        attr_type[4] = CU_POINTER_ATTRIBUTE_MEMPOOL_HANDLE;


minor: CU_POINTER_ATTRIBUTE_MEMPOOL_HANDLE and CUmemoryPool are CUDA 11.2+; every other use in the tree is guarded (#if CUDA_VERSION >= 11020 in ucm/cuda/cudamem.c, HAVE_CUDA_FABRIC/HAVE_DECL_SYS_PIDFD_GETFD in cuda_ipc_md.c / cuda_ipc_cache.c). This is the first unguarded use, so building against older CUDA headers would break — do we need a guard here too?

Added #if CUDA_VERSION >= 11020 guards around the mempool type, fifth pointer attribute, and managed-pool detection path. Older CUDA headers retain the four-attribute query.

svc-nvidia-pr-review · 2026-09-08T16:27:51Z

             * provided address and length as base address and alloc length
             * respectively */
            mem_info->type = UCS_MEMORY_TYPE_CUDA_MANAGED;
+            if (cuda_mempool != NULL) {


minor (compactness): the if can be a direct assignment:

Suggested change

if (cuda_mempool != NULL) {

/* Managed-pool allocations are stream-ordered and cannot be

* registered. */

*is_async_managed = (cuda_mempool != NULL);

Kept the branch because the managed-pool case now also sets the allocation’s GPU sys_dev, validates it, and skips the preferred-location query. A direct assignment would not express the required locality handling.

svc-nvidia-pr-review · 2026-09-08T16:53:46Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-09-08T17:00:03Z

+                /* Managed-pool allocations are stream-ordered and cannot be
+                 * registered. */
+                *is_async_managed = 1;
+                mem_info->sys_dev = uct_cuda_get_sys_dev(cuda_device);


does this need the cuda_device == CU_DEVICE_CPU handling that the common path below has? for a host-located managed pool uct_cuda_get_sys_dev() returns UNKNOWN and then the whole query fails with UCS_ERR_NO_DEVICE.

Fixed. The managed-pool path now mirrors the common CPU-device handling: it reports UCS_SYS_DEVICE_ID_UNKNOWN for CU_DEVICE_CPU instead of failing with UCS_ERR_NO_DEVICE.

svc-nvidia-pr-review · 2026-09-08T17:00:03Z

        attr_type[3] = CU_POINTER_ATTRIBUTE_CONTEXT;
        attr_data[3] = &cuda_mem_ctx;
+#if CUDA_VERSION >= 11020
+        attr_type[4] = CU_POINTER_ATTRIBUTE_MEMPOOL_HANDLE;


maybe reuse cuda_mempool for the (cuda_mem_ctx == NULL) heuristic below as well? it identifies stream-ordered allocations directly, so the context-based guess and its driver-bug comment could go away (fine as a follow-up).

Leaving this as a follow-up. CU_POINTER_ATTRIBUTE_MEMPOOL_HANDLE is unavailable before CUDA 11.2, while the existing context heuristic remains the compatibility fallback for older CUDA headers. Removing it here would broaden this bug-fix PR and reduce coverage for that configuration.

svc-nvidia-pr-review · 2026-09-08T17:00:04Z

Residual gap (previously reported): CUDART >= 13000 gating plus the concurrent-managed-access / default-managed-pool skips mean the negative tests may never actually execute in CI.

Residual gap (previously reported): PREF_LOC=gpu remains uncovered for the managed path.

Add positive coverage for ordinary managed memory so CUDA managed allocations with an owning context keep reporting UCS_MEM_FLAG_REGISTRABLE. Cover both user cudaMallocManaged memory and memory allocated through uct_mem_alloc.

Use the CUDA mempool pointer attribute to identify managed stream-ordered pool allocations. Such allocations are not registrable regardless of CUDA_COPY_ASYNC_MEM_TYPE, so skip dmabuf probing for them. Extend the regression test to cover CUDA_COPY_ASYNC_MEM_TYPE=cuda and use RAII for CUDA test allocations.

Report the GPU system device for managed pool allocations so the non-registrable cache entry does not trigger UCP memory detection on every operation. Guard mempool pointer attributes for CUDA versions before 11.2.

Preserve the common CUDA CPU-device handling for managed pool allocations and cover GPU preferred location in the managed pool regression test.

svc-nvidia-pr-review · 2026-09-10T13:29:21Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-09-10T13:35:54Z

+    if (status != UCS_OK) {
+        printf("memtype cache lookup failed: %s\n", ucs_status_string(status));
+        ++num_errors;
+    } else if (cached_mem_info.mem_flags & UCS_MEM_FLAG_REGISTRABLE) {


does this hold when dmabuf is not supported? there uct_cuda_copy_md_is_registrable() returns 1 for the symbol (driver type is cuda with a real sys_dev), so the app fails on such a machine. maybe check the flags only when dmabuf is supported?

svc-nvidia-pr-review · 2026-09-10T13:35:54Z


+    /* Managed memory is registered through ODP and is not dmabuf-exportable. */
+    if (mem_info->type == UCS_MEMORY_TYPE_CUDA_MANAGED) {
+        return UCS_MEM_FLAG_REGISTRABLE;


Suggested change

return UCS_MEM_FLAG_REGISTRABLE;

return 1;

minor: the function is a boolean predicate, the other branches return 0/1.

svc-nvidia-pr-review · 2026-09-10T13:35:55Z

+                              UCT_MD_MEM_ATTR_V2_FIELD_MEM_FLAGS;
+        EXPECT_UCS_OK(uct_md_mem_query_v2(md(), buffer.ptr(), size, &mem_attr));
+        EXPECT_EQ(UCS_MEMORY_TYPE_CUDA_MANAGED, mem_attr.mem_type);
+        EXPECT_TRUE(mem_attr.mem_flags & UCS_MEM_FLAG_REGISTRABLE);


minor: can we reuse query_managed_registrable() here? the only difference is the context check, which can move to its caller.

tomerg-nvidia · 2026-09-10T14:54:43Z

    CUdevice avail_cuda_device = CU_DEVICE_INVALID;
    ucs_memory_info_t cached_mem_info;
    ucs_memory_info_t addr_mem_info;
+    ucs_memory_info_t detected_mem_info = {};


nit: initialized variables go above unintialized variables.

Issue with symbols looks resolved

svc-nvidia-pr-review · 2026-09-10T15:18:23Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-09-10T15:26:20Z

+    if (status != UCS_OK) {
+        printf("memtype cache lookup failed: %s\n", ucs_status_string(status));
+        ++num_errors;
+    } else if (cached_mem_info.mem_flags & UCS_MEM_FLAG_REGISTRABLE) {


this expects dmabuf export to fail for the symbol, but when the build or device has no dmabuf support (CUDA < 11.7, or UCX_CUDA_COPY_DMABUF=n), uct_cuda_copy_md_is_registrable() returns 1 and this check fails. can we make the expectation independent of dmabuf support?

Rationale: uct_cuda_copy_md_is_dmabuf_supported() compiles the attribute query out below CUDA 11.7, so md->config.dmabuf_supported == 0 and is_registrable() returns 1 at the !md->config.dmabuf_supported short-circuit, i.e. the cached symbol flags would contain UCS_MEM_FLAG_REGISTRABLE. buildlib/pr/cuda/test_malloc_hook.sh runs this app unconditionally.

Removed the zero-flags assertion.

svc-nvidia-pr-review · 2026-09-10T15:26:21Z

    EXPECT_EQ(CUDA_SUCCESS, cuMemFree(dptr));
 }

+UCS_TEST_P(test_mem_alloc_device, user_managed_mem_registrable,


with PREF_LOC=cpu (which is also the default) the managed branch sets sys_dev to unknown, so is_registrable() returns 1 before reaching the new CUDA_MANAGED check - this test and uct_alloc_managed_mem_registrable pass without the fix too. can we use gpu here?

Suggested change

UCS_TEST_P(test_mem_alloc_device, user_managed_mem_registrable,

UCS_TEST_P(test_mem_alloc_device, user_managed_mem_registrable,

"CUDA_COPY_PREF_LOC=gpu")

svc-nvidia-pr-review · 2026-09-10T15:26:21Z

+
+    void test_async_managed_mem_pool_registrable()
+    {
+        constexpr size_t size         = 192;


minor: extra spaces, nothing to align with here.

Suggested change

constexpr size_t size = 192;

constexpr size_t size = 192;

svc-ucx · 2026-09-10T16:05:01Z

🤖 CI Triage Agent — UCX PR (Coverity coverity release on coverity_rh7) · commit 437da172

TL;DR: The Coverity release stage failed on a single new UNINIT defect: src/uct/cuda/cuda_copy/cuda_copy_md.c:1020 copies the whole addr_mem_info struct while its mem_flags field is still uninitialized. Fix by zero-initializing addr_mem_info (= {}) at its declaration (line 992), or by having uct_cuda_copy_md_query_attributes() set mem_flags = 0.

Full analysis

Summary: Azure Pipelines job "Coverity coverity release on coverity_rh7" (build 135586) failed with ##[error]Coverity found 1 issues — a UNINIT (uninitialized scalar variable) defect in the CUDA copy MD; the build and analysis themselves succeeded (551 units compiled, 96 total defect occurrences, gate trips on the 1 newly-formatted error).

Root cause: In uct_cuda_copy_md_mem_query(), ucs_memory_info_t addr_mem_info; is declared without an initializer (line 992). uct_cuda_copy_md_query_attributes() populates only type, sys_dev, base_address and alloc_length — it never writes mem_flags (that field is only assigned later, at line 1074). The new statement detected_mem_info = addr_mem_info; at line 1020 performs a full struct copy, which reads the still-uninitialized mem_flags. Coverity's trace is explicit:

step 2 var_decl: "Declaring variable addr_mem_info without initializer" (line 992)
step 6 uninit_use: "Using uninitialized value addr_mem_info. Field addr_mem_info.mem_flags is uninitialized." (line 1020)

Note the sibling variable detected_mem_info on line 990 is zero-initialized (= {}), and default_mem_info is zero-filled by its designated initializer — addr_mem_info is the odd one out. This is a real (if benign) uninitialized read, not a false positive.

Implicated commit: a7cbb96f — "UCT/CUDA_COPY: Preserve device symbol flags", Peter Andreas Entschev (introduced the detected_mem_info = addr_mem_info; full-struct copy); tip of branch is 437da172 "UCT/CUDA_COPY: Simplify managed memory checks", same author.

File: src/uct/cuda/cuda_copy/cuda_copy_md.c:1020 (declaration at src/uct/cuda/cuda_copy/cuda_copy_md.c:992)

Suggested fix: Zero-initialize the variable at declaration so the struct copy is well-defined:

-    ucs_memory_info_t addr_mem_info;
+    ucs_memory_info_t addr_mem_info = {};

Alternatives, either of which also silences the checker and is arguably cleaner:

Have uct_cuda_copy_md_query_attributes() explicitly initialize mem_info->mem_flags = 0 on all its return paths (including out_default_range), making the contract "fully populated" instead of "partially populated".
Avoid the full-struct copy at line 1020 and copy only the fields actually needed by uct_cuda_copy_md_detect_mem_flags() (type, sys_dev, base_address, alloc_length) into the already-zeroed detected_mem_info.

Option 1 is the most robust, since addr_mem_info is also fed to ucs_memtype_cache_update() and future field additions to ucs_memory_info_t would hit the same trap.

Related: PR #11922 (branch cuda-async-skip-dmabuf-probe); no pre-existing issue found for this defect — it is newly introduced by this PR's mem_flags/device-symbol work.

svc-nvidia-pr-review · 2026-09-10T18:38:00Z

🤖 Starting review — findings will be posted here when done.

pentschev · 2026-09-10T18:40:01Z

🤖 CI Triage Agent — UCX PR (Coverity coverity release on coverity_rh7) · commit 437da172

TL;DR: The Coverity release stage failed on a single new UNINIT defect: src/uct/cuda/cuda_copy/cuda_copy_md.c:1020 copies the whole addr_mem_info struct while its mem_flags field is still uninitialized. Fix by zero-initializing addr_mem_info (= {}) at its declaration (line 992), or by having uct_cuda_copy_md_query_attributes() set mem_flags = 0.

Fixed.

svc-nvidia-pr-review · 2026-09-10T18:47:50Z

    int is_host_located        = 0;
    CUdevice cur_cuda_device   = CU_DEVICE_INVALID;
    CUdevice avail_cuda_device = CU_DEVICE_INVALID;
+    ucs_memory_info_t detected_mem_info = {};


minor: can we drop the = {}? both structs are fully assigned before any field is read, and the initializers break the = alignment of the block above (docs/CodeStyle.md: consecutive assignments align on one column).

Suggested change

ucs_memory_info_t detected_mem_info = {};

ucs_memory_info_t detected_mem_info;

ucs_memory_info_t addr_mem_info;

ucs_memory_info_t cached_mem_info;

#11922 (comment)

svc-ucx · 2026-09-10T20:07:52Z

🤖 CI Triage Agent — UCX PR (Tests roce on worker 0) · commit fae7ff12

TL;DR: The build failed on a single gtest, rcx/test_ucp_fault_tolerance.probe_gated_recovery/6 (rc_x/AM), which is a race-prone test unrelated to this PR's CUDA/dmabuf change — it only samples the transient recovery_arg->probe[lane].comp.func state between short_progress_loop() batches, so a fast recovery makes EXPECT_TRUE(probe_armed) fail. Fix the test to count probe arming deterministically (mock/counter) instead of polling for a window that can open and close inside one progress batch.

Full analysis

Summary: Tests roce on worker 0 (Azure build 135618) failed with make: *** [Makefile:4713: test] Error 1; 8724/8725 tests passed, the only failure being rcx/test_ucp_fault_tolerance.probe_gated_recovery/6, where GetParam() = rc_x/AM.

Root cause: Flaky assertion in the fault-tolerance gtest, not a product regression from this PR. In probe_gated_recovery the test polls with wait_for_cond(...): the condition lambda samples ep->ext->recovery_arg->probe[lane].comp.func != NULL once per iteration, and the progress action is short_progress_loop(), which runs many worker progress iterations per call. On the product side the aux probe window is very short: ucp_ep_recovery_rebuild_p2p_lane() arms the probe (ucp_ep_recovery_arm_probe, src/ucp/core/ucp_ep.c:2172/:2155) and clears it again with ucp_ep_recovery_reset_probe() (src/ucp/core/ucp_ep.c:2188, sets comp.func = NULL) as soon as connect_to_ep_v2 succeeds. Arm → uct_ep_check completion → connect → reset can therefore all happen inside a single short_progress_loop(), and wait_for_cond exits as soon as ucp_ep_get_failed_lanes(ep) == 0, so probe_armed is never observed and EXPECT_TRUE(probe_armed) << "RC p2p lane recovery completed without arming an aux probe" fires. Nothing in the log points at CUDA/dmabuf code (the job is a RoCE worker running an RC AM fault-tolerance test), and no timeout/SIGTERM/hang signature is present — the suite ran to completion in 3321 s with continuous output.

Implicated commit: db208ee "UCP/FT: probe-gated lane recovery via aux uct_ep_check (#11563)", Evgeny Leksikov (introduced both the probe gating and this test). Timing likely aggravated by fb88400 "UCP/WIREUP: Add a token trailer and an ACK message to the lane address exchange (#11843)", Evgeny Leksikov, merged the same day (2026-09-10), which shortens the reply→probe→connect path. Not caused by fae7ff1 (cuda-async-skip-dmabuf-probe).

File: test/gtest/ucp/test_ucp_fault_tolerance.cc:913-943 (assert at :941); product-side probe window src/ucp/core/ucp_ep.c:2072-2097 and src/ucp/core/ucp_ep.c:2188.

Suggested fix: Make the probe observation deterministic instead of sampled:

Wrap ep_check with the existing mock_recovery_probe() helper using a counting stub (increment a static counter, then forward/return UCS_INPROGRESS) and assert the counter is non-zero — this cannot be missed by polling; or
add a cumulative counter (e.g. worker->counters.recovery_probes incremented in ucp_ep_recovery_arm_probe()) and assert it grew, instead of inspecting probe[lane].comp.func; or
as a minimal mitigation, replace short_progress_loop() in that specific wait_for_cond with a single progress() step so the sampling granularity is finer.
Until fixed, this failure should be treated as unrelated to PR UCT/CUDA_COPY: Skip dmabuf probe for async managed memory #11922 and the job re-run.

Related: #11563 (introduced probe-gated recovery + test), #11843 (LANES_ADDR ACK/token trailer, changes recovery timing), #11397 (precedent: previously disabled a flaky fault-tolerance test)

pentschev · 2026-09-12T19:54:00Z

/ok to test

This adjusts our UCX pinning to exclude `ucx==1.22.0`. That release includes an issue that occasionally causes a segfault in some rapidsmpf and cudf tests. The issue is fixed in ucx main by openucx/ucx#11922. Closes #783 Authors: - Tom Augspurger (https://github.com/TomAugspurger) Approvers: - Bradley Dice (https://github.com/bdice) URL: #784

Use the official v1.23.2 release asset and verify its checksum. Its archive and embedded runtime version remain labeled 1.23.0. Backport openucx/ucx#11922 (2054662d38316b3688ba4a5631f3fdd12d218c51) with its upstream tests, preserving the release-specific function argument. Apply the patch only to the verified release and preserve explicit local, master, and other-version source overrides. Install GNU patch explicitly and stop on checksum or application failures, including Docker AND-list invocations that suppress Bash errexit.

pentschev self-assigned this Sep 8, 2026

pentschev added Bugfix Ready for Review labels Sep 8, 2026

pentschev mentioned this pull request Sep 8, 2026

SIGSEGV in CUDA DMA-BUF probing for CUDA 13 async-managed pool allocations rapidsai/ucxx#749

Closed

Merge remote-tracking branch 'upstream/master' into cuda-async-skip-d…

65e7d2c

…mabuf-probe

svc-nvidia-pr-review reviewed Sep 8, 2026

View reviewed changes

pentschev force-pushed the cuda-async-skip-dmabuf-probe branch from e502c49 to 1d53937 Compare September 8, 2026 14:32

svc-nvidia-pr-review reviewed Sep 8, 2026

View reviewed changes

tvegas1 reviewed Sep 8, 2026

View reviewed changes

svc-nvidia-pr-review reviewed Sep 8, 2026

View reviewed changes

pentschev added 4 commits September 8, 2026 17:36

GTEST/CUDA: Cover managed memory registration flags

458aeca

Add positive coverage for ordinary managed memory so CUDA managed allocations with an owning context keep reporting UCS_MEM_FLAG_REGISTRABLE. Cover both user cudaMallocManaged memory and memory allocated through uct_mem_alloc.

UCT/CUDA_COPY: Preserve managed pool device locality

0fb53b3

Report the GPU system device for managed pool allocations so the non-registrable cache entry does not trigger UCP memory detection on every operation. Guard mempool pointer attributes for CUDA versions before 11.2.

UCT/CUDA_COPY: Handle managed pool CPU devices

1b27615

Preserve the common CUDA CPU-device handling for managed pool allocations and cover GPU preferred location in the managed pool regression test.

pentschev force-pushed the cuda-async-skip-dmabuf-probe branch from c5fcf7b to 1b27615 Compare September 8, 2026 17:39

pentschev dismissed tvegas1’s stale review via a7cbb96 September 10, 2026 13:29

svc-nvidia-pr-review reviewed Sep 10, 2026

View reviewed changes

tomerg-nvidia reviewed Sep 10, 2026

View reviewed changes

UCT/CUDA_COPY: Simplify managed memory checks

437da17

svc-nvidia-pr-review reviewed Sep 10, 2026

View reviewed changes

UCT/CUDA_COPY: Initialize memory query result

fae7ff1

svc-nvidia-pr-review reviewed Sep 10, 2026

View reviewed changes

tvegas1 approved these changes Sep 11, 2026

View reviewed changes

tomerg-nvidia approved these changes Sep 11, 2026

View reviewed changes

tvegas1 merged commit 2054662 into openucx:master Sep 14, 2026
162 checks passed

svc-ucx mentioned this pull request Sep 20, 2026

UCP/WIREUP: Activate AM lane iface while EP config is used #11971

Merged

svc-ucx mentioned this pull request Oct 1, 2026

UCP/AM: Extract ucs_status from AM bcopy return before use #12007

Merged

svc-ucx mentioned this pull request Oct 17, 2026

CI: Jenkins integration testing (do not merge) #12094

Draft

This was referenced Oct 20, 2026

Flaky segmentation fault in cudf libcudf_streaming_ucxx_tests tests rapidsai/ucxx#783

Closed

Exclude ucx=1.22.0 from supported versions. rapidsai/ucxx#784

Merged

bdice mentioned this pull request Oct 20, 2026

[DO NOT MERGE] Diagnose arm64 L4 UCX crash in conda polars and streaming tests NVIDIA/cudf#24689

Closed

svc-ucx mentioned this pull request Oct 30, 2026

Jenkins CI: Gate Jenkins jobs on a basic compile #12166

Draft

vyasr mentioned this pull request Nov 5, 2026

[BUG] Intermittent segfault in multi-rank cudf_streaming tests on arm64 since UCX 1.22 / Open MPI 5.0.11 NVIDIA/cudf#24682

Closed

svc-ucx mentioned this pull request Nov 6, 2026

Coverity: Uinitialized variable fixes for Coverity 2026.3.0 Upgrade #12209

Open

	UCS_TEST_P(test_mem_alloc_device, user_managed_mem_registrable,
	UCS_TEST_P(test_mem_alloc_device, user_managed_mem_registrable,
	"CUDA_COPY_PREF_LOC=gpu")

Conversation

pentschev commented Sep 8, 2026

What?

Why?

How?

Uh oh!

svc-nvidia-pr-review commented Sep 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

svc-nvidia-pr-review commented Sep 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

svc-nvidia-pr-review commented Sep 8, 2026

Uh oh!

svc-nvidia-pr-review commented Sep 8, 2026

Uh oh!

pentschev commented Sep 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

svc-nvidia-pr-review commented Sep 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

svc-nvidia-pr-review commented Sep 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

svc-nvidia-pr-review commented Sep 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

svc-nvidia-pr-review commented Sep 8, 2026

Uh oh!