[rakhmets]

Why?

[ RUN      ] dcx/test_ucp_mmap.rndv_mpool_mdesc_no_rcache/1 <dc_x,cuda_copy,rocm_copy/map_nb>
          ib_md.c:323  UCX  ERROR ibv_reg_mr(address=0xfffd20000000, length=536870912, access=0xf) failed: Bad address
         ucp_mm.c:87   UCX  ERROR failed to register address 0xfffd20000000 (cuda-managed) length 536870912 on md[1]=mlx5_6: Input/output error (md supports: host|cuda|cuda-managed)
          mpool.c:246  UCX  ERROR Failed to allocate memory pool (name=ucp_rndv_frags) chunk: Input/output error
[  FAILED  ] dcx/test_ucp_mmap.rndv_mpool_mdesc_no_rcache/1, where GetParam() = dc_x,cuda_copy,rocm_copy/map_nb (337 ms)

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

No test is added. The failure only reproduces on ODP-capable HW with IB_ODP_MEM_TYPES including cuda-managed and RNDV_FRAG_MEM_TYPES including cuda-managed; no gtest currently sets a managed frag memory type (test/gtest only uses host/cuda), so a regression test is likely impractical in CI — worth confirming with the author rather than demanding a test.

[svc-nvidia-pr-review]

🤖 Starting review — findings will be posted here when done.

[svc-nvidia-pr-review]

why not handle this in ucp_memh_register_internal()? ucp_mem_map() with UCP_MEM_MAP_LOCK on cuda-managed memory hits the same failure, and a central fix would cover all callers.

[svc-nvidia-pr-review]

can we cover the other managed types here? if IB_ODP_MEM_TYPES is set to rocm-managed or ze-managed, those frag pools fail the same way, since reg_mem_types gets the ODP types in uct_ib_md_check_odp().

[svc-nvidia-pr-review]

minor: chunk_hdr is leaked here; pls free it before returning. Pre-existing, but this is exactly the failure path this PR fixes.

[tvegas1]

The fix seems odd: the frag mpool hardcodes a memory type to bypass LOCK, so other callers (e.g. ucp_mem_map() with UCP_MEM_MAP_LOCK on managed memory) still fail. Note that LOCK was never implemented by IB MD; it only uses NONBLOCK to select ODP vs pinned registration.

Suggestions WDYT:

1. to instead have the memory detector report it? cuda_copy already identifies managed memory by CU_POINTER_ATTRIBUTE_IS_MANAGED and assumes it is registered via ODP (cuda_copy_md.c:893), so it could set a new UCS_MEM_FLAG_NOT_PINNABLE in uct_cuda_copy_md_detect_mem_flags(), and ucp_memh_register_internal() would replace LOCK with NONBLOCK for such memh. This covers all callers and can be extended to rocm/ze-managed. It needs OR semantics for this flag on rcache merge (ucp_mm.c:1765), since a negative flag is lost with AND.

2. Otherwise, maybe fix it in IB MD: UCP already passes mem_type for cuda-managed, so uct_ib_verbs_mem_reg() and uct_ib_mlx5_devx_mem_reg() can do:

/* CUDA managed memory cannot be pinned, it can be registered only by ODP */
if ((mem_type == UCS_MEMORY_TYPE_CUDA_MANAGED) &&
    (md->reg_nonblock_mem_types & UCS_BIT(mem_type))) {
    flags |= UCT_MD_MEM_FLAG_NONBLOCK;
}