Repository navigation
UCT/ZE/COPY: Initialize memtype cache when opening MD - #12069
Conversation
Without an earlier cache lookup, a Level Zero allocation can occur before the UCM handler is installed. A subsequent cache miss makes UCP assume host memory, which can trigger the ucp_mm.c memory-type assertion or a crash in the host eager-short path. Create the cache at ZE copy MD open, as CUDA copy does. This fixes ZE-device transfers on paths without a DMA-BUF registration query, such as UCX_TLS=sm,ze_copy,ze_ipc. Signed-off-by: Yaser Afshar <yaser.afshar@intel.com>
|
🤖 Starting review — findings will be posted here when done. |
|
Residual coverage gaps: the test only runs on a host with a working Level Zero runtime and device, and there is no CI evidence in the diff that it was exercised, so the ZE path is validated by hardware-gated tests only. Also, only |
Signed-off-by: Yaser Afshar <yaser.afshar@intel.com>
|
🤖 Starting review — findings will be posted here when done. |
|
Residual coverage gap: only
Residual coverage gap: the whole file is hardware-gated (working Level Zero runtime + device), so there is no CI evidence in the diff that the new test actually ran. |
Signed-off-by: Yaser Afshar <yaser.afshar@intel.com>
|
🤖 Starting review — findings will be posted here when done. |
Signed-off-by: Yaser Afshar <yaser.afshar@intel.com>
Signed-off-by: Yaser Afshar <yaser.afshar@intel.com>
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: Full analysisSummary: The "Tests roce on worker 0" job's Root cause: This is a hang, not a slow test. Timeline evidence from the step log:
That is a single ~3m08s gap with no output at all, i.e. essentially the entire remaining runtime, and the cleanup section confirms The only functional change on this branch is in the ZE copy MD. /* Install the memtype cache handler before Level Zero allocations. */
ucs_memtype_cache_global_create();
return UCS_OK;Previously the global memtype cache was created lazily, on the first So Implicated commit: File: Suggested fix:
Related: PR #12069 (this PR, "UCT/ZE/COPY: Initialize memtype cache when opening MD"); adjacent ZE work in PR #11903 ("UCT/ZE/COPY: Allocate DMA-BUF-exportable memory for RDMA"). Historical context for the locking fragility: commit |
|
/azp run UCX PR |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Signed-off-by: Yaser Afshar <yaser.afshar@intel.com> Co-authored-by: Thomas Vegas <125344591+tvegas1@users.noreply.github.com>
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 Review complete — no issues found. |
|
🤖 CI Triage Agent — Summary: Unable to fetch logs — cannot diagnose without them.
|
|
/azp run UCX PR |
|
Azure Pipelines successfully started running 1 pipeline(s). |
What?
Create the global memtype cache when the
ze_copymemory domain is opened, ascuda_copyalready does (cuda_copy_md.c, "install its handler before CUDA events"). Add aze_copyMD gtest which checks that the cache exists right afteruct_md_open()and that a Level Zero allocation made by the application afterwards is found in it. Theze_copyMD tests now use the sharedtest_mdfixture.Why?
If nothing else has initialized the global memtype cache, it is created on the first lookup. A Level Zero allocation made before that has no handler to receive its UCM
MEM_TYPE_ALLOCevent, and in the reproduced case UCM's scan of/proc/self/mapsdid not recover it, since the device VA has no mapping in the process. Without a later memory query to populate the cache, the lookup misses and UCP assumes host memory. Depending on the protocol this ends in theucp_mm.cassertionmem_type == memh->mem_type(mem_type=host got memh->mem_type=ze-device) or a segfault in the host eager-short path.A successful DMA-BUF query can mask the bug, because
uct_ze_copy_md_mem_query()updates the cache as a side effect. Paths without such a query hit it;UCX_TLS=sm,ze_copy,ze_ipcis a reproducer on master today.Verified on Intel Arc Pro B60 and B70 nodes:
ucx_perftest -t tag_bw -m ze-device -s 4MwithUCX_TLS=sm,ze_copy,ze_ipc(two processes, one device, B60): asserts on master, passes with this change.UCX_TLS=rc,ze_copy, device-to-device over the NIC,tag_bw/ucp_put_bw/ucp_getat 8 B, 64 KiB, 4 MiB and-Vpingpong validation (B60): all pass, as before.ucs_memtype_cache_global_instanceis NULL afteruct_md_open()) and passes with it, on B60 and B70. Theze_cpy,ze_copyand memtype cache gtests pass on both, excepttest_memtype_cache_deferred_createfor ze-device, which already fails on master.How?
One call to
ucs_memtype_cache_global_create()at the end ofuct_ze_copy_md_open().The regression test resets the global cache (
ucs_memtype_cache_cleanup()+ucs_memtype_cache_global_init(), the same pattern astest_memtype_cache_deferred_create) in the fixture'sinit(), beforetest_md::init()opens the MD. An earlier test in the same process may already have created the cache on demand, and without the reset the test would pass regardless of the fix. The test runs withUCX_MEMTYPE_CACHE=y, so a failure to create the cache fails it instead of skipping, and it allocates throughmem_buffer, as an application would.The
ze_copyMD tests now derive fromtest_md, so they are only instantiated where aze_cpyMD exists; CI without Intel GPUs runs none of them rather than reporting skips.component_is_registeredis removed because it only compared the name stored in the component struct and would pass even if the component were not registered.query_md_resources_returns_at_least_oneis removed because undertest_mdit would only be instantiated after aze_cpyresource has been found.