DEVICE/API: Wait for wireup completion in createGpuXferReq

[michal-shalev]

What?

Add blocking wait for endpoint wireup completion in createGpuXferReq().

Why?

Previously, users had to implement workarounds in their applications to wait for wireup completion before calling createGpuXferReq() (as shown in UCX tests).
This PR moves the wireup handling into the library, simplifying the API and removing the burden from application code.

How?

• Retry ucp_device_mem_list_create() in a loop while it returns UCS_ERR_NOT_CONNECTED
• Call worker.progress() in each iteration to advance the wireup state machine
• Document the blocking behavior in the public API

[github-actions]

👋 Hi michal-shalev! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[michal-shalev]

/build

[michal-shalev]

/build

[michal-shalev]

/build

[brminich]

/build

[rakhmets]

What do you think about making it configurable via environment variable?

[rakhmets]

I think it makes sense to swap the time check and the execution of the progress on the workers. Otherwise, we may throw the exception even when the wireup is completed on this iteration.

Optional. I'd prefer to do a time loop. E.g.:

for (const auto start = std::chrono::steady_clock::now();
     std::chrono::steady_clock::now() - start <= timeout;)
{
    status = ucp_device_mem_list_create(ep.getEp(), &params, &ucx_handle);
    if (status != UCS_ERR_NOT_CONNECTED) {
        break;
    }

    for (const auto &w : workers) {
        w->progress();
    }
}

if (status == UCS_ERR_NOT_CONNECTED) {
    throw std::runtime_error("Timeout waiting for endpoint wireup completion has been exceeded");
} else if (status != UCS_OK) {
    throw std::runtime_error(std::string("Failed to create device memory list: ") +
                             ucs_status_string(ucs_status));
}

[rakhmets]

It looks like all_ is redundant for this parameter. workers would be enough.