fix(libfabric): Apply CUDA context inside progress thread loop

[dmvevents]

Summary

Fixes GPU memory operation failures in the libfabric backend by applying the CUDA context inside the progress thread loop instead of outside it.

Problem

The libfabric backend's progress thread currently doesn't properly handle CUDA context changes during operation. This causes failures when:

1. Progress thread starts in constructor BEFORE registerMem() is called
2. pthrCudaCtx_ is NULL until first GPU memory registration
3. Unlike the UCX backend, libfabric doesn't restart the thread on context change

Solution

Call vramApplyCtx() at the start of each progress loop iteration when cuda_addr_wa_ is set.

while (!progress_thread_stop_.load()) {
#ifdef HAVE_CUDA
    if (cuda_addr_wa_) {
        vramApplyCtx();
    }
#endif
    // ... rest of progress loop
}

This is safe because:

• vramApplyCtx() checks cuda_addr_wa_ internally before doing anything
• Performance overhead is ~2us per iteration, acceptable for KV cache latency
• The UCX backend already handles this by restarting the progress thread

Testing

• Tested on AWS p5.48xlarge (8x H100, 32 EFA)
• Verified with TRT-LLM disaggregated inference
• Verified with vLLM disaggregated inference
• No GPU memory errors or CUDA context issues after fix

Test Plan

• TRT-LLM disaggregated inference with GPU Direct RDMA
• vLLM disaggregated inference with GPU Direct RDMA
• Multi-rail (32 EFA) configuration
• No performance regression observed

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

👋 Hi dmvevents! Thank you for contributing to ai-dynamo/nixl.

Your PR reviewers will review your contribution then trigger the CI to test your changes.

🚀

[brminich]

cuda dependency is removed from UCX backend and is handled inside UCX now, probably this comment is not relevant anymore

[brminich]

/ok to test ae2e91c

[brminich]

/build

[dmvevents]

Closing Per Maintainer Clarification

@amitrad-aws - Thank you for clarifying the progress thread design.

You're correct that fi_cq_read() doesn't access GPU memory directly. Our concern was about thread-local CUDA context state, but after your explanation of the architecture, we understand the current design doesn't require it.

Closing this PR.

If we encounter CUDA context issues in production, we'll:

1. Capture complete stack traces with debug symbols
2. Verify the crash is in the progress thread, not the data posting path
3. Open a new issue with diagnostic data

Thank you for your time reviewing this.