Repository navigation
[release/1.5.1] fix(libfabric): fall back for context-independent CUDA allocations - #2470
Merged
Merged
Conversation
…2425) ## Problem A CUDA device pointer can have a valid device ordinal and a null `CU_POINTER_ATTRIBUTE_CONTEXT`, as with context-independent VMM allocations. In a two-node TensorRT-LLM KV cache V2 run, the failing pointer belonged to an active, mapped VMM allocation while the submitting thread had a non-null current CUDA context. A null allocation-associated context is not a null pointer or proof that the calling thread lacks a current context. The [NIXL LIBFABRIC backend](https://github.com/ai-dynamo/nixl/blob/v1.4.0/src/plugins/libfabric/libfabric_backend.cpp#L212) obtains this value by querying `CU_POINTER_ATTRIBUTE_CONTEXT`; TRT-LLM does not supply a NULL `CUcontext` argument. [CUDA documents](https://docs.nvidia.com/cuda/archive/13.0.0/cuda-driver-api/group__CUDA__CTX.html) that `cuMemCreate()` allocations are not associated with any CUDA context. `cudaUpdateCtxPtr()` currently caches that null allocation context and reports success. The subsequent `cudaSetCtx()` rejects the cached null value before calling `cuCtxSetCurrent()`, so LIBFABRIC descriptor posting returns `NIXL_ERR_BACKEND`. The error message therefore does not mean that the CUDA driver rejected an otherwise valid current context. ## Solution Treat a null allocation context as a reason to use the existing per-device `cudaSetDevice()` fallback. Keep ordinary allocation-context handling unchanged, and update the fallback log so it also describes this case. This change addresses NIXL's handling of a null allocation-associated context. EFA also needs libfabric's separate VMM synchronization fixes ([ofiwg/libfabric#12853](ofiwg/libfabric#12853)); this PR does not replace those fixes. ## Validation - A standalone CUDA-only control on B300, without TRT-LLM, NIXL or libfabric, reproduced a NULL allocation-associated context for mapped `cuMemCreate` allocations while the thread current context was non-null. Both singular and bulk pointer-attribute queries returned `CUDA_SUCCESS`; mapping and memory write/read checks passed. The `cuMemAlloc` control returned a non-null allocation context. POSIX_FD and NONE handles were exercised; the FABRIC handle case was skipped after allocation creation failed. [Source, build command and observed results](#2425 (comment)). This validates the NULL attribute's independent reproduction, not this PR revision's GPU correctness. - Compiled the actual changed context-selection function with a stub CUDA query. The original code fails the null-context assertion; the patched v1.4.0 and main implementations pass checks for null, valid, repeated and conflicting contexts, a device mismatch, host memory, and a query error. - Changed lines pass clang-format 19 and `git diff --check`. - Follow-up isolation test on TRT-LLM `eac0f787e1aab5b1d8757909345b26acd1854adb`, two B300 nodes, Qwen3-0.6B TP1/PP1: unmodified NIXL 1.4.0 still failed the same context guard with only the diagnostic libfabric VMM fixes. Keeping the same binaries and setting `NIXL_DISABLE_CUDA_ADDR_WA=1` completed 3/3 KV cache V2 inference requests using the NIXL LIBFABRIC backend over EFA RDM. Actual mapped libraries and KV cache V2 worker logs were checked. This confirms the separate NIXL failure, not this PR revision's GPU correctness. - The equivalent patch on NIXL v1.4.0, combined with the libfabric VMM fixes, completed three two-node TensorRT-LLM KV cache V2 inference requests over EFA without `NIXL_DISABLE_CUDA_ADDR_WA` or `LD_PRELOAD`. - Validation gap: a full build and GPU regression on this main-branch revision, including multiple GPUs and deferred posting, remain outstanding. (cherry picked from commit 1f4afe6)
nv-nmailhot
requested review from
akkart-aws,
amitrad-aws,
fengjica,
oa-aws and
rongbingzhou
as code owners
October 21, 2026 06:02
|
👋 Hi nv-nmailhot! Thank you for contributing to ai-dynamo/nixl. Your PR reviewers will review your contribution then trigger the CI to test your changes. 🚀 |
5 of 6 tasks
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What?
Cherry-pick of #2425 (
1f4afe6b): fall back for context-independent CUDA allocations in the libfabric backend.Why?
CUDA returns a null
CU_POINTER_ATTRIBUTE_CONTEXTfor valid VMM allocations, breaking the libfabric backend on NIXL 1.4.0+ (TRT-LLM KV cache manager V2 on B300). Needed for 1.5.1.🤖 Generated with Claude Code