docs(eks): remove Known Issues block now that libfabric + GDRCopy fixes are merged - #9831
docs(eks): remove Known Issues block now that libfabric + GDRCopy fixes are merged#9831yifjiang wants to merge 2 commits into
Conversation
github-actions
Bot
added
docs
external-contribution
dbae7d4 to
7f7a37a
Compare
yifjiang
changed the title
7f7a37a to
54cf069
Compare
yifjiang
changed the title
|
/ok to test 54cf069 |
…es are merged Removes the "Known Issues" section from docs/kubernetes/cloud-providers/eks/efa.md and prunes the two now-obsolete rows from "Common Failure Modes". Assumes ai-dynamo#9703, ai-dynamo#9704, ai-dynamo#9705, and ai-dynamo#9727 are all merged — after those land, the issues this section documented (GB200 fi_mr_reg(VRAM) failure on the EFA installer's stock libfabric, and GDRCopy v2.5.1 kmod build failure on kernel >= 6.15) no longer affect default --make-efa builds, so the inline workarounds the section provided would mislead readers. Also removes the ofiwg/libfabric#12019 reference from the bottom links list since it points at the same now-resolved upstream issue. Net diff: -34 / +1. Signed-off-by: Yifan Jiang <yifjiang@users.noreply.github.com>
|
/ok to test 78d0621 |
54cf069 to
78d0621
Compare
yifjiang
changed the title
78d0621 to
9b52cac
Compare
GitHub Actions
Inactive
|
/ok to test 066d516 |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| ## Known Issues | ||
|
|
||
| One issue currently affects default-built Dynamo EFA images. | ||
|
|
||
| ### Issue 1: libfabric on GB200 fails `fi_mr_reg` on CUDA VRAM | ||
|
|
||
| **Known affected platforms:** GB200. | ||
|
|
||
| **Symptom:** Worker pod fails at startup with `fi_mr_reg` returning EFAULT during NIXL initialization. NIXL VRAM registration fails; depending on the framework, the worker either crashes or silently falls back to TCP. | ||
|
|
||
| **Root cause:** The libfabric version (versions lower than 2.5.x) bundled with the EFA installer (up to currently latest 1.48.0) lacks a CUDA branch in the dmabuf-eligibility check in `prov/efa/src/efa_mr.c`. On x86_64 hosts the legacy `ibv_reg_mr` path handles CUDA pointers natively, so the bug doesn't surface. On arm64 64K-page kernels (GB200), the legacy path returns EFAULT for CUDA VRAM. Tracked in [ofiwg/libfabric#12019](https://github.com/ofiwg/libfabric/issues/12019). | ||
|
|
||
| **Upstream status:** The bug is resolved in `ofiwg/libfabric` main and v2.5.x via a more comprehensive rewrite of `efa_mr_reg_ibv_mr()`. AWS's `aws/libfabric` fork has not picked up the upstream rewrite; the latest EFA installer (1.48.0) still ships `v2.4.0amzn3.0` with the older code path. | ||
|
|
||
| **Workarounds:** | ||
|
|
||
| 1. **Apply the one-line patch to the bundled libfabric.** During image build, replace the `aws.Dockerfile` install step with a custom build: | ||
| ```dockerfile | ||
| RUN git clone --depth 1 --branch v2.4.0amzn3.0 https://github.com/aws/libfabric.git /tmp/libfabric && \ | ||
| cd /tmp/libfabric && \ | ||
| sed -i 's/efa_mr_is_neuron(efa_mr) || efa_mr_is_rocr(efa_mr)/efa_mr_is_neuron(efa_mr) || efa_mr_is_rocr(efa_mr) || efa_mr_is_cuda(efa_mr)/' prov/efa/src/efa_mr.c && \ | ||
| ./autogen.sh && \ | ||
| CPPFLAGS="-I/usr/local/cuda/include" \ | ||
| LDFLAGS="-L/usr/local/cuda/lib64 -L/usr/local/cuda/lib64/stubs -Wl,-rpath,/usr/local/cuda/lib64" \ | ||
| ./configure --prefix=/opt/amazon/efa --enable-efa --with-cuda=/usr/local/cuda --enable-cuda-dlopen && \ | ||
| make -j$(nproc) && make install | ||
| # Then rebuild aws-ofi-nccl from source against the patched libfabric (do not mix versions) | ||
| ``` | ||
| 2. **Replace bundled libfabric with `ofiwg/libfabric@v2.5.1`** (or newer). The upstream rewrite is already present; no patch needed. Rebuild `aws-ofi-nccl` against it. | ||
|
|
There was a problem hiding this comment.
🟡 Dangling anchor link to a removed section leaves readers clicking a dead link
The "Known Issues" section was removed (efa.md:214-243, LEFT) but the admonition at docs/kubernetes/cloud-providers/eks/efa.md:139 still references [Known Issues](#known-issues), so readers who click it land nowhere.
Impact: Users following the important callout in Step 3 hit a broken in-page link and cannot find the referenced information.
Broken anchor details and style-guide rule
Line 139 contains:
See Known Issues below for one case where the default-built image does not produce a working EFA deployment out of the box (GB200 / arm64 64K-page kernels).
The ## Known Issues heading that provided the #known-issues anchor was entirely deleted in this PR (old lines 214–243), but the > [!IMPORTANT] admonition referencing it was not updated or removed.
This also violates the docs style guide at docs/documentation-style-guide.md:169: "Every internal link and #anchor must resolve to a real file or heading."
The entire admonition block at line 139 should either be removed (since the known issue it describes was removed) or updated to point to valid content.
Prompt for agents
The PR removed the '## Known Issues' section (old lines 214-243) from docs/kubernetes/cloud-providers/eks/efa.md but left an admonition at line 139 that still references [Known Issues](#known-issues). This anchor is now broken. The admonition reads: 'See [Known Issues](#known-issues) below for one case where the default-built image does not produce a working EFA deployment out of the box (GB200 / arm64 64K-page kernels).' Since the Known Issues section and its content about the GB200 libfabric CUDA dmabuf bug have been removed, this entire admonition block (the > [!IMPORTANT] block at line 138-139) should also be removed, or rewritten to reference whatever content (if any) replaced the removed section.
Was this helpful? React with 👍 or 👎 to provide feedback.
| | TTFT ~10 s, throughput 1–5 GB/s | UCX host-staged (no GPU-Direct on kernel ≥ 6.8) | Switch to LIBFABRIC backend | | ||
| | Pod fails at startup with `fi_mr_reg` EFAULT on GB200 | Issue 1 (libfabric CUDA dmabuf bug) | Apply patch or use ofiwg/libfabric v2.5.1 | | ||
| | Pod fails at startup with `fi_mr_reg` EFAULT on x86_64 | `privileged: true` missing OR `efa_nv_peermem` missing on old kernel | Verify Step 5 security context | | ||
| | Pod fails at startup with `fi_mr_reg` EFAULT | `privileged: true` missing OR (on pre-DMA-BUF kernels) `efa_nv_peermem` not loaded | Verify Step 5 security context; verify Step 2 kernel/module state | |
There was a problem hiding this comment.
🚩 Removal of Known Issues section drops actionable workaround content
The removed "Known Issues" section (old lines 214–243) contained a concrete one-line sed patch and an alternative workaround (switching to ofiwg/libfabric@v2.5.1) for a GB200-specific fi_mr_reg EFAULT during NIXL initialization. The consolidated table row at docs/kubernetes/cloud-providers/eks/efa.md:284 now attributes the symptom generically to missing privileged: true or efa_nv_peermem, which doesn't cover the libfabric CUDA dmabuf bug that was documented. If that upstream bug is still relevant for users on older libfabric versions, removing the workaround without a replacement could leave GB200 users without guidance. If the bug has been fixed in the libfabric versions shipped by the EFA installer, the removal is correct — but that context isn't stated in the PR.
Was this helpful? React with 👍 or 👎 to provide feedback.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
WalkthroughThe EKS EFA documentation was updated to remove the "Known Issues" section detailing a libfabric CUDA dmabuf registration failure, simplify the "Common Failure Modes" table entry for fi_mr_reg EFAULT errors with updated cause/fix guidance, and remove an upstream libfabric issue reference link. ChangesEKS EFA Troubleshooting Docs Update
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~3 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Comment |
3 participants
Summary
Removes the "Known Issues" section from
docs/kubernetes/cloud-providers/eks/efa.mdand prunes the two now-obsolete rows from "Common Failure Modes". Assumes #9703, #9704, #9705, and #9727 are all merged — after those land, the issues this section documented (GB200fi_mr_reg(VRAM)failure on the EFA installer's stock libfabric, and GDRCopy v2.5.1 kmod build failure on kernel ≥ 6.15) no longer affect default--make-efabuilds, so the inline workarounds the section provided would be misleading.Depends on #9703 + #9704 + #9705 + #9727 merging first. If any of those slip, this PR should be held until they all land. Marked draft for review.
What changes
docs/kubernetes/cloud-providers/eks/efa.md:## Known Issuessection deleted entirely. Both Issue 1 (libfabric CUDA dmabuf bug) and the inline Dockerfile workaround it carried are no longer needed: feat(container): configurable libfabric repo + v2.5.1 overlay for EFA #9727 internalizes the patched libfabric into the--make-efaaws stage. (The previous text on main only documented Issue 1; the GDRCopy v2.5.1 / kernel-6.15 issue was never on main — it landed in this branch earlier as Issue 2 during my expansion. Both go away now.)## Common Failure Modestable cleaned up:fi_mr_regEFAULT on GB200 → Issue 1 → Apply patch or use ofiwg/libfabric v2.5.1" (issue is fixed in main; references a nonexistent## Known Issues > Issue 1)./dev/gdrdrvmissing on host → Issue 2 → Bumpnixl_gdrcopy_refto v2.5.2" (issue is fixed by build(container): bump nixl_gdrcopy_ref to v2.5.2 (kernel >=6.15 fix) #9705 / dynamo ships v2.5.2 source by default).fi_mr_reg EFAULTrows (GB200 and x86_64) into one more general "missingprivileged: trueorefa_nv_peermem" entry — that's the onlyfi_mr_reg EFAULTcause left once the libfabric bug is fixed upstream.[ofiwg/libfabric#12019]link (was only referenced from the deleted Issue 1).Net diff: 1 line added, 34 deleted.
Validation evidence
The v4 PR-internalized image (built from a branch that has all four merged PRs cherry-picked) passes end-to-end with no inline workaround:
scripts/probe_nixl_libfabric_vram.py(standalone NIXLregister_memory(VRAM, LIBFABRIC)probe) returns=== libfabric VRAM registration: PASSED ===on dev-01 GB200 (the cluster where the dropped Issue 1 originally manifested).using NIXL backend: LIBFABRIC.prs-internalized-v4-validation-2026-05-21.mdfor the full validation report (3-way A/B with prior images, TTFT sweep, etc.).Sequencing
Merge order required:
--has-trtllm-contextflag)(Any merge order among 1–4 works; this docs-cleanup PR must come last.)
Risk
LOW. Documentation-only. No code paths affected. If a reader builds from an older dynamo (pre-merge of the four PRs) they may hit issues this section used to document — but the appropriate fix is to upgrade dynamo, not re-add the inline workaround section.
Test plan
🤖 Generated with Claude Code
Summary by CodeRabbit
fi_mr_regEFAULT with clearer cause-and-fix steps.