Skip to content

docs(eks): remove Known Issues block now that libfabric + GDRCopy fixes are merged #9831

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
yifjiang wants to merge 2 commits into
base: main
Choose a base branch
from
Open

docs(eks): remove Known Issues block now that libfabric + GDRCopy fixes are merged #9831

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9b52cac
docs(eks): remove Known Issues block now that libfabric + GDRCopy fix…
yifjiang May 21, 2026
066d516
Merge branch 'main' into yifjiang/efa-doc-libfabric-workaround
rmccorm4 Jun 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 1 addition & 34 deletions docs/kubernetes/cloud-providers/eks/efa.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -211,37 +211,6 @@ volumes:
> [!IMPORTANT]
> `privileged: true` is required for NIXL to register CUDA VRAM with the EFA NIC via `fi_mr_reg`. `IPC_LOCK` alone is insufficient.

## Known Issues

One issue currently affects default-built Dynamo EFA images.

### Issue 1: libfabric on GB200 fails `fi_mr_reg` on CUDA VRAM

**Known affected platforms:** GB200.

**Symptom:** Worker pod fails at startup with `fi_mr_reg` returning EFAULT during NIXL initialization. NIXL VRAM registration fails; depending on the framework, the worker either crashes or silently falls back to TCP.

**Root cause:** The libfabric version (versions lower than 2.5.x) bundled with the EFA installer (up to currently latest 1.48.0) lacks a CUDA branch in the dmabuf-eligibility check in `prov/efa/src/efa_mr.c`. On x86_64 hosts the legacy `ibv_reg_mr` path handles CUDA pointers natively, so the bug doesn't surface. On arm64 64K-page kernels (GB200), the legacy path returns EFAULT for CUDA VRAM. Tracked in [ofiwg/libfabric#12019](https://github.com/ofiwg/libfabric/issues/12019).

**Upstream status:** The bug is resolved in `ofiwg/libfabric` main and v2.5.x via a more comprehensive rewrite of `efa_mr_reg_ibv_mr()`. AWS's `aws/libfabric` fork has not picked up the upstream rewrite; the latest EFA installer (1.48.0) still ships `v2.4.0amzn3.0` with the older code path.

**Workarounds:**

1. **Apply the one-line patch to the bundled libfabric.** During image build, replace the `aws.Dockerfile` install step with a custom build:
```dockerfile
RUN git clone --depth 1 --branch v2.4.0amzn3.0 https://github.com/aws/libfabric.git /tmp/libfabric && \
cd /tmp/libfabric && \
sed -i 's/efa_mr_is_neuron(efa_mr) || efa_mr_is_rocr(efa_mr)/efa_mr_is_neuron(efa_mr) || efa_mr_is_rocr(efa_mr) || efa_mr_is_cuda(efa_mr)/' prov/efa/src/efa_mr.c && \
./autogen.sh && \
CPPFLAGS="-I/usr/local/cuda/include" \
LDFLAGS="-L/usr/local/cuda/lib64 -L/usr/local/cuda/lib64/stubs -Wl,-rpath,/usr/local/cuda/lib64" \
./configure --prefix=/opt/amazon/efa --enable-efa --with-cuda=/usr/local/cuda --enable-cuda-dlopen && \
make -j$(nproc) && make install
# Then rebuild aws-ofi-nccl from source against the patched libfabric (do not mix versions)
```
2. **Replace bundled libfabric with `ofiwg/libfabric@v2.5.1`** (or newer). The upstream rewrite is already present; no patch needed. Rebuild `aws-ofi-nccl` against it.

Comment on lines -214 to -243

@devin-ai-integration devin-ai-integration Bot Jun 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Dangling anchor link to a removed section leaves readers clicking a dead link

The "Known Issues" section was removed (efa.md:214-243, LEFT) but the admonition at docs/kubernetes/cloud-providers/eks/efa.md:139 still references [Known Issues](#known-issues), so readers who click it land nowhere.

Impact: Users following the important callout in Step 3 hit a broken in-page link and cannot find the referenced information.

Broken anchor details and style-guide rule

Line 139 contains:

See Known Issues below for one case where the default-built image does not produce a working EFA deployment out of the box (GB200 / arm64 64K-page kernels).

The ## Known Issues heading that provided the #known-issues anchor was entirely deleted in this PR (old lines 214–243), but the > [!IMPORTANT] admonition referencing it was not updated or removed.

This also violates the docs style guide at docs/documentation-style-guide.md:169: "Every internal link and #anchor must resolve to a real file or heading."

The entire admonition block at line 139 should either be removed (since the known issue it describes was removed) or updated to point to valid content.

Prompt for agents 
The PR removed the '## Known Issues' section (old lines 214-243) from docs/kubernetes/cloud-providers/eks/efa.md but left an admonition at line 139 that still references [Known Issues](#known-issues). This anchor is now broken. The admonition reads: 'See [Known Issues](#known-issues) below for one case where the default-built image does not produce a working EFA deployment out of the box (GB200 / arm64 64K-page kernels).' Since the Known Issues section and its content about the GB200 libfabric CUDA dmabuf bug have been removed, this entire admonition block (the > [!IMPORTANT] block at line 138-139) should also be removed, or rewritten to reference whatever content (if any) replaced the removed section.
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.


## Verification

After deployment, confirm EFA is actually being used (not silent TCP fallback):
Expand Down Expand Up @@ -312,8 +281,7 @@ kubectl logs <decode-pod> | grep "External prefix cache hit rate"
| ------------------------------------------------------ | -------------------------------------------------------------------- | ----------------------------------------------------------------------------- |
| TTFT ~100 s, throughput ~MB/s | Silent TCP fallback — NIXL backend selection not applied | Verify Step 4 backend env var; check NIXL startup log |
| TTFT ~10 s, throughput 1–5 GB/s | UCX host-staged (no GPU-Direct on kernel ≥ 6.8) | Switch to LIBFABRIC backend |
| Pod fails at startup with `fi_mr_reg` EFAULT on GB200 | Issue 1 (libfabric CUDA dmabuf bug) | Apply patch or use ofiwg/libfabric v2.5.1 |
| Pod fails at startup with `fi_mr_reg` EFAULT on x86_64 | `privileged: true` missing OR `efa_nv_peermem` missing on old kernel | Verify Step 5 security context |
| Pod fails at startup with `fi_mr_reg` EFAULT | `privileged: true` missing OR (on pre-DMA-BUF kernels) `efa_nv_peermem` not loaded | Verify Step 5 security context; verify Step 2 kernel/module state |

@devin-ai-integration devin-ai-integration Bot Jun 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚩 Removal of Known Issues section drops actionable workaround content

The removed "Known Issues" section (old lines 214–243) contained a concrete one-line sed patch and an alternative workaround (switching to ofiwg/libfabric@v2.5.1) for a GB200-specific fi_mr_reg EFAULT during NIXL initialization. The consolidated table row at docs/kubernetes/cloud-providers/eks/efa.md:284 now attributes the symptom generically to missing privileged: true or efa_nv_peermem, which doesn't cover the libfabric CUDA dmabuf bug that was documented. If that upstream bug is still relevant for users on older libfabric versions, removing the workaround without a replacement could leave GB200 users without guidance. If the bug has been fixed in the libfabric versions shipped by the EFA installer, the removal is correct — but that context isn't stated in the PR.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

| Bandwidth halves after image rebuild | libfabric / aws-ofi-nccl ABI mismatch | Rebuild aws-ofi-nccl from source against the libfabric used in the same image |
| `rdma_write_bytes` shows 0 | **Not a failure** — EFA SRD uses SEND, not WRITE | Use Prometheus `nixl_bytes_transferred` instead |

Expand All @@ -326,4 +294,3 @@ kubectl logs <decode-pod> | grep "External prefix cache hit rate"
- [AWS — Manage EFA devices on Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/device-management-efa.html) — official EKS-side guide (DRA driver + device plugin)
- [AWS EFA documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa.html) — EC2-side EFA overview
- [`aws/eks-charts` — `aws-efa-k8s-device-plugin`](https://github.com/aws/eks-charts/tree/master/stable/aws-efa-k8s-device-plugin) — Helm chart source
- [ofiwg/libfabric#12019](https://github.com/ofiwg/libfabric/issues/12019) — CUDA dmabuf registration on EFA
Loading