ggml-rpc : bound rdma_recv completion length before copy - #24136
ggml-rpc : bound rdma_recv completion length before copy#24136palios-taey wants to merge 1 commit into
Conversation
socket_t::impl::rdma_recv copies wc.byte_len bytes into the caller buffer with no check that it fits the requested size, then does rem -= got (size_t underflow). The completion length is peer-controlled (up to the posted recv-slot size), so a peer sending more than the logical recv expects overflows dst and turns the loop into an unbounded copy; a zero-length completion makes no progress and spins. Reject both and drop the connection rather than clamp, so a malformed peer cannot silently desync the message stream. Surfaced as an ASAN stack-buffer-overflow WRITE on real RoCEv2 hardware via a single oversized post-HELLO SEND; the guard makes it fail closed.
github-actions
Bot
added
the
ggml
|
Hi @palios-taey, thanks for your contribution! Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:
Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below. |
|
Closing to respect the new-contributor 1-open-PR limit (per the automated checker — apologies for the over-submission). Keeping #24093 (already approved) as my single open PR; I'll resubmit this after that merges. The change here is verified and stands on its own. Thanks! |
1 participant
Found this fuzzing the RPC RDMA transport.
socket_t::impl::rdma_recv(ggml/src/ggml-rpc/transport.cpp:450) copieswc.byte_lenbytes into the caller's buffer with no check that it fits the requested size, then doesrem -= got— asize_tunderflow. The completion length is peer-controlled (up to the posted recv-slot size), so a peer that sends more than the logical recv expects overflows the destination and turns the loop into an unbounded copy; a zero-length completion makes no progress and spins.Hit it as an ASAN stack-buffer-overflow WRITE on real RoCEv2 hardware with a single oversized post-HELLO SEND. DoS is confirmed (deterministic abort); I have not demonstrated control-transfer, so I'm not claiming RCE — just a peer-reachable out-of-bounds write.
The fix bounds
gotbefore the copy (kills the OOB write, therem -= gotunderflow, and a 0-length spin) and drops the connection rather than clamping, so a malformed peer can't silently desync the message stream. Re-confirmed unpatched on master before filing; fix-validated on the same hardware (the oversized SEND now fails closed).This is the experimental RPC backend, which SECURITY.md places outside the covered threat model — so I'm filing it as a robustness fix, not a security-scope claim. Full PoC + a SoftRoCE repro available on request.
AI usage disclosure: AI-assisted in the RDMA fuzzing that surfaced this and in drafting the fix; I validated the overflow and the fix on real hardware myself and own the change.