Repository navigation
UCT/TCP: Validate AM receive length - #12081
Conversation
Validate wire-provided AM lengths at runtime before retaining the header for a later receive. Disconnect peers that advertise a payload larger than the configured receive segment.\n\nAdd a raw TCP regression that sends the protocol magic and an oversized AM header.
|
🤖 Starting review — findings will be posted here when done. |
| uct_tcp_ep_handle_invalid_am_length(uct_tcp_ep_t *ep, uint32_t length, | ||
| size_t max_length) | ||
| { | ||
| ucs_debug("tcp_ep %p (state %s): received AM length %u exceeds maximum %zu", |
There was a problem hiding this comment.
ucs_debug hides this entirely in release runs. Peers do not exchange segment sizes, so a peer configured with a larger RX_SEG_SIZE will legitimately hit this and the user only sees a connection reset with no hint. Can we use ucs_diag here? The connection is reset right after, so there is at most one message per connection.
| "tcp_ep %p (conn state - %s): %u vs %zu", | ||
| ep, uct_tcp_ep_cm_state[ep->conn_state].name, hdr->length, | ||
| (iface->config.rx_seg_size - sizeof(*hdr))); | ||
| if (ucs_unlikely(hdr->length > |
There was a problem hiding this comment.
hdr->length is still only ucs_assert-ed for the internal AM IDs further down (UCT_TCP_EP_PUT_REQ_AM_ID, UCT_TCP_EP_PUT_ACK_AM_ID, uct_tcp_cm_handle_conn_pkt). A crafted header passes this new check and in release builds uct_tcp_ep_handle_put_req() then memcpy's to put_req->addr. Should those lengths be validated here as well, or is that intentionally out of scope?
There was a problem hiding this comment.
This is intentionally out of scope for this change. This PR prevents an oversized AM length from being retained and used to size a later receive beyond the RX buffer. Exact semantic validation of internal PUT and CM messages is a separate parser-hardening issue that needs event-specific checks and focused tests.
| @@ -259,6 +259,30 @@ UCS_TEST_P(test_uct_tcp, listener_flood_connect_and_close) { | |||
| test_listener_flood(*m_ent, max_conn, 0); | |||
There was a problem hiding this comment.
Residual coverage gap: the new test only covers the single-header case in ACCEPTING state. The partial-message path (remaining < sizeof(*hdr) + hdr->length followed by a second uct_tcp_ep_recv()), which is the one that actually overflowed before this fix, is not directly exercised; it is only covered indirectly because the check now runs before that exit.
There was a problem hiding this comment.
The test intentionally sends only the oversized header. Without the fix it enters the partial-message path and subsequent progress calls attempt the second receive, but no payload is sent, so it fails safely by leaving the endpoint active instead of corrupting memory. The exact second-receive overwrite was reproduced separately under ASan. With the fix, every complete header is rejected before it can be retained, independent of TCP segmentation.
| @@ -259,6 +259,30 @@ UCS_TEST_P(test_uct_tcp, listener_flood_connect_and_close) { | |||
| test_listener_flood(*m_ent, max_conn, 0); | |||
There was a problem hiding this comment.
Residual coverage gap: no coverage for a TX-capable EP hitting this path (error-handler flow via uct_iface_handle_ep_err), only the RX-only destroy-from-progress flow.
There was a problem hiding this comment.
The new check only selects the existing uct_tcp_ep_handle_disconnected() path. The TX-capable branch and uct_iface_handle_ep_err() are existing peer-failure plumbing covered by the generic peer-failure tests, while this focused regression covers the new oversized-header decision and the RX-only destruction path. I do not think a duplicate TX-capable fixture is needed here.
|
|
||
| UCS_TEST_P(test_uct_tcp, reject_oversized_am) | ||
| { | ||
| const uint64_t magic_number = UCT_TCP_MAGIC_NUMBER; |
|
🤖 CI Triage Agent — TL;DR: One gtest failed — Full analysisSummary: Root cause: From the log: The test runs with This is not caused by PR #12081: commit Implicated commit: Not the PR commit File: Suggested fix:
Related: PR #12081 (this PR, TCP-only); #12039, #11985, #12032, #12041 (recent CUDA/rndv selection changes); #11858, #11861, #11864 (prior Azure-specific
|
Report rejected oversized messages at diagnostic level and use a compile-time constant for the TCP magic value in the regression test.
|
🤖 Starting review — findings will be posted here when done. |
| (iface->config.rx_seg_size - sizeof(*hdr))); | ||
| if (ucs_unlikely(hdr->length > | ||
| (iface->config.rx_seg_size - sizeof(*hdr)))) { | ||
| uct_tcp_ep_handle_invalid_am_length( |
There was a problem hiding this comment.
minor: iface->config.rx_seg_size - sizeof(*hdr) is computed twice here; maybe derive it inside the helper from the ep's iface and drop the max_length parameter.
|
Residual coverage gap: the new test only exercises the path where the EP is in |
|
🤖 Starting review — findings will be posted here when done. |
| uct_tcp_ep_handle_invalid_am_length(uct_tcp_ep_t *ep, uint32_t length, | ||
| size_t max_length) | ||
| { | ||
| ucs_diag("tcp_ep %p (state %s): received AM length %u exceeds maximum %zu", |
There was a problem hiding this comment.
maybe use ucs_error() here? diag is below the default warn log level so it is not printed by default, and this path also triggers for a legit peer configured with a bigger UCX_TCP_RX_SEG_SIZE (the sender bounds AM zcopy by its own rx_seg_size), so the user would see only a connection reset without the reason.
| "tcp_ep %p (conn state - %s): %u vs %zu", | ||
| ep, uct_tcp_ep_cm_state[ep->conn_state].name, hdr->length, | ||
| (iface->config.rx_seg_size - sizeof(*hdr))); | ||
| if (ucs_unlikely(hdr->length > max_am_length)) { |
There was a problem hiding this comment.
can we validate the length of the service AM ids here as well? hdr->length for PUT_REQ/PUT_ACK/CM is still only checked by ucs_assert, e.g. a PUT_REQ with length != sizeof(uct_tcp_ep_put_req_hdr_t) makes uct_tcp_ep_handle_put_req() read put_req->addr from stale buffer bytes and memcpy() to a garbage address in release builds.
|
🤖 CI Triage Agent — TL;DR: The gtest shard hung for the full 900 s watchdog in Full analysisSummary: "Tests roce on worker 0" (Azure build 139128) aborted: Root cause: Hang, not slowness. The log shows The backtrace pins the hung spot precisely:
Line 105 is inside This is almost certainly not caused by the PR: commit Implicated commit: none for the hang itself — branch commits File: Suggested fix:
Related: PR #12081 (this PR, TCP AM receive length validation — unrelated to the failing transport); prior FC deadlock fix in this same test: commit |
* UCT/TCP: Validate AM receive length Validate wire-provided AM lengths at runtime before retaining the header for a later receive. Disconnect peers that advertise a payload larger than the configured receive segment.\n\nAdd a raw TCP regression that sends the protocol magic and an oversized AM header. * UCT/TCP: Improve oversized AM diagnostics Report rejected oversized messages at diagnostic level and use a compile-time constant for the TCP magic value in the regression test. * UCT/TCP: Reuse maximum AM length (cherry picked from commit 413775a)
UCT/TCP: Validate AM receive length (#12081) * UCT/TCP: Validate AM receive length Validate wire-provided AM lengths at runtime before retaining the header for a later receive. Disconnect peers that advertise a payload larger than the configured receive segment.\n\nAdd a raw TCP regression that sends the protocol magic and an oversized AM header. * UCT/TCP: Improve oversized AM diagnostics Report rejected oversized messages at diagnostic level and use a compile-time constant for the TCP magic value in the regression test. * UCT/TCP: Reuse maximum AM length (cherry picked from commit 413775a)
What?
Validate UCT TCP active-message lengths at runtime and disconnect peers that
advertise a payload larger than the receive segment. Add focused regression
coverage for an oversized wire header.
Why?
Release builds disable assertions. A malformed length could therefore be
retained and used to size a later socket receive beyond the RX buffer.
How?
Perform the length check as soon as a complete active-message header is
available, reset the RX context, and use the existing TCP disconnect path on
failure.