Skip to content

UCT/TCP: Bound negative connect test timeout - #11415

Merged
yosefe merged 2 commits into
openucx:masterfrom
yosefe:codex/tcp-user-timeout-from-master
May 7, 2026
Merged

yosefe merged 2 commits into
openucx:masterfrom
yosefe:codex/tcp-user-timeout-from-master

Conversation

@yosefe

@yosefe yosefe commented May 5, 2026 •

Copy link
Copy Markdown
Member

Why

The UCT sockaddr negative-connect test connects to a non-existing IP address and waits for the client error callback. On hpc-build-ub, the TCP CM variants were taking about 135 seconds each because the socket remained in SYN retransmission timeout handling. The 9 TCP parameters in tcp/test_uct_sockaddr_err_handle_non_exist_ip.conn_to_non_exist_ip consumed about 20 minutes while still covering only the same negative-connect path.

The test attempted to shorten the wait with SYN_CNT, but the TCP CM configuration key needs the TCP_CM_ prefix when modifying the config bundle, so the setting was not applied to the CM socket.

How

Add a TCP USER_TIMEOUT config option for both TCP iface and TCP sockcm sockets. When set, UCX applies Linux TCP_USER_TIMEOUT on a best-effort basis, guarded by #ifdef TCP_USER_TIMEOUT so non-Linux builds continue to compile and keep the default behavior.

The negative-connect test now sets TCP_CM_SYN_CNT=1 and TCP_CM_USER_TIMEOUT=1s for transports that support those TCP CM options. This preserves the test coverage: it still connects to the non-existing IP and verifies the server-unavailable callback, but the kernel is asked to bound the failed connect attempt instead of waiting for the host default retry window.

Testing

On hpc-build-ub:

  • module load dev/cuda-latest dev/gdrcopy-latest tools/hpc
  • make -C src/uct -j16 libuct.la
  • make -C test/gtest -j16 gtest
  • ./test/gtest/gtest --gtest_filter="tcp/test_uct_sockaddr_err_handle_non_exist_ip.conn_to_non_exist_ip*"

Result: all 9 tests passed in 9258 ms. Before the fix, the same TCP cases took about 135 seconds each on this host.

Comment thread src/uct/tcp/tcp_base.c Resolved
@yosefe
yosefe force-pushed the codex/tcp-user-timeout-from-master branch from 50df6bb to e80a778 Compare May 5, 2026 12:16
@yosefe
yosefe merged commit b82a920 into openucx:master May 7, 2026
153 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants