Repository navigation
Use SWRX timestamps in server tests to match production - #532
Closed
leoleovich wants to merge 1 commit into
Closed
leoleovich wants to merge 1 commit into
leoleovich wants to merge 1 commit into
Conversation
Summary: TestServer fails non-deterministically (and consistently on newer kernels, e.g. Fedora rawhide where it breaks the RPM %check stage) with a UDP read timeout at server_test.go:223. Root cause: the tests configure the server with TimestampType: timestamp.SW, a value production never uses. cmd/ntpresponder defaults to SWRX and Config.Validate() only accepts SWRX/HWRX. SW routes through EnableSWTimestamps, which additionally turns on TX software timestamping (SOF_TIMESTAMPING_TX_SOFTWARE). Every response the worker sends then makes the kernel clone the packet onto the socket error queue. The responder never drains MSG_ERRQUEUE, so those clones accumulate; error-queue skbs are charged to sk_rmem_alloc and bounded by SO_RCVBUF. Once that budget fills, the kernel silently drops incoming requests, the listener never sees them, and the client's synchronous read blocks until the 10s deadline. How quickly it fills relative to the 1000-request loop depends on skb truesize / rmem accounting, which is why it passes on some kernels and fails on others; the earlier 1MB buffer bump only delays the fill. Measured difference (1MB rcvbuf, 4000 sends, no draining): | Mode | sk_rmem_alloc after sends | |------|---------------------------| | SWRX (production, RX only) | 0 bytes | | SW (test, TX+RX) | 2,096,640 bytes (entire 2MB rcvbuf) | Switch TestServer and TestListener to timestamp.SWRX so the tests exercise the real production code path and no longer accumulate TX timestamps. Differential Revision: D107533486
Contributor
|
@leoleovich has exported this pull request. If you are a Meta employee, you can view the originating Diff in D107533486. |
Contributor
|
This pull request has been merged in 67daa0f. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
TestServer fails non-deterministically (and consistently on newer kernels,
e.g. Fedora rawhide where it breaks the RPM %check stage) with a UDP read
timeout at server_test.go:223.
Root cause: the tests configure the server with TimestampType: timestamp.SW,
a value production never uses. cmd/ntpresponder defaults to SWRX and
Config.Validate() only accepts SWRX/HWRX. SW routes through
EnableSWTimestamps, which additionally turns on TX software timestamping
(SOF_TIMESTAMPING_TX_SOFTWARE). Every response the worker sends then makes
the kernel clone the packet onto the socket error queue. The responder never
drains MSG_ERRQUEUE, so those clones accumulate; error-queue skbs are charged
to sk_rmem_alloc and bounded by SO_RCVBUF. Once that budget fills, the kernel
silently drops incoming requests, the listener never sees them, and the
client's synchronous read blocks until the 10s deadline. How quickly it fills
relative to the 1000-request loop depends on skb truesize / rmem accounting,
which is why it passes on some kernels and fails on others; the earlier 1MB
buffer bump only delays the fill.
Measured difference (1MB rcvbuf, 4000 sends, no draining):
Switch TestServer and TestListener to timestamp.SWRX so the tests exercise
the real production code path and no longer accumulate TX timestamps.
Differential Revision: D107533486