investigate modifying the kernel's receive buffer size

[marten-seemann]

It turns out that the Linux kernel's UDP receive buffer is too small by default (around 200k). In order to support higher bandwidths, we need a significantly higher value.

It would be good if we could set this on a per-connection basis. Not sure if Go allows us to do this.

[dswarbrick]

https://golang.org/pkg/net/#UDPConn.SetReadBuffer should do the trick. It's implemented as a setsockopt SO_RCVBUF syscall under the hood.

[marten-seemann]

The buffer sizes can be queried by running

sysctl net.core.rmem_default
sysctl net.core.rmem_max

On the Linux system I'm testing on, both values are 212992. When querying the receive buffer from quic-go using

syscall.GetsockoptInt(int(fd.Fd()), syscall.SOL_SOCKET, syscall.SO_RCVBUF)

I get the same value. Using UDPConn.SetReadBuffer allows me increase the buffer size to 425984 (which is exactly twice the value of 212992). I assume that this has something to do with the doubling described here: http://man7.org/linux/man-pages/man7/socket.7.html (see SO_RCVBUF).

While doubling this value is nice (and leads to speedup of ~2x in my test), I'd like to increase it even further. As far as I can see, we'd need to ask people to run

sysctl -w net.core.rmem_max=xxx

in order to increase the maximum value though.

[dswarbrick]

Yup, the default Linux kernel UDP receive buffer sizes are fairly small. Applications can only set the receive buffer size up to the limit specified by the kernel. The app can read back the buffer size however, and warn the use if the requested buffer size less than desired.

[K9A2]

Shoud net.core.rmem_max be set as the BDP of current link?

[marten-seemann]

The BDP has nothing to do with this. The problem is how fast (and consistently) we’re able to read packet from the socket. We need to do that faster than the bandwidth of the connection, the RTT is completely irrelevant for this.

[K9A2]

The linux kernel also provide the ability to config udp_mem, udp_wmem_min, udp_rmem_min. Could them affect the performance of QUIC protocol?