Trying to understand UDP testing inconsistencies (loss and limits) with Windows in various scenarios

[DomingoMontoya]

Hi,

This all started with my trying to troubleshoot why I'm getting kicked from a game for unstable connection.

I started testing UDP with iperf on Windows and Linux on various (6) machines with different network hardware. I'll try to stay focused here.

Linux seems bulletproof. Windows, not so much.

• I know that for some reason I'm uncertain of, Windows uses 8192 size datagrams, and Linux 1460. I'm assuming that since I don't have jumbo frames enabled anywhere on my network that the 8192's are going to be fragmented to meet MTU's limits?
• When testing Windows (regardless of the NIC, drivers, Windows version (10/11), or System) with Linux it does not like the smaller size (1460B) and will not send to the iperf server at full speed (-b900M), and receiving 900M from the Linux iperf server will drop a lot of packets.
  • I noticed that during these tests the Windows hosts was core locked. I had 3-4 cores at 100%.
  • At first I thought that had to be it. Maybe Windows and checksum offloading wasn't working right, and it was doing it on the CPU which couldn't keep up. But then I tried on a VM on a 12th gen i9 and got the same test results, but no cores were close to maxed out, nor on the host.
  • iperf didn't seem like it was the cause of the CPU use, since it was reporting almost no CPU use, plus iperf is not multi-threaded, right? So it can't be using 3-4 cores, and it has to be kernel/driver, since no process appeared to be using any significant amount of CPU during the tests, according to resmon.
  • What would be causing the CPU use/bottleneck if UDP checksum offloading is enabled and iperf (nor any other user task) is reportedly using much CPU?
  • And if it's not CPU core limited (as in the case of the i9 VM system that never came close to peaking a core), why is Windows unable to achieve the full 900M speed with 1460 datagram size, but is able to with 8192 as the size? (QoS has been disabled BTW).

This should all be very easy to reproduce, if needed. Any Linux server as the iperf server, and any seemingly any Windows machine. I've tried on several with various different hardware, internal NICs from Realtek and Intel, external USB NICs with Realtek and ASIX chipsets, and on 2 completely different networks. All tests were flawless with Linux, but struggle/fail with Windows and smaller data sizes.

Linux Server

iperf3 -s

Results

There's a fair amount of loss here, considering if I boot that system under Linux and do the same test I can do 900M with no loss.

-----------------------------------------------------------
Server listening on 5201 (test #14)
-----------------------------------------------------------
Accepted connection from 172.17.17.158, port 51038
[  5] local 172.17.17.166 port 5201 connected to 172.17.17.158 port 58299
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec  78.8 MBytes   661 Mbits/sec  0.032 ms  0/56586 (0%)  
[  5]   1.00-2.00   sec  78.8 MBytes   661 Mbits/sec  0.019 ms  75/56679 (0.13%)  
[  5]   2.00-3.00   sec  82.0 MBytes   688 Mbits/sec  0.020 ms  8/58923 (0.014%)  
[  5]   3.00-4.00   sec  80.8 MBytes   678 Mbits/sec  0.019 ms  99/58136 (0.17%)  
[  5]   4.00-5.00   sec  80.9 MBytes   679 Mbits/sec  0.018 ms  0/58112 (0%)  
[  5]   5.00-6.00   sec  80.2 MBytes   673 Mbits/sec  0.029 ms  0/57631 (0%)  
[  5]   6.00-7.00   sec  80.4 MBytes   675 Mbits/sec  0.016 ms  103/57877 (0.18%)  
[  5]   7.00-8.00   sec  79.7 MBytes   669 Mbits/sec  0.028 ms  72/57318 (0.13%)  
[  5]   8.00-9.00   sec  77.9 MBytes   653 Mbits/sec  0.019 ms  0/55915 (0%)  
[  5]   9.00-10.00  sec  78.9 MBytes   662 Mbits/sec  0.017 ms  0/56669 (0%)  
[  5]  10.00-10.00  sec   321 KBytes   707 Mbits/sec  0.018 ms  0/225 (0%)  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   799 MBytes   670 Mbits/sec  0.018 ms  357/574071 (0.062%)  receiver

Windows tester

iperf3.exe -u -b900M -l1460

Results

As you can see, Windows is only able to output less than 700M at that data size.

PS C:\Users\junfan\Desktop\iperf-3.1.3-win64> .\iperf3.exe -u -b900M -c 172.17.17.166 -l1460
Connecting to host 172.17.17.166, port 5201
[  4] local 172.17.17.158 port 58299 connected to 172.17.17.166 port 5201
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-1.00   sec  79.0 MBytes   663 Mbits/sec  56755
[  4]   1.00-2.00   sec  79.0 MBytes   663 Mbits/sec  56753
[  4]   2.00-3.00   sec  82.0 MBytes   688 Mbits/sec  58923
[  4]   3.00-4.00   sec  80.9 MBytes   679 Mbits/sec  58123
[  4]   4.00-5.00   sec  80.9 MBytes   679 Mbits/sec  58116
[  4]   5.00-6.00   sec  80.2 MBytes   673 Mbits/sec  57631
[  4]   6.00-7.00   sec  80.6 MBytes   676 Mbits/sec  57869
[  4]   7.00-8.00   sec  79.8 MBytes   670 Mbits/sec  57340
[  4]   8.00-9.00   sec  77.8 MBytes   653 Mbits/sec  55906
[  4]   9.00-10.00  sec  78.9 MBytes   662 Mbits/sec  56657
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-10.00  sec   799 MBytes   671 Mbits/sec  0.018 ms  357/574071 (0.062%)
[  4] Sent 574071 datagrams

Using -R I get a lot more loss, because Linux is capable of sending 900M at that size, but Windows is incapable of processing it.

PS C:\Users\junfan\Desktop\iperf-3.1.3-win64> .\iperf3.exe -u -b900M -c 172.17.17.166 -l1460 -R
Connecting to host 172.17.17.166, port 5201
Reverse mode, remote host 172.17.17.166 is sending
[  4] local 172.17.17.158 port 62113 connected to 172.17.17.166 port 5201
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-1.00   sec   102 MBytes   859 Mbits/sec  0.011 ms  3555/77140 (4.6%)
[  4]   1.00-2.00   sec  95.0 MBytes   797 Mbits/sec  0.006 ms  8820/77067 (11%)
[  4]   2.00-3.00   sec   100 MBytes   840 Mbits/sec  0.007 ms  5136/77053 (6.7%)
[  4]   3.00-4.00   sec   103 MBytes   868 Mbits/sec  0.006 ms  2859/77150 (3.7%)
[  4]   4.00-5.00   sec  99.7 MBytes   836 Mbits/sec  0.023 ms  5274/76854 (6.9%)
[  4]   5.00-6.00   sec   101 MBytes   849 Mbits/sec  0.054 ms  4554/77245 (5.9%)
[  4]   6.00-7.00   sec  98.2 MBytes   823 Mbits/sec  0.020 ms  6336/76840 (8.2%)
[  4]   7.00-8.00   sec  97.2 MBytes   816 Mbits/sec  0.021 ms  7281/77106 (9.4%)
[  4]   8.00-9.00   sec  93.7 MBytes   786 Mbits/sec  0.022 ms  9688/76997 (13%)
[  4]   9.00-10.00  sec  95.7 MBytes   803 Mbits/sec  0.010 ms  8464/77172 (11%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-10.00  sec  1.05 GBytes   900 Mbits/sec  0.007 ms  61968/770678 (8%)
[  4] Sent 770678 datagrams

Thanks for any education you can bestow on me.

[bmah888]

The usual disclaimer: ESnet cannot and does not officially support iperf3 on Windows. Some people have made this work (obviously) but there could be differences between the Windows and *nix networking stacks that might cause issues or unexplained performance differences.

In your case, the difference might come from running an ancient version of iperf3 (3.1.3). It's about 7 years old. The current version of iperf3 is 3.15, and was released a few weeks ago. Note that sometime around iperf 3.2 we changed the default UDP packet size from 8192 to something based on the path MTU, precisely to avoid fragmentation and disproportionate packet loss. Your Windows binary doesn't have this change. If you want you can change iperf3's UDP send size to something smaller using the -l flag (for example -l 1460). But you really should find a newer Windows binary. Some people have had success with the files hosted on https://files.budman.pw/ but do remember that these were built and hosted by a third party.

Another possible cause of packet loss might be the socket buffer size (set, somewhat confusingly by the -w option). Sometimes UDP tests might need to have this set larger in order to be able to handle bursts of UDP packets arriving at once.

Note that there's no guarantee that any of this discussion is relevant to the behavior of the on-line game you were trying to troubleshoot originally, but hopefully it'll give some insights. Good luck!