[Bug] Fix ZMap not obeying --rate edge case

[phillip-stephens]

Resolves #903

Thanks to @huxcrux for helping to debug this issue that I was having difficulty reproducing locally.

The bug he saw was that sometimes, ZMap would not obey the --rate limit set in the CLI. After investigation, the issue is here. The adaptive timing code uses an adaptive delay to ensure the send rate stays near the user's desired rate.

This code works in most circumstances, however if some outside influence (other processes in @huxcrux's case were having contention on the core, when he used --cores= to pin ZMap, the effect was far reduced) prevents ZMap from sending at the desired rate, this adaptive delay will decrease to attempt to send faster. If ZMap is continually prevented from sending at the desired rate, the delay variable will be halved repeatedly until eventually it becomes equal to zero. With a delay = 0, it will remain locked at 0 even if in the future ZMap can send faster. The adaptive delay logic uses a multiplier to modify delay and if delay = 0, it'll stay at 0.

From this point on, the delay will be locked at delay = 0 and ZMap will send at the maximum rate supported by the NIC and CPU.

Testing

To reproduce what @huxcrux saw, I used tc to artificially limit the bandwidth of my testing VM.

main

TImeline Summary

1. Send rate looks good at --rate=24000
2. Bandwidth limiting imposed, rate drops
3. Bandwidth limits removed after a few seconds, rate spikes up to 200k pps

0:20 0% (4h58m left); send: 480519 24.0 Kp/s (24.0 Kp/s avg); recv: 603 27 p/s (30 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.13%
 0:21 0% (4h58m left); send: 504461 23.9 Kp/s (24.0 Kp/s avg); recv: 625 22 p/s (29 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:22 0% (4h58m left); send: 528408 23.9 Kp/s (24.0 Kp/s avg); recv: 651 26 p/s (29 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:23 0% (4h58m left); send: 552465 24.1 Kp/s (24.0 Kp/s avg); recv: 689 38 p/s (29 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:24 0% (5h05m left); send: 563391 10.9 Kp/s (23.4 Kp/s avg); recv: 705 16 p/s (29 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.13%
 0:25 0% (5h17m left); send: 563583 192 p/s (22.5 Kp/s avg); recv: 705 0 p/s (28 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.13%
 0:26 0% (5h30m left); send: 563711 128 p/s (21.7 Kp/s avg); recv: 705 0 p/s (27 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.13%
 0:27 0% (5h42m left); send: 563967 256 p/s (20.9 Kp/s avg); recv: 705 0 p/s (26 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.13%
 0:28 0% (5h55m left); send: 564095 128 p/s (20.1 Kp/s avg); recv: 706 1 p/s (25 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.13%
 0:29 0% (6h07m left); send: 564351 256 p/s (19.4 Kp/s avg); recv: 707 1 p/s (24 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.13%
 0:30 0% (6h20m left); send: 564607 256 p/s (18.8 Kp/s avg); recv: 707 0 p/s (23 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.13%
 0:31 0% (6h32m left); send: 564671 64 p/s (18.2 Kp/s avg); recv: 708 1 p/s (22 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.13%
 0:32 0% (6h45m left); send: 564927 256 p/s (17.6 Kp/s avg); recv: 708 0 p/s (22 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.13%
 0:33 0% (6h58m left); send: 565055 128 p/s (17.1 Kp/s avg); recv: 708 0 p/s (21 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.13%
 0:34 0% (7h10m left); send: 565311 256 p/s (16.6 Kp/s avg); recv: 708 0 p/s (20 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.13%
 0:35 0% (7h23m left); send: 565439 128 p/s (16.1 Kp/s avg); recv: 708 0 p/s (20 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.13%
 0:36 0% (7h35m left); send: 565631 192 p/s (15.7 Kp/s avg); recv: 708 0 p/s (19 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.13%
 0:37 0% (7h48m left); send: 565759 128 p/s (15.3 Kp/s avg); recv: 708 0 p/s (19 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.13%
 0:38 0% (8h00m left); send: 566015 256 p/s (14.9 Kp/s avg); recv: 709 1 p/s (18 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.13%
 0:39 0% (8h12m left); send: 566271 256 p/s (14.5 Kp/s avg); recv: 709 0 p/s (18 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.13%
 0:40 0% (8h25m left); send: 566399 128 p/s (14.1 Kp/s avg); recv: 710 1 p/s (17 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.13%
 0:41 0% (8h37m left); send: 566591 192 p/s (13.8 Kp/s avg); recv: 710 0 p/s (17 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.13%
 0:42 0% (8h50m left); send: 566719 128 p/s (13.5 Kp/s avg); recv: 711 1 p/s (16 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.13%
 0:43 0% (9h02m left); send: 566975 256 p/s (13.2 Kp/s avg); recv: 711 0 p/s (16 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.13%
 0:44 0% (6h39m left); send: 787967 221 Kp/s (17.9 Kp/s avg); recv: 904 193 p/s (20 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.11%
 0:45 0% (4h56m left); send: 1086527 298 Kp/s (24.1 Kp/s avg); recv: 1231 327 p/s (27 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.11%
 0:46 0% (3h57m left); send: 1385151 298 Kp/s (30.1 Kp/s avg); recv: 1589 358 p/s (34 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.11%
 0:47 0% (3h19m left); send: 1684799 300 Kp/s (35.8 Kp/s avg); recv: 1904 315 p/s (40 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.11%
 0:48 0% (2h52m left); send: 1983103 298 Kp/s (41.3 Kp/s avg); recv: 2164 260 p/s (45 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.11%
 0:49 1% (2h33m left); send: 2280319 297 Kp/s (46.5 Kp/s avg); recv: 2463 299 p/s (50 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.11%
 0:50 1% (2h19m left); send: 2560511 280 Kp/s (51.2 Kp/s avg); recv: 2746 283 p/s (54 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.11%
 0:51 1% (2h07m left); send: 2846783 286 Kp/s (55.8 Kp/s avg); recv: 3003 257 p/s (58 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.11%
 0:52 1% (1h58m left); send: 3131839 285 Kp/s (60.2 Kp/s avg); recv: 3267 264 p/s (62 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.10%
 0:53 1% (1h50m left); send: 3412799 281 Kp/s (64.3 Kp/s avg); recv: 3519 252 p/s (66 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.10%
 0:54 1% (1h43m left); send: 3698431 286 Kp/s (68.4 Kp/s avg); recv: 3880 361 p/s (71 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.10%

Phillip/903-force-delay-nonzero

1. Send rate looks good at --rate=24000
2. Bandwidth limiting imposed, rate drops
3. Bandwidth limits removed after a few seconds, rate returns to normal --rate=24000

 0:20 0% (4h58m left); send: 480081 24.0 Kp/s (24.0 Kp/s avg); recv: 606 26 p/s (30 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.13%
 0:21 0% (4h58m left); send: 504087 24.0 Kp/s (24.0 Kp/s avg); recv: 626 20 p/s (29 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:22 0% (4h58m left); send: 528105 24.0 Kp/s (24.0 Kp/s avg); recv: 652 26 p/s (29 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:23 0% (4h58m left); send: 552102 24.0 Kp/s (24.0 Kp/s avg); recv: 692 40 p/s (30 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.13%
 0:24 0% (4h58m left); send: 576112 24.0 Kp/s (24.0 Kp/s avg); recv: 718 26 p/s (29 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:25 0% (4h58m left); send: 600114 24.0 Kp/s (24.0 Kp/s avg); recv: 737 19 p/s (29 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:26 0% (4h58m left); send: 624111 24.0 Kp/s (24.0 Kp/s avg); recv: 764 27 p/s (29 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:27 0% (5h05m left); send: 631679 7.57 Kp/s (23.4 Kp/s avg); recv: 774 10 p/s (28 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:28 0% (5h17m left); send: 631743 64 p/s (22.5 Kp/s avg); recv: 774 0 p/s (27 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:29 0% (5h28m left); send: 631999 256 p/s (21.8 Kp/s avg); recv: 774 0 p/s (26 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:30 0% (5h39m left); send: 632127 128 p/s (21.1 Kp/s avg); recv: 774 0 p/s (25 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:31 0% (5h50m left); send: 632383 256 p/s (20.4 Kp/s avg); recv: 775 1 p/s (24 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:32 0% (6h01m left); send: 632511 128 p/s (19.8 Kp/s avg); recv: 775 0 p/s (24 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:33 0% (6h13m left); send: 632767 256 p/s (19.2 Kp/s avg); recv: 775 0 p/s (23 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:34 0% (6h24m left); send: 632831 64 p/s (18.6 Kp/s avg); recv: 775 0 p/s (22 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:35 0% (6h35m left); send: 633087 256 p/s (18.1 Kp/s avg); recv: 775 0 p/s (22 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:36 0% (6h46m left); send: 633215 128 p/s (17.6 Kp/s avg); recv: 775 0 p/s (21 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:37 0% (6h57m left); send: 633471 256 p/s (17.1 Kp/s avg); recv: 775 0 p/s (20 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:38 0% (7h08m left); send: 633727 256 p/s (16.7 Kp/s avg); recv: 775 0 p/s (20 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:39 0% (7h20m left); send: 633791 64 p/s (16.2 Kp/s avg); recv: 775 0 p/s (19 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:40 0% (7h31m left); send: 634047 256 p/s (15.8 Kp/s avg); recv: 775 0 p/s (19 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:41 0% (7h42m left); send: 634175 128 p/s (15.5 Kp/s avg); recv: 775 0 p/s (18 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:42 0% (7h53m left); send: 634431 256 p/s (15.1 Kp/s avg); recv: 775 0 p/s (18 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:43 0% (8h04m left); send: 634559 128 p/s (14.8 Kp/s avg); recv: 775 0 p/s (18 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:44 0% (8h15m left); send: 634687 128 p/s (14.4 Kp/s avg); recv: 776 1 p/s (17 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:45 0% (8h27m left); send: 634879 192 p/s (14.1 Kp/s avg); recv: 776 0 p/s (17 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:46 0% (8h38m left); send: 635007 128 p/s (13.8 Kp/s avg); recv: 777 1 p/s (16 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:47 0% (8h49m left); send: 635263 256 p/s (13.5 Kp/s avg); recv: 778 1 p/s (16 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:48 0% (9h00m left); send: 635519 256 p/s (13.2 Kp/s avg); recv: 778 0 p/s (16 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:49 0% (9h11m left); send: 635647 128 p/s (13.0 Kp/s avg); recv: 778 0 p/s (15 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:50 0% (9h22m left); send: 635839 192 p/s (12.7 Kp/s avg); recv: 780 2 p/s (15 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:51 0% (9h33m left); send: 635967 128 p/s (12.5 Kp/s avg); recv: 780 0 p/s (15 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:52 0% (9h44m left); send: 636223 256 p/s (12.2 Kp/s avg); recv: 781 1 p/s (15 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:53 0% (9h55m left); send: 636351 128 p/s (12.0 Kp/s avg); recv: 781 0 p/s (14 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:54 0% (10h left); send: 636607 256 p/s (11.8 Kp/s avg); recv: 781 0 p/s (14 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:55 0% (10h left); send: 636671 64 p/s (11.6 Kp/s avg); recv: 781 0 p/s (14 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:56 0% (10h left); send: 636927 256 p/s (11.4 Kp/s avg); recv: 782 1 p/s (13 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:57 0% (10h left); send: 637055 128 p/s (11.2 Kp/s avg); recv: 782 0 p/s (13 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:58 0% (10h left); send: 637311 256 p/s (11.0 Kp/s avg); recv: 782 0 p/s (13 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 0:59 0% (11h left); send: 637439 128 p/s (10.8 Kp/s avg); recv: 782 0 p/s (13 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 1:00 0% (11h left); send: 637631 192 p/s (10.6 Kp/s avg); recv: 782 0 p/s (13 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 1:01 0% (11h left); send: 637759 128 p/s (10.5 Kp/s avg); recv: 782 0 p/s (12 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 1:02 0% (11h left); send: 638975 1.22 Kp/s (10.3 Kp/s avg); recv: 782 0 p/s (12 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 1:03 0% (11h left); send: 669186 30.2 Kp/s (10.6 Kp/s avg); recv: 816 34 p/s (12 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 1:04 0% (11h left); send: 693196 24.0 Kp/s (10.8 Kp/s avg); recv: 843 27 p/s (13 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 1:05 0% (10h left); send: 717192 24.0 Kp/s (11.0 Kp/s avg); recv: 874 31 p/s (13 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 1:06 0% (10h left); send: 740947 23.8 Kp/s (11.2 Kp/s avg); recv: 903 29 p/s (13 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 1:07 0% (10h left); send: 764947 24.0 Kp/s (11.4 Kp/s avg); recv: 930 27 p/s (13 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 1:08 0% (10h left); send: 788942 24.0 Kp/s (11.6 Kp/s avg); recv: 954 24 p/s (14 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 1:09 0% (10h left); send: 812953 24.0 Kp/s (11.8 Kp/s avg); recv: 978 24 p/s (14 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 1:10 0% (9h57m left); send: 836976 24.0 Kp/s (12.0 Kp/s avg); recv: 1009 31 p/s (14 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 1:11 0% (9h49m left); send: 860980 24.0 Kp/s (12.1 Kp/s avg); recv: 1033 24 p/s (14 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 1:12 0% (9h41m left); send: 884985 24.0 Kp/s (12.3 Kp/s avg); recv: 1071 38 p/s (14 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 1:13 0% (9h34m left); send: 908984 24.0 Kp/s (12.4 Kp/s avg); recv: 1091 20 p/s (14 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 1:14 0% (9h26m left); send: 932991 24.0 Kp/s (12.6 Kp/s avg); recv: 1115 24 p/s (15 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%
 1:15 0% (9h20m left); send: 956364 23.4 Kp/s (12.7 Kp/s avg); recv: 1146 31 p/s (15 p/s avg); drops: 0 p/s (0 p/s avg); hitrate: 0.12%

Here you can see the bandwidth constraints imposed and the send rate drop. But after they're lifted, we just return to sending at 24000.