Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add netperf UDP stream test #2075

Merged
merged 14 commits into from
Jan 10, 2020
Merged

Add netperf UDP stream test #2075

Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
80e8e7d
add UDP_STREAM and send size flags
dPhanekham Jan 7, 2020
634014f
add flags and UDP_STREAM and pep fixes
dPhanekham Jan 7, 2020
6e4eb0d
remove debug statements
dPhanekham Jan 7, 2020
c108990
add flag to container_netperf
dPhanekham Jan 7, 2020
0c327f1
added UDP STREAM results to tests
dPhanekham Jan 7, 2020
055d047
add -M flag
dPhanekham Jan 8, 2020
01f9d22
remove unnecessary line
dPhanekham Jan 8, 2020
edbc749
add comment justifying default size
dPhanekham Jan 9, 2020
75deef9
change to not in
dPhanekham Jan 9, 2020
26c5758
add upper and lower bounds to UDP send size
dPhanekham Jan 9, 2020
28eecd7
revised comment
dPhanekham Jan 9, 2020
b6ff0ce
remove -R 1 flag for TCP_STREAM, it is implicit
dPhanekham Jan 10, 2020
e5991df
remove line
dPhanekham Jan 10, 2020
c5a2f69
Merge branch 'master' into netperf_UDP_STREAM
bvliu Jan 10, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGES.next.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,8 @@
- Added AWS DAX provider.
- Added Google Cloud Firestore ycsb benchmarks.
- Added support for un-managed data processing yarn cluster benchmarking.
- Added support for UDP_STREAM tests to netperf_benchmark
- Added flags to select stream size to netperf_benchmark
- Added placement group flag support for Azure. "cluster" will create proximity placement group.
"spread" will create an availability set.
- Added a tcpdump collector with --tcpdump flag.
Expand Down
6 changes: 6 additions & 0 deletions perfkitbenchmarker/linux_benchmarks/container_netperf_benchmark.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,11 @@

FLAGS = flags.FLAGS

# We set the default to 128KB (131072 bytes) to override the Linux default
# of 16K so that we can achieve the "link rate".
flags.DEFINE_integer('container_netperf_tcp_stream_send_size_in_bytes', 131072,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a comment as to why this default value was selected.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This value, 131072 bytes (128KB) was selected at the recommendation of @jonesrick.

"The 128K send size is deliberate - otherwise netperf will default to 16K based on Linux send socket buffer size initial values, which has enough overhead to preclude achieving "link rate" as it were. And the default receive size will be 87380 bytes for similar reasons on the receive side. That would be a good thing to change in the netperf command-lines in PKB. And it would match the default send size in iperf3. 64K may be sufficient but 128K would give additional cushion. The need for receive side (-M) isn't as clear-cut but it won't hurt."

Or, if preferred, I could give it a default of None and let netperf select the size unless this flag is set.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think setting the default is important. I'd just like a comment in the code so that we know why we selected this default. Noting that the Linux default of 16K cannot achieve "link rate" and that the higher default value will let us would be sufficient.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we also need to set -s?

Per https://hewlettpackard.github.io/netperf/doc/netperf.html#UDP_005fSTREAM
If the value of the -m option is larger than the local send socket buffer size (-s option) netperf will likely abort with an error message about how the send call failed:

 netperf -t UDP_STREAM -H 192.168.2.125
 UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.2.125 (192.168.2.125) port 0 AF_INET
 udp_send: data send error: Message too long

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a comment to clarify default size. Addressed -s flag in a comment below

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding the comment. Since comments are easier to parse if they contain punctuation and consist of complete sentences, perhaps edit it so that it reads more like the following:

We set the default to 128KB (131072 bytes) to override the Linux default

of 16K so that we can achieve the "link rate".

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm. Looks like the prefix of "#" changed my font size. I just meant to write a comment.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed the comment

'Send size to use for TCP_STREAM tests (netperf -m flag)')

BENCHMARK_NAME = 'container_netperf'
BENCHMARK_CONFIG = """
container_netperf:
Expand Down Expand Up @@ -87,6 +92,7 @@ def Run(benchmark_spec):
'-H', container_0.ip_address,
'-l', '100',
'--',
'-m', FLAGS.container_netperf_tcp_stream_send_size_in_bytes,
'-o', netperf_benchmark.OUTPUT_SELECTOR]
cluster.DeployContainer('netperf', benchmark_spec.container_specs['netperf'])
container_1 = cluster.containers['netperf'][1]
Expand Down
44 changes: 33 additions & 11 deletions perfkitbenchmarker/linux_benchmarks/netperf_benchmark.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,8 +65,15 @@
flags.DEFINE_integer('netperf_thinktime_run_length', 0,
'The number of contiguous numbers to sum at a time in the '
'thinktime array.')

ALL_BENCHMARKS = ['TCP_RR', 'TCP_CRR', 'TCP_STREAM', 'UDP_RR']
flags.DEFINE_integer('netperf_udp_stream_send_size_in_bytes', 1024,
'Send size to use for UDP_STREAM tests (netperf -m flag)',
lower_bound=1, upper_bound=65507)
# We set the default to 128KB (131072 bytes) to override the Linux default
# of 16K so that we can achieve the "link rate".
flags.DEFINE_integer('netperf_tcp_stream_send_size_in_bytes', 131072,
'Send size to use for TCP_STREAM tests (netperf -m flag)')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment here as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment added


ALL_BENCHMARKS = ['TCP_RR', 'TCP_CRR', 'TCP_STREAM', 'UDP_RR', 'UDP_STREAM']
flags.DEFINE_list('netperf_benchmarks', ALL_BENCHMARKS,
'The netperf benchmark(s) to run.')
flags.register_validator(
Expand All @@ -78,7 +85,7 @@
BENCHMARK_NAME = 'netperf'
BENCHMARK_CONFIG = """
netperf:
description: Run TCP_RR, TCP_CRR, UDP_RR and TCP_STREAM
description: Run TCP_RR, TCP_CRR, UDP_RR, TCP_STREAM and UDP_STREAM
vm_groups:
vm_1:
vm_spec: *default_single_core
Expand Down Expand Up @@ -355,8 +362,8 @@ def RunNetperf(vm, benchmark_name, server_ip, num_streams):
"""
enable_latency_histograms = FLAGS.netperf_enable_histograms or num_streams > 1
# Throughput benchmarks don't have latency histograms
enable_latency_histograms = enable_latency_histograms and \
benchmark_name != 'TCP_STREAM'
enable_latency_histograms = (enable_latency_histograms and
(benchmark_name not in ['TCP_STREAM', 'UDP_STREAM']))
# Flags:
# -o specifies keys to include in CSV output.
# -j keeps additional latency numbers
Expand All @@ -367,6 +374,14 @@ def RunNetperf(vm, benchmark_name, server_ip, num_streams):
confidence = ('-I 99,5 -i {0},3'.format(FLAGS.netperf_max_iter)
if FLAGS.netperf_max_iter else '')
verbosity = '-v2 ' if enable_latency_histograms else ''

remote_cmd_timeout = (
FLAGS.netperf_test_length * (FLAGS.netperf_max_iter or 1) + 300)

metadata = {'netperf_test_length': FLAGS.netperf_test_length,
'sending_thread_count': num_streams,
'max_iter': FLAGS.netperf_max_iter or 1}

netperf_cmd = ('{netperf_path} -p {{command_port}} -j {verbosity} '
'-t {benchmark_name} -H {server_ip} -l {length} {confidence}'
' -- '
Expand All @@ -377,7 +392,19 @@ def RunNetperf(vm, benchmark_name, server_ip, num_streams):
server_ip=server_ip,
length=FLAGS.netperf_test_length,
output_selector=OUTPUT_SELECTOR,
confidence=confidence, verbosity=verbosity)
confidence=confidence,
verbosity=verbosity)

if benchmark_name.upper() == 'UDP_STREAM':
netperf_cmd += (' -R 1 -m {send_size} -M {send_size} '.format(
dlott marked this conversation as resolved.
Show resolved Hide resolved
send_size=FLAGS.netperf_udp_stream_send_size_in_bytes))
metadata['netperf_send_size_in_bytes'] = FLAGS.netperf_udp_stream_send_size_in_bytes

elif benchmark_name.upper() == 'TCP_STREAM':
netperf_cmd += (' -m {send_size} -M {send_size} '.format(
send_size=FLAGS.netperf_tcp_stream_send_size_in_bytes))
metadata['netperf_send_size_in_bytes'] = FLAGS.netperf_tcp_stream_send_size_in_bytes

if FLAGS.netperf_thinktime != 0:
netperf_cmd += (' -X {thinktime},{thinktime_array_size},'
'{thinktime_run_length} ').format(
Expand All @@ -402,11 +429,6 @@ def RunNetperf(vm, benchmark_name, server_ip, num_streams):
json_out = json.loads(remote_stdout)
stdouts = json_out[0]

# Metadata to attach to samples
metadata = {'netperf_test_length': FLAGS.netperf_test_length,
'max_iter': FLAGS.netperf_max_iter or 1,
'sending_thread_count': num_streams}

parsed_output = [ParseNetperfOutput(stdout, metadata, benchmark_name,
enable_latency_histograms)
for stdout in stdouts]
Expand Down
10 changes: 10 additions & 0 deletions tests/data/netperf_results.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,5 +38,15 @@
"MIGRATED UDP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 20001 AF_INET to 10.240.31.117 () port 20001 AF_INET : +/-2.500% @ 99% conf. : first burst 0",
"Throughput,Throughput Units,Throughput Confidence Width (%),Confidence Iterations Run,Stddev Latency Microseconds,50th Percentile Latency Microseconds,90th Percentile Latency Microseconds,99th Percentile Latency Microseconds,Minimum Latency Microseconds,Maximum Latency Microseconds,Local Transport Retransmissions,Remote Transport Retransmissions",
"3313.49,Trans/s,7.546,20,214.64,295,330,406,200,500,0,0"
],
[
"MIGRATED UDP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 104.198.67.251 () port 20001 AF_INET : +/-2.500% @ 99% conf.",
"Throughput,Throughput Units,50th Percentile Latency Microseconds,90th Percentile Latency Microseconds,99th Percentile Latency Microseconds,Stddev Latency Microseconds,Minimum Latency Microseconds,Maximum Latency Microseconds,Confidence Iterations Run,Throughput Confidence Width (%),Local Transport Retransmissions,Remote Transport Retransmissions",
"1102.42,10^6bits/s,3,3,11,46.14,1,15144,1,-1.000,-1,-1"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this only a send throughput? Should there also be a receive throughput?

https://hewlettpackard.github.io/netperf/doc/netperf.html#UDP_005fSTREAM

The biggest of these implications is the data which is sent might not be received by the remote. For this reason, the output of a UDP_STREAM test shows both the sending and receiving throughput. On some platforms, it may be possible for the sending throughput to be reported as a value greater than the maximum rate of the link. This is common when the CPU(s) are faster than the network and there is no intra-stack flow-control.

Is do you have a pkb.log that shows the raw output of a run?

],
[
"MIGRATED UDP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 104.198.67.251 () port 20001 AF_INET : +/-2.500% @ 99% conf.",
"Throughput,Throughput Units,50th Percentile Latency Microseconds,90th Percentile Latency Microseconds,99th Percentile Latency Microseconds,Stddev Latency Microseconds,Minimum Latency Microseconds,Maximum Latency Microseconds,Confidence Iterations Run,Throughput Confidence Width (%),Local Transport Retransmissions,Remote Transport Retransmissions",
"1802.72,10^6bits/s,3,3,11,46.14,1,15144,1,-1.000,-1,-1"
]
]
5 changes: 4 additions & 1 deletion tests/linux_benchmarks/netperf_benchmark_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,10 @@ def testExternalAndInternal(self):
('UDP_RR_Latency_p99', 406.0, 'us'),
('UDP_RR_Latency_min', 200.0, 'us'),
('UDP_RR_Latency_max', 500.0, 'us'),
('UDP_RR_Latency_stddev', 214.64, 'us')],
('UDP_RR_Latency_stddev', 214.64, 'us'),
('UDP_STREAM_Throughput', 1102.42, mbps),
('UDP_STREAM_Throughput', 1802.72, 'Mbits/sec'),
],
[i[:3] for i in result])

external_meta = {'ip_type': 'external'}
Expand Down