benchmark: cronet benchmarks and comparison with dart:io #8

unsuitable001 · 2021-07-07T04:19:28Z

Benchmarks

Package Version: 0.0.1+1
Cronet Version: 86.0.4240.198
Cronet Build Type: Debug
Wrapper Version: 1
Dart SDK Version: 2.12.0

Latency
- Writing Benchmarking Code
- Gathering observations in different environments and writing reports
- Comparing with dart:io
Throughput
- Writing Benchmarking Code
- Gathering observations in different environments and writing reports
- Comparing with dart:io

May get affected by #11

Closes #3

dcharkes · 2021-07-07T11:24:43Z

benchmarks/latency.dart

+  }
+
+  static Future<double> measureFor(Function f, int minimumMillis) async {
+    var minimumMicros = minimumMillis * 1000;


To avoid writing millis and micros, you can use Duration.

Though, future delayed gives us most "time accurate" and faster results, it doesn't ensures if "late" requests are returned or not. As, we are gradually incrementing the no. of spawned tasks in each loop, we must ensure that everything is cleaned up from the previous test.

dcharkes · 2021-07-09T11:12:10Z

Some high level comments:

Documentation on how to set up local servers.
Make the benchmarks so that they don't require editing Dart files.
- The throughput benchmark should run multiple times for parallelism 1, 2, 4, ..., 2^(N+2 above the peak)
- The url tested could be a command line parameter.
We could add a benchmarks/run_all.dart script that automatically runs both JIT and AOT and outputs the markdown tables such that it is very easy to update the result if we change the performance. This would also make it easy for others to reproduce our results with invoking a single script.

unsuitable001 · 2021-07-09T16:57:49Z

@dcharkes Done 😄

mannprerak2 · 2021-07-10T05:19:55Z

benchmarks/benchmarking.md

+2. Create a new `app.py` file that contains -
+
+   ```python
+    from waitress import serve


Shouldn't we ideally put the code required for local servers in the repository?

cc @dcharkes.

Umm.. I can put that if that's what best practice is. But, any server will do the job. I just chosen these 2 servers as they are easy to setup, well known and used in production.

Should I include all the codes in the repo? (And, the resources they served too?)

mannprerak2 · 2021-07-10T05:34:19Z

benchmarks/run_all.dart

+void main(List<String> args) async {
+  var url = 'https://example.com';
+  var throughputPrallelLimit = 10;
+  switch (args.length) {


We can also consider using package:args throughout the package for handling cmd line args. But that's probably good for later if we end up with more arguments.

Yeah. Let's use package:args in other places too.

In a separate PR maybe to keep the commit history more meaningful? As, I'll have to migrate other parts of the package too. (i.e. bin/setup.dart).

dcharkes

Looking good! ✌️

dcharkes · 2021-07-12T08:38:28Z

benchmarks/latency.dart

+
+  Future<double> report() async {
+    final runtime = await measure();
+    print('Cronet(RunTime): $runtime us');


I think microseconds is a bit hard to read. Let's do milliseconds.

dcharkes · 2021-07-12T08:47:46Z

benchmarks/run_all.dart

+  final jitLatency = await CronetBenchmark.main(url);
+  print('AOT');
+  Process.runSync('dart', ['compile', 'exe', 'benchmarks/latency.dart']);
+  final aotLatency = double.parse(


Please also stream the stdout to stdout instead of just capturing it.

dcharkes · 2021-07-12T08:48:00Z

benchmarks/run_all.dart

+      url, pow(2, throughputPrallelLimit).toInt());
+  print('AOT');
+  Process.runSync('dart', ['compile', 'exe', 'benchmarks/throughput.dart']);
+  final throughputStdout = Process.runSync(


Please also stream the stdout to stdout instead of just capturing it.

dcharkes · 2021-07-12T08:49:52Z

benchmarks/latency.dart

+
+import 'package:cronet/cronet.dart';
+
+class CronetBenchmark {


LatencyBenchmark with subclasses CronetLatencyBenchmark and DartIOLatencyBenchmark.

The only difference between two will be swapping the import statement. But, how can I achieve this without 2 near duplicate file? How can I make this subclass structure?

(Maybe it's a really basic question to ask 😅 )

dcharkes · 2021-07-12T08:50:17Z

benchmarks/throughput.dart

+
+import 'package:cronet/cronet.dart';
+
+class CronetThroughputBenchmark {


ThroughputBenchmark with subclasses CronetThroughputBenchmark and DartIOThroughputBenchmark.

dcharkes · 2021-07-12T08:50:59Z

benchmarks/run_all.dart

+  });
+
+  print('Latency Test Results');
+  print('| Mode          | package:cronet |');


Add column dart:io

unsuitable001 · 2021-07-13T04:07:00Z

@dcharkes Done! Also added #11 in TODO and run_all.dart test output.

dcharkes · 2021-07-13T11:49:04Z

Latency Test Results

Mode	package:cronet	dart:io
JIT	90.435 ms	106.105 ms
AOT	89.000 ms	103.000 ms

On my machine, Cronet is faster. 😄

Throughput Test Results

Mode	package:cronet	dart:io
JIT	830 out of 1024	512 out of 512
AOT	850 out of 1024	512 out of 512

Can we make the time configurable for this benchmark? My internet seems too fast for Dart IO to hit it's limit. 🙈

unsuitable001 · 2021-07-13T12:01:43Z

Can we make the time configurable for this benchmark?

@dcharkes You can pass the value of N where 2^N is the max spawn limit as n CLI argument. But, then you have to mention the target server url also as n CLI argument.
It will look like dart run run_all.dart http://example.com 10.
And, the default is 1024 spawns for both. Seems like dart:io is maxing out on 512 and then dipping. Can you confirm this from the console outputs?

unsuitable001 · 2021-07-13T12:06:21Z

I'll give these cli args some name on the next pr as we'll move to package:args. Then, it will be more documented and friendly to use :)

dcharkes · 2021-07-13T12:00:08Z

benchmarks/latency.dart

+  }
+  // TODO: https://github.com/google/cronet.dart/issues/11
+  await CronetLatencyBenchmark.main(url);
+  if (benchmarkDartIO) {


I'd just always run both instead of adding command line argument.

dcharkes · 2021-07-13T12:00:48Z

benchmarks/latency.dart

+}
+
+class DartIOLatencyBenchmark extends LatencyBenchmark {
+  final String url;


Can this be in LatencyBenchmark?

dcharkes · 2021-07-13T12:05:29Z

benchmarks/run_all.dart

+      break;
+    case 2:
+      url = args[0];
+      throughputPrallelLimit = int.parse(args[1]).toInt();


It's more common to use flags for arg parsing so that you can provide them in any order and in any combination.

e.g. If I want to just provide a different throughputPrallelLimit I don't want to type http://example.com

dcharkes · 2021-07-13T12:07:08Z

benchmarks/throughput.dart

+}
+
+class DartIOThroughputBenchmark extends ThroughputBenchmark {
+  final String url;


Can this be in ThroughputBenchmark?

dcharkes · 2021-07-13T12:13:35Z

And, the default is 1024 spawns for both. Seems like dart:io is maxing out on 512 and then dipping. Can you confirm this from the console outputs?

500 ms results:

Throughput Test Results

Mode	package:cronet	dart:io
JIT	291 out of 1024	112 out of 128
AOT	300 out of 1024	128 out of 128

Prints for 1 second:

DartIOThroughputBenchmark: Total Spawned: 1, Returned in time: 1.
DartIOThroughputBenchmark: Total Spawned: 2, Returned in time: 2.
DartIOThroughputBenchmark: Total Spawned: 4, Returned in time: 4.
DartIOThroughputBenchmark: Total Spawned: 8, Returned in time: 8.
DartIOThroughputBenchmark: Total Spawned: 16, Returned in time: 16.
DartIOThroughputBenchmark: Total Spawned: 32, Returned in time: 32.
DartIOThroughputBenchmark: Total Spawned: 64, Returned in time: 64.
DartIOThroughputBenchmark: Total Spawned: 128, Returned in time: 128.
DartIOThroughputBenchmark: Total Spawned: 256, Returned in time: 256.
DartIOThroughputBenchmark: Total Spawned: 512, Returned in time: 512.
DartIOThroughputBenchmark: Total Spawned: 1024, Returned in time: 0.

dcharkes · 2021-07-13T12:14:02Z

Okay, moving to package:args in a next PR is okay. Just disregard my comments on the CLI for now then.

unsuitable001 · 2021-07-13T13:35:36Z

Can we make the time configurable for this benchmark?

@dcharkes Opps! Seems like I misunderstood your comment. Anyways, Duration is now configurable as a 3rd argument from CLI.

Other changes are done except the CLI ones :). Added MacOS as supported platform in README. Seems like we missed that in the other PR.

ref google#9

dcharkes · 2021-07-13T14:10:10Z

🚀 🚀 🚀

benchmark: add cronet latency benchmark

800a8c8

unsuitable001 changed the title ~~benchmark: cronet benchmars and comparison with dart:io~~ benchmark: cronet benchmarks and comparison with dart:io Jul 7, 2021

dcharkes reviewed Jul 7, 2021

View reviewed changes

unsuitable001 added 4 commits July 7, 2021 20:47

latency benchmark: milli, micro seconds -> Duration

a7aea25

benchmark: throughput test

4668399

readme: add benchmark observations

510b168

unsuitable001 requested review from dcharkes and mannprerak2 July 9, 2021 04:17

benchmark: run_all script and server setup documentation

0d9d1f4

mannprerak2 reviewed Jul 10, 2021

View reviewed changes

unsuitable001 added 2 commits July 10, 2021 13:23

benchmark readme: fix caddy installation link formatting

cb4d7d5

benchmarks: include local server code

ffbbb22

unsuitable001 requested a review from mannprerak2 July 11, 2021 08:13

dcharkes reviewed Jul 12, 2021

View reviewed changes

unsuitable001 added 2 commits July 13, 2021 09:17

add dart:io benchmarking

720dd15

benchmark: latency microsec -> millisec

9f3fc45

unsuitable001 requested a review from dcharkes July 13, 2021 04:07

dcharkes approved these changes Jul 13, 2021

View reviewed changes

dcharkes mentioned this pull request Jul 13, 2021

Use package:args #12

Closed

time configurable throughput benchmark

e170d4a

unsuitable001 requested a review from dcharkes July 13, 2021 13:37

add macos to readme

fd05e7b

ref google#9

mannprerak2 mentioned this pull request Jul 13, 2021

Support for macos. #9

Merged

remove macos note for google#13

6587faf

dcharkes merged commit a6eac31 into google:main Jul 13, 2021

unsuitable001 mentioned this pull request Jul 15, 2021

move to package:args for cli arg parsing #15

Merged

hadesss123 mentioned this pull request Jan 16, 2022

When I use 32-bit dynamic library, there is an error #36

Open

glanium mentioned this pull request Dec 10, 2022

flutter pub run cronet:setup failed with error #43

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

benchmark: cronet benchmarks and comparison with dart:io #8

benchmark: cronet benchmarks and comparison with dart:io #8

unsuitable001 commented Jul 7, 2021 •

edited

Loading

dcharkes Jul 7, 2021

unsuitable001 Jul 7, 2021

dcharkes commented Jul 9, 2021

unsuitable001 commented Jul 9, 2021

mannprerak2 Jul 10, 2021

unsuitable001 Jul 10, 2021 •

edited

Loading

unsuitable001 Jul 11, 2021

mannprerak2 Jul 10, 2021

unsuitable001 Jul 10, 2021

unsuitable001 Jul 11, 2021

dcharkes left a comment

dcharkes Jul 12, 2021

dcharkes Jul 12, 2021

dcharkes Jul 12, 2021

dcharkes Jul 12, 2021

unsuitable001 Jul 12, 2021

dcharkes Jul 12, 2021

dcharkes Jul 12, 2021

unsuitable001 commented Jul 13, 2021

dcharkes commented Jul 13, 2021

unsuitable001 commented Jul 13, 2021 •

edited

Loading

unsuitable001 commented Jul 13, 2021

dcharkes Jul 13, 2021

dcharkes Jul 13, 2021

dcharkes Jul 13, 2021

dcharkes Jul 13, 2021

dcharkes commented Jul 13, 2021

dcharkes commented Jul 13, 2021

unsuitable001 commented Jul 13, 2021 •

edited

Loading

dcharkes commented Jul 13, 2021


		import 'package:cronet/cronet.dart';

		class CronetThroughputBenchmark {

benchmark: cronet benchmarks and comparison with dart:io #8

benchmark: cronet benchmarks and comparison with dart:io #8

Conversation

unsuitable001 commented Jul 7, 2021 • edited Loading

Benchmarks

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dcharkes commented Jul 9, 2021

unsuitable001 commented Jul 9, 2021

Choose a reason for hiding this comment

unsuitable001 Jul 10, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dcharkes left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

unsuitable001 commented Jul 13, 2021

dcharkes commented Jul 13, 2021

unsuitable001 commented Jul 13, 2021 • edited Loading

unsuitable001 commented Jul 13, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dcharkes commented Jul 13, 2021

dcharkes commented Jul 13, 2021

unsuitable001 commented Jul 13, 2021 • edited Loading

dcharkes commented Jul 13, 2021

unsuitable001 commented Jul 7, 2021 •

edited

Loading

unsuitable001 Jul 10, 2021 •

edited

Loading

unsuitable001 commented Jul 13, 2021 •

edited

Loading

unsuitable001 commented Jul 13, 2021 •

edited

Loading