Benchmarks

[dcharkes]

We should get an understanding of how package:cronet stacks up against dart:io. Cronet is not integrated into the Dart VM which might create some overhead. On the other hand Cronet provides HTTP 2 and QUIC.

We should add a benchmarks/ folder. Some inspiration from setting up benchmarks can be done from https://github.com/dart-lang/sdk/blob/master/benchmarks/MapLookup/dart/MapLookup.dart.

We are interested in multiple dimensions:

• JIT (dart ...) and AOT (dart compile exe ...).
• Latency (how fast can a single request be) and throughput (how many concurrent requests per second can we do) - tested by doing only sequential requests and parallel requests respectively.

[unsuitable001]

We may reduce benchmarking boilerplate by using package:benchmark_harness.
Edit: Can't use this package as async support isn't there yet. dart-lang/benchmark_harness#44

Should we bump up the cronet version to the latest for a better QUIC support?
I was playing around with 2 different versions of chromium (v83 & v90) and I've noticed that v83 rarely uses QUIC even on the enabled sites. v90 does a far better job (though, in some cases it used QUIC after multiple visits to a site). Currently v86 is being used.

[unsuitable001]

Version Info

package:cronet Version: 0.0.1+1
Dart SDK: 2.12.0
Cronet Dynamic Library Version: 86.0.4240.198 Debug

Report 1

Environment
Server: Flask 2.0.1 with Waitress 2.0.0 on Ubuntu 20.04 LTS
Hosted: Local Server
HTTP Protocol: HTTP/1.1
TLS: No
Payload: Lorem Ipsum plain text appended with the url query parameter sent from the client. For each query, different URL query parameter is used.

Performance (JIT Mode)

Cronet: 1807.8428 us
dart:io: 1801.662115336 us

So, we can see that package:cronet and dart:io performs on par with each other in JIT Mode.

Note: These are average values derived by running the benchmarking script 5 times.

Performance (AOT Mode)

Cronet: 1441.286689427 us
dart:io: 1049.144274589 us

So, we can see that dart:io takes a slight edge over package:cronet in AOT Mode.

Note: These are average values derived by running the benchmarking script 5 times.

[unsuitable001]

Report 2

I'm surprised (and confused) by the outcome.

Environment
Server: Caddy v2.4.3 (golang version 1.15.13) on Ubuntu 20.04 LTS
Hosted: Local Server
HTTP Protocol: HTTP/2 with Experimental HTTP3 support
TLS: Yes
Compression: gzip
Payload: Visual Studio CE iso file. Size: 7.1GB

Performance (JIT Mode)
Cronet: 202.690879 sec
dart:io: 167.378507 sec

dart:io got the lead here. (A little surprised)

Performance (AOT Mode)
Cronet: 270.522353 sec
dart:io: 155.367736 sec

dart:io got the lead here too.

I tested this again. And, almost similar result came up. Shocked. Not because of dart:io. But, package:cronet's performance decreased in AOT mode? Weird. The only way I can reduce the shock is, we're using a debug build. Or, is my machine acting funny?

[dcharkes]

Well, time to dig into what is going on.

1. Is it slow because of computations or because of waiting on I/O or async. (E.g. is the CPU fully in use or not.)
2. If the CPU is fully in use, we can use tools such as Linux perf to see what is being used.

Sometimes in JIT we can do speculative optimizations that the AOT cannot do. But 202 -> 270 seconds is surprising.

Another hypothesis is that we copy data multiple times in the package:cronet implementation but not in the dart:io implementation.

All the surprise means we're going to learn something! 🦀

[unsuitable001]

Is it slow because of computations or because of waiting on I/O or async. (E.g. is the CPU fully in use or not.)

No. CPU isn't getting fully utilized. I tested again. For Cronet AOT, time got reduced to 258 sec but other observations stayed the same.
Now, there's something interesting going on when I observed resource consumption.
For Cronet, in JIT mode, the CPU utilization is 32% while in AOT, it is 21%. Memory consumption remained the same, around 20MB.
For dart:io, it is even more interesting. In both the modes, CPU utilization remained 16%. However, for memory, in JIT mode it is 30MB. While, in AOT it is 61MB of utilization.

[unsuitable001]

Sometimes in JIT we can do speculative optimizations that the AOT cannot do.

I want to hear more about this. Tell me about this when you'll have less workload.

Another hypothesis is that we copy data multiple times in the package:cronet implementation but not in the dart:io implementation.

Can be a reason.

All the surprise means we're going to learn something! 🦀

Excited 😄

[unsuitable001]

Report 3

Here we go again ᕕ( ᐛ )ᕗ with another report.

Environment
I tried to download Google Chrome Deb Package. Size: 79MB (approx).

Performance (JIT)
Cronet: 51.5 sec
dart:io: 47.02 sec

Performance (AOT)
Cronet: 47.003 sec (This time, cronet acted a bit "rational")
dart:io: 47.75 sec

This time, cronet actually crossed dart:io in AOT mode by a negligible margin. Though, this time, dart:io performed very slightly worse? But, the difference maybe due to the uncertainties come along with networking. Nonetheless, we can see that, in practical workload, they seem to perform on par in both the modes in this benchmarking round.

[unsuitable001]

Report 4 (Throughput)

Environment
Server: Caddy v2.4.3 (golang version 1.15.13) on Ubuntu 20.04 LTS
Hosted: Local Server
HTTP Protocol: HTTP/2 with Experimental HTTP3 support
TLS: Yes
Compression: gzip
Payload: example.org's index.html

Target Time Duration: 1 second.

Performance (JIT)
Cronet: Total Spawned: 4096, In Time Returns: 2982. This is the best result. After that, no. of successful request dipped slowly.
dart:io: Total Spawned: 512, In Time Returns: 512. This is the best result. After that, numbers dipped to 1 successful request. So, 512 concurrent requests are the limit for dart:io in this case.

Performance (AOT)
Cronet: Total Spawned: 4096, In Time Returns: 2883. This is the best result. After that, no. of successful request dipped slowly.
dart:io: Total Spawned: 512, In Time Returns: 512. This is the best result. After that, numbers dipped to 1 successful request. So, 512 concurrent requests are the limit for dart:io in this case.

So, in terms of concurrent requests, package:cronet surpassed dart:io in both the modes with a large margin (5.7x more request approx.)

Report 5 (Throughput)

Environment
We fetched data from example.com.

Target Time Duration: 1 second.

Performance (JIT)
Cronet: Total Spawned: 512, In Time Returns: 178. This is the best result. After that, no. of successful request dipped slowly.
dart:io: Total Spawned: 128, In Time Returns: 39. This is the best result. Then it dipped rapidly and eventually hit 0 as we reach 512 concurrent requests.

Performance (AOT)
Cronet: Total Spawned: 512, In Time Returns: 214.. This is the best result. After that, no. of successful request dipped slowly.
dart:io: Total Spawned: 128, In Time Returns: 49.. This is the best result. Then it dipped rapidly and eventually hit 0 as we reach 512 concurrent requests.

So, by pinging a remote server we observe that, package:cronet served more than 4x more requests compared to dart:io within a second.