Benchmarks

[112RG]

To start wow this library is fast I only just managed to find a web server that could beat it. To my surprise glommio didn't beat it out which I thought it would. But bytdance's monoio just started to gain ahead with only 5 threads. Any more and the performance would be degraded I suspect this is because I am running this under WSL. I plan to run the tests again on native Linux when I reboot next

The benches and graph gen can be found on my fork

https://github.com/112RG/zap/tree/hyper_bench

[112RG]

On native Linux

[renerocksai]

Hi,

First of all, thanks for the effort you put into it! I especially like that you automated graph plotting. With a bit more work, the whole thing could be integrated into a master shell script that runs all tests, stores the outputs in a defined place, and finally produces the graphs.

FYI I always ran 3 runs of each category, then took the mean of all 3 runs. Sometimes there was quite a bit of variance within the categories.

Story time: (end of technical discussion above)
Now, every time I approach it, my explanation for the comparisons gets a little better, subjectively: I did NOT intend to, and never intended even to claim that I would develop the fastest stack on the planet - and therefore, every other one would suck, and you should all stop what you're doing and use Zap instead. 😄

As you might know from the README, I needed a replacement for Python + Flask. I wanted my (then) future framework to be written in Zig because of the features and guarantees Zig provides - which are very important to me.

At some stage, I knew I would need to justify throwing away all that Python code that I had built over the years: so I would need to back up my claims and at least show if Zap was faster than Python (...). But being fast would not be enough. I could have written a Zig + Go hybrid like it was suggested by the community - so it was good to show how Zap would compare against Go in general, ignoring the garbage collection implications, etc.

I opted to begin with the most basic test for easy comparison across frameworks, with the option (in my mind) of maybe conducting "better" tests later. That's how the comparison between Zap and a simple Python HTTP server and a simple GO HTTP server came about. They proved it was a good idea to do it the way I did it, compared with the two alternatives: a) doing nothing and sticking with Python, and b) going the GO hybrid route.

Realizing that the results of Zap were rather good, not just being almost as fast as Go's quite good networking stack but even surpassing it in my tests, I eventually also tried out going the Rust route. Just to get a sense of it. I was curious.

Unfortunately, many Rustaceans were VERY sensitive when their beloved Rust wouldn't come out on top of every benchmark in the world. But instead of doing something against it, most just started complaining about unfair tests, some assumed I had compiled in debug mode, and some claimed I had "probably" or "obviously" rigged the tests, effectively trying to just piss on my project, without any evidence. Some DEMANDED that I implement alternatives in Rust using whatever they preferred. Pure emotional, childish behavior in my eyes. Not a single engineer from "the Go community" or "the Python community" (if there even are such things as representative global programming language communities) reacted like this. Sigh.

Also, I was happy. I didn't mean to compare Zap with "everything" or anything else. I had proof that my new framework is definitely worth it and was extremely satisfied I could implement all my future backends in Zig now. Three online experiments later, I am still very pleased!

So, to end this little story, I'm very happy when someone actually puts in the work and shows how Zap fares against other highly optimized (assumed) frameworks. I perceive that as added value. Even if Zap "loses" 😄.

The performance difference is not big enough for me to switch to Rust. Rust is not made for me. I am happy to stay in Zig land and can claim I made the (now even better) informed tradeoff of being just, what I would still call, "blazingly fast" versus: even faster than "blazingly fast" 😄, based on static HTTP server tests.

BTW I would TOTALLY not be surprised if someone found a simple C lib (optimized for serving static HTML from static memory) that would beat everything we've tested out so far by almost an order of magnitude. (OK, maybe not that much but you get the idea).

[renerocksai]

BTW regarding performance degrading with more than 5 threads, I highly recommend watching King Protty's talk at SYCL 2023 when it comes out. He is an expert in all things IO and concurrency in my eyes.

Just to see what level of expertise you are getting yourself into, watch his talk from last year: https://twitter.com/kingprotty/status/1586116613463736320

[112RG]

Thank you for the reply! I know many people in the Rust community who get angry when Rust is beaten because some seem to think it is the fastest language known to man. Which I don't think is. I believe a zig io_uring HTTP server could beat the rust one. And as you said a highly optimized C one could most likely beat them all out.

I am glad you didn't see my post as pissing on your project. I think it's a great project and is actually pushing me more to try Zig for myself. I am also glad you didn't decide to switch to Rust just because a language is faster doesn't mean it's worth switching to and I am a big advocate for people using tools and Languages they love rather than what other people deem better!

Thank you for the recommendation of that talk it is very interesting and they are defentitly a expert at IO

[alexpyattaev]

I am assuming you are still looking for an explanation about why your rust benchmark was not performing well. Here are the reasons:

• You create a thread pool with 4 threads. This means that at any point in time the server can only ever wait for 4 events. Max. This is quite terrible setup as you have more than 4 requests outstanding from wrk at any given moment. You must have more worker threads then you have available cores if you are going to do blocking IO (which is what you do).
• You have threads sharing a mutex guarding a receiver for a channel. This is a very silly setup as you end up using two mutexes (one around the receiver, and another inside the channel itself). The most reasonable solution would be to either have a channel for each worker (and go round-robin over them).
• Finally, your measurement setup does not confine any of the processes to use just 4 cores (or however many) If you confine the load to use 4 cores, and the bench itself to use 4 as well, you'll get much more reasonable perf out of rust implementation. On my machine I'm getting ~8 times better perf from rust than python. I'll make a PR some time later today.

[renerocksai]

Thanks! The 4 threads are likely to be the bottleneck indeed. BTW the example was taken more or less verbatim from the rust book, where it was an example for how to do network programming. Please don't blame me for making silly decisions, especially with regard to mutexes.

Looking forward to your PR!

[alexpyattaev]

No blame, I was just curious how you could get such poor perf. I'm still checking a few things here w.r.t. other implementations, as I am now unsure why go still outperforms (it should not unless rust's standard library is doing something really silly). Can you point me to specific example you have used as the basis? I could not find any particular suspects.
Update - just tried the axum example and it shows nearly 3x the performance of the threaded implementation. Granted, axum was specifically designed to do http, but such terrible performance from standard library is not OK.

[renerocksai]

Thanks for investing so much time into this! I really appreciate the professional help. The example I based the bench on, I found here.

And yes, the axum example is a "proper" example 😄.

[alexpyattaev]

Thanks for the link!
Dug a bit more, there is indeed something "odd" going on there. I can not load the "server" cores by more than ~50%, which is pathetic. The program is spending majority of time in the syscalls instead of actually doing useful work, which is very annoying for something this simple. While this may be acceptable for C++, the general expectation for rust is that it should not be this slow by default.

[alexpyattaev]

Ok, so the resume is as follows - OS threads (which is what you end up using by default in rust) are just not a good fit for this problem. No matter how you slice it, channels between OS threads are slow compared to event-based IO. As a result, without something like epoll it is actually impossible to make this run efficiently (at least on linux)

[kassane]

Similar thing can be said about cpp-beast. It claims to be async and a beast, yet its performance seems to be below what I would expect from a "fast" C++ application.

@renerocksai,

boost::beast is a add-on for boost::asio including http and websockets support.

boost::asio::thread_pool implementation is based on blocking queue and mutex/semaphores. Although using coroutines (awaitable<T> functions) (they are sequentially asynchronous) wouldn't be beneficial because of the high cost of allocating memory.

Theory

Boost.Asio is optimized for throughput. The techniques normally used for low latency are:

• Blocking API with timeout.
• Looping the same set (which needs to be small) of events you're waiting for.
• Keeping the process "hot", without discarding caches, even if it hurts throughput or is inefficient.
• CPU pinning, etc.

Boost.Asio offers blocking APIs and a lot of low-level control too, so it's fine, but then you'd be using it more (1) for the buffer abstractions (which don't even save that much work) and (2) for the cross-platform abstraction (although you'd still need to write specific code for each platform).

[renerocksai]

Thanks for the info! And sorry, I might be ignorant because of a lack of knowledge about boost::asio and boost::beast.

[kassane]

Don't worry, not even c++ developers know. (most)

[alexpyattaev]

@kassane can you provide an implementation that would be worth comparing? Not "the most performant thing possible in C++" but reasonably optimized idiomatic version.