Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Benchmark example #403

Closed
silvioprog opened this issue Jan 27, 2021 · 3 comments
Closed

Benchmark example #403

silvioprog opened this issue Jan 27, 2021 · 3 comments

Comments

@silvioprog
Copy link

Hi.

I have been trying Actix and Warp by doing some benchmarks, and noticed the Actix routing example is very slow, but I think it is related to some missing configuration.

Consider the following examples:

Actix:

use actix_web::{web, App, HttpServer};

#[actix_web::main]
async fn main() -> std::io::Result<()> {
    HttpServer::new(|| {
        App::new()
            .service(web::resource("/hello").to(|| async { "Hello, world!" }))
            .service(web::resource("/another").to(|| async { "Another route" }))
    })
    .bind("0.0.0.0:8080")?
    .run()
    .await
}

Warp:

#![deny(warnings)]

use warp::Filter;

fn main() -> std::io::Result<()> {
    let hello = warp::path("hello").map(|| "Hello, world!");
    let another = warp::path("another").map(|| "Another route");
    let routes = warp::get().and(hello.or(another));
    tokio::runtime::Builder::new_multi_thread()
        .enable_all()
        .build()?
        .block_on(async {
            warp::serve(routes).run(([0, 0, 0, 0], 3030)).await;
        });
    Ok(())
}

Supposing they were compiled using --release flag, notice the following logs from ApacheBench:

Actix:

$ ab -n 10000 -c 1000 -k http://127.0.0.1:8080/hello
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        
Server Hostname:        127.0.0.1
Server Port:            8080

Document Path:          /hello
Document Length:        13 bytes

Concurrency Level:      1000
Time taken for tests:   5.003 seconds
Complete requests:      10000
Failed requests:        77
   (Connect: 0, Receive: 0, Length: 77, Exceptions: 0)
Keep-Alive requests:    9984
Total transferred:      1537536 bytes
HTML transferred:       129792 bytes
Requests per second:    1998.83 [#/sec] (mean)
Time per request:       500.294 [ms] (mean)
Time per request:       0.500 [ms] (mean, across all concurrent requests)
Transfer rate:          300.12 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    4  13.9      0      76
Processing:    19   38 182.7     30    4602
Waiting:        0   30   2.8     30      36
Total:         19   42 183.1     30    4602

Percentage of the requests served within a certain time (ms)
  50%     30
  66%     31
  75%     33
  80%     34
  90%     62
  95%     72
  98%     87
  99%     96
 100%   4602 (longest request)

Warp:

$ ab -n 10000 -c 1000 -k http://127.0.0.1:3030/hello
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        
Server Hostname:        127.0.0.1
Server Port:            3030

Document Path:          /hello
Document Length:        13 bytes

Concurrency Level:      1000
Time taken for tests:   0.392 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    10000
Total transferred:      1540000 bytes
HTML transferred:       130000 bytes
Requests per second:    25535.74 [#/sec] (mean)
Time per request:       39.161 [ms] (mean)
Time per request:       0.039 [ms] (mean, across all concurrent requests)
Transfer rate:          3840.34 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    4  13.6      0      73
Processing:    19   29   3.0     29      73
Waiting:        0   29   3.0     29      36
Total:         19   34  14.2     29     100

Percentage of the requests served within a certain time (ms)
  50%     29
  66%     31
  75%     33
  80%     34
  90%     62
  95%     70
  98%     84
  99%     93
 100%    100 (longest request)

Notice Actix supports only 1.9k reqs/sec, while Warp supports around 25k.

So, how to get the maximum speed from Actix routing? It would be nice to provide an example for benchmarking.

cheers
@fakeshadow
Copy link
Contributor

fakeshadow commented Jan 28, 2021 •
edited
Loading

If you look at the result cleary you are having a lot of failed request. (on actix-web example)
I have not used ab but you are likely benchmarking the default timeout on error of the tool.

@silvioprog
Copy link
Author

silvioprog commented Jan 28, 2021 •
edited
Loading

@fakeshadow, the problem above happens only when running Actix on Linux, it works properly on Windows. Please notice the log below, it was generated on Windows running Actix/Warp:

Actix:

C:\Apache24\bin>ab.exe -n 10000 -c 1000 -k http://127.0.0.1:8080/hello
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:
Server Hostname:        127.0.0.1
Server Port:            8080

Document Path:          /hello
Document Length:        13 bytes

Concurrency Level:      1000
Time taken for tests:   0.313 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    10000
Total transferred:      1540000 bytes
HTML transferred:       130000 bytes
Requests per second:    31944.49 [#/sec] (mean)
Time per request:       31.304 [ms] (mean)
Time per request:       0.031 [ms] (mean, across all concurrent requests)
Transfer rate:          4804.15 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:     2   23  24.7     15     156
Waiting:        0   23  24.6     15     156
Total:          2   23  24.7     15     156

Percentage of the requests served within a certain time (ms)
  50%     15
  66%     16
  75%     19
  80%     21
  90%     28
  95%     87
  98%    127
  99%    142
 100%    156 (longest request)

Warp:

C:\Apache24\bin>ab.exe -n 10000 -c 1000 -k http://127.0.0.1:3030/hello
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:
Server Hostname:        127.0.0.1
Server Port:            3030

Document Path:          /hello
Document Length:        13 bytes

Concurrency Level:      1000
Time taken for tests:   0.312 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    10000
Total transferred:      1540000 bytes
HTML transferred:       130000 bytes
Requests per second:    32050.56 [#/sec] (mean)
Time per request:       31.201 [ms] (mean)
Time per request:       0.031 [ms] (mean, across all concurrent requests)
Transfer rate:          4820.10 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:     2   23  23.9     16     151
Waiting:        2   23  23.9     16     151
Total:          2   23  24.0     16     151

Percentage of the requests served within a certain time (ms)
  50%     16
  66%     17
  75%     17
  80%     17
  90%     34
  95%     86
  98%    123
  99%    136
 100%    151 (longest request)

@silvioprog
Copy link
Author

silvioprog commented Feb 3, 2021 •
edited
Loading

New logs generated on Linux and Windows! 🚀 (after applying many configurations suggested by Warp users here seanmonstar/warp#557 and here seanmonstar/warp#791)

The examples (both built using --release):

Warp:

#![deny(warnings)]

use warp::Filter;

fn main() -> std::io::Result<()> {
    let hello = warp::path("hello").map(|| "Hello, world!");
    let another = warp::path("another").map(|| "Another route");
    let routes = warp::get().and(hello.or(another));
    tokio::runtime::Builder::new_multi_thread()
        .enable_all()
        .build()?
        .block_on(async {
            warp::serve(routes).run(([0, 0, 0, 0], 3030)).await;
        });
    Ok(())
}

Actix:

use actix_web::{get, App, HttpResponse, HttpServer};

#[get("/hello")]
async fn hello() -> HttpResponse {
    HttpResponse::Ok()
        .content_type("text/plain")
        .body("Hello, world!")
}

#[get("/another")]
async fn another() -> HttpResponse {
    HttpResponse::Ok()
        .content_type("text/plain")
        .body("Another route")
}

#[actix_web::main]
async fn main() -> std::io::Result<()> {
    HttpServer::new(|| App::new().service(hello).service(another))
        .bind("0.0.0.0:8080")?
        .run()
        .await
}

Logs generated using Warp on Windows (best result for 5 consecutive hits):

Warp / WRK:

wrk -t10 -c1000 -d10s --latency http://172.19.80.1:3030/hello
Running 10s test @ http://172.19.80.1:3030/hello
  10 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    23.13ms   15.18ms 169.78ms   66.93%
    Req/Sec     4.41k     2.75k   31.35k    96.67%
  Latency Distribution
     50%   21.49ms
     75%   31.62ms
     90%   42.84ms
     99%   68.74ms
  409658 requests in 10.09s, 50.79MB read
Requests/sec:  40587.89
Transfer/sec:      5.03MB

Warp / AB:

ab -n 10000 -c 1000 -k http://172.19.80.1:3030/hello
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 172.19.80.1 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:
Server Hostname:        172.19.80.1
Server Port:            3030

Document Path:          /hello
Document Length:        13 bytes

Concurrency Level:      1000
Time taken for tests:   0.240 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    10000
Total transferred:      1540000 bytes
HTML transferred:       130000 bytes
Requests per second:    41657.29 [#/sec] (mean)
Time per request:       24.005 [ms] (mean)
Time per request:       0.024 [ms] (mean, across all concurrent requests)
Transfer rate:          6264.87 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   2.1      0      17
Processing:     0    4   1.6      4      21
Waiting:        0    4   1.6      4      21
Total:          0    5   2.8      4      37

Percentage of the requests served within a certain time (ms)
  50%      4
  66%      5
  75%      5
  80%      6
  90%      7
  95%      8
  98%     17
  99%     18
 100%     37 (longest request)

Actix / WRK:

wrk -t10 -c1000 -d10s --latency http://172.19.80.1:8080/hello
Running 10s test @ http://172.19.80.1:8080/hello
  10 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    30.92ms   45.25ms 769.86ms   92.65%
    Req/Sec     4.74k     1.00k   19.57k    85.77%
  Latency Distribution
     50%   17.91ms
     75%   28.51ms
     90%   48.61ms
     99%  250.68ms
  474784 requests in 10.10s, 52.07MB read
Requests/sec:  47018.85
Transfer/sec:      5.16MB

Actix / AB:

ab -n 10000 -c 1000 -k http://172.19.80.1:8080/hello
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 172.19.80.1 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:
Server Hostname:        172.19.80.1
Server Port:            8080

Document Path:          /hello
Document Length:        13 bytes

Concurrency Level:      1000
Time taken for tests:   0.243 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    10000
Total transferred:      1390000 bytes
HTML transferred:       130000 bytes
Requests per second:    41137.20 [#/sec] (mean)
Time per request:       24.309 [ms] (mean)
Time per request:       0.024 [ms] (mean, across all concurrent requests)
Transfer rate:          5584.05 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    2   7.5      0      89
Processing:     1   19  13.4     16     102
Waiting:        1   19  13.4     16     102
Total:          1   21  18.5     16     158

Percentage of the requests served within a certain time (ms)
  50%     16
  66%     17
  75%     22
  80%     24
  90%     38
  95%     52
  98%     80
  99%    119
 100%    158 (longest request)

Logs generated using Actix on Linux (best result for 5 consecutive hits):

Warp / WRK:

wrk -t10 -c1000 -d10s --latency http://192.168.100.2:3030/hello
Running 10s test @ http://192.168.100.2:3030/hello
  10 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     6.82ms    4.92ms  50.92ms   69.41%
    Req/Sec    14.79k     1.68k   25.47k    72.40%
  Latency Distribution
     50%    5.91ms
     75%    9.54ms
     90%   13.55ms
     99%   21.76ms
  1473248 requests in 10.09s, 182.65MB read
Requests/sec: 146031.21
Transfer/sec:     18.10MB

Warp / AB:

ab -n 10000 -c 1000 -k http://192.168.100.2:3030/hello
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.100.2 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        
Server Hostname:        192.168.100.2
Server Port:            3030

Document Path:          /hello
Document Length:        13 bytes

Concurrency Level:      1000
Time taken for tests:   0.383 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    10000
Total transferred:      1540000 bytes
HTML transferred:       130000 bytes
Requests per second:    26092.63 [#/sec] (mean)
Time per request:       38.325 [ms] (mean)
Time per request:       0.038 [ms] (mean, across all concurrent requests)
Transfer rate:          3924.09 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    4  12.5      0      51
Processing:    20   31   3.5     30      50
Waiting:        0   31   3.5     30      38
Total:         20   35  13.8     30      80

Percentage of the requests served within a certain time (ms)
  50%     30
  66%     33
  75%     35
  80%     35
  90%     50
  95%     76
  98%     78
  99%     78
 100%     80 (longest request)

Actix / WRK:

wrk -t10 -c1000 -d10s --latency http://192.168.100.2:8080/hello
Running 10s test @ http://192.168.100.2:8080/hello
  10 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     7.18ms    6.32ms  57.44ms   76.35%
    Req/Sec    14.53k     2.36k   30.48k    70.20%
  Latency Distribution
     50%    5.63ms
     75%   11.06ms
     90%   15.99ms
     99%   25.85ms
  1447518 requests in 10.10s, 158.75MB read
Requests/sec: 143353.46
Transfer/sec:     15.72MB

Actix / AB:

ab -n 10000 -c 1000 -k http://192.168.100.2:8080/hello
[sudo] password for silvioprog: 
This is ApacheBench, Version 2.3 <$Revision: 1879490 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.100.2 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        
Server Hostname:        192.168.100.2
Server Port:            8080

Document Path:          /hello
Document Length:        13 bytes

Concurrency Level:      1000
Time taken for tests:   0.392 seconds
Complete requests:      10000
Failed requests:        0
Keep-Alive requests:    10000
Total transferred:      1390000 bytes
HTML transferred:       130000 bytes
Requests per second:    25513.78 [#/sec] (mean)
Time per request:       39.194 [ms] (mean)
Time per request:       0.039 [ms] (mean, across all concurrent requests)
Transfer rate:          3463.30 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    4  14.0      0      74
Processing:    20   29   2.8     29      74
Waiting:        1   29   2.8     29      36
Total:         20   34  14.5     29     101

Percentage of the requests served within a certain time (ms)
  50%     29
  66%     31
  75%     32
  80%     33
  90%     62
  95%     71
  98%     85
  99%     93
 100%    101 (longest request)

Since both projects are very fast on Linux and Windows, I'm going to close this issue and take a beer! 😅🍺

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@silvioprog @fakeshadow