Skip to content
This repository has been archived by the owner on Jan 13, 2022. It is now read-only.

Investigate newer Siege versions #50

Closed
fredemmott opened this issue Aug 21, 2015 · 8 comments
Closed

Investigate newer Siege versions #50

fredemmott opened this issue Aug 21, 2015 · 8 comments

Comments

@fredemmott
Copy link
Contributor

At first glance, 3.1.0 has no issues - hopefully the blacklist can be loosened to 3.0.*

need to make sure that for major applications, RPS, byte size, and response code ratios are similar (or have an explanation)
@fredemmott
Copy link
Contributor Author

refs #48

@fredemmott fredemmott mentioned this issue Aug 21, 2015
Open
@fredemmott fredemmott mentioned this issue Dec 11, 2015
Closed
@reedy reedy mentioned this issue Dec 12, 2015
Closed
@centminmod
Copy link

would love to see the results of such an investigation .. siege 3.1.4 is the latest version now

@reedy
Copy link
Contributor

reedy commented Jan 11, 2016

Ubuntu 12.04, hhvm 3.6.6

siege 2.70

reedy@oss-perf:~/oss-performance$ hhvm perf.php --mediawiki --hhvm=`which hhvm` --no-proxygen --i-am-not-benchmarking

**************************************************
** 2016-01-11 14:37:25 UTC
** Configuration: MediaWikiTarget on /usr/bin/hhvm

**************************
** 2016-01-11 14:37:25 UTC
** Installing framework

**************************
** 2016-01-11 14:37:34 UTC
** There is no setUpTest

**************************
** 2016-01-11 14:37:34 UTC
** Starting Nginx
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)

**************************
** 2016-01-11 14:37:34 UTC
** Starting PHP Engine
running hphp...
all files saved in /tmp/hhvm-nginxmhxFtc ...
running hphp took 1'11" (71278461 us) wall time
mapping self...
mapping self took 0'00" (3070 us) wall time
loading static content...
loaded file cache from /tmp/hhvm-nginxmhxFtc/static.content
loading static content took 0'00" (9675 us) wall time

**************************
** 2016-01-11 14:38:48 UTC
** Running sanity check

*******************************************
** 2016-01-11 14:38:51 UTC
** Starting Siege for single request warmup
** SIEGE 2.70
** Preparing 1 concurrent users for battle.
The server is now under siege...
done.
Transactions:                300 hits
Availability:             100.00 %
Elapsed time:              29.09 secs
Data transferred:          53.87 MB
Response time:              0.10 secs
Transaction rate:          10.31 trans/sec
Throughput:             1.85 MB/sec
Concurrency:                1.00
Successful transactions:         285
Failed transactions:               0
Longest transaction:            7.73
Shortest transaction:           0.02


******************************************
** 2016-01-11 14:39:20 UTC
** Starting Siege for multi request warmup
** SIEGE 2.70
** Preparing 200 concurrent users for battle.
The server is now under siege...

Lifting the server siege..      done.
Transactions:               2094 hits
Availability:             100.00 %
Elapsed time:              59.80 secs
Data transferred:         442.87 MB
Response time:              5.52 secs
Transaction rate:          35.02 trans/sec
Throughput:             7.41 MB/sec
Concurrency:              193.43
Successful transactions:        1898
Failed transactions:               0
Longest transaction:            9.77
Shortest transaction:           0.36


****************************
** 2016-01-11 14:40:20 UTC
** Clearing nginx access.log

*******************************
** 2016-01-11 14:40:20 UTC
** Starting Siege for benchmark
** SIEGE 2.70
** Preparing 200 concurrent users for battle.
The server is now under siege...

Lifting the server siege..      done.
Transactions:               2451 hits
Availability:             100.00 %
Elapsed time:              59.86 secs
Data transferred:         453.24 MB
Response time:              4.68 secs
Transaction rate:          40.95 trans/sec
Throughput:             7.57 MB/sec
Concurrency:              191.68
Successful transactions:        2251
Failed transactions:               0
Longest transaction:           14.34
Shortest transaction:           1.48


**************************
** 2016-01-11 14:41:22 UTC
** Collecting results

**************************
** 2016-01-11 14:41:25 UTC
** Collecting TC/PCRE data

***************************
** 2016-01-11 14:41:25 UTC
** There is no tearDownTest
{
    "Combined": {
        "Siege requests": 2451,
        "Siege wall sec": 4.68,
        "Siege RPS": 40.95,
        "Siege successful requests": 2251,
        "Siege failed requests": 0,
        "Nginx hits": 2651,
        "Nginx avg bytes": 179834.88042248,
        "Nginx avg time": 4.0732470765749,
        "Nginx P50 time": 3.978,
        "Nginx P90 time": 5.922,
        "Nginx P95 time": 6.609,
        "Nginx P99 time": 7.927,
        "Nginx 200": 2261,
        "Nginx 499": 190,
        "Nginx 404": 200,
        "canonical": 0
    }
}

siege 3.1.4

reedy@oss-perf:~/oss-performance$ hhvm perf.php --mediawiki --hhvm=`which hhvm` --no-proxygen --i-am-not-benchmarking --skip-version-checks

**************************************************
** 2016-01-11 14:53:48 UTC
** Configuration: MediaWikiTarget on /usr/bin/hhvm

**************************
** 2016-01-11 14:53:48 UTC
** Installing framework

**************************
** 2016-01-11 14:53:56 UTC
** There is no setUpTest

**************************
** 2016-01-11 14:53:56 UTC
** Starting Nginx
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)

**************************
** 2016-01-11 14:53:56 UTC
** Starting PHP Engine
running hphp...
all files saved in /tmp/hhvm-nginxBstzPr ...
running hphp took 1'02" (62088676 us) wall time
mapping self...
mapping self took 0'00" (2477 us) wall time
loading static content...
loaded file cache from /tmp/hhvm-nginxBstzPr/static.content
loading static content took 0'00" (8268 us) wall time

**************************
** 2016-01-11 14:55:01 UTC
** Running sanity check

*******************************************
** 2016-01-11 14:55:03 UTC
** Starting Siege for single request warmup
** SIEGE 3.1.4
** Preparing 1 concurrent users for battle.
The server is now under siege...
done.

Transactions:                300 hits
Availability:             100.00 %
Elapsed time:              25.00 secs
Data transferred:          53.96 MB
Response time:              0.08 secs
Transaction rate:          12.00 trans/sec
Throughput:             2.16 MB/sec
Concurrency:                1.00
Successful transactions:         285
Failed transactions:               0
Longest transaction:            6.55
Shortest transaction:           0.01


******************************************
** 2016-01-11 14:55:28 UTC
** Starting Siege for multi request warmup
** SIEGE 3.1.4
** Preparing 200 concurrent users for battle.
The server is now under siege...

Lifting the server siege..      done.

Transactions:               2423 hits
Availability:             100.00 %
Elapsed time:              60.05 secs
Data transferred:         450.61 MB
Response time:              4.78 secs
Transaction rate:          40.35 trans/sec
Throughput:             7.50 MB/sec
Concurrency:              193.05
Successful transactions:        2224
Failed transactions:               0
Longest transaction:           13.00
Shortest transaction:           0.32


****************************
** 2016-01-11 14:56:29 UTC
** Clearing nginx access.log

*******************************
** 2016-01-11 14:56:29 UTC
** Starting Siege for benchmark
** SIEGE 3.1.4
** Preparing 200 concurrent users for battle.
The server is now under siege...

Lifting the server siege..      done.

Transactions:               2351 hits
Availability:             100.00 %
Elapsed time:              59.14 secs
Data transferred:         448.55 MB
Response time:              4.85 secs
Transaction rate:          39.75 trans/sec
Throughput:             7.58 MB/sec
Concurrency:              192.85
Successful transactions:        2151
Failed transactions:               0
Longest transaction:           14.63
Shortest transaction:           2.10


**************************
** 2016-01-11 14:57:30 UTC
** Collecting results

**************************
** 2016-01-11 14:57:33 UTC
** Collecting TC/PCRE data

***************************
** 2016-01-11 14:57:33 UTC
** There is no tearDownTest
{
    "Combined": {
        "Siege requests": 2351,
        "Siege wall sec": 4.85,
        "Siege RPS": 39.75,
        "Siege successful requests": 2151,
        "Siege failed requests": 0,
        "Nginx hits": 2551,
        "Nginx avg bytes": 184646.40846727,
        "Nginx avg time": 4.1381387691102,
        "Nginx P50 time": 4.062,
        "Nginx P90 time": 5.578,
        "Nginx P95 time": 6.198,
        "Nginx P99 time": 7.993,
        "Nginx 200": 2153,
        "Nginx 499": 198,
        "Nginx 404": 200,
        "canonical": 0
    }
}

@reedy reedy mentioned this issue Jan 11, 2016
Merged
@reedy
Copy link
Contributor

reedy commented Jan 11, 2016

Result 2.70 3.1.4
Siege requests 2451 2351
Siege wall sec 4.68 4.85
Siege RPS 40.95 39.75
Siege successful requests 2251 2151
Siege failed requests 0 0
Nginx hits 2651 2551
Nginx avg bytes 179834.88042248 184646.40846727
Nginx avg time 4.0732470765749 4.1381387691102
Nginx P50 time 3.978 4.062
Nginx P90 time 5.922 5.578
Nginx P95 time 6.609 6.198
Nginx P99 time 7.927 7.993
Nginx 200 2261 2153
Nginx 499 190 198
Nginx 404 200 200
canonical 0 0

@fredemmott
Copy link
Contributor Author

It looks promising, but I'll need to check all of the supported frameworks

@mofarrell
Copy link
Contributor

The update to the blacklist is in master already.

@reedy
Copy link
Contributor

reedy commented Feb 12, 2017

Um... Are you sure? https://github.com/hhvm/oss-performance/blame/master/base/Siege.php#L31

The patch was to blacklist 3.0 versions, not just 3 versions

Isn't this still going to complain for 3.1, unless you pass --skip-version-checks ?

@reedy
Copy link
Contributor

reedy commented Feb 12, 2017

See also JoeDog/siege#15 (comment)

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@reedy @fredemmott @mofarrell @centminmod