Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Benchmarks #336

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

feat: Benchmarks #336

wants to merge 8 commits into from

Conversation

ianpurvis
Copy link
Collaborator

Adds support for building and running benchmarks.

Bullet 2.82 benchmarks can be run with:

$ node builds/benchmarks.js
$ node builds/benchmarks.wasm.js

A new benchmark tool bammo can be run with:

$ node builds/bammo.js
$ node builds/bammo.wasm.js

Non-emscripten versions of each tool can also be built from inside the bammo folder:

$ cd bammo
$ cmake -B native
$ cmake --build native
$ ./native/benchmarks
$ ./native/bammo

@ianpurvis
Copy link
Collaborator Author

@kripken This was a quite a journey... per your recommendation in #322, I started working with the existing Bullet 2.82 benchmarks. They were not hard to get running in ammo master, but I couldn't build them against newer Bullet. It turns out that cli benchmarking has been very broken in Bullet since v2.83, when the gui example browser was introduced.

This PR is the result of fixing headless examples in Bullet 2.89, adding a new google benchmark cli, and then porting everything back to Bullet 2.82 and ammo master. My thought is that once this is merged, the remaining fixes for Bullet >= 2.83 benchmarking can be added to the chore/bullet-2.89 branch. Then we can benchmark each branche with the new cli and compare with the google benchmark compare.py tool.

I am calling this new cli bammo, short for benchmark ammo. I took care to match the bullet 2.82 benchmark functionality as closely as possible. It runs each benchmark for 200 frames, sampling physics step duration every 25 frames. I guess there is more useful data to be had, but hopefully this provides some continuity to the old benchmarks. One nice addition is FPS....

$ node builds/bammo.wasm.js --benchmark_counters_tabular=true
failed to open /proc/cpuinfo
2020-09-16T21:18:03-05:00
Running /Users/ian/Work/ianpurvis/ammo.js/builds/bammo.wasm.js
Run on (-1 X 1000.07 MHz CPU )
-------------------------------------------------------------------------------------------------------
Benchmark                            Time             CPU   Iterations        fps     frames      start
-------------------------------------------------------------------------------------------------------
3000 Boxes/iterations:1           5.46 ms         5.44 ms            1  183.715/s          1          0
3000 Boxes/iterations:1           3.93 ms         3.91 ms            1  255.432/s          1         25
3000 Boxes/iterations:1           5.02 ms         5.02 ms            1  199.315/s          1         50
3000 Boxes/iterations:1           5.24 ms         5.23 ms            1  191.358/s          1         75
3000 Boxes/iterations:1           7.87 ms         7.87 ms            1  127.125/s          1        100
3000 Boxes/iterations:1           12.1 ms         12.1 ms            1  82.4542/s          1        125
3000 Boxes/iterations:1           14.9 ms         14.9 ms            1  67.1437/s          1        150
3000 Boxes/iterations:1           19.5 ms         19.5 ms            1   51.393/s          1        175
3000 Boxes                        2003 ms         2003 ms            1  99.8686/s        200          0
1000 Stack/iterations:1           8.19 ms         8.19 ms            1  122.173/s          1          0
1000 Stack/iterations:1           13.0 ms         13.0 ms            1  77.2172/s          1         25
1000 Stack/iterations:1           14.3 ms         14.3 ms            1  69.7976/s          1         50
1000 Stack/iterations:1           14.5 ms         14.5 ms            1  69.1041/s          1         75
1000 Stack/iterations:1           14.1 ms         14.1 ms            1  70.7752/s          1        100
1000 Stack/iterations:1           14.0 ms         14.0 ms            1  71.4107/s          1        125
1000 Stack/iterations:1           14.0 ms         14.0 ms            1  71.6023/s          1        150
1000 Stack/iterations:1           14.0 ms         14.0 ms            1  71.5597/s          1        175
1000 Stack                        2754 ms         2754 ms            1  72.6148/s        200          0
Ragdolls/iterations:1             6.67 ms         6.65 ms            1  150.481/s          1          0
Ragdolls/iterations:1             5.47 ms         5.48 ms            1  182.642/s          1         25
Ragdolls/iterations:1             5.49 ms         5.48 ms            1  182.502/s          1         50
Ragdolls/iterations:1             5.76 ms         5.75 ms            1  173.955/s          1         75
Ragdolls/iterations:1             6.23 ms         6.22 ms            1  160.794/s          1        100
Ragdolls/iterations:1             6.66 ms         6.65 ms            1  150.375/s          1        125
Ragdolls/iterations:1             7.10 ms         7.09 ms            1  141.044/s          1        150
Ragdolls/iterations:1             7.98 ms         7.97 ms            1  125.464/s          1        175
Ragdolls                          1426 ms         1426 ms            1  140.292/s        200          0
Convex Stack/iterations:1         1.32 ms         1.31 ms            1  760.579/s          1          0
Convex Stack/iterations:1         1.12 ms         1.12 ms            1  896.561/s          1         25
Convex Stack/iterations:1         4.25 ms         4.25 ms            1  235.466/s          1         50
Convex Stack/iterations:1         11.7 ms         11.7 ms            1  85.5411/s          1         75
Convex Stack/iterations:1         17.5 ms         17.4 ms            1  57.3236/s          1        100
Convex Stack/iterations:1         18.0 ms         18.0 ms            1  55.6565/s          1        125
Convex Stack/iterations:1         18.1 ms         18.1 ms            1  55.3589/s          1        150
Convex Stack/iterations:1         20.7 ms         20.7 ms            1  48.2936/s          1        175
Convex Stack                      2630 ms         2630 ms            1  76.0345/s        200          0
Prim vs Mesh/iterations:1         1.87 ms         1.86 ms            1  537.228/s          1          0
Prim vs Mesh/iterations:1         1.37 ms         1.36 ms            1  734.915/s          1         25
Prim vs Mesh/iterations:1         1.24 ms         1.23 ms            1  810.056/s          1         50
Prim vs Mesh/iterations:1         1.27 ms         1.26 ms            1  795.866/s          1         75
Prim vs Mesh/iterations:1         1.44 ms         1.43 ms            1  701.269/s          1        100
Prim vs Mesh/iterations:1         1.84 ms         1.83 ms            1  547.825/s          1        125
Prim vs Mesh/iterations:1         3.00 ms         3.00 ms            1  333.549/s          1        150
Prim vs Mesh/iterations:1         4.05 ms         4.05 ms            1  247.141/s          1        175
Prim vs Mesh                       435 ms          435 ms            2  459.314/s        200          0
Convex vs Mesh/iterations:1       2.78 ms         2.76 ms            1   362.47/s          1          0
Convex vs Mesh/iterations:1       1.12 ms         1.12 ms            1  896.032/s          1         25
Convex vs Mesh/iterations:1       1.12 ms         1.11 ms            1  897.461/s          1         50
Convex vs Mesh/iterations:1       1.21 ms         1.19 ms            1  841.296/s          1         75
Convex vs Mesh/iterations:1       2.32 ms         2.31 ms            1   432.05/s          1        100
Convex vs Mesh/iterations:1       2.36 ms         2.35 ms            1  426.054/s          1        125
Convex vs Mesh/iterations:1       4.12 ms         4.12 ms            1  242.863/s          1        150
Convex vs Mesh/iterations:1       8.92 ms         8.91 ms            1  112.195/s          1        175
Convex vs Mesh                     581 ms          581 ms            1  344.088/s        200          0
Raycast/iterations:1              5.18 ms         5.17 ms            1  193.512/s          1          0
Raycast/iterations:1              3.87 ms         3.86 ms            1  258.979/s          1         25
Raycast/iterations:1              4.26 ms         4.25 ms            1  235.051/s          1         50
25500 rays in 147 ms 147 147 147.000000
Raycast/iterations:1              4.50 ms         4.49 ms            1  222.725/s          1         75
25500 rays in 140 ms 140 140 140.000000
Raycast/iterations:1              4.56 ms         4.55 ms            1  219.819/s          1        100
25500 rays in 151 ms 151 151 151.000000
25500 rays in 172 ms 151 172 161.500000
Raycast/iterations:1              5.80 ms         5.84 ms            1  171.305/s          1        125
25500 rays in 148 ms 148 148 148.000000
25500 rays in 173 ms 148 173 160.500000
Raycast/iterations:1              8.84 ms         8.83 ms            1   113.23/s          1        150
25500 rays in 156 ms 156 156 156.000000
25500 rays in 175 ms 156 175 165.500000
25500 rays in 175 ms 156 175 168.666672
Raycast/iterations:1              10.2 ms         10.2 ms            1  97.6976/s          1        175
25500 rays in 156 ms 156 156 156.000000
25500 rays in 179 ms 156 179 167.500000
25500 rays in 176 ms 156 179 170.333328
Raycast                           1313 ms         1313 ms            1  152.364/s        200          0
blinky:ammo.js ian$

@kripken
Copy link
Owner

kripken commented Sep 21, 2020

Wow, very impressive!

What are the build instructions for builds/bammo.js? (Are they not from the bammo/ subdir that the non-emscripten version uses?)

@ianpurvis
Copy link
Collaborator Author

I tried to keep emscripten targets in the top level CMakeLists.txt (not sure if that's the best approach, tho). So builds/bammo.js and builds/bammo.wasm.js are automatically built when you build all. Just make sure to bump up TOTAL_MEMORY for the raycast benchmark:

$ cmake -B builds -D CLOSURE=1 -D TOTAL_MEMORY=134217728
$ cmake --build builds

I think I will submit the native code to upstream bullet. Maybe if Erwin accepts it we can revise this PR to provide that stuff via the bullet subtree.

@ianpurvis
Copy link
Collaborator Author

Upstream PR here: bulletphysics/bullet3#3074

@kripken
Copy link
Owner

kripken commented Sep 24, 2020

Just make sure to bump up TOTAL_MEMORY for the raycast benchmark:

I think the bammo target in CMakeLists sets the size of memory, so why is this needed?

@ianpurvis
Copy link
Collaborator Author

I think the bammo target in CMakeLists sets the size of memory, so why is this needed?

Ah- no good reason, just avoiding the change to the CMakeLists... shall we up it?

@kripken
Copy link
Owner

kripken commented Sep 30, 2020

Hmm, I guess I don't feel strongly either way. As you prefer.

@Galadirith Galadirith mentioned this pull request Oct 11, 2020
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kripken kripken kripken approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@ianpurvis @kripken