Added initial benchmarks #551

ThierryBerger · 2024-07-02T16:02:48Z

Objective

bevy_rapier currently has no benchmarks, making it difficult to be confident with refactoring, or consider performance improvements. This PR aims to implement foundation to address this.

objective 1: take inspiration from bevy_xpbd/avian benchmarks, I think it's useful to aim for comparable benchmarks with other bevy physics crates. That being said, bevy_xpbd has simplistic tests.
objective 2: implement the same benchmarks as rapier, to have a baseline to compare with.
- Using criterion for that is not adapted because the benches are very heavy and tailored for 1 run, to check the update rapier step timings. So I dropped criterion:
  - Divan is used for quick benchmarking, running the same App multiple times, and outputting statistics on update times.
  - A custom bench is used for more granular benchmarks, it's able to compute difference between rapier inner update and total bevy update.

Avian comparison

I doubt the benchmarks are comparable with avian's, as the time management from avian is customized, it might not be running full speed in avian.

bevy_rapier

results as of 6e61318:

Benchmarking cubes 3x3, 30 steps: Warming up for 3.0000 s

Warning: Unable to complete 100 samples in 20.0s. You may wish to increase target time to 22.0s, enable flat sampling, or reduce sample count to 60.
cubes 3x3, 30 steps     time:   [2.8347 ms 2.8435 ms 2.8534 ms]
Found 11 outliers among 100 measurements (11.00%)
  1 (1.00%) low mild
  3 (3.00%) high mild
  7 (7.00%) high severe

cubes 5x5, 30 steps     time:   [8.0240 ms 8.0784 ms 8.1379 ms]
Found 6 outliers among 100 measurements (6.00%)
  5 (5.00%) high mild
  1 (1.00%) high severe

cubes 10x10, 30 steps   time:   [105.89 ms 107.98 ms 110.40 ms]
Found 8 outliers among 100 measurements (8.00%)
  4 (4.00%) high mild
  4 (4.00%) high severe

Avian

results as of avianphysics/avian@8129042

cubes 3x3, 30 steps     time:   [49.782 ms 50.203 ms 50.708 ms]
                        change: [+1638.8% +1656.7% +1676.8%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) high mild
  3 (3.00%) high severe

cubes 5x5, 30 steps     time:   [109.87 ms 110.38 ms 111.03 ms]
                        change: [+1254.3% +1266.3% +1278.4%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 12 outliers among 100 measurements (12.00%)
  7 (7.00%) high mild
  5 (5.00%) high severe

Benchmarking cubes 10x10, 30 steps: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 20.0s. You may wish to increase target time to 39.4s, or reduce sample count to 50.
cubes 10x10, 30 steps   time:   [390.36 ms 391.82 ms 393.67 ms]
                        change: [+254.70% +262.86% +270.08%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 9 outliers among 100 measurements (9.00%)
  3 (3.00%) high mild
  6 (6.00%) high severe

Suspicious bad performance

I looked into pipeline.counters.step_time.time(), originally to compare with the total update time, in order to get an idea of the overhead from bevy/bevy_rapier, but I had surprising results:

on rapier testbed, a rapier step takes 116ms on first iteration then goes steadily towards ~4ms.
on bevy_rapier, I had to tune down the pyramids to 3 (opposed to 40 ⚠️), and still got 160ms 😱 for a rapier step.

Flamegraph

flamegraph.zip

^ above numbers reported with f3a3d79

investigating performance is not the objective of this Pull Request so I created another issue for follow ups: Investigate performance overhead #553

…ypyramid

bevy_rapier3d/Cargo.toml

bevy_rapier3d/benches/cubes.rs

bevy_rapier3d/benches/many_pyramids3.rs

sebcrozet · 2024-07-05T09:11:31Z

Both approaches make sense: using divan for smaller benchmarks to quickly detect performance regression; and the custom bench for larger simulations and per-step analysis (ideally outputting some sort of graph).

Is there any way to initialize the scene, and then have divan benchmark each frame of the bevy world update?

benches_common/src/lib.rs

ThierryBerger · 2024-07-10T09:38:04Z

Is there any way to initialize the scene, and then have divan benchmark each frame of the bevy world update?

I did implement that with 2cda622

sebcrozet · 2024-07-11T08:41:56Z

benches_common/src/lib.rs

Instead of benches_common be its own create, it should just be a subdirectory in the bevy_rapier3d/benches folder.

I followed the same hierarchy of avian : https://github.com/Jondolf/avian/tree/main/crates ; I'd have preferred a way to enable feature on benches_common to support either 2d or 3d, but I didn't try that.

Either way, it's not too much code, and we can probably focus on 3d for now.

That actually allows me to share some logic with the custom benchmark.

I did add this logic in the custom benchmark, and call into this custom benchmark library from the "simpler" benches.

custom_benches/src/main.rs

sebcrozet · 2024-07-11T08:48:57Z

custom_benches/README.md

The cargo bench section should be removed to the last position in this file. The Custom benches section should be a bit more wordy by saying that it runs benchmarks manually, without bench harnesses, to obtain more detailed results on long-running simulations.

The cargo bench section should say something like "For short-lived benchmarks based on statistical analysis, benchmarks can be run through the divan bench harness.`

side note: I imagine we could use the bench approach (with our own custom bench harness) for better control over custom benchmarks, i.e. run specific custom benches, but I'd need to research that a bit, that can probably be a follow up.

ThierryBerger · 2024-07-11T10:03:03Z

custom_benches/src/main.rs

+//! Translated from rapier benchmark.
+//!
+//! <https://github.com/dimforge/rapier/blob/87ada34008f4a1a313ccf8c3040040bab4f10e69/benchmarks3d/many_pyramids3.rs>
+
+use benches_common::default_app;
+use benches_common::wait_app_start;
+use bevy::prelude::*;
+use bevy_rapier3d::dynamics::RigidBody;
+use bevy_rapier3d::geometry::Collider;
+use bevy_rapier3d::math::Vect;
+use bevy_rapier3d::plugin::RapierContext;
+
+pub fn create_pyramid(commands: &mut Commands, offset: Vect, stack_height: usize, rad: f32) {


I'm not a fan of code duplication with bevy_rapier3d/benches ; but I'm not sure how to solve it.

I added shared setup code to the custom benchmark bevy_rapier_benches3d

bevy_rapier_benches3d/Cargo.toml

sebcrozet

Looking good, thanks!

ThierryBerger added 6 commits July 2, 2024 18:02

added initial benchmarks, similar to bevy_xpbd

6e61318

fix ci ?

89186b6

use divan + try a custom bench strategy

35c6630

bigger test for pyramid to have comparable performances to rapier man…

f3a3d79

…ypyramid

Merge branch 'master' into benchmarks

08df777

custom benches

c0be019

ThierryBerger force-pushed the benchmarks branch from 710e899 to c0be019 Compare July 4, 2024 07:59

ThierryBerger commented Jul 4, 2024

View reviewed changes

bevy_rapier3d/Cargo.toml Outdated Show resolved Hide resolved

ThierryBerger mentioned this pull request Jul 4, 2024

Investigate performance overhead #553

Open

ThierryBerger added 4 commits July 4, 2024 11:06

cleanup duplicate test; add a readme for benches

64e2e1e

simpler cargo.toml for custom_benches

52b3a35

licence for benches_common

82aed44

custom bench version

aecf7ea

ThierryBerger mentioned this pull request Jul 4, 2024

Rapier context as a Component #545

Merged

minor fixes

dd19b7e

ThierryBerger force-pushed the benchmarks branch from 7b848ca to 4032208 Compare July 4, 2024 12:27

please cargo deny

775ab1f

ThierryBerger force-pushed the benchmarks branch from 4032208 to 775ab1f Compare July 4, 2024 12:53

ThierryBerger marked this pull request as ready for review July 4, 2024 13:12

ThierryBerger requested a review from sebcrozet July 4, 2024 13:18

sebcrozet reviewed Jul 5, 2024

View reviewed changes

bevy_rapier3d/benches/cubes.rs Show resolved Hide resolved

bevy_rapier3d/benches/many_pyramids3.rs Show resolved Hide resolved

ThierryBerger commented Jul 5, 2024

View reviewed changes

benches_common/src/lib.rs Outdated Show resolved Hide resolved

ThierryBerger added 2 commits July 10, 2024 09:18

added permalinks to benches

4e6b753

divan benchmarks update timings independently

2cda622

ThierryBerger added 2 commits July 10, 2024 11:38

Merge branch 'master' into benchmarks

2e7266f

fix cargo doc

74315eb

ThierryBerger requested a review from sebcrozet July 10, 2024 14:15

sebcrozet requested changes Jul 11, 2024

View reviewed changes

ThierryBerger commented Jul 11, 2024

View reviewed changes

rearrange hierarchy and readme

3bb2b1a

ThierryBerger requested a review from sebcrozet July 12, 2024 09:19

ThierryBerger commented Jul 12, 2024

View reviewed changes

bevy_rapier_benches3d/Cargo.toml Outdated Show resolved Hide resolved

ThierryBerger removed the request for review from sebcrozet July 12, 2024 09:21

fix compilation ; removed some duplicate code

7bf425b

ThierryBerger requested a review from sebcrozet July 12, 2024 12:38

sebcrozet approved these changes Jul 18, 2024

View reviewed changes

ThierryBerger merged commit 188dcc2 into dimforge:master Jul 18, 2024

ThierryBerger changed the title ~~added initial benchmarks, similar to bevy_xpbd~~ added initial benchmarks Jul 18, 2024

ThierryBerger changed the title ~~added initial benchmarks~~ Added initial benchmarks Jul 18, 2024

ThierryBerger added a commit to ThierryBerger/bevy_rapier that referenced this pull request Aug 8, 2024

added initial benchmarks, similar to bevy_xpbd (dimforge#551)

3f266fe

ThierryBerger mentioned this pull request Aug 13, 2024

Fix benchmark #577

Open

Uh oh!

Added initial benchmarks #551

Added initial benchmarks #551

Uh oh!

Conversation

ThierryBerger commented Jul 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Objective

Avian comparison

Suspicious bad performance

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sebcrozet commented Jul 5, 2024

Uh oh!

Uh oh!

ThierryBerger commented Jul 10, 2024

Uh oh!

sebcrozet Jul 11, 2024

Choose a reason for hiding this comment

Uh oh!

ThierryBerger Jul 11, 2024

Choose a reason for hiding this comment

Uh oh!

ThierryBerger Jul 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sebcrozet Jul 11, 2024

Choose a reason for hiding this comment

Uh oh!

ThierryBerger Jul 11, 2024

Choose a reason for hiding this comment

Uh oh!

ThierryBerger Jul 11, 2024

Choose a reason for hiding this comment

Uh oh!

ThierryBerger Jul 12, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sebcrozet left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ThierryBerger commented Jul 2, 2024 •

edited

Loading

ThierryBerger Jul 12, 2024 •

edited

Loading