High RAM usage with physics

[lawnjelly]

Godot version:
3.2.4 beta 4

OS/device including version:
Linux Mint 20
Intel Core i5-7500T

Issue description:
While running a small physics benchmarking project for developing the new BVH (#44901), I noticed that memory use was very high with just 1000 dynamic objects.

Bullet 711 megs
Godot physics (octree) 554 megs
Godot physics (BVH) 446 megs

In contrast, drawing 10,000 objects in a rendering only benchmark uses only 64 megs (for BVH). This does seem a large discrepancy.

Steps to reproduce:
Create a bunch of rigid bodies in the same place.

Minimal reproduction project:

octree_physics_test.zip

Discussion
The BVH is pretty good on memory, but it seems like something is using a whole bunch of memory for physics.
Could be because the collision shapes are spheres. Or maybe it's the collision pairs. Maybe something for @pouleyKetchoupp to investigate in Godot 4.0.

I have measured the pairing memory use with the BVH. As a result of all 1000 objects being created in the same place initially, they are all initially paired with each other (so 1000 x 999). This comes out at approx 10 megs for BVH, possibly up to 20 megs due to the use of vectors. So while this is a lot, it's nothing like the total memory use, maybe something else within the physics is using a load of memory.

It does suggest a couple of things that could be worth considering though:

• Compaction strategy for pairing vectors. Even though memory use isn't huge, it could be worth compacting the pairing if say the number reduced to 1/8 or 1/16th of the maximum. This may (or may not) result in better cache use, as well as saving a bit of memory. I'm not totally convinced but worth a test.
• Limiting the amount of pairs per object. At first glance this seems useful, if an object is in a crowd with 1000 others, does it really matter whether the physics collides with 1000 or just say 16. This would also make physics much faster. There is a problem though .. if one of those objects that gets ignored is the floor, or a wall, you could get objects falling through the floor.

This is however something that could be looked at in the long term in Godot 4.0. If colliders could be marked as either vital (floor, walls etc) or optional (players, boxes), this could result in much better performance in worst case scenarios.

EDIT : I can now confirm it is to do with the initial overlap. By setting the objects to random positions at startup, the memory use drops to 63 megs. My best guess is this is due to vectors being increased to large sizes at startup then never decreasing.

[NuclearPhoenixx]

Hope this is relevant: Just had a look at your example project in 3.2.3 stable and after the initial collisions the static memory usage drops down from ~460MiB to level of at about 40MiB according to the debug monitor. So at least in this version memory usage goes down again, in contrast to your edit.

[lawnjelly]

Ah that is interesting I'll try this out and try and find out what has changed since 3.2.3. 👍

Well bizarrely, I can't seem to replicate it today. All the memory measurements seem much closer to 60-70 megs, with both debug and release. I have no idea what might have been causing this, perhaps something in the debugger, so I'll put down as a false alarm unless I can replicate when I'm more awake.