Garbage collection should be aware of app_id/recording_id semantics

[teh-cmc]

We've seen plenty of reports of users that start a Rerun instance and then run their algorithm a bunch of times as they go through their iterative improvement cycle.

It ends up looking like a little this:

Now obviously at some point these users run out of memory, at which point they learn about --memory-limit.
That's all fine except that garbage collection is completely unaware of these app_id/recording_id semantics, and so will only purge the currently active datastore (which likely contains the only data that the user still cares about at this point) while the old recordings are left untouched forever.

[emilk]

Related:

• Show all loaded recordings as tabs #1905

[emilk]

There are many ways to solve this:

• GC the oldest recording first
• GC all at the same rate
• …

Different use cases have different requirements. What is obvious that the current behavior sucks.

I suggest we just GC every open recording, OR drop the oldest recording, whatever is easier.

[nikolausWest]

Related ?

• datastore: bake latest-at semantics into the garbage collector #1803

[teh-cmc]

As far as I'm aware we now evenly distribute GC pass across all recordings:

pub fn purge_fraction_of_ram(&mut self, fraction_to_purge: f32) {
    re_tracing::profile_function!();

    for store_db in self.store_dbs.values_mut() {
        store_db.purge_fraction_of_ram(fraction_to_purge);
    }
}

which is overall an improvement, but there probably should be a blueprint setting to configure whether you want to distribute evenly, or prioritize cleaning up previous recordings of the same app_id first.

[nikolausWest]

I think we should make the default behavior be to drop old data in a way that if you are running serial experiments, old recordings get dropped first, and if you are doing parallel experiments, we drop data from all recordings evenly. That is, we should drop based on time.

[nikolausWest]

As it stands, I think this is a blocker for the hugging face spaces demo

[emilk]

A good starting strategy: only run gc on oldest recording. When it is empty, close it.

Werinkle: row-protection