Parallelize data ingestion

[teh-cmc]

(Random thoughts as I'm struggling with slow ingestion times)

Speaking of multi-threading: further down the line the bottleneck for real-time use cases is probably gonna be the speed of ingestion when using connect(), which is currently bounded by the time it takes to render the UI since they both run serially on the same thread (right?).

There are two main fronts that I can think of to improve things, though they are both far more challenging than parallelizing space-views:

1. Run insertion and UI rendering in parallel.
   This requires a consistent view of the datastore for the entire duration of a frame, so results are consistent across views.
   The standard approach would be to implement some kind of snapshotting/MVCC for the store and its views (which would actually be possible thanks to our RowIds), but thankfully unnecessary in our case if we follow our plan of handling all query work in one centralized place (just lock everything, get the data out, unlock, then render UI as usual).
   This asynchronicity between the store and the UI rendering is something we need to move towards to in any case in order to ultimately get the database out in its own process (the "hub").

2. Parallelize insertions into the store.
   This should not be too hard as long as we partition on (EntityPath, Timeline) so that it matches the natural indexing of the store (though Im sure we're going to discover some nasty race conditions on the way).
   Similarly, most of our builtin store-subscribers should be capable of updating in parallel when following this partition scheme.

On the bright side, this also means that the upcoming work to multi-thread the UI rendering should actually improve ingestion speeds too by side-effect:

• Parallelize space views #1325

[teh-cmc]

• Blocked on Store data in dense chunks #6066

[teh-cmc]

We really need to move ingestion off of the UI thread -- this really makes using Rerun hard on some OSes / window managers (including mine).

Reminder: somehow all of this needs to work on the web too.

[emilk]

We really need to move ingestion off of the UI thread -- this really makes using Rerun hard on some OSes / window managers (including mine).

An alternative (at least in the short term) is to ensure data ingestion still happens when Rerun is hidden:

• Data ingestion paused when viewer is invisible on Wayland and Web #7427

This may be an easier fix, though I agree proper parallel data ingestion is more desirable.

[teh-cmc]

Not blocked anymore: we have the chunks, and we have the storage handles. That doesn't mean it'll be easy though, far from it.