New data APIs 11: port all range-only views (plots, logs…) #5992
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
teh-cmc
added
📺 re_viewer
This was referenced Apr 16, 2024
teh-cmc
force-pushed
the
cmc/data_apis_10_cache_stats
branch
from
9da77dc to
e8fe7e9
Compare
teh-cmc
force-pushed
the
cmc/data_apis_11_full_views
branch
from
ad387f5 to
909038b
Compare
This was referenced Apr 18, 2024
teh-cmc
force-pushed
the
cmc/data_apis_10_cache_stats
branch
from
e8fe7e9 to
1aac73b
Compare
teh-cmc
force-pushed
the
cmc/data_apis_11_full_views
branch
from
909038b to
b7f669d
Compare
teh-cmc
force-pushed
the
cmc/data_apis_10_cache_stats
branch
from
1aac73b to
27ce535
Compare
teh-cmc
force-pushed
the
cmc/data_apis_11_full_views
branch
2 times, most recently
from
eff4db7 to
6291c69
Compare
emilk
approved these changes
Apr 23, 2024
It's much more than resolving promises, the view is now in charge of:
As I've mentioned before, I'm fairly confident that we're going to find out that the extra flexibility outweighs the extra verbosity. That being said, what makes it very hard to abstract some of the boilerplate away is the deserialization cache in particular (because of lifetimes, and mutex guards, and typed values, etc). |
|
Managed to squeeze a tiny bit more abstraction for the spatial views (next PR), and now it's basically 1:1. I'm pleasantly surprised. |
teh-cmc
force-pushed
the
cmc/data_apis_10_cache_stats
branch
from
b02426b to
c8150d3
Compare
teh-cmc
force-pushed
the
cmc/data_apis_11_full_views
branch
from
6291c69 to
b35a330
Compare
teh-cmc
added a commit
that referenced
this pull request
Apr 26, 2024
Static-aware, key-less, component-based, cached range APIs.
```rust
let caches = re_query_cache2::Caches::new(&store);
// First, get the raw results for this query.
//
// They might or might not already be cached. We won't know for sure until we try to access
// each individual component's data below.
let results: CachedRangeResults = caches.range(
&store,
&query,
&entity_path.into(),
MyPoints::all_components().iter().copied(), // no generics!
);
// Then, grab the results for each individual components.
// * `get_required` returns an error if the component batch is missing
// * `get_or_empty` returns an empty set of results if the component if missing
// * `get` returns an option
//
// At this point we still don't know whether they are cached or not. That's the next step.
let all_points: &CachedRangeComponentResults = results.get_required(MyPoint::name())?;
let all_colors: &CachedRangeComponentResults = results.get_or_empty(MyColor::name());
let all_labels: &CachedRangeComponentResults = results.get_or_empty(MyLabel::name());
// Then comes the time to resolve/convert and deserialize the data.
// These steps have to be done together for efficiency reasons.
//
// That's when caching comes into play.
// If the data has already been accessed in the past, then this will just grab the
// pre-deserialized, pre-resolved/pre-converted result from the cache.
// Otherwise, this will trigger a deserialization and cache the result for next time.
let all_points = all_points.to_dense::<MyPoint>(&resolver);
let all_colors = all_colors.to_dense::<MyColor>(&resolver);
let all_labels = all_labels.to_dense::<MyLabel>(&resolver);
// The cache might not have been able to resolve and deserialize the entire dataset across all
// available timestamps.
//
// We can use the following APIs to check the status of the front and back sides of the data range.
//
// E.g. it is possible that the front-side of the range is still waiting for pending data while
// the back-side has been fully loaded.
assert!(matches!(
all_points.status(),
(PromiseResult::Ready(()), PromiseResult::Ready(()))
));
// Zip the results together using a stateful time-based join.
let all_frames = range_zip_1x2(
all_points.range_indexed(),
all_colors.range_indexed(),
all_labels.range_indexed(),
);
// Then comes the time to resolve/convert and deserialize the data, _for each timestamp_.
// These steps have to be done together for efficiency reasons.
//
// Both the resolution and deserialization steps might fail, which is why this returns a `Result<Result<T>>`.
// Use `PromiseResult::flatten` to simplify it down to a single result.
eprintln!("results:");
for ((data_time, row_id), points, colors, labels) in all_frames {
let colors = colors.unwrap_or(&[]);
let color_default_fn = || {
static DEFAULT: MyColor = MyColor(0xFF00FFFF);
&DEFAULT
};
let labels = labels.unwrap_or(&[]).iter().cloned().map(Some);
let label_default_fn = || None;
// With the data now fully resolved/converted and deserialized, the joining logic can be
// applied.
//
// In most cases this will be either a clamped zip, or no joining at all.
let results = clamped_zip_1x2(points, colors, color_default_fn, labels, label_default_fn)
.collect_vec();
eprintln!("{data_time:?} @ {row_id}:\n {results:?}");
}
```
---
Part of a PR series to completely revamp the data APIs in preparation
for the removal of instance keys and the introduction of promises:
- #5573
- #5574
- #5581
- #5605
- #5606
- #5633
- #5673
- #5679
- #5687
- #5755
- #5990
- #5992
- #5993
- #5994
- #6035
- #6036
- #6037
Builds on top of the static data PR series:
- #5534
teh-cmc
force-pushed
the
cmc/data_apis_10_cache_stats
branch
from
76c92ea to
4c5f37a
Compare
teh-cmc
added a commit
that referenced
this pull request
Apr 26, 2024
Title. The new cache being natively component-based makes things much smoothier than before. --- Part of a PR series to completely revamp the data APIs in preparation for the removal of instance keys and the introduction of promises: - #5573 - #5574 - #5581 - #5605 - #5606 - #5633 - #5673 - #5679 - #5687 - #5755 - #5990 - #5992 - #5993 - #5994 - #6035 - #6036 - #6037 Builds on top of the static data PR series: - #5534
teh-cmc
force-pushed
the
cmc/data_apis_11_full_views
branch
from
b35a330 to
ddb0441
Compare
teh-cmc
added a commit
that referenced
this pull request
Apr 26, 2024
Migrate all spatial views that were using the old cache APIs to the new ones. Instance keys are not queried at all anymore. All views are now range-aware by default. Also took the opportunity to somewhat streamline everything. The 10min air-traffic example with full visible range is about 2-2.5x faster than before. I'm sure I broke a few things here and there, I'll run a full check suite once everything's said and done. --- Part of a PR series to completely revamp the data APIs in preparation for the removal of instance keys and the introduction of promises: - #5573 - #5574 - #5581 - #5605 - #5606 - #5633 - #5673 - #5679 - #5687 - #5755 - #5990 - #5992 - #5993 - #5994 - #6035 - #6036 - #6037 Builds on top of the static data PR series: - #5534
teh-cmc
added a commit
that referenced
this pull request
Apr 26, 2024
`re_query_cache` is gone, `re_query_cache2` takes its place -- simple as that. --- Part of a PR series to completely revamp the data APIs in preparation for the removal of instance keys and the introduction of promises: - #5573 - #5574 - #5581 - #5605 - #5606 - #5633 - #5673 - #5679 - #5687 - #5755 - #5990 - #5992 - #5993 - #5994 - #6035 - #6036 - #6037 Builds on top of the static data PR series: - #5534
teh-cmc
added a commit
that referenced
this pull request
Apr 26, 2024
Migrate every little thing that didn't use to go through the cached APIs. `Image` and `Mesh3D` are temporarily cached even though they shouldn't be, that's taken care of in a follow-up PR. Once again, I probably broke a million edge cases -- I want to get as fast as possible to removing instance keys before doing an in-depth quality pass. --- Part of a PR series to completely revamp the data APIs in preparation for the removal of instance keys and the introduction of promises: - #5573 - #5574 - #5581 - #5605 - #5606 - #5633 - #5673 - #5679 - #5687 - #5755 - #5990 - #5992 - #5993 - #5994 - #6035 - #6036 - #6037 Builds on top of the static data PR series: - #5534
teh-cmc
added a commit
that referenced
this pull request
Apr 26, 2024
There is now only one way to query data: `re_query` (well you can still query the datastore directly if you're a monster, but that's for another PR). All queries go through both the query cache and the deserialization cache. There will be a follow-up PR to disable the deserialization cache for specific components. Most of this is just (re)moving stuff around except for the last two commits which take care of porting the cached test suites since they cannot depend on uncached APIs to do comparisons anymore. - Closes #6018 - Closes #3320 --- Part of a PR series to completely revamp the data APIs in preparation for the removal of instance keys and the introduction of promises: - #5573 - #5574 - #5581 - #5605 - #5606 - #5633 - #5673 - #5679 - #5687 - #5755 - #5990 - #5992 - #5993 - #5994 - #6035 - #6036 - #6037 Builds on top of the static data PR series: - #5534
teh-cmc
added a commit
that referenced
this pull request
Apr 26, 2024
Make it possible to not cache some components, all while pretending really hard that they've been cached. - Related: #5974 --- Part of a PR series to completely revamp the data APIs in preparation for the removal of instance keys and the introduction of promises: - #5573 - #5574 - #5581 - #5605 - #5606 - #5633 - #5673 - #5679 - #5687 - #5755 - #5990 - #5992 - #5993 - #5994 - #6035 - #6036 - #6037 Builds on top of the static data PR series: - #5534
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Text logs, line plots and scatter plots.
A bit faster than
main, with a bit less memory overhead.Part of a PR series to completely revamp the data APIs in preparation for the removal of instance keys and the introduction of promises:
ClampedZipiterator machinery #5573Component, once and for all #5605RangeZipiterator machinery #5679Builds on top of the static data PR series:
TimeInt#5534Checklist
mainbuild: app.rerun.ionightlybuild: app.rerun.io