Static data 1: static-aware datastore, caches and queries #5535

teh-cmc · 2024-03-15T15:40:19Z

Introduces the concept of static data into the data APIs.

Static data is a on a per-entity per-component basis. If it exists, it unconditionally shadows any temporal data of the same type. It is never garbage collected.
When static data is returned, it is indicated via TimeInt::STATIC.

The terminology has been normalized all over the place: data is either static or temporal, and nothing else.

Static data cannot have more than one cell per-entity per-component. Trying to write more than one cells will trigger last-write-wins semantics, as defined by RowId ordering.

Timeless fallbacks just don't exist anymore, which simplifies out a lot of code in the datastore and query cache.

Note: static data is in many subtle ways incompatible with our legacy InstanceKey-based model, which results in a couple hacks in this PR. Those hacks will be gone as soon as the new data APIs land and instance keys go away.

Part of a PR series that removes the concept of timeless data in favor of the much simpler concept of static data:

Checklist

I have read and agree to Contributor Guide and the Code of Conduct
I've included a screenshot or gif (if applicable)
I have tested the web demo (if applicable):
- Using newly built examples: app.rerun.io
- Using examples from latest main build: app.rerun.io
- Using full set of examples from nightly build: app.rerun.io
The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG
If applicable, add a new check to the release checklist!

teh-cmc · 2024-03-18T09:45:34Z

crates/re_data_store/src/store.rs

-    /// If enabled, the store will throw an error if and when it notices that a single component
-    /// type maps to more than one arrow datatype.
-    ///
-    /// Enabled by default in debug builds.
-    pub enable_typecheck: bool,


Removing in anticipation of promises.

crates/re_data_store/src/store.rs

emilk · 2024-03-21T17:28:52Z

crates/re_data_store/src/store_helpers.rs

@@ -6,6 +6,8 @@ use crate::{DataStore, LatestAtQuery};

 // --- Read ---

+// TODO(cmc): these helpers have got to go once the new APIs land.


maybe add a issue number to this TODO

there's already a PR open for it!

crates/re_data_store/src/store_read.rs

crates/re_query_cache/src/cache.rs

crates/re_entity_db/src/entity_db.rs

crates/re_space_view_text_log/src/visualizer_system.rs

_Commits make no sense, review the final changelog directly._ _All the interesting bits happen in `re_log_types/time_point` & `re_sdk` -- everything else is just change propagation._ - `TimeInt` now ranges from `i64:MIN + 1` to `i64::MAX`. - `TimeInt::STATIC`, which takes the place of the now illegal `TimeInt(i64::MIN)`, is now _the only way_ of identifying static data. - It is impossible to create `TimeInt::STATIC` inadvertently -- users of the SDK cannot set the clock to that value. - Similarly, it is impossible to create a `TimeRange`, a `TimePoint`, a `LatestAtQuery` or a `RangeQuery` that includes `TimeInt::STATIC`. If static data exists, that's what will be returned, unconditionally -- there's no such thing as querying for it explicitely. - `TimePoint::timeless` is gone -- we already have `TimePoint::default` that we use all over the place, we don't need two ways of doing the same thing. There still exists a logical mapping between an empty `TimePoint` and static data, as that is how one represents static data on the wire -- terminology wise: "a timeless timepoint results in static data". Similar to the "ensure `RowId`s are unique" refactor from back when, this seemingly tiny change on the surface will vastly simplify downstream code that finally has some invariants to rely on. - Fixes #4832 - Related to #5264 --- Part of a PR series that removes the concept of timeless data in favor of the much simpler concept of static data: - #5534 - #5535 - #5536 - #5537 - #5540

Just exposing all the new static stuff to the Python SDK, and trying to kill the "timeless" terminology in the process. --- Part of a PR series that removes the concept of timeless data in favor of the much simpler concept of static data: - #5534 - #5535 - #5536 - #5537 - #5540

Just exposing all the new static stuff to the C & C++ SDKs, and trying to kill the "timeless" terminology in the process. --- Part of a PR series that removes the concept of timeless data in favor of the much simpler concept of static data: - #5534 - #5535 - #5536 - #5537 - #5540

Just exposing all the new static stuff to the Rust SDK, and trying to kill the "timeless" terminology in the process. --- Part of a PR series that removes the concept of timeless data in favor of the much simpler concept of static data: - #5534 - #5535 - #5536 - #5537 - #5540

jleibs · 2024-06-08T00:17:04Z

crates/re_entity_db/tests/clear.rs

-        // EntityTree should be empty again when we end since everything was GC'd
-        assert_eq!(db.tree().num_children_and_fields(), 0);
-    }


@teh-cmc why was this test removed -- we were depending on this functionality.

Don't quite remember -- probably a mistake, a lot of tests became redundant with each other as the model got vastly simplified

teh-cmc added ⛃ re_datastore affects the datastore itself 🔍 re_query affects re_query itself do-not-merge Do not merge this PR include in changelog 🔩 data model labels Mar 15, 2024

teh-cmc marked this pull request as draft March 15, 2024 15:40

This was referenced Mar 15, 2024

Static data 4: static-aware Rust SDK #5540

Merged

Static data 0: revamped TimeInt #5534

Merged

Static data 2: static-aware Python SDK #5536

Merged

Static data 3: static-aware C & C++ SDKs #5537

Merged

teh-cmc force-pushed the cmc/static_0_timeint_shenanigans branch from 8fd761d to 2bd5760 Compare March 15, 2024 16:41

teh-cmc force-pushed the cmc/static_1_store branch from 6f14a59 to e23c89a Compare March 15, 2024 16:52

teh-cmc force-pushed the cmc/static_0_timeint_shenanigans branch from 2bd5760 to f338850 Compare March 18, 2024 09:39

teh-cmc force-pushed the cmc/static_1_store branch from 821d2b5 to 42c201b Compare March 18, 2024 09:42

teh-cmc commented Mar 18, 2024

View reviewed changes

teh-cmc force-pushed the cmc/static_1_store branch from 42c201b to 5b699f2 Compare March 18, 2024 10:05

teh-cmc marked this pull request as ready for review March 18, 2024 10:08

teh-cmc force-pushed the cmc/static_0_timeint_shenanigans branch from 30eed52 to ca6dfca Compare March 19, 2024 16:18

teh-cmc force-pushed the cmc/static_1_store branch from 0f896b9 to 5af06be Compare March 19, 2024 16:27

emilk self-requested a review March 21, 2024 17:22

emilk approved these changes Mar 21, 2024

View reviewed changes

teh-cmc force-pushed the cmc/static_0_timeint_shenanigans branch from 36f58cd to bd0c3d9 Compare April 5, 2024 14:04

teh-cmc removed the do-not-merge Do not merge this PR label Apr 5, 2024

Base automatically changed from cmc/static_0_timeint_shenanigans to main April 5, 2024 14:20

static-aware datastore

023fccf

teh-cmc added 6 commits April 5, 2024 16:27

static-aware queries

61f1209

propagate changes everywhere

9cd47df

fbs shenaniganeries

fba7f5b

update doc

060b4b2

backported store improvements from the future

e1a4bf3

lints

3a2d207

teh-cmc force-pushed the cmc/static_1_store branch from 5af06be to 3a2d207 Compare April 5, 2024 14:30

teh-cmc added 2 commits April 5, 2024 16:34

post-rebase fixes

c89da2b

review

20d1082

teh-cmc force-pushed the cmc/static_1_store branch from 6dcaa7c to 20d1082 Compare April 5, 2024 15:09

teh-cmc merged commit 08109da into main Apr 5, 2024
10 of 13 checks passed

teh-cmc deleted the cmc/static_1_store branch April 5, 2024 15:09

jleibs reviewed Jun 8, 2024

View reviewed changes

jleibs mentioned this pull request Jun 8, 2024

Removed space views are not GC'd from the EntityTree #6521

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Static data 1: static-aware datastore, caches and queries #5535

Static data 1: static-aware datastore, caches and queries #5535

teh-cmc commented Mar 15, 2024 •

edited by github-actions bot

Loading

teh-cmc Mar 18, 2024

emilk Mar 21, 2024

teh-cmc Apr 5, 2024

jleibs Jun 8, 2024

teh-cmc Jun 10, 2024

		@@ -6,6 +6,8 @@ use crate::{DataStore, LatestAtQuery};

		// --- Read ---

		// TODO(cmc): these helpers have got to go once the new APIs land.

Static data 1: static-aware datastore, caches and queries #5535

Static data 1: static-aware datastore, caches and queries #5535

Conversation

teh-cmc commented Mar 15, 2024 • edited by github-actions bot Loading

Checklist

teh-cmc Mar 18, 2024

Choose a reason for hiding this comment

emilk Mar 21, 2024

Choose a reason for hiding this comment

teh-cmc Apr 5, 2024

Choose a reason for hiding this comment

jleibs Jun 8, 2024

Choose a reason for hiding this comment

teh-cmc Jun 10, 2024

Choose a reason for hiding this comment

teh-cmc commented Mar 15, 2024 •

edited by github-actions bot

Loading