Don't allow mixing timeless and timeful data on a single Entity-Component path #5264
Assignees
Labels
🔩 data model
🪵 Log & send APIs
Affects the user-facing API for all languages
⛃ re_datastore
affects the datastore itself
Comments
jleibs
added
🔩 data model
🪵 Log & send APIs
jleibs
changed the title
jleibs
changed the title
This was referenced Mar 18, 2024
5 tasks
teh-cmc
added a commit
that referenced
this issue
Apr 5, 2024
_Commits make no sense, review the final changelog directly._ _All the interesting bits happen in `re_log_types/time_point` & `re_sdk` -- everything else is just change propagation._ - `TimeInt` now ranges from `i64:MIN + 1` to `i64::MAX`. - `TimeInt::STATIC`, which takes the place of the now illegal `TimeInt(i64::MIN)`, is now _the only way_ of identifying static data. - It is impossible to create `TimeInt::STATIC` inadvertently -- users of the SDK cannot set the clock to that value. - Similarly, it is impossible to create a `TimeRange`, a `TimePoint`, a `LatestAtQuery` or a `RangeQuery` that includes `TimeInt::STATIC`. If static data exists, that's what will be returned, unconditionally -- there's no such thing as querying for it explicitely. - `TimePoint::timeless` is gone -- we already have `TimePoint::default` that we use all over the place, we don't need two ways of doing the same thing. There still exists a logical mapping between an empty `TimePoint` and static data, as that is how one represents static data on the wire -- terminology wise: "a timeless timepoint results in static data". Similar to the "ensure `RowId`s are unique" refactor from back when, this seemingly tiny change on the surface will vastly simplify downstream code that finally has some invariants to rely on. - Fixes #4832 - Related to #5264 --- Part of a PR series that removes the concept of timeless data in favor of the much simpler concept of static data: - #5534 - #5535 - #5536 - #5537 - #5540
teh-cmc
added a commit
that referenced
this issue
Apr 5, 2024
Introduces the concept of static data into the data APIs. Static data is a on a per-entity per-component basis. If it exists, it unconditionally shadows any temporal data of the same type. It is never garbage collected. When static data is returned, it is indicated via `TimeInt::STATIC`. The terminology has been normalized all over the place: data is either static or temporal, and nothing else. Static data cannot have more than one cell per-entity per-component. Trying to write more than one cells will trigger last-write-wins semantics, as defined by `RowId` ordering. Timeless fallbacks just don't exist anymore, which simplifies out _a lot_ of code in the datastore and query cache. Note: static data is in many subtle ways incompatible with our legacy InstanceKey-based model, which results in a couple hacks in this PR. Those hacks will be gone as soon as the new data APIs land and instance keys go away. - Fixes #5264 - Fixes #2074 - Fixes #5447 - Fixes #1766 --- Part of a PR series that removes the concept of timeless data in favor of the much simpler concept of static data: - #5534 - #5535 - #5536 - #5537 - #5540
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Context
Today timeless data acts as a "fallback" for all timelines. Any time we do a latest-at or range-query, we resolve the query results as if there is an event at -Inf on the timeline and include it in the query results as appropriate.
This query-level merging of timeless and timefull data adds significant complexity to the book-keeping and query processing for very little practical gain.
Proposal
A vast simplification for timeless data is that once an (Entity, Component) path is set to timeless, it becomes an override rather than a fallback. This means during any query we can check the timeless map first. If we find timeless data, we return it, and otherwise we can do regular query logic, but without the complexity of a fallback.
Conceptually timeless data becomes a single hashmap mapping:
(Entitypath, ComponentName) => DataCell.We will define it as an error / undefined behavior if a user logs a mixture of timeless and timeful data for the same Entity-Component Path.
Logging timeless data multiple times to the same path still only saves a single copy of the data. We make no attempt to version the timeless data or provide multi-valued results in a range query.
Changes in Behavior for Users
If users are currently using timeless data for it's "fallback" properties by logging timeless data first, and later logging timeful data, they will need to update their code. In most cases something very similar can be accomplished by logging a piece of data to -Inf on an appropriate set of timelines.
Timeless data will no longer show up implicitly as an event on the timeline at -Inf in the same way. We will need to design an alternative representation for this timeless data to accommodate for the lack of an event-marker.
Additional Considerations
Related:
TimeInt::BEGINNINGvs.TimeInt::MINvs.Option<TimeInt>#4832log_tickcannot coexist #2074The text was updated successfully, but these errors were encountered: