DataStore changelog 1: let DataStores know about their StoreId
#4202
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
teh-cmc
force-pushed
the
cmc/store_changelog_1_self_awareness
branch
from
76422fd to
05a71ef
Compare
teh-cmc
added
⛃ re_datastore
This was referenced Nov 13, 2023
teh-cmc
force-pushed
the
cmc/store_changelog_1_self_awareness
branch
from
05a71ef to
88af0d0
Compare
teh-cmc
force-pushed
the
cmc/store_changelog_1_self_awareness
branch
2 times, most recently
from
b0b1603 to
b00c552
Compare
teh-cmc
force-pushed
the
cmc/store_changelog_1_self_awareness
branch
from
b00c552 to
1f9b9c6
Compare
teh-cmc
added a commit
that referenced
this pull request
Nov 15, 2023
Introduces `StoreEvent`, an event that describes the atomic unit of
change in the Rerun `DataStore`: a row has been added to or removed from
the store.
`StoreEvent`s are fired on both the insertion and garbage collection
paths, enabling listeners to build arbitrary, always up-to-date views &
trigger systems.
```rust
/// The atomic unit of change in the Rerun [`DataStore`].
///
/// A [`StoreEvent`] describes the changes caused by the addition or deletion of a
/// [`re_log_types::DataRow`] in the store.
///
/// Methods that mutate the [`DataStore`], such as [`DataStore::insert_row`] and [`DataStore::gc`],
/// return [`StoreEvent`]s that describe the changes.
///
/// Refer to field-level documentation for more details and check out [`StoreDiff`] for a precise
/// definition of what an event involves.
#[derive(Debug, Clone, PartialEq)]
pub struct StoreEvent {
/// Which [`DataStore`] sent this event?
pub store_id: StoreId,
/// What was the store's generation when it sent that event?
pub store_generation: StoreGeneration,
/// Monotonically increasing ID of the event.
///
/// This is on a per-store basis.
///
/// When handling a [`StoreEvent`], if this is the first time you process this [`StoreId`] and
/// the associated `event_id` is not `1`, it means you registered late and missed some updates.
pub event_id: u64,
/// What actually changed?
///
/// Refer to [`StoreDiff`] for more information.
pub diff: StoreDiff,
}
/// Describes an atomic change in the Rerun [`DataStore`]: a row has been added or deleted.
///
/// From a query model standpoint, the [`DataStore`] _always_ operates one row at a time:
/// - The contents of a row (i.e. its columns) are immutable past insertion, by virtue of
/// [`RowId`]s being unique and non-reusable.
/// - Similarly, garbage collection always removes _all the data_ associated with a row in one go:
/// there cannot be orphaned columns. When a row is gone, all data associated with it is gone too.
///
/// Refer to field-level documentation for more information.
#[derive(Debug, Clone, PartialEq)]
pub struct StoreDiff {
/// Addition or deletion?
///
/// The store's internals are opaque and don't necessarily reflect the query model (e.g. there
/// might be data in the store that cannot by reached by any query).
///
/// A [`StoreDiff`] answers a logical question: "does there exist a query path which can return
/// data from that row?".
pub kind: StoreDiffKind,
/// What's the row's [`RowId`]?
///
/// [`RowId`]s are guaranteed to be unique within a single [`DataStore`].
///
/// Put another way, the same [`RowId`] can only appear twice in a [`StoreDiff`] event:
/// one addition and (optionally) one deletion (in that order!).
pub row_id: RowId,
/// The [`TimePoint`] associated with that row.
///
/// Since insertions and deletions both work on a row-level basis, this is guaranteed to be the
/// same value for both the insertion and deletion events (if any).
pub timepoint: TimePoint,
/// The [`EntityPath`] associated with that row.
///
/// Since insertions and deletions both work on a row-level basis, this is guaranteed to be the
/// same value for both the insertion and deletion events (if any).
pub entity_path: EntityPath,
/// All the [`DataCell`]s associated with that row.
///
/// Since insertions and deletions both work on a row-level basis, this is guaranteed to be the
/// same set of values for both the insertion and deletion events (if any).
pub cells: IntMap<ComponentName, DataCell>,
}
```
---
`DataStore` changelog PR series:
- #4202
- #4203
- #4205
- #4206
- #4208
- #4209
teh-cmc
added a commit
that referenced
this pull request
Nov 15, 2023
Introducing the `StoreView` trait and registration system, allowing
anybody to subscribe to `DataStore` changes, even from external code.
`StoreView`s global scope: by registering a view you subscribe to
changes to _all_ `DataStore`s, including those that are yet to be
created.
This is very powerful as it allows views & triggers implementers to
build cross-recording indices as well as be notified as soon as new
recordings come in and go out.
```rust
/// A [`StoreView`] subscribes to atomic changes in one or more [`DataStore`]s through [`StoreEvent`]s.
///
/// [`StoreView`]s can be used to build both secondary indices and trigger systems.
pub trait StoreView: std::any::Any + Send + Sync {
/// Arbitrary name for the view.
///
/// Does not need to be unique.
fn name(&self) -> String;
/// Workaround for downcasting support, simply return `self`:
/// ```ignore
/// fn as_any(&self) -> &dyn std::any::Any {
/// self
/// }
/// ```
fn as_any(&self) -> &dyn std::any::Any;
/// Workaround for downcasting support, simply return `self`:
/// ```ignore
/// fn as_any_mut(&mut self) -> &mut dyn std::any::Any {
/// self
/// }
/// ```
fn as_any_mut(&mut self) -> &mut dyn std::any::Any;
/// The core of this trait: get notified of changes happening in one or more [`DataStore`]s.
///
/// This will be called automatically by the [`DataStore`] itself if the view has been
/// registered: [`DataStore::register_view`].
/// Or you might want to feed it [`StoreEvent`]s manually, depending on your use case.
///
/// ## Example
///
/// ```ignore
/// fn on_events(&mut self, events: &[StoreEvent]) {
/// use re_arrow_store::StoreDiffKind;
/// for event in events {
/// match event.kind {
/// StoreDiffKind::Addition => println!("Row added: {}", event.row_id),
/// StoreDiffKind::Deletion => println!("Row removed: {}", event.row_id),
/// }
/// }
/// }
/// ```
fn on_events(&mut self, events: &[StoreEvent]);
}
```
---
`DataStore` changelog PR series:
- #4202
- #4203
- #4205
- #4206
- #4208
- #4209
teh-cmc
added a commit
that referenced
this pull request
Nov 15, 2023
…4206) Standalone example of how to implement and register custom `StoreView`s, even from external code. <picture> <img src="https://static.rerun.io/custom_store_view/f7258673486f91d944180bd4a83307bce09b741e/full.png" alt=""> <source media="(max-width: 480px)" srcset="https://static.rerun.io/custom_store_view/f7258673486f91d944180bd4a83307bce09b741e/480w.png"> <source media="(max-width: 768px)" srcset="https://static.rerun.io/custom_store_view/f7258673486f91d944180bd4a83307bce09b741e/768w.png"> <source media="(max-width: 1024px)" srcset="https://static.rerun.io/custom_store_view/f7258673486f91d944180bd4a83307bce09b741e/1024w.png"> <source media="(max-width: 1200px)" srcset="https://static.rerun.io/custom_store_view/f7258673486f91d944180bd4a83307bce09b741e/1200w.png"> </picture> --- `DataStore` changelog PR series: - #4202 - #4203 - #4205 - #4206 - #4208 - #4209
teh-cmc
added a commit
that referenced
this pull request
Nov 15, 2023
This is mostly preliminary work for #4209, which makes this PR a bit weird. Basically just trying to offload complexity from #4209. `TimesPerTimeline` as well as `TimeHistogramPerTimeline` are now living on their own and are maintained as `StoreView`s, i.e. they react to changes to the `DataStore` rather than constructing alternate truths. This is the first step towards turning the `EntityTree` giga-structure into an event-driven view in the next PR. --- `DataStore` changelog PR series: - #4202 - #4203 - #4205 - #4206 - #4208 - #4209
teh-cmc
added a commit
that referenced
this pull request
Nov 15, 2023
Turns the `EntityTree` giga-datastructure into a `StoreView`, meaning it now reacts to `StoreEvent`s rather than creating alternate truths. This introduces the notion of cascading side-effects, and more specifically `ClearCascade`s. When the `EntityTree` reacts to changes in the store, this might cause cascading effects (e.g. pending clears), that in turn need to write back to the store, which in turn sends more events to react to! The cycle is guaranteed finite because "clears don't get cleared"! Cascading side-effects have an interesting requirement: they need to log their cascaded data using a `RowId` _similar_ to the one used in the original event that caused the cascade (so they get GC'd at roughly the same pace). "Similar" in this cases means that their `TUID` shares the same timestamp and that the new `RowId` is strictly greater than the old one. `PathOp` has finally been annihilated. According to our new "Clears" & "Time Histograms" test suites, this behaves exactly like the `main` branch. --- `DataStore` changelog PR series: - #4202 - #4203 - #4205 - #4206 - #4208 - #4209
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The upcoming
StoreViewworks in global scope: by registering a view you subscribe to changes to allDataStores, including those that are yet to be created.This is very powerful as it allows views & triggers implementers to build cross-recording indices as well as be notified as soon as new recordings come in and go out.
But it means that
StoreEvents must indicate whichDataStorethey originate from, which isn't possible today since the stores themselves don't know who they are to begin with.This trivial PR plumbs the
StoreIdall the way through soDataStores know about their own ID.Also made
StoreGenerationaccount for the GC counter while I was at it.Requires:
DataStorechangelog PR series:DataStorechangelog 1: letDataStores know about theirStoreId#4202DataStorechangelog 2: introduceStoreEvents #4203DataStorechangelog 3: introduceStoreViews #4205DataStorechangelog 4: add standalone "Custom StoreView" example #4206DataStorechangelog 5: event-driven time histograms #4208DataStorechangelog 6: event-drivenEntityTree#4209Checklist