Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DataStore changelog 2: introduce StoreEvents #4203

Merged
merged 6 commits into from
Nov 15, 2023
+743 −112
Merged

DataStore changelog 2: introduce StoreEvents #4203

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
4005c8b
introduce StoreEvent and StoreDiff
teh-cmc Nov 12, 2023
625abfb
integrate StoreEvents on the insertion path + tests
teh-cmc Nov 12, 2023
d4fd011
integrate StoreEvents on the GC path + tests
teh-cmc Nov 12, 2023
959269d
propagate changes, lints, etc
teh-cmc Nov 12, 2023
5e69b22
dead code
teh-cmc Nov 14, 2023
93d6739
pr comments
teh-cmc Nov 15, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions crates/re_arrow_store/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ polars-ops = { workspace = true, optional = true, features = [


[dev-dependencies]
re_log_types = { workspace = true, features = ["testing"] }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

at the very least in this pr this doesn't seem to happen. Why is it needed now?
(also, makes me sad!)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably needed at some point, dont remember... in any case it can never hurt and will probably save someone some pain in the future so I'll leave it there 🤷

re_types = { workspace = true, features = ["datagen"] }

anyhow.workspace = true
Expand Down
2 changes: 2 additions & 0 deletions crates/re_arrow_store/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ mod arrow_util;
mod store;
mod store_arrow;
mod store_dump;
mod store_event;
mod store_format;
mod store_gc;
mod store_helpers;
Expand All @@ -37,6 +38,7 @@ pub mod test_util;

pub use self::arrow_util::ArrayExt;
pub use self::store::{DataStore, DataStoreConfig, StoreGeneration};
pub use self::store_event::{StoreDiff, StoreDiffKind, StoreEvent};
pub use self::store_gc::{Deleted, GarbageCollectionOptions, GarbageCollectionTarget};
pub use self::store_helpers::VersionedComponent;
pub use self::store_read::{LatestAtQuery, RangeQuery};
Expand Down
7 changes: 6 additions & 1 deletion crates/re_arrow_store/src/store.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,13 @@ use ahash::HashMap;
use arrow2::datatypes::DataType;
use nohash_hasher::IntMap;
use parking_lot::RwLock;
use re_types_core::{ComponentName, ComponentNameSet, SizeBytes};
use smallvec::SmallVec;

use re_log_types::{
DataCell, DataCellColumn, EntityPath, EntityPathHash, ErasedTimeVec, NumInstancesVec, RowId,
RowIdVec, StoreId, TimeInt, TimePoint, TimeRange, Timeline,
};
use re_types_core::{ComponentName, ComponentNameSet, SizeBytes};

// --- Data store ---

Expand Down Expand Up @@ -229,6 +229,9 @@ pub struct DataStore {

/// Monotonically increasing ID for GCs.
pub(crate) gc_id: u64,

/// Monotonically increasing ID for store events.
pub(crate) event_id: AtomicU64,
}

impl Clone for DataStore {
Expand All @@ -245,6 +248,7 @@ impl Clone for DataStore {
insert_id: Default::default(),
query_id: Default::default(),
gc_id: Default::default(),
event_id: Default::default(),
}
}
}
Expand All @@ -264,6 +268,7 @@ impl DataStore {
insert_id: 0,
query_id: AtomicU64::new(0),
gc_id: 0,
event_id: AtomicU64::new(0),
}
}

Expand Down
Loading
Loading