Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rename re_arrow_store to re_data_store #4672

Merged
merged 3 commits into from
Jan 4, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/reusable_bench.yml
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ jobs:
run: |
cargo bench \
--all-features \
-p re_arrow_store \
-p re_data_store \
-p re_entity_db \
-p re_log_encoding \
-p re_query \
Expand Down
16 changes: 8 additions & 8 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ NOTE: `.rrd` files do not yet guarantee any backwards or forwards compatibility.

## Technologies we use
### Apache Arrow
[Apache Arrow](https://arrow.apache.org/) is a language-independent columnar memory format for arbitrary data. We use it to encode the log data when transmitting it over the network or storing it in an `.rrd` file. We also use it in our in-RAM data store, [`re_arrow_store`](crates/re_arrow_store/README.md).
[Apache Arrow](https://arrow.apache.org/) is a language-independent columnar memory format for arbitrary data. We use it to encode the log data when transmitting it over the network or storing it in an `.rrd` file. We also use it in our in-RAM data store, [`re_data_store`](crates/re_data_store/README.md).

In rust, we use the [`arrow2` crate](https://crates.io/crates/arrow2).

Expand Down Expand Up @@ -88,11 +88,11 @@ Of course, this will only take us so far. In the future we plan on caching queri
Here is an overview of the crates included in the project:

<picture>
<img src="https://static.rerun.io/crates/1a5eff8b1577097cab249a751b658ba79c34396c/full.png" alt="">
<source media="(max-width: 480px)" srcset="https://static.rerun.io/crates/1a5eff8b1577097cab249a751b658ba79c34396c/480w.png">
<source media="(max-width: 768px)" srcset="https://static.rerun.io/crates/1a5eff8b1577097cab249a751b658ba79c34396c/768w.png">
<source media="(max-width: 1024px)" srcset="https://static.rerun.io/crates/1a5eff8b1577097cab249a751b658ba79c34396c/1024w.png">
<source media="(max-width: 1200px)" srcset="https://static.rerun.io/crates/1a5eff8b1577097cab249a751b658ba79c34396c/1200w.png">
<img src="https://static.rerun.io/crates/4352fe16466a95828cd9724e094acee909480f96/full.png" alt="">
<source media="(max-width: 480px)" srcset="https://static.rerun.io/crates/4352fe16466a95828cd9724e094acee909480f96/480w.png">
<source media="(max-width: 768px)" srcset="https://static.rerun.io/crates/4352fe16466a95828cd9724e094acee909480f96/768w.png">
<source media="(max-width: 1024px)" srcset="https://static.rerun.io/crates/4352fe16466a95828cd9724e094acee909480f96/1024w.png">
<source media="(max-width: 1200px)" srcset="https://static.rerun.io/crates/4352fe16466a95828cd9724e094acee909480f96/1200w.png">
</picture>

<!-- !!! IMPORTANT!!!
Expand Down Expand Up @@ -148,7 +148,7 @@ Update instructions:
| Crate | Description |
|-----------------|-----------------------------------------------------------------|
| re_entity_db | In-memory storage of Rerun entities |
| re_query | Querying data in the re_arrow_store |
| re_query | Querying data in the re_data_store |
| re_query_cache | Caching datastructures for re_query |
| re_types | The built-in Rerun data types, component types, and archetypes. |
| re_log_encoding | Helpers for encoding and transporting Rerun log messages |
Expand All @@ -158,7 +158,7 @@ Update instructions:

| Crate | Description |
|----------------|-----------------------------------------------------------------------------|
| re_arrow_store | An in-memory time series database for Rerun log data, based on Apache Arrow |
| re_data_store | An in-memory time series database for Rerun log data, based on Apache Arrow |
| re_log_types | The basic building blocks of the Rerun data types and tables. |
| re_types_core | The core traits and types that power Rerun's data model. |

Expand Down
94 changes: 47 additions & 47 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 4 additions & 4 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -28,15 +28,15 @@ version = "0.12.0-alpha.3"
# In particular: if we compile rerun 0.3.0-alpha.0 we only want it to use
# re_log_types 0.3.0-alpha.0, NOT 0.3.0-alpha.4 even though it is newer and semver-compatible.
re_analytics = { path = "crates/re_analytics", version = "=0.12.0-alpha.3", default-features = false }
re_arrow_store = { path = "crates/re_arrow_store", version = "=0.12.0-alpha.3", default-features = false }
re_build_examples = { path = "crates/re_build_examples", version = "=0.12.0-alpha.3", default-features = false }
re_build_info = { path = "crates/re_build_info", version = "=0.12.0-alpha.3", default-features = false }
re_build_tools = { path = "crates/re_build_tools", version = "=0.12.0-alpha.3", default-features = false }
re_build_web_viewer = { path = "crates/re_build_web_viewer", version = "=0.12.0-alpha.3", default-features = false }
re_crash_handler = { path = "crates/re_crash_handler", version = "=0.12.0-alpha.3", default-features = false }
re_data_source = { path = "crates/re_data_source", version = "=0.12.0-alpha.3", default-features = false }
re_entity_db = { path = "crates/re_entity_db", version = "=0.12.0-alpha.3", default-features = false }
re_data_store = { path = "crates/re_data_store", version = "=0.12.0-alpha.3", default-features = false }
re_data_ui = { path = "crates/re_data_ui", version = "=0.12.0-alpha.3", default-features = false }
re_entity_db = { path = "crates/re_entity_db", version = "=0.12.0-alpha.3", default-features = false }
re_error = { path = "crates/re_error", version = "=0.12.0-alpha.3", default-features = false }
re_format = { path = "crates/re_format", version = "=0.12.0-alpha.3", default-features = false }
re_int_histogram = { path = "crates/re_int_histogram", version = "=0.12.0-alpha.3", default-features = false }
Expand All @@ -55,16 +55,16 @@ re_space_view_bar_chart = { path = "crates/re_space_view_bar_chart", version = "
re_space_view_dataframe = { path = "crates/re_space_view_dataframe", version = "=0.12.0-alpha.3", default-features = false }
re_space_view_spatial = { path = "crates/re_space_view_spatial", version = "=0.12.0-alpha.3", default-features = false }
re_space_view_tensor = { path = "crates/re_space_view_tensor", version = "=0.12.0-alpha.3", default-features = false }
re_space_view_text_log = { path = "crates/re_space_view_text_log", version = "=0.12.0-alpha.3", default-features = false }
re_space_view_text_document = { path = "crates/re_space_view_text_document", version = "=0.12.0-alpha.3", default-features = false }
re_space_view_text_log = { path = "crates/re_space_view_text_log", version = "=0.12.0-alpha.3", default-features = false }
re_space_view_time_series = { path = "crates/re_space_view_time_series", version = "=0.12.0-alpha.3", default-features = false }
re_string_interner = { path = "crates/re_string_interner", version = "=0.12.0-alpha.3", default-features = false }
re_time_panel = { path = "crates/re_time_panel", version = "=0.12.0-alpha.3", default-features = false }
re_tracing = { path = "crates/re_tracing", version = "=0.12.0-alpha.3", default-features = false }
re_tuid = { path = "crates/re_tuid", version = "=0.12.0-alpha.3", default-features = false }
re_types_core = { path = "crates/re_types_core", version = "=0.12.0-alpha.3", default-features = false }
re_types = { path = "crates/re_types", version = "=0.12.0-alpha.3", default-features = false }
re_types_builder = { path = "crates/re_types_builder", version = "=0.12.0-alpha.3", default-features = false }
re_types_core = { path = "crates/re_types_core", version = "=0.12.0-alpha.3", default-features = false }
re_ui = { path = "crates/re_ui", version = "=0.12.0-alpha.3", default-features = false }
re_viewer = { path = "crates/re_viewer", version = "=0.12.0-alpha.3", default-features = false }
re_viewer_context = { path = "crates/re_viewer_context", version = "=0.12.0-alpha.3", default-features = false }
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[package]
name = "re_arrow_store"
name = "re_data_store"
authors.workspace = true
description = "An in-memory time series database for Rerun log data, based on Apache Arrow"
edition.workspace = true
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,4 @@ Part of the [`rerun`](https://github.com/rerun-io/rerun) family of crates.

[Apache Arrow](https://arrow.apache.org/) is a language-independent columnar memory format for arbitrary data.

The `re_arrow_store` crate is an in-memory time series database for Rerun log data. It is indexed by Entity path, component, timeline, and time. It supports out-of-order insertions, and fast `O(log(N))` queries.
The `re_data_store` crate is an in-memory time series database for Rerun log data. It is indexed by Entity path, component, timeline, and time. It supports out-of-order insertions, and fast `O(log(N))` queries.
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
use arrow2::array::{Array as _, StructArray, UnionArray};
use criterion::{criterion_group, criterion_main, Criterion};

use re_arrow_store::{
use re_data_store::{
DataStore, DataStoreConfig, GarbageCollectionOptions, GarbageCollectionTarget, LatestAtQuery,
RangeQuery, TimeInt, TimeRange,
};
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
use criterion::{criterion_group, criterion_main, BatchSize, Criterion};

use itertools::Itertools;
use re_arrow_store::{
use re_data_store::{
DataStore, DataStoreConfig, GarbageCollectionOptions, GarbageCollectionTarget,
};
use re_log_types::{
Expand Down
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
//! Demonstrates usage of [`DataStore::to_dataframe`].
//!
//! ```text
//! POLARS_FMT_MAX_ROWS=100 cargo r -p re_arrow_store --example dump_dataframe
//! POLARS_FMT_MAX_ROWS=100 cargo r -p re_data_store --example dump_dataframe
//! ```

use re_arrow_store::{test_row, DataStore};
use re_data_store::{test_row, DataStore};
use re_log_types::{build_frame_nr, build_log_time, EntityPath, Time};
use re_types::datagen::{build_some_instances, build_some_instances_from, build_some_positions2d};
use re_types::{components::InstanceKey, testing::build_some_large_structs};
Expand Down
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
//! Demonstrates usage of [`re_arrow_store::polars_util::latest_component`].
//! Demonstrates usage of [`re_data_store::polars_util::latest_component`].
//!
//! ```text
//! POLARS_FMT_MAX_ROWS=100 cargo r -p re_arrow_store --example latest_component
//! POLARS_FMT_MAX_ROWS=100 cargo r -p re_data_store --example latest_component
//! ```

use re_arrow_store::polars_util::latest_component;
use re_arrow_store::{test_row, DataStore, LatestAtQuery, TimeType, Timeline};
use re_data_store::polars_util::latest_component;
use re_data_store::{test_row, DataStore, LatestAtQuery, TimeType, Timeline};
use re_log_types::{build_frame_nr, EntityPath};
use re_types::datagen::build_some_positions2d;
use re_types::{
Expand Down
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
//! Demonstrates usage of [`re_arrow_store::polars_util::latest_components`].
//! Demonstrates usage of [`re_data_store::polars_util::latest_components`].
//!
//! ```text
//! POLARS_FMT_MAX_ROWS=100 cargo r -p re_arrow_store --example latest_components
//! POLARS_FMT_MAX_ROWS=100 cargo r -p re_data_store --example latest_components
//! ```

use polars_core::prelude::*;

use re_arrow_store::polars_util::latest_components;
use re_arrow_store::{test_row, DataStore, LatestAtQuery, TimeType, Timeline};
use re_data_store::polars_util::latest_components;
use re_data_store::{test_row, DataStore, LatestAtQuery, TimeType, Timeline};
use re_log_types::{build_frame_nr, EntityPath};
use re_types::datagen::build_some_positions2d;
use re_types::{
Expand Down
Loading
Loading