Skip to content

Commit

Permalink
Rename re_arrow_store to re_data_store
Browse files Browse the repository at this point in the history
Closes #4437
  • Loading branch information
emilk committed Jan 4, 2024
1 parent 098e601 commit 87346b3
Show file tree
Hide file tree
Showing 124 changed files with 283 additions and 285 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/reusable_bench.yml
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ jobs:
run: |
cargo bench \
--all-features \
-p re_arrow_store \
-p re_data_store \
-p re_entity_db \
-p re_log_encoding \
-p re_query \
Expand Down
16 changes: 8 additions & 8 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ NOTE: `.rrd` files do not yet guarantee any backwards or forwards compatibility.

## Technologies we use
### Apache Arrow
[Apache Arrow](https://arrow.apache.org/) is a language-independent columnar memory format for arbitrary data. We use it to encode the log data when transmitting it over the network or storing it in an `.rrd` file. We also use it in our in-RAM data store, [`re_arrow_store`](crates/re_arrow_store/README.md).
[Apache Arrow](https://arrow.apache.org/) is a language-independent columnar memory format for arbitrary data. We use it to encode the log data when transmitting it over the network or storing it in an `.rrd` file. We also use it in our in-RAM data store, [`re_data_store`](crates/re_data_store/README.md).

In rust, we use the [`arrow2` crate](https://crates.io/crates/arrow2).

Expand Down Expand Up @@ -88,11 +88,11 @@ Of course, this will only take us so far. In the future we plan on caching queri
Here is an overview of the crates included in the project:

<picture>
<img src="https://static.rerun.io/crates/1a5eff8b1577097cab249a751b658ba79c34396c/full.png" alt="">
<source media="(max-width: 480px)" srcset="https://static.rerun.io/crates/1a5eff8b1577097cab249a751b658ba79c34396c/480w.png">
<source media="(max-width: 768px)" srcset="https://static.rerun.io/crates/1a5eff8b1577097cab249a751b658ba79c34396c/768w.png">
<source media="(max-width: 1024px)" srcset="https://static.rerun.io/crates/1a5eff8b1577097cab249a751b658ba79c34396c/1024w.png">
<source media="(max-width: 1200px)" srcset="https://static.rerun.io/crates/1a5eff8b1577097cab249a751b658ba79c34396c/1200w.png">
<img src="https://static.rerun.io/crates/4352fe16466a95828cd9724e094acee909480f96/full.png" alt="">
<source media="(max-width: 480px)" srcset="https://static.rerun.io/crates/4352fe16466a95828cd9724e094acee909480f96/480w.png">
<source media="(max-width: 768px)" srcset="https://static.rerun.io/crates/4352fe16466a95828cd9724e094acee909480f96/768w.png">
<source media="(max-width: 1024px)" srcset="https://static.rerun.io/crates/4352fe16466a95828cd9724e094acee909480f96/1024w.png">
<source media="(max-width: 1200px)" srcset="https://static.rerun.io/crates/4352fe16466a95828cd9724e094acee909480f96/1200w.png">
</picture>

<!-- !!! IMPORTANT!!!
Expand Down Expand Up @@ -148,7 +148,7 @@ Update instructions:
| Crate | Description |
|-----------------|-----------------------------------------------------------------|
| re_entity_db | In-memory storage of Rerun entities |
| re_query | Querying data in the re_arrow_store |
| re_query | Querying data in the re_data_store |
| re_query_cache | Caching datastructures for re_query |
| re_types | The built-in Rerun data types, component types, and archetypes. |
| re_log_encoding | Helpers for encoding and transporting Rerun log messages |
Expand All @@ -158,7 +158,7 @@ Update instructions:

| Crate | Description |
|----------------|-----------------------------------------------------------------------------|
| re_arrow_store | An in-memory time series database for Rerun log data, based on Apache Arrow |
| re_data_store | An in-memory time series database for Rerun log data, based on Apache Arrow |
| re_log_types | The basic building blocks of the Rerun data types and tables. |
| re_types_core | The core traits and types that power Rerun's data model. |

Expand Down
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -1267,7 +1267,7 @@ We now host an experimental and unpolished web-viewer at <https://app.rerun.io/>
- Fix arrows requiring a radius to be visible [#1720](https://github.com/rerun-io/rerun/pull/1720)
#### 🚀 Performance Improvements
- Add re_arrow_store profile scopes [#1546](https://github.com/rerun-io/rerun/pull/1546)
- Add re_data_store profile scopes [#1546](https://github.com/rerun-io/rerun/pull/1546)
- datastore: early exit missing components at table level [#1554](https://github.com/rerun-io/rerun/pull/1554)
- datastore: track bucket count in store stats & mem panel [#1555](https://github.com/rerun-io/rerun/pull/1555)
- LogDb: dont split on index bucket size [#1558](https://github.com/rerun-io/rerun/pull/1558)
Expand Down
94 changes: 47 additions & 47 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ version = "0.12.0-alpha.3"
# In particular: if we compile rerun 0.3.0-alpha.0 we only want it to use
# re_log_types 0.3.0-alpha.0, NOT 0.3.0-alpha.4 even though it is newer and semver-compatible.
re_analytics = { path = "crates/re_analytics", version = "=0.12.0-alpha.3", default-features = false }
re_arrow_store = { path = "crates/re_arrow_store", version = "=0.12.0-alpha.3", default-features = false }
re_data_store = { path = "crates/re_data_store", version = "=0.12.0-alpha.3", default-features = false }
re_build_examples = { path = "crates/re_build_examples", version = "=0.12.0-alpha.3", default-features = false }
re_build_info = { path = "crates/re_build_info", version = "=0.12.0-alpha.3", default-features = false }
re_build_tools = { path = "crates/re_build_tools", version = "=0.12.0-alpha.3", default-features = false }
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[package]
name = "re_arrow_store"
name = "re_data_store"
authors.workspace = true
description = "An in-memory time series database for Rerun log data, based on Apache Arrow"
edition.workspace = true
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,4 @@ Part of the [`rerun`](https://github.com/rerun-io/rerun) family of crates.

[Apache Arrow](https://arrow.apache.org/) is a language-independent columnar memory format for arbitrary data.

The `re_arrow_store` crate is an in-memory time series database for Rerun log data. It is indexed by Entity path, component, timeline, and time. It supports out-of-order insertions, and fast `O(log(N))` queries.
The `re_data_store` crate is an in-memory time series database for Rerun log data. It is indexed by Entity path, component, timeline, and time. It supports out-of-order insertions, and fast `O(log(N))` queries.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
use arrow2::array::{Array as _, StructArray, UnionArray};
use criterion::{criterion_group, criterion_main, Criterion};

use re_arrow_store::{
use re_data_store::{
DataStore, DataStoreConfig, GarbageCollectionOptions, GarbageCollectionTarget, LatestAtQuery,
RangeQuery, TimeInt, TimeRange,
};
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
use criterion::{criterion_group, criterion_main, BatchSize, Criterion};

use itertools::Itertools;
use re_arrow_store::{
use re_data_store::{
DataStore, DataStoreConfig, GarbageCollectionOptions, GarbageCollectionTarget,
};
use re_log_types::{
Expand Down
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
//! Demonstrates usage of [`DataStore::to_dataframe`].
//!
//! ```text
//! POLARS_FMT_MAX_ROWS=100 cargo r -p re_arrow_store --example dump_dataframe
//! POLARS_FMT_MAX_ROWS=100 cargo r -p re_data_store --example dump_dataframe
//! ```
use re_arrow_store::{test_row, DataStore};
use re_data_store::{test_row, DataStore};
use re_log_types::{build_frame_nr, build_log_time, EntityPath, Time};
use re_types::datagen::{build_some_instances, build_some_instances_from, build_some_positions2d};
use re_types::{components::InstanceKey, testing::build_some_large_structs};
Expand Down
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
//! Demonstrates usage of [`re_arrow_store::polars_util::latest_component`].
//! Demonstrates usage of [`re_data_store::polars_util::latest_component`].
//!
//! ```text
//! POLARS_FMT_MAX_ROWS=100 cargo r -p re_arrow_store --example latest_component
//! POLARS_FMT_MAX_ROWS=100 cargo r -p re_data_store --example latest_component
//! ```
use re_arrow_store::polars_util::latest_component;
use re_arrow_store::{test_row, DataStore, LatestAtQuery, TimeType, Timeline};
use re_data_store::polars_util::latest_component;
use re_data_store::{test_row, DataStore, LatestAtQuery, TimeType, Timeline};
use re_log_types::{build_frame_nr, EntityPath};
use re_types::datagen::build_some_positions2d;
use re_types::{
Expand Down
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
//! Demonstrates usage of [`re_arrow_store::polars_util::latest_components`].
//! Demonstrates usage of [`re_data_store::polars_util::latest_components`].
//!
//! ```text
//! POLARS_FMT_MAX_ROWS=100 cargo r -p re_arrow_store --example latest_components
//! POLARS_FMT_MAX_ROWS=100 cargo r -p re_data_store --example latest_components
//! ```
use polars_core::prelude::*;

use re_arrow_store::polars_util::latest_components;
use re_arrow_store::{test_row, DataStore, LatestAtQuery, TimeType, Timeline};
use re_data_store::polars_util::latest_components;
use re_data_store::{test_row, DataStore, LatestAtQuery, TimeType, Timeline};
use re_log_types::{build_frame_nr, EntityPath};
use re_types::datagen::build_some_positions2d;
use re_types::{
Expand Down
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
//! Demonstrates usage of [`re_arrow_store::polars_util::range_components`].
//! Demonstrates usage of [`re_data_store::polars_util::range_components`].
//!
//! ```text
//! POLARS_FMT_MAX_ROWS=100 cargo r -p re_arrow_store --example range_components
//! POLARS_FMT_MAX_ROWS=100 cargo r -p re_data_store --example range_components
//! ```
use polars_core::prelude::JoinType;
use re_arrow_store::{polars_util, test_row, DataStore, RangeQuery, TimeRange};
use re_data_store::{polars_util, test_row, DataStore, RangeQuery, TimeRange};
use re_log_types::{build_frame_nr, EntityPath, TimeType, Timeline};
use re_types::datagen::build_some_positions2d;
use re_types::{
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
Loading

0 comments on commit 87346b3

Please sign in to comment.