Skip to content

Commit

Permalink
Rename re_arrow_store to re_data_store (#4672)
Browse files Browse the repository at this point in the history
Closes #4437

### Checklist
* [x] I have read and agree to [Contributor
Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and
the [Code of
Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md)
* [x] I've included a screenshot or gif (if applicable)
* [x] I have tested the web demo (if applicable):
* Using newly built examples:
[app.rerun.io](https://app.rerun.io/pr/4672/index.html)
* Using examples from latest `main` build:
[app.rerun.io](https://app.rerun.io/pr/4672/index.html?manifest_url=https://app.rerun.io/version/main/examples_manifest.json)
* Using full set of examples from `nightly` build:
[app.rerun.io](https://app.rerun.io/pr/4672/index.html?manifest_url=https://app.rerun.io/version/nightly/examples_manifest.json)
* [x] The PR title and labels are set such as to maximize their
usefulness for the next release's CHANGELOG

- [PR Build Summary](https://build.rerun.io/pr/4672)
- [Docs
preview](https://rerun.io/preview/13d64ac10bc5bc0a712082dac786ac4849276b6d/docs)
<!--DOCS-PREVIEW-->
- [Examples
preview](https://rerun.io/preview/13d64ac10bc5bc0a712082dac786ac4849276b6d/examples)
<!--EXAMPLES-PREVIEW-->
- [Recent benchmark results](https://build.rerun.io/graphs/crates.html)
- [Wasm size tracking](https://build.rerun.io/graphs/sizes.html)
  • Loading branch information
emilk authored Jan 4, 2024
1 parent 7093d12 commit 790f391
Show file tree
Hide file tree
Showing 123 changed files with 285 additions and 287 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/reusable_bench.yml
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ jobs:
run: |
cargo bench \
--all-features \
-p re_arrow_store \
-p re_data_store \
-p re_entity_db \
-p re_log_encoding \
-p re_query \
Expand Down
16 changes: 8 additions & 8 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ NOTE: `.rrd` files do not yet guarantee any backwards or forwards compatibility.

## Technologies we use
### Apache Arrow
[Apache Arrow](https://arrow.apache.org/) is a language-independent columnar memory format for arbitrary data. We use it to encode the log data when transmitting it over the network or storing it in an `.rrd` file. We also use it in our in-RAM data store, [`re_arrow_store`](crates/re_arrow_store/README.md).
[Apache Arrow](https://arrow.apache.org/) is a language-independent columnar memory format for arbitrary data. We use it to encode the log data when transmitting it over the network or storing it in an `.rrd` file. We also use it in our in-RAM data store, [`re_data_store`](crates/re_data_store/README.md).

In rust, we use the [`arrow2` crate](https://crates.io/crates/arrow2).

Expand Down Expand Up @@ -88,11 +88,11 @@ Of course, this will only take us so far. In the future we plan on caching queri
Here is an overview of the crates included in the project:

<picture>
<img src="https://static.rerun.io/crates/1a5eff8b1577097cab249a751b658ba79c34396c/full.png" alt="">
<source media="(max-width: 480px)" srcset="https://static.rerun.io/crates/1a5eff8b1577097cab249a751b658ba79c34396c/480w.png">
<source media="(max-width: 768px)" srcset="https://static.rerun.io/crates/1a5eff8b1577097cab249a751b658ba79c34396c/768w.png">
<source media="(max-width: 1024px)" srcset="https://static.rerun.io/crates/1a5eff8b1577097cab249a751b658ba79c34396c/1024w.png">
<source media="(max-width: 1200px)" srcset="https://static.rerun.io/crates/1a5eff8b1577097cab249a751b658ba79c34396c/1200w.png">
<img src="https://static.rerun.io/crates/4352fe16466a95828cd9724e094acee909480f96/full.png" alt="">
<source media="(max-width: 480px)" srcset="https://static.rerun.io/crates/4352fe16466a95828cd9724e094acee909480f96/480w.png">
<source media="(max-width: 768px)" srcset="https://static.rerun.io/crates/4352fe16466a95828cd9724e094acee909480f96/768w.png">
<source media="(max-width: 1024px)" srcset="https://static.rerun.io/crates/4352fe16466a95828cd9724e094acee909480f96/1024w.png">
<source media="(max-width: 1200px)" srcset="https://static.rerun.io/crates/4352fe16466a95828cd9724e094acee909480f96/1200w.png">
</picture>

<!-- !!! IMPORTANT!!!
Expand Down Expand Up @@ -148,7 +148,7 @@ Update instructions:
| Crate | Description |
|-----------------|-----------------------------------------------------------------|
| re_entity_db | In-memory storage of Rerun entities |
| re_query | Querying data in the re_arrow_store |
| re_query | Querying data in the re_data_store |
| re_query_cache | Caching datastructures for re_query |
| re_types | The built-in Rerun data types, component types, and archetypes. |
| re_log_encoding | Helpers for encoding and transporting Rerun log messages |
Expand All @@ -158,7 +158,7 @@ Update instructions:

| Crate | Description |
|----------------|-----------------------------------------------------------------------------|
| re_arrow_store | An in-memory time series database for Rerun log data, based on Apache Arrow |
| re_data_store | An in-memory time series database for Rerun log data, based on Apache Arrow |
| re_log_types | The basic building blocks of the Rerun data types and tables. |
| re_types_core | The core traits and types that power Rerun's data model. |

Expand Down
94 changes: 47 additions & 47 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 4 additions & 4 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -28,15 +28,15 @@ version = "0.12.0-alpha.3"
# In particular: if we compile rerun 0.3.0-alpha.0 we only want it to use
# re_log_types 0.3.0-alpha.0, NOT 0.3.0-alpha.4 even though it is newer and semver-compatible.
re_analytics = { path = "crates/re_analytics", version = "=0.12.0-alpha.3", default-features = false }
re_arrow_store = { path = "crates/re_arrow_store", version = "=0.12.0-alpha.3", default-features = false }
re_build_examples = { path = "crates/re_build_examples", version = "=0.12.0-alpha.3", default-features = false }
re_build_info = { path = "crates/re_build_info", version = "=0.12.0-alpha.3", default-features = false }
re_build_tools = { path = "crates/re_build_tools", version = "=0.12.0-alpha.3", default-features = false }
re_build_web_viewer = { path = "crates/re_build_web_viewer", version = "=0.12.0-alpha.3", default-features = false }
re_crash_handler = { path = "crates/re_crash_handler", version = "=0.12.0-alpha.3", default-features = false }
re_data_source = { path = "crates/re_data_source", version = "=0.12.0-alpha.3", default-features = false }
re_entity_db = { path = "crates/re_entity_db", version = "=0.12.0-alpha.3", default-features = false }
re_data_store = { path = "crates/re_data_store", version = "=0.12.0-alpha.3", default-features = false }
re_data_ui = { path = "crates/re_data_ui", version = "=0.12.0-alpha.3", default-features = false }
re_entity_db = { path = "crates/re_entity_db", version = "=0.12.0-alpha.3", default-features = false }
re_error = { path = "crates/re_error", version = "=0.12.0-alpha.3", default-features = false }
re_format = { path = "crates/re_format", version = "=0.12.0-alpha.3", default-features = false }
re_int_histogram = { path = "crates/re_int_histogram", version = "=0.12.0-alpha.3", default-features = false }
Expand All @@ -55,16 +55,16 @@ re_space_view_bar_chart = { path = "crates/re_space_view_bar_chart", version = "
re_space_view_dataframe = { path = "crates/re_space_view_dataframe", version = "=0.12.0-alpha.3", default-features = false }
re_space_view_spatial = { path = "crates/re_space_view_spatial", version = "=0.12.0-alpha.3", default-features = false }
re_space_view_tensor = { path = "crates/re_space_view_tensor", version = "=0.12.0-alpha.3", default-features = false }
re_space_view_text_log = { path = "crates/re_space_view_text_log", version = "=0.12.0-alpha.3", default-features = false }
re_space_view_text_document = { path = "crates/re_space_view_text_document", version = "=0.12.0-alpha.3", default-features = false }
re_space_view_text_log = { path = "crates/re_space_view_text_log", version = "=0.12.0-alpha.3", default-features = false }
re_space_view_time_series = { path = "crates/re_space_view_time_series", version = "=0.12.0-alpha.3", default-features = false }
re_string_interner = { path = "crates/re_string_interner", version = "=0.12.0-alpha.3", default-features = false }
re_time_panel = { path = "crates/re_time_panel", version = "=0.12.0-alpha.3", default-features = false }
re_tracing = { path = "crates/re_tracing", version = "=0.12.0-alpha.3", default-features = false }
re_tuid = { path = "crates/re_tuid", version = "=0.12.0-alpha.3", default-features = false }
re_types_core = { path = "crates/re_types_core", version = "=0.12.0-alpha.3", default-features = false }
re_types = { path = "crates/re_types", version = "=0.12.0-alpha.3", default-features = false }
re_types_builder = { path = "crates/re_types_builder", version = "=0.12.0-alpha.3", default-features = false }
re_types_core = { path = "crates/re_types_core", version = "=0.12.0-alpha.3", default-features = false }
re_ui = { path = "crates/re_ui", version = "=0.12.0-alpha.3", default-features = false }
re_viewer = { path = "crates/re_viewer", version = "=0.12.0-alpha.3", default-features = false }
re_viewer_context = { path = "crates/re_viewer_context", version = "=0.12.0-alpha.3", default-features = false }
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[package]
name = "re_arrow_store"
name = "re_data_store"
authors.workspace = true
description = "An in-memory time series database for Rerun log data, based on Apache Arrow"
edition.workspace = true
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,4 @@ Part of the [`rerun`](https://github.com/rerun-io/rerun) family of crates.

[Apache Arrow](https://arrow.apache.org/) is a language-independent columnar memory format for arbitrary data.

The `re_arrow_store` crate is an in-memory time series database for Rerun log data. It is indexed by Entity path, component, timeline, and time. It supports out-of-order insertions, and fast `O(log(N))` queries.
The `re_data_store` crate is an in-memory time series database for Rerun log data. It is indexed by Entity path, component, timeline, and time. It supports out-of-order insertions, and fast `O(log(N))` queries.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
use arrow2::array::{Array as _, StructArray, UnionArray};
use criterion::{criterion_group, criterion_main, Criterion};

use re_arrow_store::{
use re_data_store::{
DataStore, DataStoreConfig, GarbageCollectionOptions, GarbageCollectionTarget, LatestAtQuery,
RangeQuery, TimeInt, TimeRange,
};
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ static GLOBAL: mimalloc::MiMalloc = mimalloc::MiMalloc;
use criterion::{criterion_group, criterion_main, BatchSize, Criterion};

use itertools::Itertools;
use re_arrow_store::{
use re_data_store::{
DataStore, DataStoreConfig, GarbageCollectionOptions, GarbageCollectionTarget,
};
use re_log_types::{
Expand Down
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
//! Demonstrates usage of [`DataStore::to_dataframe`].
//!
//! ```text
//! POLARS_FMT_MAX_ROWS=100 cargo r -p re_arrow_store --example dump_dataframe
//! POLARS_FMT_MAX_ROWS=100 cargo r -p re_data_store --example dump_dataframe
//! ```
use re_arrow_store::{test_row, DataStore};
use re_data_store::{test_row, DataStore};
use re_log_types::{build_frame_nr, build_log_time, EntityPath, Time};
use re_types::datagen::{build_some_instances, build_some_instances_from, build_some_positions2d};
use re_types::{components::InstanceKey, testing::build_some_large_structs};
Expand Down
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
//! Demonstrates usage of [`re_arrow_store::polars_util::latest_component`].
//! Demonstrates usage of [`re_data_store::polars_util::latest_component`].
//!
//! ```text
//! POLARS_FMT_MAX_ROWS=100 cargo r -p re_arrow_store --example latest_component
//! POLARS_FMT_MAX_ROWS=100 cargo r -p re_data_store --example latest_component
//! ```
use re_arrow_store::polars_util::latest_component;
use re_arrow_store::{test_row, DataStore, LatestAtQuery, TimeType, Timeline};
use re_data_store::polars_util::latest_component;
use re_data_store::{test_row, DataStore, LatestAtQuery, TimeType, Timeline};
use re_log_types::{build_frame_nr, EntityPath};
use re_types::datagen::build_some_positions2d;
use re_types::{
Expand Down
Original file line number Diff line number Diff line change
@@ -1,13 +1,13 @@
//! Demonstrates usage of [`re_arrow_store::polars_util::latest_components`].
//! Demonstrates usage of [`re_data_store::polars_util::latest_components`].
//!
//! ```text
//! POLARS_FMT_MAX_ROWS=100 cargo r -p re_arrow_store --example latest_components
//! POLARS_FMT_MAX_ROWS=100 cargo r -p re_data_store --example latest_components
//! ```
use polars_core::prelude::*;

use re_arrow_store::polars_util::latest_components;
use re_arrow_store::{test_row, DataStore, LatestAtQuery, TimeType, Timeline};
use re_data_store::polars_util::latest_components;
use re_data_store::{test_row, DataStore, LatestAtQuery, TimeType, Timeline};
use re_log_types::{build_frame_nr, EntityPath};
use re_types::datagen::build_some_positions2d;
use re_types::{
Expand Down
Loading

0 comments on commit 790f391

Please sign in to comment.