Skip to content

Commit

Permalink
Document our crate organization in ARCHITECTURE.md (#4458)
Browse files Browse the repository at this point in the history 
### What

Add a list of crates and an org diagram in ARCHITECTURE.md. Also add a
lint to minimise the chance of rot.

Source FigJam:
https://www.figma.com/file/Umob8ztK1HmYKLUMSq8aPb/Crates-org


![](https://static.rerun.io/crates/eaea8b78fd7efbefd76c0d6a09086ef9cd742c8b/1200w.png)

### Checklist
* [x] I have read and agree to [Contributor
Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and
the [Code of
Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md)
* [x] I've included a screenshot or gif (if applicable)
* [x] I have tested the web demo (if applicable):
  * Full build: [app.rerun.io](https://app.rerun.io/pr/4458/index.html)
* Partial build:
[app.rerun.io](https://app.rerun.io/pr/4458/index.html?manifest_url=https://app.rerun.io/version/nightly/examples_manifest.json)
- Useful for quick testing when changes do not affect examples in any
way
* [x] The PR title and labels are set such as to maximize their
usefulness for the next release's CHANGELOG

- [PR Build Summary](https://build.rerun.io/pr/4458)
- [Docs
preview](https://rerun.io/preview/61649d5f937007affd7624aa6b75d9a72dc654c3/docs)
<!--DOCS-PREVIEW-->
- [Examples
preview](https://rerun.io/preview/61649d5f937007affd7624aa6b75d9a72dc654c3/examples)
<!--EXAMPLES-PREVIEW-->
- [Recent benchmark results](https://build.rerun.io/graphs/crates.html)
- [Wasm size tracking](https://build.rerun.io/graphs/sizes.html)

---------

Co-authored-by: Emil Ernerfeldt <[email protected]>
  • Loading branch information
@abey79 @emilk
abey79 and emilk authored Dec 10, 2023
1 parent 6458d6d commit c824555
Show file tree
Hide file tree
Showing 3 changed files with 179 additions and 19 deletions.
154 changes: 136 additions & 18 deletions ARCHITECTURE.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,24 +37,6 @@ The web viewer can load `.rrd` files (just drag-drop them into the browser), or
NOTE: `.rrd` files do not yet guarantee any backwards or forwards compatibility. One version of Rerun will likely not be able to open an `.rrd` file generated by another Rerun version.


## Crates
In order to get an overview of our in-house crates and how they depend on each other, we recommend you run:

```
cargo install cargo-depgraph
cargo depgraph --all-deps --workspace-only --all-features --dedup-transitive-deps | dot -Tpng > deps.png
open deps.png
```

and:

```
cargo doc --no-deps --open
```

and then browse through the `re_*` crates.


## Technologies we use
### Apache Arrow
[Apache Arrow](https://arrow.apache.org/) is a language-independent columnar memory format for arbitrary data. We use it to encode the log data when transmitting it over the network or storing it in an `.rrd` file. We also use it in our in-RAM data store, [`re_arrow_store`](crates/re_arrow_store/README.md).
Expand Down Expand Up @@ -99,3 +81,139 @@ Immediate mode is also a forcing function, forcing us to relentlessly optimize o
This leads to a very responsive GUI, where there is no "hickups" when switching data source or doing time scrubbing.

Of course, this will only take us so far. In the future we plan on caching queries and work submitted to the renderer so that we don't perform unnecessary work each frame. We also plan on doing larger operation in background threads. This will be necessary in order to support viewing large datasets, e.g. several million points. The plan is still to do so within an immediate mode framework, retaining most of the advantages of stateless code.


## Crates

Here is an overview of the crates included in the project:

<picture>
<img src="https://static.rerun.io/crates/eaea8b78fd7efbefd76c0d6a09086ef9cd742c8b/full.png" alt="">
<source media="(max-width: 480px)" srcset="https://static.rerun.io/crates/eaea8b78fd7efbefd76c0d6a09086ef9cd742c8b/480w.png">
<source media="(max-width: 768px)" srcset="https://static.rerun.io/crates/eaea8b78fd7efbefd76c0d6a09086ef9cd742c8b/768w.png">
<source media="(max-width: 1024px)" srcset="https://static.rerun.io/crates/eaea8b78fd7efbefd76c0d6a09086ef9cd742c8b/1024w.png">
<source media="(max-width: 1200px)" srcset="https://static.rerun.io/crates/eaea8b78fd7efbefd76c0d6a09086ef9cd742c8b/1200w.png">
</picture>

<!-- !!! IMPORTANT!!!
This image must be updated each time a crate is added/removed/updated.
FigJam document: https://www.figma.com/file/Umob8ztK1HmYKLUMSq8aPb/Crates-org
Update instructions:
1) Update the FigJame document
2) Select all -> right-click -> Copy as PNG
3) `just upload --name crates`
4) Copy/paste the resulting HTML
-->

### SDK/CLI/Wasm top-level crates

| Crate | Description |
|-----------|--------------------------------------|
| rerun-cli | Rerun native CLI binary crate |
| rerun | Rerun Rust SDK and viewer shim crate |
| rerun_c | Rerun C SDK |
| rerun_py | Rerun Python SDK |
| re_sdk | Rerun logging SDK |


### Viewer crates

##### UI crates

| Crate | Description |
|-----------------------------|----------------------------------------------------------------------------------------|
| re_viewer | The Rerun viewer |
| re_viewport | The central viewport panel of the Rerun viewer. |
| re_time_panel | The time panel of the Rerun Viewer, allowing to control the displayed timeline & time. |
| re_data_ui | Provides ui elements for Rerun component data for the Rerun Viewer. |
| re_viewer_context | Rerun viewer state that is shared with the viewer's code components. |
| re_ui | Rerun GUI theme and helpers, built around egui |
| re_renderer | A wgpu-based renderer for all your visualization needs. |
| re_space_view | Types & utilities for defining Space View classes and communicating with the Viewport. |
| re_space_view_bar_chart | A Space View that shows a single bar chart. |
| re_space_view_spatial | Space Views that show entities in a 2D or 3D spatial relationship. |
| re_space_view_tensor | A Space View dedicated to visualizing tensors with arbitrary dimensionality. |
| re_space_view_text_document | A simple Space View that shows a single text box. |
| re_space_view_text_log | A Space View that shows text entries in a table and scrolls with the active time. |
| re_space_view_time_series | A Space View that shows plots over Rerun timelines. |


### Application-level store

| Crate | Description |
|-----------------|-----------------------------------------------------------------|
| re_data_store | In-memory storage of Rerun log data, indexed for fast queries. |
| re_query | Querying data in the re_arrow_store |
| re_types | The built-in Rerun data types, component types, and archetypes. |
| re_log_encoding | Helpers for encoding and transporting Rerun log messages |


### Low-level store

| Crate | Description |
|----------------|-----------------------------------------------------------------------------|
| re_arrow_store | An in-memory time series database for Rerun log data, based on Apache Arrow |
| re_log_types | The basic building blocks of the Rerun data types and tables. |
| re_types_core | The core traits and types that power Rerun's data model. |


### Data flow

| Crate | Description |
|----------------------|--------------------------------------------------------------------------------------------------------|
| re_sdk_comms | TCP communication between Rerun SDK and Rerun Server |
| re_web_viewer_server | Serves the Rerun web viewer (Wasm and HTML) over HTTP |
| re_ws_comms | WebSocket communication library (encoding, decoding, client, server) between a Rerun server and viewer |
| re_data_source | Handles loading of Rerun data |


### Build support

| Crate | Description |
|----------------------------|---------------------------------------------------------------|
| re_build_info | Information about the build. Use together with re_build_tools |
| re_build_tools | build.rs helpers for generating build info |
| re_types_builder | Generates code for Rerun's SDKs from flatbuffers definitions. |
| re_build_examples | Build rerun example RRD files |
| re_build_examples_manifest | Build the rerun examples manifest JSON file |
| re_build_web_viewer | Build the rerun web-viewer Wasm from source |


### Utilities

| Crate | Description |
|--------------------|--------------------------------------------------------------------------------------|
| re_analytics | Rerun's analytics SDK |
| re_log | Helpers for setting up and doing text logging in the Rerun crates. |
| re_error | Helpers for handling errors. |
| re_format | Miscellaneous tools to format and parse numbers, durations, etc. |
| re_tuid | 128-bit Time-based Unique Identifier |
| re_string_interner | Yet another string interning library |
| re_tracing | Helpers for tracing/spans/flamegraphs and such. |
| re_crash_handler | Detect panics and signals, logging them and optionally sending them to analytics. |
| re_smart_channel | A channel that keeps track of latency and queue length. |
| re_int_histogram | A histogram with `i64` keys and `u32` counts, supporting both sparse and dense uses. |
| re_memory | Run-time memory tracking and profiling. |



### Dependencies and docs

In order to get a dependency graph for our in-house crates and their docs, we recommend you run:

```
cargo install cargo-depgraph
cargo depgraph --all-deps --workspace-only --all-features --dedup-transitive-deps | dot -Tpng > deps.png
open deps.png
```

and:

```
cargo doc --no-deps --open
```

and then browse through the `re_*` crates.
1 change: 1 addition & 0 deletions docs/cspell.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,7 @@
"icosphere",
"imgmsg",
"impls",
"interner",
"intrinsics",
"ipynb",
"ipython",
Expand Down
43 changes: 42 additions & 1 deletion scripts/lint.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@
import os
import re
import sys
from typing import Any, Iterator
from pathlib import Path
from typing import Any, Callable, Iterator

import frontmatter
from gitignore_parser import parse_gitignore
Expand Down Expand Up @@ -748,6 +749,43 @@ def lint_file(filepath: str, args: Any) -> int:
return num_errors


def lint_crate_docs(should_ignore: Callable[[Any], bool]) -> int:
"""Make sure ARCHITECTURE.md talks about every single crate we have."""

crates_dir = Path("crates")
architecture_md_file = Path("ARCHITECTURE.md")

architecture_md = architecture_md_file.read_text()

# extract all crate names ("re_...") from ARCHITECTURE.md to ensure they actually exist
listed_crates: dict[str, int] = {}
for i, line in enumerate(architecture_md.split("\n"), start=1):
for crate_name in re.findall(r"\bre_\w+", line):
if crate_name not in listed_crates:
listed_crates[crate_name] = i

error_count = 0
for cargo_toml in crates_dir.glob("**/Cargo.toml"):
crate = cargo_toml.parent
crate_name = crate.name

if crate_name in listed_crates:
del listed_crates[crate_name]

if should_ignore(crate):
continue

if not re.search(r"\b" + crate_name + r"\b", architecture_md):
print(f"{architecture_md_file}: missing documentation for crate {crate.name}")
error_count += 1

for crate_name, line_nr in sorted(listed_crates.items(), key=lambda x: x[1]):
print(f"{architecture_md_file}:{line_nr}: crate name {crate_name} does not exist")
error_count += 1

return error_count


def main() -> None:
# Make sure we are bug free before we run:
test_lint_line()
Expand Down Expand Up @@ -832,6 +870,9 @@ def main() -> None:
continue
num_errors += lint_file(filepath, args)

# Since no files have been specified, we also run the global lints.
num_errors += lint_crate_docs(should_ignore)

if num_errors == 0:
print(f"{sys.argv[0]} finished without error")
sys.exit(0)
Expand Down

0 comments on commit c824555

Please sign in to comment.