`DataLoader`s 2: add text-based `DataLoader` (`.txt`, `.md`) #4518

teh-cmc · 2023-12-13T18:57:03Z

What the title says.

Checks:

cargo r -p rerun-cli --no-default-features --features native_viewer -- examples/assets/example.{glb,gltf,obj,jpg,png,rrd,txt,md}
Native: File > Open > examples/assets/*
Native: Drag-n-drop > examples/assets/*
Web: File > Open > examples/assets/*
Web: Drag-n-drop > examples/assets/*

Part of a series of PRs to make it possible to load any file from the local filesystem, by any means, on web and native:

DataLoaders 0: utility for hierarchical EntityPath from file path #4516
DataLoaders 1: introduce, and migrate to, DataLoaders #4517
DataLoaders 2: add text-based DataLoader (.txt, .md) #4518
DataLoaders 3: add 3D point cloud DataLoader (.ply) #4519
DataLoaders 4: add generic folder DataLoader #4520
DataLoaders 5: add support for external binary DataLoaders (PATH) #4521
TODO: register custom loaders
TODO: high level docs and guides for everything related to loading files

Checklist

I have read and agree to Contributor Guide and the Code of Conduct
I've included a screenshot or gif (if applicable)
I have tested the web demo (if applicable):
- Full build: app.rerun.io
- Partial build: app.rerun.io - Useful for quick testing when changes do not affect examples in any way
The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG

teh-cmc · 2023-12-15T08:45:32Z

I forgot to update the DataLoader docstring to link the new loader, will do so in the docs/howto guide PR.

emilk · 2023-12-15T09:44:33Z

crates/re_data_source/src/lib.rs

@@ -49,8 +45,11 @@ pub const SUPPORTED_MESH_EXTENSIONS: &[&str] = &["glb", "gltf", "obj"];

 pub const SUPPORTED_RERUN_EXTENSIONS: &[&str] = &["rrd"];

+pub const SUPPORTED_TEXT_EXTENSIONS: &[&str] = &["txt", "md"];


We should support .rs, .yaml, .json, … etc too. Anything with the text mediatype basically.

Hmmm... I'd rather not list every single language-specific file extension in existence, and we'd get very little value out of it: our rendering of plain text is pretty dull and non-interactive.

Same thing with structured formats (toml, json...), if we cannot make use of their structure, there's very little value in having opening them in plain-text, and I'd rather not list them all...

I think I'll open an issue to introduce a builtin catch-all DataLoader that accepts all things of this nature and logs them as markdown with the proper syntax hint: that way you at least get syntax coloring out of this.

crates/re_types/src/archetypes/text_document_ext.rs

emilk · 2023-12-15T09:46:09Z

crates/re_types/src/archetypes/text_document_ext.rs

+        media_type: Option<impl Into<MediaType>>,
+    ) -> anyhow::Result<Self> {
+        let media_type = media_type.map(Into::into);
+        let media_type = MediaType::or_guess_from_data(media_type, &contents);


What text types can be guessed from data? O.o

$ head whatami <!DOCTYPE html>  <html> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />  <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=no" /> $ file --mime-type whatami ./whatami: text/html

🤷

…#4516) A trivial PR that I extracted into its own in case this ends up being controversial. --- Part of a series of PRs to make it possible to load _any_ file from the local filesystem, by any means, on web and native: - #4516 - #4517 - #4518 - #4519 - #4520 - #4521

**COMMIT PER COMMIT!** _That's the nasty one... there's always a nasty one 😬_ Introduces the `DataLoader` trait, which can load any file, either from a filepath by reading from the local filesystem, or from a pre-loaded file's contents. `DataLoader`s are run in parallel where possible, and have complete say over what data they are interested in or not (i.e. they are not registered based on an extension, which is very limited in practice). I've decided to commit small example assets for the types we support, cause I feel we really need a fast way of checking whether everything still works from time to time. It's pretty light but might be controversial, I figure. Checks: - [x] `cargo r -p rerun-cli --no-default-features --features native_viewer -- examples/assets/example.{glb,gltf,obj,jpg,png,rrd}` - [x] Native: `File > Open > examples/assets/*` - [x] Native: `Drag-n-drop > examples/assets/*` - [x] Web: `File > Open > examples/assets/*` - [x] Web: `Drag-n-drop > examples/assets/*` --- Part of a series of PRs to make it possible to load _any_ file from the local filesystem, by any means, on web and native: - #4516 - #4517 - #4518 - #4519 - #4520 - #4521 - TODO: register custom loaders - TODO: high level docs and guides for everything related to loading files

`.ply` support, but point clouds only, at least for now. No feature flag: I had some, and tbh it only made thing needlessly hard to work with for very little gains. `.ply` are so common anyway, they always come up in user requests. Here's an example from [there](https://github.com/HuangCongQing/Point-Clouds-Visualization/blob/master/2open3D/data/fragment.ply): ![image](https://github.com/rerun-io/rerun/assets/2910679/8242c5a2-037d-46c6-9816-ab9b240a7473) Checks: - [x] `cargo r -p rerun-cli --no-default-features --features native_viewer -- examples/assets/example.{glb,gltf,obj,jpg,png,rrd,txt,md,ply}` - [x] Native: `File > Open > examples/assets/*` - [x] Native: `Drag-n-drop > examples/assets/*` - [x] Web: `File > Open > examples/assets/*` - [x] Web: `Drag-n-drop > examples/assets/*` --- Part of a series of PRs to make it possible to load _any_ file from the local filesystem, by any means, on web and native: - #4516 - #4517 - #4518 - #4519 - #4520 - #4521 - TODO: register custom loaders - TODO: high level docs and guides for everything related to loading files

Load folders filled with whatever. Go nuts. The loader has no particular semantics (e.g. HIVE partitioning): it just loads everything in a folder using the appropriate loader(s) for each file. Checks: - [x] `cargo r -p rerun-cli --no-default-features --features native_viewer -- examples/assets` - [x] Native: `File > Open > examples/assets/` - [x] Native: `Drag-n-drop > examples/assets/` --- Part of a series of PRs to make it possible to load _any_ file from the local filesystem, by any means, on web and native: - #4516 - #4517 - #4518 - #4519 - #4520 - #4521 - TODO: register custom loaders - TODO: high level docs and guides for everything related to loading files

…#4521) Turn any executable in your `PATH` into a `DataLoader` by logging data to stdout. Includes examples for all supported languages. ![image](https://github.com/rerun-io/rerun/assets/2910679/82391961-9ac1-4bd8-bdbe-a7378ed7a3fa) Checks: - [x] `rerun-loader-rust-file` just prints help message - [x] `rerun-loader-rust-file examples/rust/external_data_loader/src/main.rs` successfully does nothing - [x] `rerun-loader-rust-file examples/rust/external_data_loader/src/main.rs | rerun -` works - [x] `rerun-loader-python-file` just prints help message - [x] `rerun-loader-python-file examples/python/external_data_loader/main.py` successfully does nothing - [x] `rerun-loader-python-file examples/python/external_data_loader/main.py | rerun -` works - [x] `rerun-loader-cpp-file` just prints help message - [x] `rerun-loader-cpp-file examples/cpp/external_data_loader/main.cpp` successfully does nothing - [x] `rerun-loader-cpp-file examples/cpp/external_data_loader/main.cpp | rerun -` works - [x] `cargo r -p rerun-cli --no-default-features --features native_viewer -- examples/assets` - [x] Native: `File > Open > examples/assets/` - [x] Native: `Drag-n-drop > examples/assets/` --- Part of a series of PRs to make it possible to load _any_ file from the local filesystem, by any means, on web and native: - #4516 - #4517 - #4518 - #4519 - #4520 - #4521 - TODO: register custom loaders - TODO: high level docs and guides for everything related to loading files

Introduce a first-class protocol for `DataLoader`s -- whether they are builtin, custom, or external -- to announce that they don't support loading a given piece of data. This fixes one of the oldest issue when loading files in Rerun: loading unsupported data would always bottom out in either the image or websocket paths, leading to unrelated errors very confusing to users. The loading process is now a two-pass process: 1. Dispatch the data to be loaded to all loaders, which will respond ASAP whether they can do it or not. 1.1. If at least one compatible loader is found, go to 2 1.2. Otherwise, fail early with a nice error message for the end user 2. Dispatch the actual data loading. This has important/subtle ramifications on the threading model, but other than that is what you'd expect. Checks: - [x] Native: CLI examples/assets/* - [x] Native: CLI examples/assets - [x] Native: CLI examples/assets/* containing unsupported files - [x] Native: CLI examples/assets containing unsupported files - [x] Native: File>Open examples/assets/* - [x] Native: File>Open examples/assets - [x] Native: File>Open examples/assets/* containing unsupported files - [x] Native: File>Open examples/assets containing unsupported files - [x] Native: Drag-n-drop examples/assets/* - [x] Native: Drag-n-drop examples/assets - [x] Native: Drag-n-drop examples/assets/* containing unsupported files - [x] Native: Drag-n-drop examples/assets containing unsupported files - [x] Web: File>Open examples/assets/* - [x] Web: Drag-n-drop examples/assets/* - [x] Web: File>Open examples/assets/* containing unsupported files - [x] Web: Drag-n-drop examples/assets/* containing unsupported files --- Part of a series of PRs to make it possible to load _any_ file from the local filesystem, by any means, on web and native: - #4516 - #4517 - #4518 - #4519 - #4520 - #4521 - #4565 - #4566 - #4567

Add support for registering custom loaders, the same way we offer support for registering custom space views and store subscribers. Checks: - [x] cargo r -p custom_data_loader -- Cargo.toml --- Part of a series of PRs to make it possible to load _any_ file from the local filesystem, by any means, on web and native: - #4516 - #4517 - #4518 - #4519 - #4520 - #4521 - #4565 - #4566 - #4567

Some docs for the whole thing. It heavily relies on linking to the examples for exact details and to avoid rot. It's probably extremely bad. Help welcome. --- Part of a series of PRs to make it possible to load _any_ file from the local filesystem, by any means, on web and native: - #4516 - #4517 - #4518 - #4519 - #4520 - #4521 - #4565 - #4566 - #4567 Co-authored-by: Nikolaus West <[email protected]>

teh-cmc added 📺 re_viewer affects re_viewer itself do-not-merge Do not merge this PR include in changelog CLI Related to the Rerun CLI labels Dec 13, 2023

teh-cmc changed the title ~~DataLoaders 3: add text-based DataLoader (.txt, .md)~~ DataLoaders 2: add text-based DataLoader (.txt, .md) Dec 13, 2023

This was referenced Dec 14, 2023

DataLoaders 0: utility for hierarchical EntityPath from file path #4516

Merged

DataLoaders 1: introduce, and migrate to, DataLoaders #4517

Merged

teh-cmc force-pushed the cmc/dataloaders_1_introduction branch from a035d19 to c830876 Compare December 14, 2023 07:05

teh-cmc force-pushed the cmc/dataloaders_2_text_docs branch 2 times, most recently from 3b58604 to 7b1b30a Compare December 14, 2023 10:52

teh-cmc marked this pull request as ready for review December 14, 2023 10:53

This was referenced Dec 14, 2023

DataLoaders 3: add 3D point cloud DataLoader (.ply) #4519

Merged

DataLoaders 4: add generic folder DataLoader #4520

Merged

DataLoaders 5: add support for external binary DataLoaders (PATH) #4521

Merged

teh-cmc force-pushed the cmc/dataloaders_1_introduction branch from f4c4781 to b0a47fe Compare December 14, 2023 17:25

teh-cmc force-pushed the cmc/dataloaders_2_text_docs branch from 7b1b30a to 21a7fe2 Compare December 14, 2023 17:27

emilk approved these changes Dec 15, 2023

View reviewed changes

teh-cmc force-pushed the cmc/dataloaders_1_introduction branch from b0a47fe to ee5ed0b Compare December 15, 2023 12:09

Base automatically changed from cmc/dataloaders_1_introduction to main December 15, 2023 13:01

teh-cmc added 6 commits December 15, 2023 14:15

add test file example assets

74f5e72

implement supoport for loading text files (txt, md)

7bffbec

update dialogs

36bfd69

cspell shenanigans

2afcb91

more lint shenanigans

643b912

docs

bae6752

teh-cmc force-pushed the cmc/dataloaders_2_text_docs branch from 21a7fe2 to bae6752 Compare December 15, 2023 13:33

teh-cmc mentioned this pull request Dec 15, 2023

Add catch-all builtin DataLoader for text files #4555

Open

issue link

eaf39c6

teh-cmc removed the do-not-merge Do not merge this PR label Dec 15, 2023

shady feature flag business

3c202c8

teh-cmc merged commit 8f4f4e9 into main Dec 15, 2023
21 of 28 checks passed

teh-cmc deleted the cmc/dataloaders_2_text_docs branch December 15, 2023 13:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`DataLoader`s 2: add text-based `DataLoader` (`.txt`, `.md`) #4518

`DataLoader`s 2: add text-based `DataLoader` (`.txt`, `.md`) #4518

teh-cmc commented Dec 13, 2023 •

edited by github-actions bot

Loading

teh-cmc commented Dec 15, 2023

emilk Dec 15, 2023

teh-cmc Dec 15, 2023

emilk Dec 15, 2023

teh-cmc Dec 15, 2023

		@@ -49,8 +45,11 @@ pub const SUPPORTED_MESH_EXTENSIONS: &[&str] = &["glb", "gltf", "obj"];

		pub const SUPPORTED_RERUN_EXTENSIONS: &[&str] = &["rrd"];

		pub const SUPPORTED_TEXT_EXTENSIONS: &[&str] = &["txt", "md"];

DataLoaders 2: add text-based DataLoader (.txt, .md) #4518

DataLoaders 2: add text-based DataLoader (.txt, .md) #4518

Conversation

teh-cmc commented Dec 13, 2023 • edited by github-actions bot Loading

Checklist

teh-cmc commented Dec 15, 2023

emilk Dec 15, 2023

Choose a reason for hiding this comment

teh-cmc Dec 15, 2023

Choose a reason for hiding this comment

emilk Dec 15, 2023

Choose a reason for hiding this comment

teh-cmc Dec 15, 2023

Choose a reason for hiding this comment

`DataLoader`s 2: add text-based `DataLoader` (`.txt`, `.md`) #4518

`DataLoader`s 2: add text-based `DataLoader` (`.txt`, `.md`) #4518

teh-cmc commented Dec 13, 2023 •

edited by github-actions bot

Loading