Stream `.rrd` files #2412

jprochazk · 2023-06-13T15:17:38Z

What

Instead of waiting for the .rrd file to finish downloading, stream_rrd_from_http now uses ehttp::streaming and parses messages as data is streamed in.

As part of this change, compression now uses the LZ4 block format instead of the LZ4 frame format. This is a breaking change to the RRD format.

example	`main`	`jan/stream-rrd`	change
signed_distance_fields	21.328 MB	21.420 MB	+0.43%
deep_sdf	21.329 MB	21.329 MB	0.00%
dicom	67.657 MB	67.657 MB	0.00%
raw_mesh	109.070 KB	110.960 KB	+1.73%
clock	54.901 KB	80.439 KB	+46.52%
dicom_mri	67.656 MB	66.434 MB	-1.81%
api_demo	320.549 KB	388.021 KB	+21.05%
structure_from_motion	252.978 MB	253.735 MB	+0.30%
car	402.542 KB	391.056 KB	-2.85%
rgbd	168.420 MB	167.382 MB	-0.62%
colmap	253.054 MB	253.054 MB	0.00%
plots	149.360 KB	211.155 KB	+41.37%
nyud	168.467 MB	168.467 MB	0.00%
text_logging	2.732 KB	3.091 KB	+13.14%

Checklist

I have read and agree to Contributor Guide and the Code of Conduct
I've included a screenshot or gif (if applicable)

PR Build Summary: https://build.rerun.io/pr/2412

Docs preview: https://rerun.io/preview/0c82335/docs
Examples preview: https://rerun.io/preview/0c82335/examples

emilk · 2023-06-14T20:14:38Z

That nyud change looks very suspicious - this branch hasn't merged in main with the new jpeg_quality flag that has been added to nyud and colmap.

emilk · 2023-06-14T20:24:58Z

I think its time to add a docstring to re_log_encoding/src/lib.rs explaining how messages are encoded on the wire

emilk

I've done a partial review, and it looks really good so far!

But I think ChunkBuffer has some nasty bugs in it, and I suspect it can be simplified quite a bit.

emilk · 2023-06-14T15:17:46Z

crates/re_log_encoding/Cargo.toml

+ehttp = { git = "https://github.com/jprochazk/ehttp.git", features = [
+  "streaming",
+] }


We should update this to point to the main repo before merge

emilk/ehttp#28 has been merged, so we can update this now

crates/re_log_encoding/src/decoder.rs

crates/re_log_encoding/src/lib.rs

crates/re_log_encoding/src/decoder.rs

crates/re_log_encoding/src/decoder/stream.rs

emilk · 2023-06-14T21:00:43Z

crates/re_log_encoding/src/decoder/stream.rs

+        let mut decoder = StreamDecoder::new();
+        for chunk in data.chunks(1) {
+            decoder.push_chunk(chunk.to_vec());
+        }
+
+        for _ in 0..16 {
+            assert_message_ok!(decoder.try_read());
+        }
+    }


you should also test that you can:

call decoder.try_read() without first pushing any chunks (and get None)

push a few chunks, get a few things out of decoder.try_read(), then None, and then push the rest of the chunks and read the rest of the messages (i.e. actually stream!)

push chunks of various sizes (not just the edge-case of size 1, which never tests the Cursor part of Chunk

call decoder.try_read() without first pushing any chunks (and get None)

This was added to the stream_something_chunks_{compressed,uncompressed} tests

push a few chunks, get a few things out of decoder.try_read(), then None, and then push the rest of the chunks and read the rest of the messages (i.e. actually stream!)

push chunks of various sizes (not just the edge-case of size 1, which never tests the Cursor part of Chunk

I added the following tests for this:

stream_3x16_chunks, which pushes 3 chunks of 16 bytes, then tries to read a message. It continues to do this until it reads all the messages, which are then compared against the input.

stream_irregular_chunks, which does the same thing, but pushes 2 chunks of N bytes at a time, where N comes from a repeating pattern.

crates/re_log_encoding/src/stream_rrd_from_http.rs

Co-authored-by: Emil Ernerfeldt <[email protected]>

emilk · 2023-06-15T12:35:35Z

It would be nice with a debug-level log when we reach the end of the HTTP stream. It will make testing the streaming behavior easier

emilk · 2023-06-15T12:38:30Z

I'm testing this with:

./scripts/build_demo_app.py
(cd web_demo && basic-http-server .)
open http://127.0.0.1:4000/examples/structure_from_motion/

and I get:

[re_viewer::app] re_viewer/src/app.rs:583: data source has left unexpectedly msg.source=http:///examples/structure_from_motion/data.rrd err=Failed to fetch .rrd file from /examples/structure_from_motion/data.rrd: Found an .rrd file from a Rerun version from 0.5.1 or earlier

However, cargo r -p objectron -- --serve works correctly

emilk

Code LGTM (except for wrong error message, see above), but there is something going wrong when testing this (again, see above). I added the do-no-merge label until the testing works.

Also: please update the PR description with the latest (correct) table, and mention the breaking change to the RRD stream format.

crates/re_log_encoding/src/decoder/stream.rs

jprochazk · 2023-06-15T13:24:08Z

I'm testing this with:

I just tested this and it seems to work for me. The build_demo_app script does not build the SDK, but uses it to render the data files. It seems like that could be the problem here, could you run the SDK build, rm -rf web_demo and then try again?

Co-authored-by: Emil Ernerfeldt <[email protected]>

jprochazk · 2023-06-15T13:31:06Z

The error message of OldRrdVersion is outdated

Updated to say 0.6.0 instead of 0.5.1 - I hope that's the right version

It would be nice with a debug-level log when we reach the end of the HTTP stream. It will make testing the streaming behavior easier
...
please update the PR description with the latest (correct) table, and mention the breaking change to the RRD stream format

✅

crates/re_log_encoding/Cargo.toml

emilk · 2023-06-15T14:45:25Z

Seems to be working as advertised 🥳

emilk · 2023-06-15T14:53:53Z

Let's wait for the ehttp release before merging. I will do it momentarily.

…wasm is loading (#2421) ### What Resolves: - #2261 According to the new design, here is how the loading state should work: 1. HTML and CSS loads. It renders the skeleton content without any text. 1. WASM loads. It renders the exact same skeleton content without any text. The user doesn’t notice that the rendering method switched from HTML => WASM. 1. If loading takes long (more than 3 seconds) text fades in and provides additional information. 1. When the App UI is ready, the skeleton content fades away (200ms duration) and the app fades in. The first deployed test of this ended up with 300% progress during the download step, so I've added some extra handling for progress related to compressed streams.At the moment we default to an adhoc compression ratio, but once we land: - #2422 then the progress should be completely accurate again. In order to make all the waiting variants more consistent, this also changes the RRD loading page. The normal waiting page looks like: ![image](https://github.com/rerun-io/rerun/assets/3312232/a8eb2473-7b7c-433a-a0b2-09c6d209c44e) Before: ![image](https://github.com/rerun-io/rerun/assets/3312232/adcf27dc-5c59-4642-8fda-18d0de544442) After: ![image](https://github.com/rerun-io/rerun/assets/3312232/cd4104db-be28-4c32-8171-2b9a9408cf23) This PR does not handle progress indicator during the loading of the RRD itself. The hooks necessary to get access to progress feedback depend on essentially the same work already happening as part of: #2412 and just directly loading the rrd file is generally going to be better than showing a progress indicator. It could be worth revisiting this for things like blueprints with large static data. This needs to be tested manually on our major OS / browser combinations: Builds to test: - https://app.rerun.io/commit/e15ebb5 - https://demo.rerun.io/commit/e15ebb5 If your internet connection is too fast, use developer tools to throttle connection and enable cache. ![image](https://github.com/rerun-io/rerun/assets/3312232/8ba99668-5441-4652-b1f3-9c82c2b6cbb3) - Linux: - [x] Chrome - [x] Firefox - Mac: - [x] Chrome - [x] Firefox - [x] Safari - Windows: - [x] Chrome - [x] Firefox - [x] Edge ### Checklist * [x] I have read and agree to [Contributor Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and the [Code of Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md) * [x] I've included a screenshot or gif (if applicable)  PR Build Summary: https://build.rerun.io/pr/2421  Docs preview: https://rerun.io/preview/934925b/docs Examples preview: https://rerun.io/preview/934925b/examples

…wasm is loading (#2421) Resolves: - #2261 According to the new design, here is how the loading state should work: 1. HTML and CSS loads. It renders the skeleton content without any text. 1. WASM loads. It renders the exact same skeleton content without any text. The user doesn’t notice that the rendering method switched from HTML => WASM. 1. If loading takes long (more than 3 seconds) text fades in and provides additional information. 1. When the App UI is ready, the skeleton content fades away (200ms duration) and the app fades in. The first deployed test of this ended up with 300% progress during the download step, so I've added some extra handling for progress related to compressed streams.At the moment we default to an adhoc compression ratio, but once we land: - #2422 then the progress should be completely accurate again. In order to make all the waiting variants more consistent, this also changes the RRD loading page. The normal waiting page looks like: ![image](https://github.com/rerun-io/rerun/assets/3312232/a8eb2473-7b7c-433a-a0b2-09c6d209c44e) Before: ![image](https://github.com/rerun-io/rerun/assets/3312232/adcf27dc-5c59-4642-8fda-18d0de544442) After: ![image](https://github.com/rerun-io/rerun/assets/3312232/cd4104db-be28-4c32-8171-2b9a9408cf23) This PR does not handle progress indicator during the loading of the RRD itself. The hooks necessary to get access to progress feedback depend on essentially the same work already happening as part of: #2412 and just directly loading the rrd file is generally going to be better than showing a progress indicator. It could be worth revisiting this for things like blueprints with large static data. This needs to be tested manually on our major OS / browser combinations: Builds to test: - https://app.rerun.io/commit/e15ebb5 - https://demo.rerun.io/commit/e15ebb5 If your internet connection is too fast, use developer tools to throttle connection and enable cache. ![image](https://github.com/rerun-io/rerun/assets/3312232/8ba99668-5441-4652-b1f3-9c82c2b6cbb3) - Linux: - [x] Chrome - [x] Firefox - Mac: - [x] Chrome - [x] Firefox - [x] Safari - Windows: - [x] Chrome - [x] Firefox - [x] Edge * [x] I have read and agree to [Contributor Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and the [Code of Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md) * [x] I've included a screenshot or gif (if applicable)  PR Build Summary: https://build.rerun.io/pr/2421  Docs preview: https://rerun.io/preview/934925b/docs Examples preview: https://rerun.io/preview/934925b/examples

@kpreid

…nts - 2023-07-27 (#2842) ## [0.8.0](v0.7.0...v0.8.0) - Infrastructure investments and more transform improvements - 2023-07-27 [Rerun](https://www.rerun.io/) is an easy-to-use visualization toolbox for computer vision and robotics. * Python: `pip install rerun-sdk` * Rust: `cargo add rerun` and `cargo install rerun-cli` * Online demo: <https://demo.rerun.io/version/0.8.0/> ### Overview & Highlights - `log_pinhole` is now easier to use in simple cases and supports non-RDF camera coordinates. [#2614](#2614) - You only need to set focal length and optional principal point instead of setting the full 3x3 matrix. - There is also a new argument: `camera_xyz` for setting the coordinate system. The default is RDF (the old default). This affects the visible camera frustum, how rays are projected when hovering a 2D image, and how depth clouds are projected. - The visualizer can now show coordinate arrows for all affine transforms within the view. [#2577](#2577) - Linestrips and oriented bounding boxes can now be logged via batch APIs in python. - See: `log_linestrips_2d`, `log_linestrips_3d`, [#2822](#2822) and `log_obbs` [#2823](#2823) - Rust users that build their own Viewer applications can now add fully custom Space Views. Find more information [here](https://www.rerun.io/docs/howto/extend-ui#custom-space-views-classes). - New optional `flush_timeout` specifies how long Rerun will wait if a TCP stream is disconnected during a flush. [#2821](#2821) - In Rust, `RecordingStream::connect` now requires `flush_timeout` specified as an `Option<Duration>`. - To keep default behavior, this can be specified using the `rerun::default_flush_time()` helper. - In Python `flush_init_sec` is now an optional argument to `rr.connect()` - In Rust, the `RecordingStream` now offers a stateful time API, similar to the Python APIs. [#2506](#2506) - You can now call `set_time_sequence`, `set_time_seconds`, and `set_time_nanos` directly on the `RecordingStream`, which will set the time for all subsequent logs using that stream. - This can be used as an alternative to the the previous `MsgSender::with_time` APIs. - The Rerun SDK now defaults to 8ms long microbatches instead of 50ms. This makes the default behavior more suitable for use-cases like real-time video feeds. [#2220](#2220) - Check out [the microbatching docs](https://www.rerun.io/docs/reference/sdk-micro-batching) for more information on fine-tuning the micro-batching behavior. - The web viewer now incremental loads `.rrd` files when streaming over HTTP. [#2412](#2412) ![Open Photogrammetry Preview](https://static.rerun.io/9fa26e73a197690e0403cd35f29e31c2941dea36_release_080_photogrammetry_full.png) ### Ongoing Refactors - There have been a number of significant internal changes going on during this release with little visible impact. This work will land across future releases, but is highlighted here since much of it is visible through the changelog. - The layout of the Viewer is now controlled by a Blueprint datastore. In the future this will allow for direct API control of the layout and configuration of the Viewer. A very early prototype of this functionality is available via the `rerun.experimental` module in Python. - An entirely new code-generation framework has been brought online for Rust, Python and C++. This will eventually enable new object-centric APIs with a more scalable, consistent, and ergonomic experience. - Bringup of C++ support is now underway and will eventually become our third officially supported SDK language. ### Known Regressions - Due to the Blueprint storage migration, blueprint persistence on web is currently broken. Will be resolved in: [#2579](#2579) ### In Detail #### 🐍 Python SDK - Clean up warnings printed when `rr.init` hasn't been called [#2209](#2209) - Normalise Python typing syntax to 3.8+ [#2361](#2361) - Simpler, sturdier stateful time tracking in both SDKs [#2506](#2506) - Fix not taking np.array for single colors [#2569](#2569) - Add a basic pyright config [#2610](#2610) - Improve `log_pinhole` and support non-RDF pinholes [#2614](#2614) - Expose batch APIs for linestrips [#2822](#2822) - Expose batch APIs for oriented bounding boxes [#2823](#2823) #### 🦀 Rust SDK - Add example for adding custom Space Views [#2328](#2328) - Simpler, sturdier stateful time tracking in both SDKs [#2506](#2506) - Automagic flush when `take()`ing a `MemorySinkStorage` [#2632](#2632) - Logging SDK: Log warnings if user data is dropped [#2630](#2630) - Add support for `RecordingStream::serve` [#2815](#2815) #### 🌁 Viewer Improvements - Better handle scroll-to-zoom in 3D views [#1764](#1764) - Add command to screenshot the application [#2293](#2293) - Show layout in blueprint tree view [#2465](#2465) - Double-click to select entity [#2504](#2504) - Add Rerun.io link/text in top bar [#2540](#2540) - New auto-layout of space views [#2558](#2558) - Add 'Dump datastore' command to palette [#2564](#2564) - Support any `dtype` for depth images [#2602](#2602) - Change "Save Selection" command to Cmd+Alt+S [#2631](#2631) - Consistent transform visualization for all entities with transforms [#2577](#2577) - Improve `log_pinhole` and support non-RDF pinholes [#2614](#2614) #### 🚀 Performance Improvements - Flush the batches every 8ms instead of 50 ms [#2220](#2220) - Replace `image` crate jpeg decoder with zune-jpeg [#2376](#2376) - Stream `.rrd` files when loading via http [#2412](#2412) #### 🪳 Bug Fixes - Fix deadlock when misusing the Caches [#2318](#2318) - Fix unstable order/flickering of "shown in" space view list on selection [#2327](#2327) - Fix transforms not applied to connections from transform context [#2407](#2407) - Fix texture clamping and color gradient selection being displayed incorrectly [#2394](#2394) - Fix projected ray length [#2482](#2482) - Tweak the depth bias multiplier for WebGL [#2491](#2491) - Clip image zoom rectangle [#2505](#2505) - Fix missing feature flags for benchmarks [#2515](#2515) - `run_all.py` script fixes [#2519](#2519) - Update egui_tiles with fix for drag-and-drop-panic [#2555](#2555) - Convert objectron proto.py back to using typing.List [#2559](#2559) - Exclude from `objectron/proto/objectron/proto.py` from `just py-format` [#2562](#2562) - Fix pinhole visualization not working with camera extrinsics & intrinsics on the same path [#2568](#2568) - Fix: always auto-layout spaceviews until the user interveens [#2583](#2583) - Fix freeze/crash when logging large times [#2588](#2588) - Update egui_tiles to fix crash [#2598](#2598) - Fix clicking object with single instance (of every component) selecting instance instead of entity [#2573](#2573) - Cleanup internal data-structures when process has been forked [#2676](#2676) - Fix shutdown race-condition by introducing a flush_timeout before dropping data [#2821](#2821) - Fix ui-scale based point/line sizes incorrectly scaled when zooming based on horizontal dimension [#2805](#2805) - Fix visibility toggle for maximized Space Views [#2806](#2806) - Fix loading file via CLI [#2807](#2807) - Fix disconnected space APIs in Python SDK [#2832](#2832) - Avoid unwrap when generating authkey [#2804](#2804) #### 🧑‍🏫 Examples - Add example template [#2392](#2392) - Show hidden url search param in `app.rerun.io` [#2455](#2455) - Minimal example of running an intel realsense depth sensor live [#2541](#2541) - Add a simple example to display Open Photogrammetry Format datasets [#2512](#2512) - Move `examples/api_demo` -> `tests/test_api` [#2585](#2585) #### 📚 Docs - Docs: link to `rr.save` and suggest `rerun` instead of `python -m rerun` [#2586](#2586) - Update docs about transforms [#2496](#2496) - Fixup remaining usages of log_rigid3 in docs [#2831](#2831) #### 🎨 Renderer Improvements - Expose type erased draw data that can be consumed directly [#2300](#2300) - Use less `mut` when using `RenderContext` [#2312](#2312) #### 🧑‍💻 Dev-experience - Better error messages in build.rs [#2173](#2173) - Recommend sccache in CONTRIBUTING.md [#2245](#2245) - introduce `re_tracing` [#2283](#2283) - lint: standardize formatting of let-else-return statements [#2297](#2297) - Centralized build tools in `re_build_tools` [#2331](#2331) - Lint for explicit quotes [#2332](#2332) - Added example screenshot instructions in `just upload --help` [#2454](#2454) - Added support for puling image from an URL to `upload_image.py` [#2462](#2462) - `setup_dev.sh` now installs pngcrush [#2470](#2470) - Added docs/code-examples to the directories checked by py-lint and py-format [#2476](#2476) - Link to demo in PR + check checkboxes [#2543](#2543) - Add script to find external issues we haven't commented on [#2532](#2532) - Move CI-related scripts to its own folder [#2561](#2561) - Render PR description as template [#2563](#2563) - Add basic testing automation against all version of Python using nox [#2536](#2536) - Run clippy on public API too [#2596](#2596) - Bump all `py-lint`-related package versions [#2600](#2600) - Crates publishing script [#2604](#2604) - Fix rust docs deploy [#2615](#2615) - Add support for .gitignore to scripts/lint.py [#2666](#2666) #### 🗣 Refactors - Refactor space-view dependencies: - Move spatial space view to its own crate [#2286](#2286) - Separate crate for bar chart space view [#2322](#2322) - Separate crate for time series space view [#2324](#2324) - Separate crate for tensor space view [#2334](#2334) - Separate viewport related files out to a new re_viewport crate [#2251](#2251) - Remove timepanel dependency from viewport [#2256](#2256) - New trait system for SpaceViews: - Initial Space View trait & port of text space views to the new Space View trait system [#2281](#2281) - Extend/iterate on SpaceViewClass framework with SceneContext & port SpatialSpaceView scene parts [#2304](#2304) - Finalize move of SpatialSpaceView to SpaceViewClass trait framework [#2311](#2311) - Typename cleanup in SpaceViewClass framework [#2321](#2321) - Automatic fallback for unrecognized Space View Class, start removing old ViewCategory [#2357](#2357) - Rename ScenePart -> ViewPartSystem + related renamings [#2674](#2674) - Dynamically registered space view (part/context) systems [#2688](#2688) - Viewer's command queue is now a channel, allowing to queue commands without mutable access [#2339](#2339) - Break up app.rs into parts [#2303](#2303) - Break out `re_log_types::component_types` as `re_components` [#2258](#2258) - Introduce StoreHub and rename Recording->Store [#2301](#2301) - Move StoreHub out of the Viewer during Update [#2330](#2330) - Expand CommandSender to support SystemCommand [#2344](#2344) - Use `camino` crate for UTF8 paths in `re_types_builder` [#2637](#2637) - Separate 2d & 3d spaceview classes, removal of `ViewCategory`, `SpaceViewClass` driven spawn heuristics [#2716](#2716) - Move object property heuristics to heuristics.rs [#2764](#2764) #### 📦 Dependencies - Version `rand` & friends at workspace level [#2508](#2508) - Update to PyO3 0.19 [#2350](#2350) - Pin `half` to `2.2.1` [#2587](#2587) #### 📘 Blueprint Changes - Drive blueprints off of a DataStore [#2010](#2010) - Split SpaceView -> SpaceViewState + SpaceViewBlueprint [#2188](#2188) - Split the Blueprint into AppBlueprint and ViewportBlueprint [#2358](#2358) - Swap the naming of Viewport and ViewportBlueprint [#2595](#2595) - Basic persistence for blueprints [#2578](#2578) #### 🏭 New Codegen Framework - Codegen/IDL 1: add more build tools [#2362](#2362) - Codegen/IDL 2: introduce `re_types_builder` [#2363](#2363) - Codegen/IDL 3: introduce `re_types` [#2369](#2369) - Codegen/IDL 4: definitions for a `Points2D` archetype [#2370](#2370) - Codegen/IDL 5: auto-generated Python code for `Points2D` [#2374](#2374) - Codegen/IDL 7: handwritten Python tests and extensions for `Points2D` [#2410](#2410) - Codegen/IDL 6: auto-generated Rust code for `Points2D` [#2375](#2375) - Codegen/IDL 8: handwritten Rust tests and extensions for `Points2D` [#2432](#2432) - Codegen'd Rust/Arrow 1: upgrading to actual `TokenStream`s [#2484](#2484) - Codegen'd Rust/Arrow 2: matching legacy definitions [#2485](#2485) - Codegen'd Rust/Arrow 3: misc fixes & improvements [#2487](#2487) - Codegen'd Rust/Arrow 4: out-of-sync definitions CI detection [#2545](#2545) - Codegen'd Rust/Arrow 5: doc, definitions and regression tests for combinatorial affixes [#2546](#2546) - Codegen'd Rust/Arrow 6: serialization [#2549](#2549) - Codegen'd Rust/Arrow 7: deserialization [#2554](#2554) - Codegen'd Rust/Arrow 8: carry extension metadata across transparency layers [#2570](#2570) - Codegen'd Rust/Arrow 9: Rust backport! [#2571](#2571) - End-to-end cross-language roundtrip tests for our archetypes [#2601](#2601) - Automatically derive `Debug` and `Clone` in Rust backend [#2613](#2613) - Generating (de)serialization code for dense unions in Rust backend [#2626](#2626) - Fix `FixedSizeList` deserialization edge-case + trivial optimizations [#2673](#2673) - Make `Datatype` & `Component` both inherit from `Loggable` [#2677](#2677) - Roundtrip-able `Transform3D`s [#2669](#2669) - Don't inline recursive datatypes in Rust backend [#2760](#2760) - Automatically derive `tuple_struct` attr and trivial `From` impls where possible [#2772](#2772) - Introduce roundtrip-able `Points3D` archetype (py + rs) [#2774](#2774) - Add `fmt::Debug` implementations to various types. [#2784](#2784) (thanks [@kpreid](https://github.com/kpreid)!) - Isolate testing types in Rust backend [#2810](#2810) - Fix out-of-sync codegen hash [#2567](#2567) - Python backport: add `log_any()` [#2581](#2581) - Integrate unit examples into codegen stack [#2590](#2590) - Disable codegen on windows [#2592](#2592) - Python codegen: big cleaning and paving the way towards transforms [#2603](#2603) - Automatically assume arrow transparency for components [#2608](#2608) - Fix wrong path being `rerun_if_changed()` in `compute_dir_hash` [#2612](#2612) - Support transparency at the semantic layer [#2611](#2611) - Don't use builtin `required` anymore, introduce `nullable` instead [#2619](#2619) - Rust codegen: generate proper docstrings [#2668](#2668) - Support nullable Arrow unions using virtual union arms [#2708](#2708) - Introduce support for querying Archetypes [#2743](#2743) - Introduce legacy shims and migrate DataCell to re_types::Component [#2752](#2752) #### 🌊 Starting work on C++ - Seed of C and C++ SDKs [#2594](#2594) - Move C++ SDK to own folder [#2624](#2624) - C++ codegen [#2678](#2678) - C++ codegen for reporting arrow data type for structs [#2756](#2756) - Don't inline recursive datatypes in C++ backend [#2765](#2765) - C++ codegen to_arrow_data_type for unions [#2766](#2766) - C++ codegen arrow serialize non-union components/datatypes without nested rerun types [#2820](#2820) - C++ codegen of structs and unions [#2707](#2707) - Fix cpp formatter differences [#2773](#2773) #### 🤷‍♂️ Other - test_api: set different app_id based on what test is run [#2599](#2599) - Introduce `rerun compare` to check whether 2 rrd files are functionally equivalent [#2597](#2597) - Remove `files.exclude` in vscode settings [#2621](#2621) - Support feature-gated rust attributes [#2813](#2813) ### Checklist * [x] I have read and agree to [Contributor Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and the [Code of Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md) * [x] I've included a screenshot or gif (if applicable) * [x] I have tested [demo.rerun.io](https://demo.rerun.io/pr/2842) (if applicable) - [PR Build Summary](https://build.rerun.io/pr/2842) - [Docs preview](https://rerun.io/preview/pr%3Arelease-0.8/docs) - [Examples preview](https://rerun.io/preview/pr%3Arelease-0.8/examples)

jprochazk added 5 commits June 12, 2023 14:26

temp

46c1a34

Merge branch 'main' into jan/stream-rrd

1643efc

Merge branch 'main' into jan/stream-rrd

1d2efde

start implementing stream decoder

15d726c

remove prints

f0a5aad

jleibs mentioned this pull request Jun 13, 2023

When loading on web, match style and show a progress indicator while wasm is loading #2421

Merged

10 tasks

jprochazk added 10 commits June 14, 2023 09:20

Merge branch 'main' into jan/stream-rrd

46e1432

update ehttp

4461bd4

use ops::ControlFlow

ace4ab8

finish each chunk during encoding

d27c313

read_options from unsized slice

18ab421

simplify stream decoder

ecf93d8

do not filter tests in launch.json

999a3ff

update ehttp commit

b222755

use lz4 blocks and simplify stream decoder further

41e4ce2

Merge branch 'main' into jan/stream-rrd

ec52b1c

jprochazk marked this pull request as ready for review June 14, 2023 15:08

jprochazk added enhancement New feature or request 📉 performance Optimization, memory use, etc labels Jun 14, 2023

jprochazk added 2 commits June 14, 2023 17:12

fix lints

c60c2be

update ehttp after rebase on upstream master

c89074f

emilk reviewed Jun 14, 2023

View reviewed changes

emilk requested changes Jun 15, 2023

View reviewed changes

crates/re_log_encoding/src/stream_rrd_from_http.rs Show resolved Hide resolved

crates/re_log_encoding/src/stream_rrd_from_http.rs Outdated Show resolved Hide resolved

jprochazk and others added 5 commits June 15, 2023 10:08

Merge branch 'main' into jan/stream-rrd

daf3f75

update to RRF2

d51f9fd

update naming + add comment about message header len equality

65fcd39

some comments about state machine

b536af7

Update crates/re_log_encoding/src/decoder/stream.rs

f3c661e

Co-authored-by: Emil Ernerfeldt <[email protected]>

jprochazk added 2 commits June 15, 2023 15:07

update old rrd version message

31f4cae

add stream end debug log

3a87459

emilk approved these changes Jun 15, 2023

View reviewed changes

crates/re_log_encoding/src/decoder/stream.rs Show resolved Hide resolved

crates/re_log_encoding/src/decoder/stream.rs Outdated Show resolved Hide resolved

emilk added the do-not-merge Do not merge this PR label Jun 15, 2023

jprochazk and others added 3 commits June 15, 2023 15:27

switch ehttp to emilk/ehttp@master

0e91dc2

Update crates/re_log_encoding/src/decoder/stream.rs

f3d2bf5

Co-authored-by: Emil Ernerfeldt <[email protected]>

early-out on empty chunks

b80731a

Merge branch 'main' into jan/stream-rrd

38d3403

emilk reviewed Jun 15, 2023

View reviewed changes

crates/re_log_encoding/Cargo.toml Outdated Show resolved Hide resolved

emilk added 4 commits June 15, 2023 16:35

pin the ehttp version

ae5664c

update Cargo.lock

8b2cf30

Improve OldRrdVersion error message

14efd77

build_demo_app.py: build Python SDK before running examples

12c645f

emilk removed the do-not-merge Do not merge this PR label Jun 15, 2023

emilk approved these changes Jun 15, 2023

View reviewed changes

jprochazk added 2 commits June 15, 2023 16:47

update formatting

f5f44f3

Merge branch 'main' into jan/stream-rrd

e5026f0

update ehttp

0c82335

jprochazk merged commit 2514b72 into main Jun 15, 2023

jprochazk deleted the jan/stream-rrd branch June 15, 2023 15:39

jleibs mentioned this pull request Jul 27, 2023

Release 0.8 - Infrastructure investments and more transform improvements - 2023-07-27 #2842

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Stream `.rrd` files #2412

Stream `.rrd` files #2412

jprochazk commented Jun 13, 2023 •

edited by github-actions bot

Loading

emilk commented Jun 14, 2023 •

edited

Loading

emilk commented Jun 14, 2023

emilk left a comment

emilk Jun 14, 2023

emilk Jun 15, 2023

emilk Jun 14, 2023

jprochazk Jun 15, 2023 •

edited

Loading

emilk commented Jun 15, 2023

emilk commented Jun 15, 2023

emilk left a comment •

edited

Loading

jprochazk commented Jun 15, 2023

jprochazk commented Jun 15, 2023

emilk commented Jun 15, 2023

emilk commented Jun 15, 2023

Stream .rrd files #2412

Stream .rrd files #2412

Conversation

jprochazk commented Jun 13, 2023 • edited by github-actions bot Loading

What

Checklist

emilk commented Jun 14, 2023 • edited Loading

emilk commented Jun 14, 2023

emilk left a comment

Choose a reason for hiding this comment

emilk Jun 14, 2023

Choose a reason for hiding this comment

emilk Jun 15, 2023

Choose a reason for hiding this comment

emilk Jun 14, 2023

Choose a reason for hiding this comment

jprochazk Jun 15, 2023 • edited Loading

Choose a reason for hiding this comment

emilk commented Jun 15, 2023

emilk commented Jun 15, 2023

emilk left a comment • edited Loading

Choose a reason for hiding this comment

jprochazk commented Jun 15, 2023

jprochazk commented Jun 15, 2023

emilk commented Jun 15, 2023

emilk commented Jun 15, 2023

Stream `.rrd` files #2412

Stream `.rrd` files #2412

jprochazk commented Jun 13, 2023 •

edited by github-actions bot

Loading

emilk commented Jun 14, 2023 •

edited

Loading

jprochazk Jun 15, 2023 •

edited

Loading

emilk left a comment •

edited

Loading