Stream .rrd files loaded over http #2262
Assignees
Labels
examples
Issues relating to the Rerun examples
🚀 performance
Optimization, memory use, etc
🕸️ web
regarding running the viewer in a browser
Comments
emilk
added
🚀 performance
emilk
added a commit
that referenced
this issue
Jun 14, 2023
Closes #2260 ### What I've updated the list of examples to use at demo.rerun.io. Sizes are pretty good now with JPEG compression, and all of these will be streamed once we have #2262 (except DICOM, which is just one big tensor) ``` arkitscenes: 42.7 MB colmap: 7.6 MB dicom: 67.7 MB mp_pose: 58.2 MB plots: 0.1 MB tracking_hf_opencv: 58.2 MB ``` The CSS here could use some work though in a folllow-up PR:  Some of the names here have been updated, following the lead of #2416 ### How to test it ``` scripts/build_demo_app.py (cd web_demo && basic-http-server -a 0.0.0.0:8080 .) ``` This will also require a manual deploy to `demo.rerun.io` at some point. ### Checklist * [x] I have read and agree to [Contributor Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and the [Code of Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md) <!-- This line will get updated when the PR build summary job finishes. --> PR Build Summary: https://build.rerun.io/pr/2420 <!-- pr-link-docs:start --> Docs preview: https://rerun.io/preview/07dd16b/docs Examples preview: https://rerun.io/preview/07dd16b/examples <!-- pr-link-docs:end -->
emilk
added a commit
that referenced
this issue
Jun 15, 2023
Closes #2260 ### What I've updated the list of examples to use at demo.rerun.io. Sizes are pretty good now with JPEG compression, and all of these will be streamed once we have #2262 (except DICOM, which is just one big tensor) ``` arkitscenes: 42.7 MB colmap: 7.6 MB dicom: 67.7 MB mp_pose: 58.2 MB plots: 0.1 MB tracking_hf_opencv: 58.2 MB ``` The CSS here could use some work though in a folllow-up PR:  Some of the names here have been updated, following the lead of #2416 ### How to test it ``` scripts/build_demo_app.py (cd web_demo && basic-http-server -a 0.0.0.0:8080 .) ``` This will also require a manual deploy to `demo.rerun.io` at some point. ### Checklist * [x] I have read and agree to [Contributor Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and the [Code of Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md) <!-- This line will get updated when the PR build summary job finishes. --> PR Build Summary: https://build.rerun.io/pr/2420 <!-- pr-link-docs:start --> Docs preview: https://rerun.io/preview/07dd16b/docs Examples preview: https://rerun.io/preview/07dd16b/examples <!-- pr-link-docs:end -->
jprochazk
added a commit
that referenced
this issue
Jun 15, 2023
<!-- Open the PR up as a draft until you feel it is ready for a proper review. Do not make PR:s from your own `main` branch, as that makes it difficult for reviewers to add their own fixes. Add any improvements to the branch as new commits to make it easier for reviewers to follow the progress. All commits will be squashed to a single commit once the PR is merged into `main`. Make sure you mention any issues that this PR closes in the description, as well as any other related issues. To get an auto-generated PR description you can put "copilot:summary" or "copilot:walkthrough" anywhere. --> ### What Closes #2262 Instead of waiting for the `.rrd` file to finish downloading, `stream_rrd_from_http` now uses `ehttp::streaming` and parses messages as data is streamed in. As part of this change, compression now uses the LZ4 block format instead of the LZ4 frame format. **This is a breaking change to the RRD format**. | example | `main` | `jan/stream-rrd` | change | | ---------------------- | ---------- | ---------------- | ------- | | signed_distance_fields | 21.328 MB | 21.420 MB | +0.43% | | deep_sdf | 21.329 MB | 21.329 MB | 0.00% | | dicom | 67.657 MB | 67.657 MB | 0.00% | | raw_mesh | 109.070 KB | 110.960 KB | +1.73% | | clock | 54.901 KB | 80.439 KB | +46.52% | | dicom_mri | 67.656 MB | 66.434 MB | -1.81% | | api_demo | 320.549 KB | 388.021 KB | +21.05% | | structure_from_motion | 252.978 MB | 253.735 MB | +0.30% | | car | 402.542 KB | 391.056 KB | -2.85% | | rgbd | 168.420 MB | 167.382 MB | -0.62% | | colmap | 253.054 MB | 253.054 MB | 0.00% | | plots | 149.360 KB | 211.155 KB | +41.37% | | nyud | 168.467 MB | 168.467 MB | 0.00% | | text_logging | 2.732 KB | 3.091 KB | +13.14% | ### Checklist * [ ] I have read and agree to [Contributor Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and the [Code of Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md) * [ ] I've included a screenshot or gif (if applicable) <!-- This line will get updated when the PR build summary job finishes. --> PR Build Summary: https://build.rerun.io/pr/2412 <!-- pr-link-docs:start --> Docs preview: https://rerun.io/preview/0c82335/docs Examples preview: https://rerun.io/preview/0c82335/examples <!-- pr-link-docs:end --> --------- Co-authored-by: Emil Ernerfeldt <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We currently download the entire
.rrdbefore we start parsing it. This means it can be a long delay before a user sees anything on e.g. https://demo.rerun.ioWe are using my
ehttpcrate (since it works on both native and on wasm), so either we find an alternative to it, or we improve it to support chunked downloads.Our
re_log_encodingcrate already supports streaming decoding of the .rrd file format.The text was updated successfully, but these errors were encountered: