Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DataLoader drag-n-drop streaming support #4554

Open
teh-cmc opened this issue Dec 15, 2023 · 0 comments
Open

DataLoader drag-n-drop streaming support #4554

teh-cmc opened this issue Dec 15, 2023 · 0 comments
Labels
🚀 performance Optimization, memory use, etc 📺 re_viewer affects re_viewer itself

Comments

@teh-cmc
Copy link
Member

teh-cmc commented Dec 15, 2023

Currently, the FileContents path taken when drag-n-dropping on the web (I think it's only ever taken on the web, right? hard to speak for all platforms and desktop environments out there) loads the entire file at once instead of streaming it in piece by piece when possible (e.g. rrd).

We should consider how to stream-load data. The naive approach fn load_from_reader(&self, read: &mut dyn std::io::Read, …) doesn't work easily on the web, since you cannot block on a read. You can take a look at crates/re_log_encoding/src/stream_rrd_from_http.rs for the horror complexity required in streaming data on the web. That uses ehttp, which wraps an JS-async reader, by registering callbacks on each read chunk of data.

When drag-dropping a file into eframe we also get an JS-async reader, but eframe "helpfully" loads the entire things for us (https://github.com/emilk/egui/blob/c8dd3dd01a7ea40b95925b8de11ecfa981eaa738/crates/eframe/src/web/events.rs#L531-L538).

It should all probably be doable with an async function (async traits coming soon!) reading from an async reader.

Anyways, I am NOT suggesting we handle that now, but perhaps add a note for future generations who are interested in streaming data on drag-drop instead of waiting for the full file to load before it is shown.

@teh-cmc teh-cmc added 📺 re_viewer affects re_viewer itself 🚀 performance Optimization, memory use, etc labels Dec 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🚀 performance Optimization, memory use, etc 📺 re_viewer affects re_viewer itself
Projects
None yet
Development

No branches or pull requests

1 participant