DataLoaders 1: introduce, and migrate to, DataLoaders

[teh-cmc]

COMMIT PER COMMIT!

That's the nasty one... there's always a nasty one 😬

Introduces the DataLoader trait, which can load any file, either from a filepath by reading from the local filesystem, or from a pre-loaded file's contents.

DataLoaders are run in parallel where possible, and have complete say over what data they are interested in or not (i.e. they are not registered based on an extension, which is very limited in practice).

I've decided to commit small example assets for the types we support, cause I feel we really need a fast way of checking whether everything still works from time to time.
It's pretty light but might be controversial, I figure.

Checks:

• cargo r -p rerun-cli --no-default-features --features native_viewer -- examples/assets/example.{glb,gltf,obj,jpg,png,rrd}
• Native: File > Open > examples/assets/*
• Native: Drag-n-drop > examples/assets/*
• Web: File > Open > examples/assets/*
• Web: Drag-n-drop > examples/assets/*

---

Part of a series of PRs to make it possible to load any file from the local filesystem, by any means, on web and native:

• DataLoaders 0: utility for hierarchical EntityPath from file path #4516
• DataLoaders 1: introduce, and migrate to, DataLoaders #4517
• DataLoaders 2: add text-based DataLoader (.txt, .md) #4518
• DataLoaders 3: add 3D point cloud DataLoader (.ply) #4519
• DataLoaders 4: add generic folder DataLoader #4520
• DataLoaders 5: add support for external binary DataLoaders (PATH) #4521
• TODO: register custom loaders
• TODO: high level docs and guides for everything related to loading files

Checklist

• I have read and agree to Contributor Guide and the Code of Conduct
• I've included a screenshot or gif (if applicable)
• I have tested the web demo (if applicable):
  • Full build: app.rerun.io
  • Partial build: app.rerun.io - Useful for quick testing when changes do not affect examples in any way
• The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG

• PR Build Summary
• Docs preview
• Examples preview
• Recent benchmark results
• Wasm size tracking

[teh-cmc]

If I log these times, the space views disappear and cannot even be created manually :(

cc @Wumpf @jleibs

[emilk]

Very cool!

I am a bit unclear how a specific loader is chosen, if there are several.
Relatedly, I am also concerned by the use of Ok(()) to mean to different things (successful load and unsupported file extension).

The DataLoader trait doesn't at all support streaming loads from e.g. HTTP, which is a bit of a shame. Of course, not everything needs to be solved right away, but it would be nice to at least acknowledge that up as a shortcoming somewhere (e.g. for the doscstring of DataLoader).

[emilk]

We could have a default implementation that opens the file, then calls load_from_file_contents

[teh-cmc]

I don't want to go there because the former accepts folder paths while the latter doesn't, and I don't want to do some automagic filtering of folders on behalf of the user...

[emilk]

it seems wrong to me that we use Ok(()) to mean two very different things:

• I successfully loaded the file
• Unsupported file format / I didn't try to load the file

In the egui data loaders we have a specific error message for unsupported file extensions/types. That's in fact how we find the correct loader: try them all until one doesn't return the NotSupported error

[emilk]

For reference: https://docs.rs/egui/latest/egui/load/index.html

[teh-cmc]

Note: we don't have a notion of the one correct loader in our case: there can be more than one.

I'll make a follow up PR with unsupported errors (for builtin) and exit codes (for externals), with some extra logic to return an actual error in the end if no loader whatsoever could be found.

[teh-cmc]

I am a bit unclear how a specific loader is chosen, if there are several.

All applicable loaders are run, at which point they might:

• write to the same recording, or not,
• write to the same entity, or not,
• write to the same entity and component, or not,
• etc

I'll try and document this.

Relatedly, I am also concerned by the use of Ok(()) to mean to different things (successful load and unsupported file extension).

Hmm, good point. I didn't want to go there until external loaders were in; now that we have the full picture in place, I think I can make an extra PR to make it happen for both builtin/custom and external loaders.

The DataLoader trait doesn't at all support streaming loads from e.g. HTTP, which is a bit of a shame. Of course, not everything needs to be solved right away, but it would be nice to at least acknowledge that up as a shortcoming somewhere (e.g. for the doscstring of DataLoader).

Yes DataLoader is only for local filesystem stuff as a start, HTTP (and more) is part of #4525; i'll make that clearer in the docstring.

• DataLoaders should support arbitrary URIs #4525