Improve compile times

[Wumpf]

(edited by @emilk)

Improve build times, and aspire to reduce number of dependencies and code bloat in the process.

Some useful tools include:

• cargo clean && cargo build --timings -p rerun
• cargo tree
• cargo machete
• https://github.com/google/bloaty
• https://github.com/RazrFalcon/cargo-bloat
• https://github.com/dtolnay/cargo-llvm-lines
• cargo udeps to find unused dependencies

See also:

• https://fasterthanli.me/articles/why-is-my-rust-build-so-slow

Reducing dependencies

Some ideas:

• replace clap with something simpler, like pico-args
• replace sha2 with a simpler hash
• replace chrono with time
• replace image crate with https://github.com/etemesi254/zune-image
• re_web_server should be able to use something must simpler than hyper+tokio
• replace lazy_static with once_cell ?
• remove strum
• enumset can be replaced with something simpler (maybe by macro_rules!)
• reduce naga features?
• crossbeam (std::sync::mpsc is now using the same implementation) - though crossbeam compiles in 2s, so probably not worth it

Done:

• Remove extreneous image features and dependencies #1425
• Replace reqwest with ureq #1407
• Remove derive_more crate #1406

Other tasks

• Get everything to link dynamically on native debug builds #347
• Track build time and number of dependencies over time like we do for benchmarks
• Add cargo-deny to the CI of other crates, like arboard, to prevent duplicate dependencies of e.g. nix
• In puffin, add support for lz4_flex: Compression benchmarks EmbarkStudios/puffin#130 (comment)

Other

We have some instructions for improving build times in https://github.com/rerun-io/rerun/blob/main/BUILD.md#improving-compile-times

[Wumpf]

rebuild on dev with M1 max via cargo clean && cargo build --timings

cargo-timing-20230215T130337Z.html.zip

Quite surprised about zlib being so slow here! Investigating alternatives!

[Wumpf]

very similar results on second run

[Wumpf]

Getting rid of zstd is somewhat hard - it is just really really good it seems EmbarkStudios/puffin#130

[emilk]

I think we can switch from zstd to lz4_flex. To get the benefits we must first remove zstd from our puffin dependency:

• Let user choose compression algorithm EmbarkStudios/puffin#131

EDIT:

• Switch default compression algorithm to lz4 EmbarkStudios/puffin#135
• Switch compression algorithm from zstd to lz4 #2112

[emilk]

Current state on my M1 MacBook Pro

sccache --stop-server && rm -rf /Users/emilk/Library/Caches/Mozilla.sccache && sccache --start-server
cargo clean && cargo build --timings -p rerun

2m 12s

[emilk]

This halves compilation times for users only using the rerun crate as a logging library:

• cargo install rerun-cli #2183

[emilk]

Things has improved - cargo clean && cargo build --timings -p rerun on my M1 MacBook Pro:

78.8s and 353 dependencies

[Wumpf]

Status update: On c5f8176 (somewhere between 0.16 and 0.17 ;)

M1max MacBook Pro (very similar to emils machine above)
66.5s and 336 dependencies

re_types is the biggest unit with quite some distance, otherwise the next logical thing would be to try to get rid of serde (should be in reach for the sdk!)