Make ensure_sufficient_stack() non-generic, using cargo-llvm-lines

[Julian-Wollersberger]

Inspired by this blog post from @nnethercote, I used cargo-llvm-lines on the rust compiler itself, to improve it's compile time. This PR contains only one low-hanging fruit, but I also want to share some measurements.

The function ensure_sufficient_stack() was monomorphized 1500 times, and with it the stacker and psm crates, for a total of 1.5% of all llvm IR lines. With some trickery I convert the generic closure into a dynamic one, and thus all that code is only monomorphized once.

Measurements

Getting these numbers took some fiddling with CLI flags and I modified cargo-llvm-lines to read from a folder instead of invoking cargo. Commands I used:

./x.py clean
RUSTFLAGS="--emit=llvm-ir -C link-args=-fuse-ld=lld -Z self-profile=profile" CARGOFLAGS_BOOTSTRAP="-Ztimings" RUSTC_BOOTSTRAP=1 ./x.py build -i --stage 1 library/std

# Then manually copy all .ll files into a folder I hardcoded in cargo-llvm-lines in main.rs#L115
cd ../cargo-llvm-lines
cargo run llvm-lines

The result is this list (see first 500 lines ), before the change:

  Lines            Copies        Function name
  -----            ------        -------------
  16894211 (100%)  58417 (100%)  (TOTAL)
   2223855 (13.2%)   502 (0.9%)  rustc_query_system::query::plumbing::get_query_impl::{{closure}}
   1331918 (7.9%)   1287 (2.2%)  hashbrown::raw::RawTable<T>::reserve_rehash
    774434 (4.6%)  12043 (20.6%) core::ptr::drop_in_place
    294170 (1.7%)    499 (0.9%)  rustc_query_system::dep_graph::graph::DepGraph<K>::with_task_impl
    245410 (1.5%)   1552 (2.7%)  psm::on_stack::with_on_stack
    210311 (1.2%)      1 (0.0%)  rustc_target::spec::load_specific
    200962 (1.2%)    513 (0.9%)  rustc_query_system::query::plumbing::get_query_impl
    190704 (1.1%)      1 (0.0%)  rustc_middle::ty::query::<impl rustc_middle::ty::context::TyCtxt>::alloc_self_profile_query_strings
    180272 (1.1%)    468 (0.8%)  rustc_query_system::query::plumbing::load_from_disk_and_cache_in_memory
    177396 (1.1%)    114 (0.2%)  rustc_query_system::query::plumbing::force_query_impl
    161134 (1.0%)    445 (0.8%)  rustc_query_system::dep_graph::graph::DepGraph<K>::with_anon_task
    141551 (0.8%)    186 (0.3%)  rustc_query_system::query::plumbing::incremental_verify_ich
    110191 (0.7%)      7 (0.0%)  rustc_middle::ty::context::_DERIVE_rustc_serialize_Decodable_D_FOR_TypeckResults::<impl rustc_serialize::serialize::Decodable<__D> for rustc_middle::ty::context::TypeckResults>::decode::{{closure}}
    108590 (0.6%)    420 (0.7%)  core::ops::function::FnOnce::call_once
     88488 (0.5%)     21 (0.0%)  rustc_query_system::dep_graph::graph::DepGraph<K>::try_mark_previous_green
     86368 (0.5%)      1 (0.0%)  rustc_middle::ty::query::stats::query_stats
     85654 (0.5%)   3973 (6.8%)  <&T as core::fmt::Debug>::fmt
     84475 (0.5%)      1 (0.0%)  rustc_middle::ty::query::Queries::try_collect_active_jobs
     81220 (0.5%)    862 (1.5%)  <hashbrown::raw::RawIterHash<T> as core::iter::traits::iterator::Iterator>::next
     77636 (0.5%)     54 (0.1%)  core::slice::sort::recurse
     66484 (0.4%)    461 (0.8%)  <hashbrown::raw::RawIter<T> as core::iter::traits::iterator::Iterator>::next

All .ll files together had 4.4GB. After my change they had 4.2GB. So a few percent less code LLVM has to process. Hurray!
Sadly, I couldn't measure an actual wall-time improvement. Watching YouTube while compiling added to much noise...

Here is the top of the list after the change:

  16460866 (100%)  58341 (100%)  (TOTAL)
   1903085 (11.6%)   504 (0.9%)  rustc_query_system::query::plumbing::get_query_impl::{{closure}}
   1331918 (8.1%)   1287 (2.2%)  hashbrown::raw::RawTable<T>::reserve_rehash
    777796 (4.7%)  12031 (20.6%) core::ptr::drop_in_place
    551462 (3.4%)   1519 (2.6%)  rustc_data_structures::stack::ensure_sufficient_stack::{{closure}}

Note that the total was reduced by 430 000 lines and psm::on_stack::with_on_stack has disappeared. Instead rustc_data_structures::stack::ensure_sufficient_stack::{{closure}} appeared. I'm confused about that one, but it seems to consist of inlined calls to rustc_query_system::* stuff.

Further note the other two big culprits in this list: rustc_query_system and hashbrown. These two are monomorphized many times, the query system summing to more than 20% of all lines, not even counting code that's probably inlined elsewhere.
Assuming compile times scale linearly with llvm-lines, that means a possible 20% compile time reduction.

Reducing eg. get_query_impl would probably need a major refactoring of the qery system though. Everything in there is generic over multiple types, has associated types and passes generic Self arguments by value. Which means you can't simply make things dyn.

---

This PR is a small step to make rustc compile faster and thus make contributing to rustc less painful. Nonetheless I love Rust and I find the work around rustc fascinating :)

[rust-highfive]

r? @estebank

(rust_highfive has picked a reviewer for you, use r? to override)

[jyn514]

@bors try @rust-timer queue

I wish perf measured compile times for the rust compiler itself :(

[rust-timer]

Awaiting bors try build completion

[bors]

⌛ Trying commit d744dc857ba132dfc9016717864aec7f5d2f2a24 with merge f242334c1e2e719cf1cba923923ad8ec62affb71...

[jyn514]

Inlining is done by llvm, right? Is inline(never) necessary here?

[Julian-Wollersberger]

I don't know. It has no effect on llvm-lines, but I was afraid that it would all be inlined again and thus negating the benefit of this change.
Now thinking about it, that probably won't happen though.

[Julian-Wollersberger]

I removed the #[inline(never)].

[bors]

☀️ Try build successful - checks-actions, checks-azure
Build commit: f242334c1e2e719cf1cba923923ad8ec62affb71 (f242334c1e2e719cf1cba923923ad8ec62affb71)

[rust-timer]

Queued f242334c1e2e719cf1cba923923ad8ec62affb71 with parent 7402a39, future comparison URL.

[rust-timer]

Finished benchmarking try commit (f242334c1e2e719cf1cba923923ad8ec62affb71): comparison url.

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. Please note that if the perf results are neutral, you should likely undo the rollup=never given below by specifying rollup- to bors.

Importantly, though, if the results of this run are non-neutral do not roll this PR up -- it will mask other regressions or improvements in the roll up.

@bors rollup=never

[jyn514]

Oh wow! -2.5% on hello world, -1.5-2% on several other benchmarks.

[jyn514]

I'm curious to see if it would be improved by removing the inline(never) ...

[Marwes]

It might be better to apply this change in stacker itself.

It's internal _grow function could be changed to take a &mut dyn FnMut() https://github.com/rust-lang/stacker/blob/7a1d421a91c3f83d307bbdd86237dc6c9121b130/src/lib.rs#L64-L66 . That way this is only a dynamic call if the stack actually needs to grow (when the overhead is unlikely to matter) but keeps the common path fast (no dynamic call) when the stack does not grow.

[jyn514]

@Marwes won't that prevent using FnOnce with stacker?

[Marwes]

No, because it would be the exact same trick as you do here, wrap the FnOnce in an Option, then take and unwrap in the FnMut.

[Julian-Wollersberger]

It might be better to apply this change in stacker itself.

I'll try that too and we can compare it.

And wow, I'm surprised this is a perf improvement ^^

[Julian-Wollersberger]

I've now moved the conversion to a dyn closure into stacker.
For that I temporarily set the stacker dependency to my fork on GitHub. (I have no idea if CI allows such a hack.)
I also don't have time today to measure llvm-lines again.

If this version turns out to be better, I will open a PR to stacker, and probably change this PR to be just a version bump.

With that, can I have another perf run?

[jyn514]

Relevant commit: Julian-Wollersberger/stacker@71993c7

@bors try @rust-timer queue

[rust-timer]

Awaiting bors try build completion

[bors]

⌛ Trying commit 2b1badd692a5f92acd49ac01bdccb0a92b0020ae with merge 3a9349835672afaadc58a1327d44e3453b34a415...

[Julian-Wollersberger]

(I have no idea if CI allows such a hack.)

No, Tidy doesn't. But bors is still going? I guess it doesn't run tidy?
Anyway, I'm going to bed now.

[jyn514]

No, bors doesn't run tidy (otherwise you'd have to wait to run perf runs until the author ran x.py fmt, which is annoying).

[rust-log-analyzer]

The job dist-x86_64-linux of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

   Compiling num-traits v0.2.12
   Compiling num-integer v0.1.43
   Compiling rustc_lexer v0.1.0 (/checkout/compiler/rustc_lexer)
   Compiling unicode-normalization v0.1.13
   Compiling psm v0.1.11 (https://github.com/Julian-Wollersberger/stacker#71993c7a)
   Compiling stacker v0.1.11 (https://github.com/Julian-Wollersberger/stacker#71993c7a)
   Compiling jemalloc-sys v0.3.2
   Compiling crossbeam-queue v0.1.2
   Compiling rustc_version v0.2.3
   Compiling ena v0.14.0
---
   Compiling num-traits v0.2.12
   Compiling num-integer v0.1.43
   Compiling unicode-normalization v0.1.13
   Compiling crossbeam-queue v0.1.2
   Compiling psm v0.1.11 (https://github.com/Julian-Wollersberger/stacker#71993c7a)
   Compiling stacker v0.1.11 (https://github.com/Julian-Wollersberger/stacker#71993c7a)
   Compiling jemalloc-sys v0.3.2
   Compiling rustc_version v0.2.3
   Compiling smallvec v0.6.13
   Compiling ena v0.14.0
---
   Compiling rustc_version v0.2.3
    Checking smallvec v0.6.13
    Checking rustc_apfloat v0.0.0 (/checkout/compiler/rustc_apfloat)
 Documenting rustc_apfloat v0.0.0 (/checkout/compiler/rustc_apfloat)
   Compiling psm v0.1.11 (https://github.com/Julian-Wollersberger/stacker#71993c7a)
   Compiling stacker v0.1.11 (https://github.com/Julian-Wollersberger/stacker#71993c7a)
    Checking ena v0.14.0
    Checking polonius-engine v0.12.1
    Checking tracing-log v0.1.1
    Checking aho-corasick v0.7.13

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @rust-lang/infra. (Feature Requests)

[jyn514]

On a side note, I have three weeks of free time in October coming up, and plan to spend it programming on rustc. I enjoy optimising things, like in this PR.

I'm just reading Discuss results of 2020 Contributor Survey, where rustc's compile times are mentioned multiple times. And with the llvm-lines measurements in the top comment in mind, and cargo-timings saying that 95% of the compile time of rustc_mir, rustc_middle and others are spent in LLVM, I think optimising llvm-lines has great potential.

More specific, rustc_query_system has way to many generics, but I'd need help tackling that beast. Do you know who I could ping, or were to ask (GitHub, Zulip, Discord)?

Some more places to look at after this:

• Decentralize queries compiler-team#277
• The rustc_query_impl crate is too big, which hurts compile times for the compiler itself #65031
• https://rust-lang.zulipchat.com/#narrow/stream/247081-t-compiler.2Fperformance

Let me know if any of those sound interesting, I don't know how to help with all of them but I know who does ;)

[ecstatic-morse]

Hi! This PR showed up in the weekly perf triage report. It resulted in a moderate improvement in instruction counts (up to -3.2% on incr-full builds of coercions-debug).

A really nice result @Julian-Wollersberger, especially since the main goal of this PR seems to have been to speed up rustc. Interestingly, rustdoc seems to have slowed down somewhat, but historically we don't care very much about its performance compared to actual builds.

Also, thank you for iterating on this, since the first version was indeed a small regression on most "real-world" benchmarks. @jyn514, here's an example of a perf run for a PR that doesn't affect perf (i.e. it's just noise). It's important to look past the first few numbers and see if there's a larger trend, especially if those results aren't for real-world benchmarks.