Linker-plugin-based LTO: give an explanation how to use linker-plugin-lto with full LTO#151933
Conversation
|
r? @fee1-dead rustbot has assigned @fee1-dead. Use |
|
r? @nnethercote |
There was a problem hiding this comment.
Reasonable info to add, but some extra clarity is needed.
I will cc @bjorn3 also as someone who knows more about LTO than me.
| all the object files being linked were created by LLVM-based toolchains | ||
| using the **same** LTO mode: either ThinLTO or full (also known as fat). | ||
| The examples would be linking Rust code together with | ||
| Clang-compiled C/C++ code and LLVMFlang-compiled Fortran code. |
There was a problem hiding this comment.
| Clang-compiled C/C++ code and LLVMFlang-compiled Fortran code. | |
| Clang-compiled C/C++ code and LLVM Flang-compiled Fortran code. |
This seems to be how it's commonly written?
There was a problem hiding this comment.
Also, I would keep "prime" for C/C++, and then mention that Fortran can also work.
| the C/C++ code with `-flto=thin` so that object files are emitted as LLVM bitcode. | ||
| C/C++/Fortran code compiled with `-flto=full` is also emitted as LLVM bitcode; | ||
| however, this bitcode can be used for interlanguage optimizations only if `rustc` | ||
| is invoked with both `-C linker-plugin-lto` and `-C lto`, enabling full LTO. |
There was a problem hiding this comment.
The standard library is always compiled with thin LTO bitcode afaik. If doing fat lto with linker-plugin-lto needs -Clto=fat, then that did mean that the standard library would not be included in the LTO, which is bad. Why exactly do you need -Clto=fat?
Also please write -Clto=fat rather than -Clto to clarify that it enables fat LTO rather than just any LTO and to future proof for if we ever change the default to Thin LTO.
There was a problem hiding this comment.
The standard library is always compiled with thin LTO bitcode afaik. If doing fat lto with linker-plugin-lto needs -Clto=fat, then that did mean that the standard library would not be included in the LTO, which is bad.
That's actually a very interesting detail for FatLTO users. If this is true, then FatLTO users at least in the linker-based-lto case cannot get full power of the FatLTO.
Why exactly do you need -Clto=fat?
Cannot say for @foxtran but can for my use cases - enable as much as possible optimizations for production binaries for use cases, where increased CI time and memory consumption of FatLTO is totally fine. My use case is C++ + Rust binary built on my beefy CI machines, that later is delivered to thousands of users - I am completely okay with spending more time on building production binary since we have pretty rare releases.
There was a problem hiding this comment.
I meant to say why does -Clto=thin not work for doing fat LTO with a linker plugin? Rustc can do fat LTO with dependencies compiled for thin LTO just fine.
By the way fat LTO may not actually be producing faster executables. Have you actually measured it? Rustc for example is faster when it was compiled with thin LTO than when it was compiled with fat LTO. As I understand, this is in part because thin LTO does a bunch of optimizations that would have been way too slow to do for fat LTO, but are fine for thin LTO as it is already much faster anyway. The only clear cut case where fat LTO wins is on binary size which tends to be a couple percent smaller with fat LTO, though enabling ICF in the linker does reduce the gap a bit.
There was a problem hiding this comment.
I meant to say why does -Clto=thin not work for doing fat LTO with a linker plugin?
That's a good question - I don't know, maybe @foxtran found something during his experiments. From my understanding it should work too.
By the way fat LTO may not actually be producing faster executables. Have you actually measured it?
Specifically for that reason I didn't say "faster" in my comment ;) For my use case the main concern was the binary size, yep. Regarding optimization differences between Thin and Fat - ofc it depends on an application, and should be benched for each case (as it should be with any other compiler optimization).
thin LTO does a bunch of optimizations that would have been way too slow to do for fat LTO, but are fine for thin LTO as it is already much faster anyway
Interesting detail. Is there any place to read about it? Never heard about this difference before. Because FatLTO is not that slow for relatively small binaries, and its runtime overhead is fine for such cases - so disabling some specific optimization due to "too slow" reason sounds a bit like "missed optimization opportunity".
though enabling ICF in the linker does reduce the gap a bit.
Thanks, I forgot (once again) about this linker optimization. For lld seems like it works fine, but I am not sure regarding Mold + ICF (which is my default linker at the moment).
There was a problem hiding this comment.
Why exactly do you need -Clto=fat?
For me that is necessary since LLVM Flang does not support -flto=thin (at least it was so for 21.1.6).
The standard library is always compiled with thin LTO bitcode afaik. If doing fat lto with linker-plugin-lto needs -Clto=fat, then that did mean that the standard library would not be included in the LTO, which is bad.
Oh... Very interesting detail. I will check it at some time. But should it be solved with -Zbuild-std=... passed to cargo?
Also please write -Clto=fat rather than -Clto to clarify that it enables fat LTO rather than just any LTO and to future proof for if we ever change the default to Thin LTO.
Done.
I meant to say why does -Clto=thin not work for doing fat LTO with a linker plugin?
I have no idea. By some reason linker in FatLTO mode can not work with binaries in ThinLTO mode. As far as I could understand, ThinLTO bitcode just have more information than FatLTO bitcode, so, it seems to me that it should not be a problem for linker just delete this information and use such bitcode for FatLTO.
So, ThinLTO + ThinLTO or FatLTO + FatLTO works, other combinations do not.
Also, when I tried manually port bitcode from ThinLTO to FatLTO using their human-readable representation, I found several of bugs in LLVM.
There was a problem hiding this comment.
The standard library is always compiled with thin LTO bitcode afaik.
I took data from .rustup/toolchains/1.93.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib. Hope, it is a correct place where stdlib is located.
There, I see libstd-6455781cbbf0c25c.rlib which is an ar archive. I've extracted it and I see the following:
lib.rmeta: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped
std-6455781cbbf0c25c.std.5414f88c956f4157-cgu.0.rcgu.o: ELF 64-bit LSB relocatable, x86-64, version 1 (GNU/Linux), with debug_info, not stripped
Other files also does not contain LLVM bitcode.
So, looks like by default stdlib is not compiled into LLVM bitcode (maybe, before linking it was so, but not after).
There was a problem hiding this comment.
The object file contains the LLVM bitcode in the .llvmbc section.
There was a problem hiding this comment.
The object file contains the LLVM bitcode in the .llvmbc section.
Thank you! I found it. Really looks like ThinLTO bitcode.
|
Looping in @bjorn3 was the right thing to do :) |
|
@rustbot ready |
There was a problem hiding this comment.
Thanks for the update. Two remaining comments:
- The commits should be squashed into a single commit, because there is no good reason to keep them separate.
- For consistency, please use either "thin LTO"/"fat LTO" or "ThinLTO"/"FatLTO"; I have a slight preference for the former given that this is an English prose document, not code.
5aa8153 to
56df9fe
Compare
|
Some changes occurred to MIR optimizations cc @rust-lang/wg-mir-opt |
This comment has been minimized.
This comment has been minimized.
|
Oops.. Works strange... Sorry.. |
56df9fe to
23c5c60
Compare
|
This PR was rebased onto a different main commit. Here's a range-diff highlighting what actually changed. Rebasing is a normal part of keeping PRs up to date, so no action is needed—this note is just to help reviewers. |
23c5c60 to
cd50e62
Compare
|
@nnethercote, it is ready.
|
|
@bors r+ rollup |
…-lto, r=nnethercote Linker-plugin-based LTO: give an explanation how to use linker-plugin-lto with full LTO Closes rust-lang#138910 The existing linker-plugin-based LTO documentation does not describe the correct usage of full LTO. Specifically, when invoking `rustc` with full LTO, the `-C lto` flag must be passed in addition to `-C linker-plugin-lto`. Also, this PR documents the use of full LTO when linking Rust with Fortran. Unfortunately, LLVM `flang` does not currently support ThinLTO, so full LTO is the only viable option in this case. Toolchain combinations were slightly updated. TODO: - [x] check swiftc compiler. Almost unusable. - [x] check how std lib is actually compiled - [x] add note about LLD and bitcode - [x] report bug to LLVM: llvm/llvm-project#179800 <details> <summary>Swiftc is unusable</summary> https://www.swift.org/install/ gave me LLVM-17. During playing with swift main + rust static library, LLVM-23 removed main :D ```console # thin LTO Rust: rustc --crate-type=staticlib -Clinker-plugin-lto -Copt-level=3 ./ftn.rs # full LTO swift: swiftc -static libftn.a main.swift -lto=llvm-full -O -use-ld=/tmp/test/llvm-project/install/bin/ld.lld -Xlinker --gc-sections -Xlinker --as-needed -o sr ./sr > ftn() returned: 77 # thin LTO swift: swiftc -static libftn.a main.swift -lto=llvm-thin -O -use-ld=/tmp/test/llvm-project/install/bin/ld.lld -Xlinker --gc-sections -Xlinker --as-needed -o sr ./sr > No output ``` </details>
…uwer Rollup of 8 pull requests Successful merges: - #148590 (Stabilize `atomic_try_update`and deprecate `fetch_update` starting 1.99.0) - #150522 (Stabilize new inclusive range type and iterator type) - #152235 (Convert to inline diagnostics in `rustc_parse`) - #152267 (feat: Implement `int_from_ascii` for `NonZero<T>`) - #151576 (Stabilize `core::hint::cold_path`) - #151933 (Linker-plugin-based LTO: give an explanation how to use linker-plugin-lto with full LTO) - #152010 (Ignore all debuginfo tests for LLDB that we do not run in CI) - #152199 (Move `rustc_query_system::cache`.)
Rollup merge of #151933 - foxtran:doc/linker-plugin-lto-full-lto, r=nnethercote Linker-plugin-based LTO: give an explanation how to use linker-plugin-lto with full LTO Closes #138910 The existing linker-plugin-based LTO documentation does not describe the correct usage of full LTO. Specifically, when invoking `rustc` with full LTO, the `-C lto` flag must be passed in addition to `-C linker-plugin-lto`. Also, this PR documents the use of full LTO when linking Rust with Fortran. Unfortunately, LLVM `flang` does not currently support ThinLTO, so full LTO is the only viable option in this case. Toolchain combinations were slightly updated. TODO: - [x] check swiftc compiler. Almost unusable. - [x] check how std lib is actually compiled - [x] add note about LLD and bitcode - [x] report bug to LLVM: llvm/llvm-project#179800 <details> <summary>Swiftc is unusable</summary> https://www.swift.org/install/ gave me LLVM-17. During playing with swift main + rust static library, LLVM-23 removed main :D ```console # thin LTO Rust: rustc --crate-type=staticlib -Clinker-plugin-lto -Copt-level=3 ./ftn.rs # full LTO swift: swiftc -static libftn.a main.swift -lto=llvm-full -O -use-ld=/tmp/test/llvm-project/install/bin/ld.lld -Xlinker --gc-sections -Xlinker --as-needed -o sr ./sr > ftn() returned: 77 # thin LTO swift: swiftc -static libftn.a main.swift -lto=llvm-thin -O -use-ld=/tmp/test/llvm-project/install/bin/ld.lld -Xlinker --gc-sections -Xlinker --as-needed -o sr ./sr > No output ``` </details>
Closes #138910
The existing linker-plugin-based LTO documentation does not describe the correct usage of full LTO. Specifically, when invoking
rustcwith full LTO, the-C ltoflag must be passed in addition to-C linker-plugin-lto.Also, this PR documents the use of full LTO when linking Rust with Fortran. Unfortunately, LLVM
flangdoes not currently support ThinLTO, so full LTO is the only viable option in this case.Toolchain combinations were slightly updated.
TODO:
Swiftc is unusable
https://www.swift.org/install/ gave me LLVM-17. During playing with swift main + rust static library, LLVM-23 removed main :D