Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rust 1.59 rustc greatly increased compile time with include_str! #94390

Open
sstangl opened this issue Feb 26, 2022 · 8 comments
Open

Rust 1.59 rustc greatly increased compile time with include_str! #94390

sstangl opened this issue Feb 26, 2022 · 8 comments
Assignees
Labels
A-LLVM Area: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues. C-bug Category: This is a bug. I-compiletime Issue: Problems and improvements with respect to compile times. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.

Comments

@sstangl
Copy link
Contributor

sstangl commented Feb 26, 2022

The following repo/hash compiles with rustc v1.58, but loops forever in v1.59 and Nightly (1.61): sstangl/openpowerlifting@ac057e3

The problem appears to be with the compilation of a very small crate (modules/langpack) that uses include_str! to include some JSON files. The problematic method is the implementation of default():

    fn default() -> LangInfo {
        LangInfo {
            cz: json::from_str(include_str!("../translations/cz.json")).expect("cz"),
            de: json::from_str(include_str!("../translations/de.json")).expect("de"),
            el: json::from_str(include_str!("../translations/el.json")).expect("el"),
            en: json::from_str(include_str!("../translations/en.json")).expect("en"),
            eo: json::from_str(include_str!("../translations/eo.json")).expect("eo"),
            es: json::from_str(include_str!("../translations/es.json")).expect("es"),
            fi: json::from_str(include_str!("../translations/fi.json")).expect("fi"),
            fr: json::from_str(include_str!("../translations/fr.json")).expect("fr"),
            hr: json::from_str(include_str!("../translations/hr.json")).expect("hr"),
            hu: json::from_str(include_str!("../translations/hu.json")).expect("hu"),
            it: json::from_str(include_str!("../translations/it.json")).expect("it"),
            ja: json::from_str(include_str!("../translations/ja.json")).expect("ja"),
            ko: json::from_str(include_str!("../translations/ko.json")).expect("ko"),
            lt: json::from_str(include_str!("../translations/lt.json")).expect("lt"),
            nl: json::from_str(include_str!("../translations/nl.json")).expect("nl"),
            pl: json::from_str(include_str!("../translations/pl.json")).expect("pl"),
            pt: json::from_str(include_str!("../translations/pt.json")).expect("pt"),
            sk: json::from_str(include_str!("../translations/sk.json")).expect("sk"),
            sl: json::from_str(include_str!("../translations/sl.json")).expect("sl"),
            sr: json::from_str(include_str!("../translations/sr.json")).expect("sr"),
            sv: json::from_str(include_str!("../translations/sv.json")).expect("sv"),
            ru: json::from_str(include_str!("../translations/ru.json")).expect("ru"),
            tr: json::from_str(include_str!("../translations/tr.json")).expect("tr"),
            uk: json::from_str(include_str!("../translations/uk.json")).expect("uk"),
            vi: json::from_str(include_str!("../translations/vi.json")).expect("vi"),
            zh_hant: json::from_str(include_str!("../translations/zh-Hant.json")).expect("zh_hant"),
            zh_hans: json::from_str(include_str!("../translations/zh-Hans.json")).expect("zh_hans"),
        }
    }

When this function is removed, compilation succeeds.

GDB shows a thread stuck within LLVM, doing various optimizations, in a seemingly-endless loop (I waited several minutes. Previously this compilation with 1.58.1 took ~32s):

  6    Thread 0x7f6b9fbff640 (LWP 171415) "LTO langpack.a3" 0x00007f6bbeb4cd3f in llvm::SSAUpdater::GetValueAtEndOfBlockInternal(llvm::BasicBlock*) ()
  5    Thread 0x7f6ba03ff640 (LWP 171218) "opt langpack.a3" 0x00007f6bbe3f34cf in llvm::removeUnreachableBlocks(llvm::Function&, llvm::DomTreeUpdater*, llvm::MemorySSAUpdater*) ()
  5    Thread 0x7fa58c3ff640 (LWP 170073) "opt langpack.a3" 0x00007fa5aa4f21e4 in llvm::simplifyCFG(llvm::BasicBlock*, llvm::TargetTransformInfo const&, llvm::DomTreeUpdater*, llvm::SimplifyCFGOptions const&, llvm::ArrayRef<llvm::WeakVH>) ()

Steps to Reproduce

  1. git clone https://github.com/sstangl/openpowerlifting.git
  2. git checkout ac057e388ac911fda2e93acd8689d13f6df23034
  3. cd modules/langpack
  4. cargo build

Meta

rustc --version --verbose:

rustc 1.59.0 (9d1b2106e 2022-02-23)
binary: rustc
commit-hash: 9d1b2106e23b1abd32fce1f17267604a5102f57a
commit-date: 2022-02-23
host: x86_64-unknown-linux-gnu
release: 1.59.0
LLVM version: 13.0.0

The problem also exists with Nightly:

rustc 1.61.0-nightly (4b043faba 2022-02-24)
binary: rustc
commit-hash: 4b043faba34ccc053a4d0110634c323f6c03765e
commit-date: 2022-02-24
host: x86_64-unknown-linux-gnu
release: 1.61.0-nightly
LLVM version: 14.0.0
@sstangl sstangl added the C-bug Category: This is a bug. label Feb 26, 2022
@sstangl sstangl changed the title Rust 1.59 rustc ~infinite loop with include_str! Rust 1.59 rustc greatly increased compile time with include_str! Feb 26, 2022
@MarkDDR
Copy link

MarkDDR commented Feb 26, 2022

From poking at the codebase a bit I have determined that include_str!() itself is not the direct cause of the issue, rather it is the optimizer at opt-level = 2 trying to optimize the serde_json::from_str::<Translations>() function given compile time knowledge of what the json string looks like. The issue also occurs if you manually inline the string

@MarkDDR
Copy link

MarkDDR commented Feb 26, 2022

I have created a much smaller code sample that recreates this bug https://github.com/MarkDDR/long_compile_rustc_1.59.0

@nikic nikic added A-LLVM Area: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues. I-compiletime Issue: Problems and improvements with respect to compile times. labels Feb 26, 2022
@ehuss
Copy link
Contributor

ehuss commented Feb 26, 2022

Thanks for putting together the reproductions! This seems to have started with #92419. cc @erikdesjardins

@erikdesjardins
Copy link
Contributor

Presumably there are more issues with runaway inlining (possibly the same thing as #89524 / https://reviews.llvm.org/D98481 - serde almost certainly has mutual recursion somewhere).

It is unlikely that there will be a trivial fix, and I don't have time to investigate it deeply right now. I'll open a revert in a bit.

dae added a commit to ankitects/anki that referenced this issue Mar 1, 2022
Rust version has been pinned, as 1.59 gets stuck compiling pyo3_macros,
and appears to have other issues:
rust-lang/rust#94390
bors added a commit to rust-lang-ci/rust that referenced this issue Mar 1, 2022
Revert "Auto merge of rust-lang#92419 - erikdesjardins:coldland, r=nagisa"

Should fix (untested) rust-lang#94390

Reopens rust-lang#46515, rust-lang#87055

r? `@ehuss`
dae added a commit to ankitects/anki that referenced this issue Mar 3, 2022
Rust version has been pinned, as 1.59 gets stuck compiling pyo3_macros,
and appears to have other issues:
rust-lang/rust#94390
@nikic nikic self-assigned this Mar 9, 2022
@nikic
Copy link
Contributor

nikic commented Mar 17, 2022

I looked into this a bit, and I'm not sure this qualifies as a catastrophic inlining issue. The input IR for "_ZN189_$LT$long_compile_time..big_struct.._..$LT$impl$u20$serde..de..Deserialize$u20$for$u20$long_compile_time..big_struct..BigStruct$GT$..deserialize..__Visitor$u20$as$u20$serde..de..Visitor$GT$9visit_map17h6f5c5ed07f452137E" is already more than 50k lines to start with and "only" grows to 140k lines during optimization. The result is huge because the input is huge.

Huge functions have a high chance of hitting one or the other super-linear optimization behavior in LLVM. I fixed one issue this hits in llvm/llvm-project@1dbeb64, which drops compile-time from 3:55 to 2:45.

I personally wouldn't block reapplication of the patch on this issue, as the interaction here is somewhat incidental -- large machine-generated code will always run into these sorts of issues, and minor perturbations in the generated code, rustc or LLVM can cause major variations in compile-time. Serde should avoid generating a large amount of code into a single function.

@mati865
Copy link
Contributor

mati865 commented Jun 30, 2022

Pinging Serde maintainer @dtolnay

Serde should avoid generating a large amount of code into a single function.

Would it be feasible to apply such change to Serde?

@Noratrieb Noratrieb added the T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. label Apr 5, 2023
TaKO8Ki added a commit to TaKO8Ki/rust that referenced this issue Jun 27, 2023
…=nikic

Rebased: Mark drop calls in landing pads cold instead of noinline

I noticed that certain inlining optimizations were missing while staring at some compiled code output. I'd like to see this relanded, so I rebased the PR from ``@erikdesjardins`` (PR rust-lang#94823).

This PR reapplies rust-lang#92419, which was reverted in rust-lang#94402 due to rust-lang#94390.

Fixes rust-lang#46515, fixes rust-lang#87055.

Update: fixes rust-lang#97217.
bors added a commit to rust-lang-ci/rust that referenced this issue Oct 2, 2023
…ikic

Rebased: Mark drop calls in landing pads cold instead of noinline

I noticed that certain inlining optimizations were missing while staring at some compiled code output. I'd like to see this relanded, so I rebased the PR from `@erikdesjardins` (PR rust-lang#94823).

This PR reapplies rust-lang#92419, which was reverted in rust-lang#94402 due to rust-lang#94390.

Fixes rust-lang#46515, fixes rust-lang#87055.

Update: fixes rust-lang#97217.
bors added a commit to rust-lang-ci/rust that referenced this issue Oct 2, 2023
…ikic

Rebased: Mark drop calls in landing pads cold instead of noinline

I noticed that certain inlining optimizations were missing while staring at some compiled code output. I'd like to see this relanded, so I rebased the PR from `@erikdesjardins` (PR rust-lang#94823).

This PR reapplies rust-lang#92419, which was reverted in rust-lang#94402 due to rust-lang#94390.

Fixes rust-lang#46515, fixes rust-lang#87055.

Update: fixes rust-lang#97217.
@RalfJung
Copy link
Member

RalfJung commented Oct 3, 2023

The revert got reverted when #102099 landed. Would be good to make sure that this bug does not come back.

@InnovativeInventor
Copy link
Contributor

On my mac, with

-> rustc --version --verbose
rustc 1.75.0-nightly (2e5a9dd6c 2023-10-02)
binary: rustc
commit-hash: 2e5a9dd6c9eaa42f0684b4b760bd68fc27cbe51b
commit-date: 2023-10-02
host: aarch64-apple-darwin
release: 1.75.0-nightly
LLVM version: 17.0.2

cargo build, following the reproduction steps listed in the first comment of this issue takes 5 minutes and 18 seconds.

antoyo pushed a commit to rust-lang/rustc_codegen_gcc that referenced this issue Oct 8, 2023
Rebased: Mark drop calls in landing pads cold instead of noinline

I noticed that certain inlining optimizations were missing while staring at some compiled code output. I'd like to see this relanded, so I rebased the PR from `@erikdesjardins` (PR #94823).

This PR reapplies rust-lang/rust#92419, which was reverted in rust-lang/rust#94402 due to rust-lang/rust#94390.

Fixes rust-lang/rust#46515, fixes rust-lang/rust#87055.

Update: fixes #97217.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-LLVM Area: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues. C-bug Category: This is a bug. I-compiletime Issue: Problems and improvements with respect to compile times. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Development

No branches or pull requests

9 participants