Rust 1.59 rustc greatly increased compile time with include_str!

The following repo/hash compiles with rustc v1.58, but loops forever in v1.59 and Nightly (1.61): sstangl/openpowerlifting@ac057e3

The problem appears to be with the compilation of a very small crate (modules/langpack) that uses include_str! to include some JSON files. The problematic method is the implementation of default():

    fn default() -> LangInfo {
        LangInfo {
            cz: json::from_str(include_str!("../translations/cz.json")).expect("cz"),
            de: json::from_str(include_str!("../translations/de.json")).expect("de"),
            el: json::from_str(include_str!("../translations/el.json")).expect("el"),
            en: json::from_str(include_str!("../translations/en.json")).expect("en"),
            eo: json::from_str(include_str!("../translations/eo.json")).expect("eo"),
            es: json::from_str(include_str!("../translations/es.json")).expect("es"),
            fi: json::from_str(include_str!("../translations/fi.json")).expect("fi"),
            fr: json::from_str(include_str!("../translations/fr.json")).expect("fr"),
            hr: json::from_str(include_str!("../translations/hr.json")).expect("hr"),
            hu: json::from_str(include_str!("../translations/hu.json")).expect("hu"),
            it: json::from_str(include_str!("../translations/it.json")).expect("it"),
            ja: json::from_str(include_str!("../translations/ja.json")).expect("ja"),
            ko: json::from_str(include_str!("../translations/ko.json")).expect("ko"),
            lt: json::from_str(include_str!("../translations/lt.json")).expect("lt"),
            nl: json::from_str(include_str!("../translations/nl.json")).expect("nl"),
            pl: json::from_str(include_str!("../translations/pl.json")).expect("pl"),
            pt: json::from_str(include_str!("../translations/pt.json")).expect("pt"),
            sk: json::from_str(include_str!("../translations/sk.json")).expect("sk"),
            sl: json::from_str(include_str!("../translations/sl.json")).expect("sl"),
            sr: json::from_str(include_str!("../translations/sr.json")).expect("sr"),
            sv: json::from_str(include_str!("../translations/sv.json")).expect("sv"),
            ru: json::from_str(include_str!("../translations/ru.json")).expect("ru"),
            tr: json::from_str(include_str!("../translations/tr.json")).expect("tr"),
            uk: json::from_str(include_str!("../translations/uk.json")).expect("uk"),
            vi: json::from_str(include_str!("../translations/vi.json")).expect("vi"),
            zh_hant: json::from_str(include_str!("../translations/zh-Hant.json")).expect("zh_hant"),
            zh_hans: json::from_str(include_str!("../translations/zh-Hans.json")).expect("zh_hans"),
        }
    }

When this function is removed, compilation succeeds.

GDB shows a thread stuck within LLVM, doing various optimizations, in a seemingly-endless loop (I waited several minutes. Previously this compilation with 1.58.1 took ~32s):

  6    Thread 0x7f6b9fbff640 (LWP 171415) "LTO langpack.a3" 0x00007f6bbeb4cd3f in llvm::SSAUpdater::GetValueAtEndOfBlockInternal(llvm::BasicBlock*) ()

  5    Thread 0x7f6ba03ff640 (LWP 171218) "opt langpack.a3" 0x00007f6bbe3f34cf in llvm::removeUnreachableBlocks(llvm::Function&, llvm::DomTreeUpdater*, llvm::MemorySSAUpdater*) ()

  5    Thread 0x7fa58c3ff640 (LWP 170073) "opt langpack.a3" 0x00007fa5aa4f21e4 in llvm::simplifyCFG(llvm::BasicBlock*, llvm::TargetTransformInfo const&, llvm::DomTreeUpdater*, llvm::SimplifyCFGOptions const&, llvm::ArrayRef<llvm::WeakVH>) ()

Steps to Reproduce

git clone https://github.com/sstangl/openpowerlifting.git
git checkout ac057e388ac911fda2e93acd8689d13f6df23034
cd modules/langpack
cargo build

Meta

rustc --version --verbose:

rustc 1.59.0 (9d1b2106e 2022-02-23)
binary: rustc
commit-hash: 9d1b2106e23b1abd32fce1f17267604a5102f57a
commit-date: 2022-02-23
host: x86_64-unknown-linux-gnu
release: 1.59.0
LLVM version: 13.0.0

The problem also exists with Nightly:

rustc 1.61.0-nightly (4b043faba 2022-02-24)
binary: rustc
commit-hash: 4b043faba34ccc053a4d0110634c323f6c03765e
commit-date: 2022-02-24
host: x86_64-unknown-linux-gnu
release: 1.61.0-nightly
LLVM version: 14.0.0

From poking at the codebase a bit I have determined that include_str!() itself is not the direct cause of the issue, rather it is the optimizer at opt-level = 2 trying to optimize the serde_json::from_str::<Translations>() function given compile time knowledge of what the json string looks like. The issue also occurs if you manually inline the string

I have created a much smaller code sample that recreates this bug https://github.com/MarkDDR/long_compile_rustc_1.59.0

@erikdesjardins

Thanks for putting together the reproductions! This seems to have started with #92419. cc @erikdesjardins

Presumably there are more issues with runaway inlining (possibly the same thing as #89524 / https://reviews.llvm.org/D98481 - serde almost certainly has mutual recursion somewhere).

It is unlikely that there will be a trivial fix, and I don't have time to investigate it deeply right now. I'll open a revert in a bit.

Update to latest rules_rust, which unblocks the tokio upgrade

Auto merge of rust-lang#94402 - erikdesjardins:revert-coldland, r=nagisa

Update to latest rules_rust, which unblocks the tokio upgrade

I looked into this a bit, and I'm not sure this qualifies as a catastrophic inlining issue. The input IR for "_ZN189_$LT$long_compile_time..big_struct.._..$LT$impl$u20$serde..de..Deserialize$u20$for$u20$long_compile_time..big_struct..BigStruct$GT$..deserialize..__Visitor$u20$as$u20$serde..de..Visitor$GT$9visit_map17h6f5c5ed07f452137E" is already more than 50k lines to start with and "only" grows to 140k lines during optimization. The result is huge because the input is huge.

Huge functions have a high chance of hitting one or the other super-linear optimization behavior in LLVM. I fixed one issue this hits in llvm/llvm-project@1dbeb64, which drops compile-time from 3:55 to 2:45.

I personally wouldn't block reapplication of the patch on this issue, as the interaction here is somewhat incidental -- large machine-generated code will always run into these sorts of issues, and minor perturbations in the generated code, rustc or LLVM can cause major variations in compile-time. Serde should avoid generating a large amount of code into a single function.

@dtolnay

Pinging Serde maintainer @dtolnay

Serde should avoid generating a large amount of code into a single function.

Would it be feasible to apply such change to Serde?

Rollup merge of rust-lang#102099 - InnovativeInventor:re-cold-land, r…

Auto merge of rust-lang#102099 - InnovativeInventor:re-cold-land, r=n…

The revert got reverted when #102099 landed. Would be good to make sure that this bug does not come back.

On my mac, with

-> rustc --version --verbose
rustc 1.75.0-nightly (2e5a9dd6c 2023-10-02)
binary: rustc
commit-hash: 2e5a9dd6c9eaa42f0684b4b760bd68fc27cbe51b
commit-date: 2023-10-02
host: aarch64-apple-darwin
release: 1.75.0-nightly
LLVM version: 17.0.2

cargo build, following the reproduction steps listed in the first comment of this issue takes 5 minutes and 18 seconds.

Auto merge of #102099 - InnovativeInventor:re-cold-land, r=nikic

Navigation Menu

Repository navigation

Rust 1.59 rustc greatly increased compile time with include_str! #94390

Steps to Reproduce

Meta

5 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Rust 1.59 rustc greatly increased compile time with include_str! #94390

Description

Steps to Reproduce

Meta

Activity

MarkDDR commented on Feb 26, 2022

MarkDDR commented on Feb 26, 2022

ehuss commented on Feb 26, 2022

erikdesjardins commented on Feb 26, 2022

nikic commented on Mar 17, 2022

mati865 commented on Jun 30, 2022

5 remaining items

RalfJung commented on Oct 3, 2023

InnovativeInventor commented on Oct 3, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions