Compilation of large project taking much longer after 1.84 (monomorphization)

@rustbot

Code

I have a big private project and we try to stay current on rust versions. Upon trying 1.84 I saw compilation times grow about 3 times. I already seen this behavior in other rust versions which made me have to skip it, for example 1.82.

I self-profiled the compile in 1.83 and 1.84 and diffed, so I got to this (https://gist.github.com/lsunsi/7d301c7e332f50a734647d3aff0efbdc).
I'm not sure how useful it is, but there we go. I can post the prof data as well if it's useful.

Further, I'll try to bisect and get back with more information.

Version it worked on

It most recently worked on: 1.83

rustc 1.83.0 (90b35a623 2024-11-26)
binary: rustc
commit-hash: 90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf
commit-date: 2024-11-26
host: x86_64-unknown-linux-gnu
release: 1.83.0
LLVM version: 19.1.1

Version with regression

rustc --version --verbose:

rustc 1.84.0 (9fc6b4312 2025-01-07)
binary: rustc
commit-hash: 9fc6b43126469e3858e2fe86cafb4f0fd5068869
commit-date: 2025-01-07
host: x86_64-unknown-linux-gnu
release: 1.84.0
LLVM version: 19.1.5

@rustbot modify labels: +regression-from-stable-to-stable-regression-untriaged

While the bisection results as discussed in the other issue will be helpful, I'm sure you know this but we won't be able to do much without some code to reproduce and analyze the issue.

@lqd

@lqd, I work with @lsunsi.
I tried to use https://github.com/rust-lang/cargo-bisect-rustc to try to bisect and discover which commit between 1.83.0 and 1.84.0 could be the problem, but even the nightly that the cargo-bisect-rustc uses to try the start of the range (which is the nightly-2024-10-13 in that case) also takes the unusual long time to run cargo build.

Right now, I am thinking that because it uses a nightly instead of a "stable build" it does not really reproduces the same behaviour as if I was running cargo build with the 1.83.0-stable. Does it make any sense? The summary is that any nightly build that I try has the longer compilation time for our project.

Super thanks!

but even the nightly that the cargo-bisect-rustc uses to try the start of the range (which is the nightly-2024-10-13 in that case) also takes the unusual long time to run cargo build.

Here, you should try to go back further than 1.83.0 to see if the difference starts to appear at some earlier point.

Right now, I am thinking that because it uses a nightly instead of a "stable build" it does not really reproduces the same behaviour as if I was running cargo build with the 1.83.0-stable. Does it make any sense? The summary is that any nightly build that I try has the longer compilation time for our project.

Interesting. Normally this should be quite unusual, because nightlies and stable have basically the same code, only gated differently. You could try using the stable release but unlocking the gating so it behaves as a nightly. With the env var RUSTC_BOOTSTRAP=1 your stable release should "think it's a nightly".

If that behaves differently than the nightly that was promoted to 1.83.0 it will be quite strange. And if it behaves differently than without the env var, it will also be quite strange. Both cases would help you narrow down the source of the issue.

And also the other way around, you could try RUSTC_BOOTSTRAP=-1 on the nightly to make it think it is a stable rustc, but I suspect that should not cause observable differences (outside of you can't use nightly features).

Ok, I have tried again starting at 1.81.0, and cargo-bisect-rustc could find nightlies with "normal" comptime and nightlies with big comptime.

Here is the full log , but the summary is:

searched toolchains nightly-2024-07-20 through nightly-2024-11-22
********************************************************************************
Regression in nightly-2024-07-25
********************************************************************************

(...)

looking for regression commit between 2024-07-24 and 2024-07-25
fetching (via remote github) commits from max(8bfcae730a5db2438bbda72796175bba21427be1, 2024-07-22) to c1a6199e9d92bb785c17a6d7ffd8b8b552f79c10
ending github query because we found starting sha: 8bfcae730a5db2438bbda72796175bba21427be1
get_commits_between returning commits, len: 9
  commit[0] 2024-07-23: Auto merge of #128109 - matthiaskrgr:rollup-gc7kopi, r=matthiaskrgr
  commit[1] 2024-07-24: Auto merge of #127153 - NobodyXu:pipe, r=ChrisDenton
  commit[2] 2024-07-24: Auto merge of #128123 - ehuss:tidy-rustbook-submodule, r=Mark-Simulacrum
  commit[3] 2024-07-24: Auto merge of #128128 - matthiaskrgr:rollup-jz6w0ck, r=matthiaskrgr
  commit[4] 2024-07-24: Auto merge of #128127 - tmandry:you-wouldnt-bump-a-fuchsia, r=Kobzol
  commit[5] 2024-07-24: Auto merge of #127524 - oli-obk:feed_item_attrs2, r=petrochenkov
  commit[6] 2024-07-24: Auto merge of #126024 - oli-obk:candidate_key_caching_is_unsound_yay, r=lcnr
  commit[7] 2024-07-24: Auto merge of #128142 - matthiaskrgr:rollup-rep8ofv, r=matthiaskrgr
  commit[8] 2024-07-24: Auto merge of #128146 - notriddle:notriddle/natsortfixes, r=GuillaumeGomez
ERROR: no CI builds available between 8bfcae730a5db2438bbda72796175bba21427be1 and c1a6199e9d92bb785c17a6d7ffd8b8b552f79c10 within last 167 days

Which is a bit strange for me, considering that 1.83.0 is "fast" and we noticed the problem only with the 1.84.0 of last week.

And, probably the perception that I had that "a nigthly does not reproduce the problem like a stable" does not make sense. It was that I was not looking for nightlies "older enough". Sorry for that hallucination :)

@orlp

@orlp For the specific crate we were testing before, here is the profile for two significant commits: 3a6da61357aca9fbf7b3017ed9d795cab46b57dd 1m42s
d9b4598d7e8ce233b8bd4fe89182b759238ea246 3m45s

The results were so exciting that I wanted to test against the whole project.
3a6da61357aca9fbf7b3017ed9d795cab46b57dd 3m50s
d9b4598d7e8ce233b8bd4fe89182b759238ea246 9m30s

@lsunsi

@lsunsi How does 3a6da61 compare against rustc 1.83.0 (90b35a623 2024-11-26), since you said that was a stable version before the regression? Might want to build 90b35a623 like you are doing for the rest, if you get it from the public binary list it might've been built in a subtly different way / with different optimizations giving biased results.

1.83 stable just one crate
1.83 stable all crates 2m50s

EDIT sorry I was already running against 1.83 public before I read your message properly. I'll build 90b35a6 and run it, just a sec.

@lsunsi

@lsunsi FYI your 'whole project' profiles are missing debug symbols again.

@orlp

90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf (1.83) just one crate 1m43s
90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf (1.83) all crates 3m46s

@orlp It seems like the parameters for public release really made a difference, great catch. The times are basically identical from your fixed branch!

Just to have another datapoint, I also ran 90b35a6 vs d694429, building our project, and they took, respectively 2m 29s vs 2m 30s, in my machine. 😃

@steffahn

@steffahn Do you want me to test #136106 ? I'd be happy to if it helps anything

There's no real need for that, because it didn't aim to fix any of your performance in the first place. It should be just as bad as nightly. (The point was to prove/check that it didn't become any worse.)

@lochetti

Quoting @lochetti from rust-lang/rustc-hash#55 (comment), since it’s relevant to this issue:

Hello all! We have created a reproducer for this scenario. You can check it here: https://github.com/lochetti/rustc-hash-pr55-reproducer

The code is basically a "loop" of calls to diesel's table macro. We noticed, doing the reproducer, that when we start to call the macro more than 512 times, we start to get more than 2^20 GenericArgs interner and we start to see the compile time degradation. At 2048 calls, the difference in compile time is super noticiable.

Since the code is basically one macro that calls another macro in loop, it is very easy to "configure" how many times you want to call the inner (diesel's) macro and experiment the behaviour for other numbers of GernicArgs interners.

We hope that this helps :) And thanks again for all the help so far!

@steffahn

Thanks @steffahn and just adding to that, it seems like there's a mitigation to be done for our case and a better solution for the general case. There's more information about the possibilities in the mentioned issue on rustc-hash!

Auto merge of rust-lang#136605 - lsunsi:update-rustc-hash, r=<try>

Auto merge of rust-lang#136605 - lsunsi:update-rustc-hash, r=lqd

@steffahn

@steffahn @lqd @orlp @Noratrieb Thanks for all the help with this issue. the current compile time of my project on nightly went from 9m54s to 2m56s, which is in line with 1.83 performence. Besides that, I learned A LOT and gained a lot from this experience, both in debugging perf issues and learning about how Rust is cooked. Again, thanks a lot to you all!

Navigation Menu

Repository navigation

Compilation of large project taking much longer after 1.84 (monomorphization) #135477

Code

Version it worked on

Version with regression

56 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Compilation of large project taking much longer after 1.84 (monomorphization) #135477

Description

Code

Version it worked on

Version with regression

Activity

lqd commented on Jan 14, 2025

lochetti commented on Jan 14, 2025

lqd commented on Jan 14, 2025

jieyouxu commented on Jan 14, 2025

lochetti commented on Jan 14, 2025

56 remaining items

lsunsi commented on Jan 26, 2025

orlp commented on Jan 26, 2025

lsunsi commented on Jan 26, 2025

orlp commented on Jan 26, 2025

lsunsi commented on Jan 26, 2025

lochetti commented on Jan 26, 2025

lsunsi commented on Jan 27, 2025

steffahn commented on Jan 27, 2025

steffahn commented on Feb 4, 2025

lsunsi commented on Feb 5, 2025

lsunsi commented on Feb 12, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions