Skip to content

build --release takes forever on latest rust release #119822

Closed
Closed
build --release takes forever on latest rust release#119822
Labels
I-compiletimeIssue: Problems and improvements with respect to compile times.S-needs-reproStatus: This issue has no reproduction and needs a reproduction to make progress.T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.regression-from-stable-to-stablePerformance or correctness regression from one stable version to another.

Description

@Qumeric
Qumeric
opened on Jan 10, 2024

I have a project of around 10k lines and 1000 dependency crates. I cannot share it because it's closed source.

I was working on it for a few months and previously compiled it with stable, beta and nightly rust.

I decided to update rust to latest version. I run cargo build --release and it stuck at

   Compiling reth-block-validator v0.1.0 (/path/...)
   Building [=====================> ] 1157/1161: my-crate-name

I tried to build my-crate-name solely (full project is workspace) but it stuck in a same way.

I tried to do it with macOS + lld, linux + mold, linux + ld. Linux were 3 different machines with 5x00X Ryzen cpus.

It always stuck (I waited at least 10 and up to 60 minutes). Both in latest (installed with rustup) stable, beta and nightly.

I tried different standard things like cargo clean but nothing helped.

The project builds without --release just fine and builds with --release in rust 1.74.1 (and some previous versions of nightly and beta). It takes ~4 minutes for clean --release build and ~1.5 minutes for clean debug build.

I may try git bisect on compiler versions later if I will have time for that. Although I am not completely sure if it's compiler issue, could be cargo issue as well?

Activity

  1. rustbot
    added
    needs-triageThis issue may need triage. Remove when done. See docs forge.rust-lang.org/release/issue-triaging
    on Jan 10, 2024

  2. ehuss commented on Jan 10, 2024

    @ehuss
    ehuss
    on Jan 10, 2024
    Contributor

    Thanks for the report!

    I may try git bisect on compiler versions later if I will have time for that. Although I am not completely sure if it's compiler issue, could be cargo issue as well?

    We have a tool called cargo-bisect-rustc specifically for bisecting these issues. There is a --timeout option for diagnosing slow/hung builds (documented here, though it currently does not work on macOS).

  3. saethlin
    added
    I-compiletimeIssue: Problems and improvements with respect to compile times.
    T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.
    regression-from-stable-to-stablePerformance or correctness regression from one stable version to another.
    and removed
    needs-triageThis issue may need triage. Remove when done. See docs forge.rust-lang.org/release/issue-triaging
    on Jan 11, 2024
  4. rustbot
    added
    I-prioritizeIssue needs a team member to assess the impact. Will be replaced by P-{low,medium,high,critical}
    on Jan 11, 2024
  5. apiraino
    added
    E-needs-bisectionCall for participation: This issue needs bisection: https://github.com/rust-lang/cargo-bisect-rustc
    E-needs-mcveCall for participation: This issue has a repro, but needs a Minimal Complete and Verifiable Example
    S-needs-reproStatus: This issue has no reproduction and needs a reproduction to make progress.
    on Jan 11, 2024
  6. Noratrieb
    removed
    E-needs-mcveCall for participation: This issue has a repro, but needs a Minimal Complete and Verifiable Example
    on Jan 11, 2024

  7. Qumeric commented on Feb 6, 2024

    @Qumeric
    Qumeric
    on Feb 6, 2024
    Author

    I finally run bisect tool, thanks for pointing on it.

    Report:

    ********************************************************************************
Regression in 2e5a9dd6c9eaa42f0684b4b760bd68fc27cbe51b
********************************************************************************

Attempting to search unrolled perf builds
ERROR: couldn't find perf build comment
==================================================================================
= Please file this regression report on the rust-lang/rust GitHub repository     =
=        New issue: https://github.com/rust-lang/rust/issues/new                 =
=     Known issues: https://github.com/rust-lang/rust/issues                     =
= Copy and paste the text below into the issue report thread.  Thanks!           =
==================================================================================

    searched nightlies: from nightly-2023-09-01 to nightly-2024-01-01
    regressed nightly: nightly-2023-10-03
    searched commit range: e0d7ed1...2e5a9dd
    regressed commit: 2e5a9dd

    bisected with cargo-bisect-rustc v0.6.8

    Host triple: x86_64-unknown-linux-gnu
    Reproduce with:
    bash
    cargo bisect-rustc --end=2024-01-01 --timeout 300 -- build --release

    Also, A bit of feedback on this tool:
    RESULT: nightly-2023-09-01, ===> No is a bit confusing, I am not sure what "No" means? Does it mean success or fail or it depends on what it's trying to check? I suggest changing it to something like "Timeout", "ICE", "Success" etc. (perhaps in addition to Yes/No).

    I guess that "No" means "No failure" and "Yes" means "Failure" but I only understood it after the run.

  8. Qumeric commented on Feb 6, 2024

    @Qumeric
    Qumeric
    on Feb 6, 2024
    Author

    Tried to build manually with nightly-2023-10-02 and can confirm it works.

    Also tried the latest nightly and beta and it is still broken. Pretty sure the latest stable is also still broken but I did not check it.

  9. ehuss commented on Feb 6, 2024

    @ehuss
    ehuss
    on Feb 6, 2024
    Contributor

    Thanks for investigating.

    cc @nikic @InnovativeInventor, this seems to be caused by #102099. Is this kind of regression expected? Are there additional tasks for investigating why it appears to be getting stuck?

    One possibility that comes to mind in this situation is that maybe it is doing significantly more inlining, and possibly running out of memory and hitting swap.

  10. ehuss
    removed
    E-needs-bisectionCall for participation: This issue needs bisection: https://github.com/rust-lang/cargo-bisect-rustc
    on Feb 6, 2024

  11. nikic commented on Feb 6, 2024

    @nikic
    nikic
    on Feb 6, 2024
    Contributor

    Yes, this is "expected", in the sense that it can result in pathological compile-times for pathological inputs -- most likely, you have a proc macro somewhere creating a huge function (say, hundreds of thousands instructions), and this change perturbs things enough for it to trigger some super-linear optimization behavior.

    Beyond that, there's not much I can say here without a reproducer or even a profile.

  12. Qumeric commented on Feb 6, 2024

    @Qumeric
    Qumeric
    on Feb 6, 2024
    Author

    I don't think it's OOM, I had 256GB of RAM on some machines I tried this.

    In terms of macro I can think of only async_trait, everything else doesn't look heavy. And in general I don't have an idea what can be pathological in my project, there is nothing unusual to my eye.

    I will try to remove async_trait and see if it helps, also I will try to run compilation overnight, maybe it will finish.

    Do you have any advice on what I can do to figure out what's going on?

  13. nikic commented on Feb 6, 2024

    @nikic
    nikic
    on Feb 6, 2024
    Contributor

    I'd start by adding -v to the cargo command and finding the rustc command that hangs.

    Then run it under perf record (it's okay to just ctrl+c after a while) and then check perf report -- I'd expect to see most time being taken up somewhere in LLVM. If we're very lucky it may be possible to identify what in LLVM causes this just from that.

    The other thing I'd try is adding --emit=llvm-ir -C no-prepopulate-passes and seeing whether there are any large functions in the .ll file.

  14. Qumeric commented on Mar 14, 2024

    @Qumeric
    Qumeric
    on Mar 14, 2024
    Author

    I think this issue was fixed, at least my project (with library updates) compiles at latest stable with --release

  15. apiraino commented on Mar 14, 2024

    @apiraino
    apiraino
    on Mar 14, 2024
    Contributor

    thanks @Qumeric for the feedback. I don't see other comments (or comment from other issues) so feel free to close this issue if everything's fine on your end

  16. Qumeric
    closed this as completedon Mar 15, 2024
  17. apiraino
    removed
    I-prioritizeIssue needs a team member to assess the impact. Will be replaced by P-{low,medium,high,critical}
    on Mar 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    I-compiletimeIssue: Problems and improvements with respect to compile times.S-needs-reproStatus: This issue has no reproduction and needs a reproduction to make progress.T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.regression-from-stable-to-stablePerformance or correctness regression from one stable version to another.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions