Skip to content

Linker hangs with no output 9/10 times. #88704

Closed
Closed
Linker hangs with no output 9/10 times.#88704
Labels
C-bugCategory: This is a bug.I-hangIssue: The compiler never terminates, due to infinite loops, deadlock, livelock, etc.O-windows-gnuToolchain: GNU, Operating system: Windows

Description

@hydra
hydra
opened on Sep 6, 2021

When building various example projects, I find that the rust-lld.exe linker will just stall instead of creating the .elf file. When it works, it takes about 0.4seconds, most of the time it fails and you have to ctrl+c the command and retry, it's obviously ridiculously frustrating, especially during a pair programming session.

To make sure there was nothing odd on my PC that might be affecting it, I did a complete fresh install of the OS, rust and the tools, yet the same thing still happens.

I was able to record detailed logs using ProcessMonitor of the failure. Attached to this issue are 3 files:

  1. good.CSV, a log from when it works.

good.CSV

The output was this:

PS D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor> cargo build -j1
   Compiling stm32f4_pwm_input v0.1.0 (D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor)
    Finished dev [unoptimized + debuginfo] target(s) in 0.70s
  1. bad.CSV and 3) bad-after-killing.CSV. A trace was started and saved as bad.CSV when rust-lld.exe hanged, then a new trace was started, ctrl+c was pressed and the new trace was saved as bad-after-killing.CSV so that you can see what happens before and after ctrl+c was pressed.

bad.CSV
bad-after-killing.CSV

The output, when it fails is this:

...
Compiling stm32f4_pwm_input v0.1.0 (D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor)
    Building [=========================> ] 98/99: stm32f4_pwm_input(bin)

I'm compiling for an STM32F446RE CPU.

I tried with stable and with nightly

Meta

Project source: rtic-examples.zip

Nightly:

> rustup show
Default host: x86_64-pc-windows-gnu
rustup home:  D:\Users\Hydra\.rustup

installed toolchains
--------------------

stable-x86_64-pc-windows-gnu
nightly-x86_64-pc-windows-gnu (default)

installed targets for active toolchain
--------------------------------------

thumbv7em-none-eabihf
x86_64-pc-windows-gnu

active toolchain
----------------

nightly-x86_64-pc-windows-gnu (default)
rustc 1.57.0-nightly (e30b68353 2021-09-05)

Stable:

> rustup show
Default host: x86_64-pc-windows-gnu
rustup home:  D:\Users\Hydra\.rustup

installed toolchains
--------------------

stable-x86_64-pc-windows-gnu (default)
nightly-x86_64-pc-windows-gnu

installed targets for active toolchain
--------------------------------------

thumbv7em-none-eabihf
x86_64-pc-windows-gnu

active toolchain
----------------

stable-x86_64-pc-windows-gnu (default)
rustc 1.54.0 (a178d0322 2021-07-26)

Hardware:

Processor	AMD Ryzen 9 5950X 16-Core Processor               3.40 GHz
Installed RAM	64.0 GB

OS:

Edition	Windows 10 Pro
Version	21H1
Installed on	‎05/‎09/‎2021
OS build	19043.928
Experience	Windows Feature Experience Pack 120.2212.551.0

If you need more information/logs/etc, just tell me what commands you want me to run and what output/files to provide.

Activity

  1. hydra
    added
    C-bugCategory: This is a bug.
    on Sep 6, 2021
  2. jonas-schievink
    added
    I-hangIssue: The compiler never terminates, due to infinite loops, deadlock, livelock, etc.
    on Sep 6, 2021

  3. mati865 commented on Sep 7, 2021

    @mati865
    mati865
    on Sep 7, 2021
    Member

    Could you obtain stacktraces for the hanged process? I though we fixed all hangs like that in LLD 13 but maybe Rust's ancient GCC is even more broken than more recent versions.

  4. mati865 commented on Sep 7, 2021

    @mati865
    mati865
    on Sep 7, 2021 · Hidden as resolved
  5. rustbot
    added
    O-windows-gnuToolchain: GNU, Operating system: Windows
    on Sep 7, 2021

  6. hydra commented on Sep 7, 2021

    @hydra
    hydra
    on Sep 7, 2021
    Author

    @mati865 Sure, but how do I obtain a stacktrace for the hung process?

  7. mati865 commented on Sep 7, 2021

    @mati865
    mati865
    on Sep 7, 2021
    Member

    The easiest way is to attach debugger to the process (this sometimes fixes the hang so you cannot get the trace) or create dump by right clicking on the process in task manager and load it into the debugger.
    Anyway I'm back home so I've obtained the trace myself:

    [0x0]   ntdll!NtWaitForSingleObject + 0x14   
[0x1]   KERNELBASE!WaitForSingleObjectEx + 0x8e   
[0x2]   libwinpthread_1!pthread_cond_init + 0x22c   
[0x3]   libwinpthread_1!pthread_cond_init + 0x36e   
[0x4]   libwinpthread_1!pthread_cond_signal + 0xcf   
[0x5]   rust_lld + 0x282fcd9   
[0x6]   rust_lld + 0x1f6f14c   
[0x7]   rust_lld + 0xdda465   
[0x8]   rust_lld + 0xcc9958   
[0x9]   rust_lld + 0xdb497c   
[0xa]   rust_lld + 0xdc70ab   
[0xb]   rust_lld + 0xdc059a   
[0xc]   rust_lld + 0xe32940   
[0xd]   rust_lld + 0x3cc319   
[0xe]   rust_lld + 0x2b22910   
[0xf]   rust_lld + 0x13f8   
[0x10]   rust_lld + 0x151b   
[0x11]   kernel32!BaseThreadInitThunk + 0x14   
[0x12]   ntdll!RtlUserThreadStart + 0x21

    Indeed it looks like bug in old GCC or mingw-w64 used by Rust to build this target.

  8. hydra commented on Sep 7, 2021

    @hydra
    hydra
    on Sep 7, 2021
    Author

    @mati865 Awesome! Thanks ever so much for digging into it, was glad you were able to replicate the issue. Keep us all posted. I and my colleagues will be so happy when this is fixed.

    Is it possible to upgrade the tools used by rust to workaround this issue?

  9. mati865 commented on Sep 8, 2021

    @mati865
    mati865
    on Sep 8, 2021
    Member

    Unfortunately there is no easy fix, last time I tried to upgrade the tools it couldn't make it on the CI: #51989
    Mingw-builds haven't provided any new build since then, I'll try to provide more up to date toolchain but cannot give any ETA.

    I don't know how rust-lld is called here but as a workaround you could try using ld.lld from official LLVM builds or MSYS2.

  10. hydra commented on Sep 8, 2021

    @hydra
    hydra
    on Sep 8, 2021
    Author

    rust-lld is called via cargo -> rustc -> rust-lld IIRC.

    Can you reach out the mingw developers to see if they can provide an updated build?

    How would I go about using ld.lld ? overwrite the rust-lld.exe with a different .exe or?

  11. hydra commented on Sep 9, 2021

    @hydra
    hydra
    on Sep 9, 2021
    Author

    I found the documentation for changing the linker:

    https://doc.rust-lang.org/rustc/codegen-options/index.html#linker-flavor

    I will give that a try as time permits and report back.

  12. hydra commented on May 24, 2022

    @hydra
    hydra
    on May 24, 2022
    Author

    This is still an issue, I changed my linker to the gnu-ld linker for other reasons (map file generation, printing memory usage stats, etc) and forgot about this issue. Today when working on a new project I encountered the error again. I'm using a much newer version of rust since I opened this issue.

    rustup show reports the following:

    Default host: x86_64-pc-windows-gnu
rustup home:  D:\Users\Hydra\.rustup

installed toolchains

--------------------

stable-x86_64-pc-windows-gnu
nightly-2019-07-01-x86_64-pc-windows-gnu
nightly-2021-08-23-x86_64-pc-windows-gnu
nightly-2021-09-07-x86_64-pc-windows-gnu
nightly-2022-03-27-x86_64-pc-windows-gnu (default)
nightly-2022-04-24-x86_64-pc-windows-gnu
nightly-x86_64-pc-windows-gnu

installed targets for active toolchain
--------------------------------------

thumbv7em-none-eabihf
x86_64-pc-windows-gnu

active toolchain
----------------

nightly-2022-03-27-x86_64-pc-windows-gnu (default)
rustc 1.61.0-nightly (1d9c262ee 2022-03-26)

    Has there been any update to the tools? what's the current status of this? are there any other workarounds?

  13. hydra commented on Jul 19, 2022

    @hydra
    hydra
    on Jul 19, 2022
    Author

    Any update on this? I find it a bit crazy that such a fundamental issue, i.e. it randomly fails 9/10 times, hasn't been prioritized over other changes.

  14. hydra commented on Jul 19, 2022

    @hydra
    hydra
    on Jul 19, 2022
    Author

    This still happens on a very recent nightly:

    Default host: x86_64-pc-windows-gnu
rustup home:  D:\Users\Hydra\.rustup

installed toolchains
--------------------

nightly-2019-07-01-x86_64-pc-windows-gnu
nightly-2021-08-23-x86_64-pc-windows-gnu
nightly-2021-09-07-x86_64-pc-windows-gnu
nightly-2022-03-27-x86_64-pc-windows-gnu
nightly-2022-04-24-x86_64-pc-windows-gnu
nightly-2022-05-31-x86_64-pc-windows-gnu
nightly-2022-06-17-x86_64-pc-windows-gnu
nightly-2022-07-13-x86_64-pc-windows-gnu (default)

installed targets for active toolchain
--------------------------------------

thumbv7em-none-eabihf
thumbv7m-none-eabi
x86_64-pc-windows-gnu

active toolchain
----------------

nightly-2022-07-13-x86_64-pc-windows-gnu (default)
rustc 1.64.0-nightly (1c7b36d4d 2022-07-12)

    example, frustrating, user experience:

    D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor>cargo build
   Compiling stm32f4_pwm_input v0.1.0 (D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor)
^C  Building [=========================> ] 97/98: stm32f4_pwm_input(bin)
D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor>cargo build
   Compiling stm32f4_pwm_input v0.1.0 (D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor)
^C  Building [=========================> ] 97/98: stm32f4_pwm_input(bin)
D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor>cargo build
   Compiling stm32f4_pwm_input v0.1.0 (D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor)
^C  Building [=========================> ] 97/98: stm32f4_pwm_input(bin)
D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor>cargo build
   Compiling stm32f4_pwm_input v0.1.0 (D:\Users\Hydra\Documents\dev\playground\rust\rtic\rtic-examples\rtic_v6\stm32f4_pwm_monitor)
    Finished dev [unoptimized + debuginfo] target(s) in 0.45s
  15. bors
    added a commit that references this issue on Nov 24, 2022

    Merge #83

    ed69005

  16. mati865 commented on Feb 17, 2023

    @mati865
    mati865
    on Feb 17, 2023
    Member

    The MinGW version had been updated, does this issue still happen with recent nightly?

  17. hydra commented on Feb 21, 2023

    @hydra
    hydra
    on Feb 21, 2023
    Author

    I'll check with the latest nightly and report back ASAP.

  18. hydra commented on Jun 17, 2023

    @hydra
    hydra
    on Jun 17, 2023
    Author

    The MinGW version had been updated, does this issue still happen with recent nightly?

    Hi, Sorry for not getting back to you sooner.

    I confirm this doesn't happen with the nightly from 2023-06-17, installed via rustup. To verify I ran rustup default 1.54.0 and the problem occurred again. I again switched back to the new nightly with rustup default nightly-2023-06-17-x86_64-pc-windows-gnu and the problem doesn't occur again.

    Happy days!

    Do we know exactly which nightly and/or or stable version this issue issue was fixed in so that others coming back to this issue can ensure they are on known-working fixed version?

  19. mati865 commented on Jun 17, 2023

    @mati865
    mati865
    on Jun 17, 2023
    Member

    It was fixed by #100178 because this was GCC/winpthreads bug.

  20. hydra commented on Jun 17, 2023

    @hydra
    hydra
    on Jun 17, 2023
    Author

    ok, closing since it seems to be fixed now.

  21. hydra
    closed this as completedon Jun 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    C-bugCategory: This is a bug.I-hangIssue: The compiler never terminates, due to infinite loops, deadlock, livelock, etc.O-windows-gnuToolchain: GNU, Operating system: Windows

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions