Repository navigation
cargo run causes linking problems (dynamic and static) on i686-pc-windows-gnu #8990
Description
In the context of generating Rust bindings for R via bindgen (libR-sys project, https://github.com/extendr/libR-sys), we have encountered a strange problem for the target i686-pc-windows-gnu that seems to trace back to something cargo run does when calling an executable it has just built.
In brief, we noticed that when trying to run our bindgen build script on 32-bit Windows, the program would abort with a linking error in libclang. We first thought it was a packaging problem with the mingw package providing libclang, and we reported it as such: msys2/MINGW-packages#7442. However, further investigation has shown that this is unlikely to be the case. Instead, we now suspect there's a problem with cargo itself.
@Ilia-Kosenkov has created a minimal example of a Rust program calling libclang, to mimic what bindgen does:
https://github.com/Ilia-Kosenkov/rust_clang_run. This program compiles and runs just fine on 64-bit Windows, and it aborts with a linking error on 32-bit Windows, regardless of whether libclang is linked statically or dynamically. But here's the kicker: This happens only when we run the program with cargo run. If we build the program with cargo build and then run the executable by calling it directly, everything works fine. You can see this behavior in the GitHub Actions for the minimal example, e.g. here: https://github.com/Ilia-Kosenkov/rust_clang_run/runs/1565696019?check_suite_focus=true
In the minimal example, we can of course work around the problem by not using cargo run. But in the actual application, libclang is linked into the build script, and thus the compiled executable is always run by cargo and the linking error occurs.
Related libR-sys issue with discussion and background: extendr/libR-sys#9
Note: We're using the i686-pc-windows-gnu target because that's a hard requirement set by the R project. We cannot use i686-pc-windows-msvc as a workaround, and we have not tested whether the problem exists on i686-pc-windows-msvc.
Activity
alexcrichton commentedon Dec 17, 2020
Thanks for the report! I suspect though that that this may be an issue for the compiler itself rather than Cargo? Cargo seems like it may be unlikely to be too involved in what's going on here since it's just calling rustc perhaps?
clauswilke commentedon Dec 17, 2020
I don't know what happens under the hood when I enter
cargo run, so it may be a compiler issue. I'd just want to emphasize one more time that the executable that is generated is fine and runs, as long as it isn't started viacargo run.clauswilke commentedon Dec 17, 2020
If you could point us to the relevant code in Cargo that starts an executable we could dig a little deeper and see if we can narrow down the culprit further.
alexcrichton commentedon Dec 17, 2020
Ah sorry I missed that!
In that case it does indeed sound like a Cargo issue. Cargo will adjust
PATHfor native libraries built as part of the build since that's how dynamic libraries are found on Windows. Perhaps those adjustments are causing the wrong dll to get loaded at some point?I'm not entirely sure if we have a great location to point to since the adjustment for the target process command happens in a few locations, but for this sort of error it seems like the
PATHadjustments are probably causing the issues?clauswilke commentedon Dec 17, 2020
Maybe. I don't know anything about Windows, so I'm flying blind here, but @Ilia-Kosenkov has been very good at drilling down and trying to find what exactly goes wrong.
Here is a WinDbg log from a case where the problem occurs (previously posted here):
Not sure if the paths look correct or not.
@Ilia-Kosenkov Could you regenerate this log for your simple example program, and could you also generate a comparable log for when the linking succeeds? Maybe we can pinpoint a path difference or a difference in the libraries that get loaded.
mati865 commentedon Dec 17, 2020
It's quite likely.
Maybe system libraries get mess-up when running 32-bit cargo on 64-bit OS due to SysWOW64 thing? Guess nobody has 32-bit OS to test these days?
Ilia-Kosenkov commentedon Dec 17, 2020
Here is part of the output of
WinDbgX.I first compiled the project using
cargo build --release --target=i686-pc-windows-gnu --features runtimeThen attached a debugger to
cargo run ...Details
This looks like the problematic fragment, as it starts with loading
libclanand ends withc0000139error, which isENTRY_POINT_NOT_FOUNDthat we get.After this error, all libraries are being unloaded and execution terminates.
For comparison, this is what happens if I just debug the produced executable:
Details
Again, I am not
WinDbgguru, but if needed, I can try digging deeper.Note that this is executed on a real system used every day, so
PATHis contaminated by other apps.So far the only difference I see is that
cargouses two libraries from its toolchain, while executable uses correctmingw32libs.clauswilke commentedon Dec 17, 2020
@Ilia-Kosenkov Could you check whether the same difference exists on the 64-bit system? It probably does but doesn't cause problems there. Maybe the two libraries that get pulled in from the toolchain have a conflict with zlib1.dll provided by msys2 on i686.
In general it makes me uncomfortable to mix libraries from different sources (rust toolchain vs msys2), but maybe there have been other bugs in the past that required this sort of behavior?
Ilia-Kosenkov commentedon Dec 17, 2020
@clauswilke,
Actually, I am unable to, as the WinDbg output of 64-bit
cargo runis drastically different from what I posted above, and I have to yet learn how to navigate this debugger.It feels like
cargodoes something completely different when doing64bit compared to32bit.For instance, there is no
libclangamong pulled libraries that I see, may be it runs executable out of process somehow?However, if I debug just the executable, I see similar libraries referenced (notice that I have both
mingw32andmingw64on myPATH, yet still correct libraries are found):Details
mati865 commentedon Dec 18, 2020
Rust ships mingw-w64 GCC libs from version 7.X while MSYS2 is already at 10.2.
Could you try on the CI to run this command after installing Rust? (posted from my mobile so there can a typo).
rustup component remove rust-mingwIlia-Kosenkov commentedon Dec 18, 2020
@mati865,
Seems there is no change, see GHA report
https://github.com/Ilia-Kosenkov/rust_clang_run/runs/1575413067?check_suite_focus=true
mati865 commentedon Dec 18, 2020
@Ilia-Kosenkov can you try replacing Rust shipped library with ones provided by MSYS2 (bash syntax):
clauswilke commentedon Dec 18, 2020
@mati865 Should we be able to avoid this problem by not using rustup at all and instead installing rustc and cargo via MSYS2? I'm also running into other linking problems that seem to be caused by interference between the two toolchains, and it seems to me not having two toolchains at all would be best.
alexcrichton commentedon Dec 18, 2020
Yeah if it's toolchain differences than a Rust msys2 package, if one exists, likely solves this since it'll be guaranteed to use the same version of gcc you have on your system (and won't have the bundled copy like Rust releases). The only way I know of around this on Windows is to use an MSVC toolchain to cross-compile to the MinGW target.
Ilia-Kosenkov commentedon Dec 18, 2020
@mati865,
Sorry for the delay. Your solution worked -- I moved
mingwlibs torusttoolchain folder and nowcargo run ...finishses successfully.I have tested this so far only on my machine, I will reproduce this in GHA shortly.
Ilia-Kosenkov commentedon Dec 18, 2020
@alexcrichton, @mati865, @clauswilke,
I reproduced all steps in GHA. Both solutions work -- either swapping in
mingw32libs or usingstable-msvctoolchain to compile targetingi686-pc-windows-gnu.This is the latest GHA report, the last job uses
msvc, otheri686jobs replace libraries -- evrything is green.mati865 commentedon Dec 19, 2020
@clauswilke MSYS2 Rust package is not always up to date and the main reason for it is packaging Rust crates into MSYS2 repo.
Since rust-lang/rust#76167 there is no conflict between Rust shipped and system mingw-w64 toolchains. You can test it by removing toolchain shipped by Rust (
rustup component remove rust-mingw).The conflict here comes from runtime libraries.
I think cross-compiling from
x86_64-gnutoi686-gnushould also work.When it comes to fixing:
I'm not sure if it's possible for Cargo to avoid adding
<sysroot>/binto the beginning of PATH when running the binaries since crates usingrustc-devcomponent could depend on it.Statically linking
libgcc(to avoid shipping it as a DLL) would very likely breakrustcown backtraces (backtraces in crates compiled withrustcwould still work).Updating mingw-w64 used to build Rust would workaround the issue here, I have some WIP but it's still long time from completion due to lack of time.
What Cargo (or maybe bindgen?) could do to fix this issue is copying (hard-linking if possible?) runtime dependencies next to created DLL. That would work since Windows first looks for the libraries next to your DLL/executable then searches PATH.
clauswilke commentedon Dec 19, 2020
@mati865 If I try to build an R package using Rust with
x86_64-gnuas default host I get a linker error:See e.g. here: https://github.com/clauswilke/extendr/runs/1578651175?check_suite_focus=true#step:8:145
This goes away when I use
x86_64-msvc. Note that this build process pulls in yet another toolchain, the Rtools tool chain which ships its own version of mingw64. I'm not sure how all of this fits together.mati865 commentedon Jan 4, 2021
For some unknown to me reason it tried to build
winapi-x86_64-pc-windows-gnubut there was no x86_64 mingw-w64 toolchain in the path?Ilia-Kosenkov commentedon Jan 14, 2021
mati865 commentedon Jan 15, 2021
@Ilia-Kosenkov sounds like you don't have i686 linker in
PATH:Ilia-Kosenkov commentedon Jan 15, 2021
mati865 commentedon Jan 15, 2021
@Ilia-Kosenkov that is right,
windows-gnuhosts ship native linker but don't do that for the targets. It's user responsibility to provide linker when cross-compiling.I'd suggest moving this discussion somewhere else to avoid off-topic in this issue.
mati865 commentedon Aug 18, 2021
I think there is nothing to do from Cargo perspective, if there are still issues they should be reported to main Rust repository.
Ilia-Kosenkov commentedon Aug 23, 2021
@mati865, while there can still be some issues with cross-compilation (or
mingwtoolchain) on Windows, we were able to find a way to bypass these restrictions in our project, and we are happy with that.Feel free to close this issue.
Thank you for the discussion, it helped us a lot.