Repository navigation
provide a way of overriding the MinGW library path #68887
Description
The recently landed #67429 introduced a heuristic to search for the system MinGW libraries relative to the cross-gcc (used as linker) itself. However, there are systems that do not follow this pattern. For example, on my Gentoo system:
/usr/bin/x86_64-w64-mingw32-gccis a symlink to/usr/x86_64-pc-linux-gnu/x86_64-w64-mingw32/gcc-bin/9.2.0/x86_64-w64-mingw32-gcc/usr/x86_64-w64-mingw32/usr/lib/is the directory with the actual libraries in the/usr/x86_64-w64-mingw32sysroot
It would be useful to have a way of overriding the heuristics and informing rustc about the correct library path manually. In the current configuration even RUSTFLAGS="-L /usr/x86_64-w64-mingw32/usr/lib/" does not help because the rustc-provided libraries have higher priority.
Activity
mati865 commentedon Feb 7, 2020
cc @alexcrichton
My comment in the PR:
alexcrichton commentedon Feb 7, 2020
My hope is that this would be of far lesser urgency than #67429 because this only affects the startup object files which ideally change far less in the gcc toolchain than the rest of the toolchain. The main issue was that we'd prefer our own import libraries and such, which really was only effective if they came from the same toolchain as the host toolchain, which they rarely did.
mati865 commentedon Feb 7, 2020
It'd be nice to have Rust's MinGW follow latest MinGW release but it's not backwards compatible: https://www.diffchecker.com/mjMudsm6
Although it's worth mentioning the last breaking change to the CRT happened 2 years ago. This means using newer MinGW would already increase amount of systems where it just works.
How does
*-linux-gnutarget work in regard of CRT? It doesn't ship startup objects unlike*-linux-musland*-windows-gnutargets.alexcrichton commentedon Feb 7, 2020
The toolchain used for MinGW has historically been "whatever is on the builders", so it's not strictly kept in sync with something or intentionally held back. It should be updateable at any time!
The non-MinGW targets all rely on the linker to pull in the CRT. None of them work without a preinstalled toolchain, though, whereas one of the purposes of the MinGW toolchain is to work without a preinstalled toolchain.
mati865 commentedon Feb 8, 2020
@alexcrichton there is another case of nonstandard path. 😔
This time on Windows when MinGW is installed through Chocolatey #68872 (comment)
I'll check if it is using symlink which we follow but I'm at a loss.Chocolatey install wrappers so there is really no way to get location of original executable.
alexcrichton commentedon Feb 10, 2020
Sorry I don't know of a great way to fix things, I know very little about MinGW myself.
rivy commentedon Feb 15, 2020
@mati865 , On AppVeyor CI (which uses a choco install of mingw), while testing, I used the relative path "..\lib\mingw\tools\install\mingw64" from the wrapper to the main directory of the install. I believe that should be a relatively stable path. It should be at least testable.
rivy commentedon Feb 15, 2020
@mati865 ,
After a bit of googling and stackoverflow search, I found a second, more robust, though more complicated/slower, method of finding the installation main directory (ref: SE). Using the
--shimgen-noopoption togccreturns a reference to the true installation path (seepath to executable).I hope that's helpful.
mati865 commentedon Feb 15, 2020
@rivy Rust could get library path from
gccitself but calling executables on Windows is slow.I was about to add Chocolatey structure as one of the paths (and fix Cygwin as well) but there was a lot going on for me this week.
rivy commentedon Feb 17, 2020
@mati865 , yep, I think using the "usual" relative path (ie, "..\lib\mingw\tools\install\mingw64") would be the better choice. I guess you could fall back to parsing the
gcc --shimgen-noopoutput, but that's a lot of effort for a low percentage occurrence.I do want to say thanks for your efforts on this problem.
tanriol commentedon Feb 17, 2020
Still, how about a fallback to some config item or environment variable so that it can be made work on unsupported layouts too without rebuilding rustc?
25 remaining items
saurik commentedon Jul 2, 2020
-Z link-self-contained=noworked so well--for both 64-bit and 32-bit--that I was even able to use clang as a wrapper for the linker instead of having to directly specify mingw's ld! \o/ I guess the question I have now is: if the user is specifying LINKER, shouldn't link-self-contained default to no? (I'm assuming the idea of having this self-contained environment in the first place is to avoid people having to bring their own install of MinGW, and so one is being shipped with rustc?) If you have an overridden LINKER, it seems like the probability that the libraries that come with rustc will be exactly compatible is "slim" (and so I'd expect "user brought their own linker, but wants to roll the die with the rustc embedded libraries" to be the corner case scenario that should require a flag override, not "user brought their own linker and thereby probably the linker works"; I didn't bring this up before because I had assumed there was some reason all these flags had to be passed manually, and thereby the paths needed to be known, but now seeing that "just call the linker" was an option from the beginning it seems like it should be the default in cases when the user provided the linker).mati865 commentedon Jul 2, 2020
@saurik the plan is to always disable self contained mode when cross compiling and on Windows when the linker location is outside of Rust sysroot.
To do it properly we need to stabilise
link-self-containedfirst but that requires MCP.There is open MCP already for what #72999 did and if it's not approved all the changes made so far will be probably reverted: rust-lang/compiler-team#310
mati865 commentedon Jul 16, 2020
Status update:
MCP was seconded and since the beta was cut I've opened #74410 to completely ignore shipped CRT objects when cross compiling.
saurik commentedon Jul 16, 2020
@mati865 To verify, by "cross compiling" here we mean "when the user is using a toolchain that isn't the system default toolchain (and thereby has provided their own LINKER)", not "when the user happens to be compiling for a host/target architecture that is different from the build/host architecture" (to use x/y as gnu/cargo naming)? (Like, if I am on Win32 compiling for Win32, many people would not call that "cross compiling"--though I can see an argument that it is, and it is obvious to see that when one considers one things like "I am on Linux and am compiling a binary for Linux but I am on a system with glibc 2.23 and need the resulting binary to run on a system with glibc 2.17 and so I have a sysroot for an old version of CentOS that is my host/target system while a very recent build of Ubuntu is my build/host system": in some sense I am "cross compiling" from Ubuntu bionic to CentOS 7--but if I am specifying the linker and thereby have a sysroot I would need to not have these shipped objects injected no matter the architectures involved. I mostly ask this question as I ran into a really annoying bug in Cargo rust-lang/cargo#8147 wherein it incorrectly makes assumptions about toolchains based on architecture.)
mati865 commentedon Jul 16, 2020
@saurik that's a good point. Until
link-self-containedoption is stabilised and heuristic for automatic external linker detection is added, it's expected when user provides own linker self-contained mode makes no sense. Do you want to open PR yourself?saurik commentedon Jul 16, 2020
@mati865 FWIW, "it's expected when user provides own linker self-contained mode makes no sense" is what I agree with / want, so if you are saying that will be the behavior I have no issue? My belief is just that "self contained mode" is the "narrow" (even if very common) case that only makes sense if the user is using the rust-provided linker (and that that has nothing to do with "cross compiling").
mati865 commentedon Jul 16, 2020
My proposal for now:
-Z link-self-contained=yes/notakes precedence over anything else, it will become stable-Cflag at some pointWould you like to implement and open PR for "user manually specified the linker: for now just disable self-contained mode" (since it's your idea) or want me to do it?
saurik commentedon Jul 17, 2020
My proposal would be:
if LINKER is set, then use that linker and assume it isn't self-contained (as the user should not even know where that linker is located to specify it manually... and if they do, honestly, I bet they are doing something epic and want all the magic behavior to go away);
otherwise if target is a platform for which rust ships a self-contained toolchain (I know of Win32 and Linux+musl or whatever) then use self-contained toolchain;
otherwise (if the user didn't specify LINKER and is expecting rust to just guess at a linker, but rust also doesn't ship its own preferable linker) if the user has a useful-looking linker on their path then use that linker without self-contained and hope (really hard) that it works well enough to make a useful binary (and if the user wants this linker over the default preferable self-contained one they should specify it manually with LINKER, which would throw them into the earlier case);
otherwise, throw an error to the user telling them they need to provide a linker.
I don't see why host != target is relevant at all... :(. But, I will note that as I have said in passing a couple times, I am making an assumption about what this self-contained mode is even for and who is using it: my assumption is that this is essentially a compete toolchain with sysroot/libraries that ships with rust so the compiler can do builds for a handful of platforms that are common targets for which the user is unlikely to have a toolchain installed on their current computer, such as for non-standard libc on Linux, Win32, or maybe emscripten or something. If so, I would expect this to be purely a "fallback" option that would always be available for these targets regardless of the host and the tooling for which might cause "unwelcome" command line arguments if the user manually specified LINKER to use a linker they brought, and which would cause mix-and-march issues--like we are seeing in these various ones I have run into--were you to attempt to use the rust libraries with a linker that came from someone else, no matter where that linker is located... like, my confusion on the use case here is so strong I honestly don't see why you would have this as a -C flag: either you are using the self-container linker or you aren't, and there doesn't seem like there should be any grey area where we have to ask the user.
Is the issue that rust isn't shipping a linker, and is only shipping the libraries? If so, I feel like that just needs to be fixed, as otherwise that is going to just keep causing problems forever; like, there is simply no forward compatibility guarantee for libraries: even on macOS old linkers can't link against new libraries as the load commands at the head of the binary are extended with new features every year. A linker and a sysroot aren't exactly glued together, but there is at least a loose level of coupling between them and it isn't safe to just mix and match them: either don't ship libraries and expect the user to bring them or ship a full toolchain.
My confusion with PATH might be that I don't know if you are saying this is for linkers found on the path by rust without LINKER or linkers found on the path that were manually specified with LINKER. In the former case I see the relevance, but then "later" is strange as both rules are relevant, as custom linkers are often--but not always--not on your path and adding them to the path to be found might not work a seat the name might be strange. In the latter case, I am just really confused as most of my time on this issue I was actually using the system copy of MinGW that I installed from homebrew/Ubuntu: I only switched to an off-PATH linker at the very end when I "could" (as I happen to prefer using an off-path copy of clang from the Android SDK as a wrapper for the on-PATH copy of mingw-ld... but I could just as easily be using the distribution's default install of clang and maybe even lld instead of mingw and now everything is both on path and strange); but even to the extent to which my toolchain is off-PATH today doesn't mean I might not be setting my PATH differently tomorrow (and sometimes I do this) so that weird folder is on my path.
Taking a further step back, once I saw how the command line being passed to my linker was complete "vanilla" and it worked, I don't even understand what this option does: why not pass the vanilla options to your own self-contained linker? If the paths are internally wrong on that linker and it can't find it's sysroot, why not fix that in the self-contained linker? I would expect the self-contained linker to work just like my linker does, and my linker "just works". Why does the rust self contained linker fail to "just work" in a way that requires the crt objects to be manually passed in?
(As for a PR, I am not in a position to test this without some maybe-hopefully-minimal instruction. I would be willing to accept said instruction, and maybe that will mean you get more patches out of me in the future ;P, but I just don't know what all I need to recompile to make this kind of change or then how to use that alternative copy: I have been "using" the rust toolchain, but I have so far not developed on it myself. Note that I continue to be willing to provide some compensation for you doing changes--assuming you agree with them, of course--in addition to paying for your time on this already that I might have caused, as me doing my job to pay you to do these tasks probably is arguably efficient for the world than me doing this directly; but again: if there is some minimal-investment-to-you way to tell me what I need to rebuild to make a change and test it, given that I seem to be resourceful enough to understand how to use the tooling itself and have been a maintainer of compilers in the past, I am definitely willing to do that and maybe seeing what I am suggesting manifested concretely will make more sense, or the process of me editing the code will help me understand the use case and thereby come up with a more reasonable suggestion for what to change.)
mati865 commentedon Jul 17, 2020
I'll get to that wall of text later 😉.
For now I'll address only
For MinGW the linker is shipped only when it's the host compiler and is preferred over external linkers in the PATH. So we assume that if user overridden the linker they don't want self contained mode.
However when cross compiling users don't need to specify the linker because it will be taken from the PATH. Hence self contained mode is disabled entirely.
mati865 commentedon Jul 30, 2020
@saurik Apologies, I was quite busy recently.
The change itself is trivial and comes to adding condition that check if linker option is none, similarly to:
#74410
There is a (big) catch however, testing this change is complicated. You have to modify https://github.com/mati865/rust/blob/08990e5c7554d4d6a0440debc2edc99c8e9565c8/src/librustc_codegen_ssa/back/link.rs#L1017 to always return
None, then copy and paste few lines in CI config to create dist archive for windows-gnu.Finally you need to run it on Windows machine with MinGW compiler added to the PATH and test that:
-C linker=gccwas not specified rustc picks CRT objects from the sysroot-C linker=gccwas specified rustc picks CRT objects from the MinGW in the systemAuto merge of rust-lang#74410 - mati865:mingw-no-self-contained-when-…
Rollup merge of rust-lang#76167 - mati865:mingw-self-contained-heuris…
mati865 commentedon Sep 3, 2020
-C link-self-containedhas been stabilised in nightly and MinGW detection was reworked in the next nightly.If somebody still encounters the issue with tomorrow's nightly, please open new report.