Repository navigation
Distribute libc.a as a separate library instead of bundling it into liblibc.rlib on musl and wasi #72274
Description
Right now the libc crate links native libc with #[link(kind = "static")] for two targets - musl and wasi.
kind = "static" means that object files from native libc.a are bundled into liblibc.rlib and distributed with rustc toolchain in this form.
Then if user builds something with +crt-static then the bundled object files are used, and if user builds with -crt-static then they are "unbundled" and dynamic linking with libc.so available at user site happens.
Experience with windows-gnu targets (#67429) and porting rust to musl-native targets (#40113) showed that it's strongly preferable to use the native toolchain available at user site to using anything bundled with rustc toolchain, and that includes libc.a as well.
At the same time rustc historically provides more "self-contained" experience for selected targets (windows-gnu, musl, wasm), so we must continue shipping libc.a in some form in case the user doesn't have a native toolchain available.
That we means we need to fall back from using the native toolchain if it's available to using bundled libc.a (and other libraries) if it's not.
Such scheme is already used relatively successfully for windows-gnu, but not for musl and wasi.
So, the plan of the work should be roughly like this:
- Link libc as
kind = "static-nobundle"instead ofkind = "static"in the libc crate (Don't bundle static libc. libc#1327). - Modify rustbuild to distribute libc.a for musl (and whatever is its equivalent for wasi) similarly to how we distribute crt1.o objects for the same targets.
- Organize library search directories during linking in such way that they find the native toolchain first and then fall back to the rust toolchain's sysroot where the bundled libc.a resides (should be similar to windows-gnu: prefer system crt libraries if they are available #67429).
- Eliminate the "unbundling" logic from the compiler.
Once this is done #65760 should become fixed as well.
UPD: libunwind (https://github.com/rust-lang/rust/tree/master/src/libunwind) on musl also bundles libunwind.a, it should probably have the same treatment.
Activity
retep998 commentedon May 17, 2020
If only we could also eliminate
kind = "static"with fire...alexcrichton commentedon May 18, 2020
FWIW It's hard for me personally at least to read comments like "Eliminate the unbundling logic from the compiler (with fire)." and "If only we could also eliminate kind = static with fire..." as semi-attacks on me personally since I originally implemented all of this. I really do feel that much ire about these features is directed personally at me since I've tried to review patches and provide feedback on other designs, and often times things don't work out. I did the best I could with the resources and knowledge I had at the time. I'm sorry if things didn't work out.
The general idea here is "things just work". I don't think it really matters how they work, so long as they work. The "thing" here for targets like musl and wasi is "you don't need to install a native toolchain". I think anyone has much of a preference how things works so long as it works.
I'm gonna unsubscribe from this issue because I don't think I have much to offer and I'm likely to just get stressed out staying on this. Feel free to cc me if there's a specific question (e.g. about wasi), however.
petrochenkov commentedon May 18, 2020
@alexcrichton
Apologies if some of this sounded like an attack.
This post is written with 5 years of hindsight available, but when this was originally implemented nobody could predict the future.
retep998 commentedon May 18, 2020
I'd also like to apologize for my comment. You've done a lot of fantastic work on Rust over the years, and I don't want any of my comments on specific features to feel like they're directed at you!
We've all done stuff in Rust that later turned out to not be the greatest (myself included), but such is hindsight.
kind=statichas some unfortunate consequences such as LLVM mangling certain sections it doesn't understand (like Microsoft's LTCG stuff) and not being able to link to static libraries provided by the system. Unless they need to distribute rlibs with C libraries bundled in (which is rare outside of the rust toolchain) there's very little reason for users to not usestatic-nobundleoverstatic. That it has taken the easy to remember name ofstaticwhilestatic-nobundlehas been left to languish in unstable land for years is somewhat frustrating, and I wish the situation could improve.petrochenkov commentedon May 18, 2020
FWIW, I want to write an RFC for general syntax for linking modifiers for native libraries that would cover things like
--whole-archiveand--as-needed, and could cover bundling as well.Perhaps it could be less controversial to stabilize it as a modifier to
staticrather than a separate linking kind.(This will hopefully happen this or next month, otherwise I think
static-nobundlecan be stabilized as is.)UPD: Done in rust-lang/rfcs#2951.
mati865 commentedon Jun 6, 2020
That would also fix #58163
petrochenkov commentedon Jul 22, 2020
List of libraries that need to be shipped with Rust rather than bundled in rlibs:
Additionally:
crt-staticflag) on fuchsia and linux whenllvm-libunwindfeature of libstd is enabled, seesrc\libunwind\libunwind.rs.joshtriplett commentedon Sep 4, 2020
Would this make it easier to decide at build time whether to statically or dynamically link libc, for targets that currently always dynamically link libc?
petrochenkov commentedon Sep 4, 2020
@joshtriplett
I don't think so?
This issue applies to two specific groups of targets, and both of them can currently link libc statically.
joshtriplett commentedon Sep 5, 2020
@petrochenkov I realize that. I'm more wondering, if we can defer linking against libc until the final link, could we potentially decide at the final link whether to link
libc.aorlibc.so, even on glibc?petrochenkov commentedon Sep 5, 2020
@joshtriplett
Ah, yes, sure.
Whether libc is linked statically or dynamically is already decided during the final link (on targets that give this choice).
glibc-based targets don't give it right now, but changing that is a matter of switching a flag in the target spec + adding a couple of
cfglines into libc crate.(I wanted to do this for bionic libc for months, but still didn't because other tasks were more important and contributing to libc is annoying.)
smaeul commentedon Sep 5, 2020
@joshtriplett To clarify the functional impact of this issue:
For the listed targets, in scenarios where we have already decided to link statically using
libc.a(and only for those scenarios), this allows choosing wherelibc.acomes from at the time of the final link. Currently,libc.ais always the version that was inmusl-rootorwasi-rootwhenliblibc/rustcwas built. With this change, we would be able to dynamically decide between thatlibc.aand thelibc.aprovided by the native toolchain.The expectation is that the native toolchain's
libc.amay be newer (security fixes) and will match the native headers.joshtriplett commentedon Sep 6, 2020
@petrochenkov
That'd be incredible. I'd be happy to submit the necessary changes, if you can outline what needs doing. I have a need for static glibc-based Rust binaries.
mingweishih commentedon Feb 21, 2021
Is there any follow-up on this issue?
petrochenkov commentedon Feb 21, 2021
@mingweishih
This is still waiting on someone to implement it.
With item 3 done this should actually be a pretty small task at this point.
Auto merge of #2272 - 12101111:musl-bundle, r=Amanieu
Auto merge of #2272 - 12101111:musl-bundle, r=Amanieu
Auto merge of #2272 - 12101111:musl-bundle, r=Amanieu
petrochenkov commentedon Nov 24, 2021
The only thing required for implementing the first item now is bumping libc version used by this repo.
petrochenkov commentedon Nov 29, 2021
libc has been updated in #90681.
petrochenkov commentedon Feb 15, 2023
The last item ('Eliminate the "unbundling" logic from the compiler') was implemented in #105601, closing.