Repository navigation
Allow restricting the number of parallel linker invocations #9157
Description
In CI at my work, we ran into a situation where rustc would get OOM-killed while linking example binaries:
error: linking with `cc` failed: exit code: 1
|
= note: "cc" <…>
= note: collect2: fatal error: ld terminated with signal 9 [Killed]
compilation terminated.
We were able to mitigate this by using a builder with more available memory, but it's unfortunate. We could dial down the parallelism of the whole build by explicitly passing -jN, but that would make the non-linking parts of the build slower by leaving CPU cores idle.
It would be ideal if we could explicitly ask cargo to lower the number of parallel linker invocations it will spawn. Compile steps are generally CPU-intensive, but linking is usually much more memory-intensive. In the extreme case, for large projects like Firefox and Chromium where the vast majority of code gets linked into a single binary, that link step far outweighs any other part of the build in terms of memory usage.
In terms of prior art, ninja has a concept of "pools" that allow expressing this sort of restriction in a more generic way:
Pools allow you to allocate one or more rules or edges a finite number of concurrent jobs which is more tightly restricted than the default parallelism.
This can be useful, for example, to restrict a particular expensive rule (like link steps for huge executables), or to restrict particular build statements which you know perform poorly when run concurrently.
The Ninja feature was originally motivated by Chromium builds switching to Ninja and wanting to support distributed builds, in which there might be capacity to spawn way more compile jobs in parallel since they can be run on distributed build nodes, but link jobs, needing to be run on the local machine, would want a lower limit.
If this were implemented, one could imagine a further step whereby cargo could estimate how heavy individual linker invocations are by the number of crates they link, and attempt to set a reasonable default value based on that and the amount of available system memory.
Activity
luser commentedon Feb 9, 2021
I believe this would also be useful for people using
sccachein distributed compilation mode, as they could have an exaggerated version of this problem similar to what's described in that message about Chromium, with more build capacity for compiling than linking.Be-ing commentedon Feb 27, 2021
I have no idea if this would be practical, but could cargo automatically monitor memory usage to adjust how many concurrent threads to use?
levkk commentedon Oct 26, 2021
I've managed to work around this by enabling swap. Linking time did not suffer visibly. On Ubuntu, I followed this guide.
sagudev commentedon Mar 16, 2023
Proposed Solution
Adding
--link-jobsoption to specify number of jobs for linking. The option would default to number of parallel jobs.Here what would help look like (
-joption is displayed for comparison):weihanglo commentedon Oct 31, 2023
@rustbot claim
epage commentedon Nov 2, 2023
@weihanglo see also #7480
weihanglo commentedon Nov 2, 2023
FWIW, Cabel community had a discussion a while back: haskell/cabal#1529
weihanglo commentedon Nov 3, 2023
Potentially the unstable rustc flag
-Zno-linkcan separate linking phase from others (see #9019), and then Cargo can control the parallelism of linker invocations. Somebody needs to take a look at the status of-Zno-link/-Zlink-onlyin rustc (and that is very likely me).epage commentedon Nov 3, 2023
As this is focusing on the problem of OOM, I'm going to close in favor of #12912 so we keep the conversations in one place.
3 remaining items
weihanglo commentedon Nov 7, 2023
Thanks. Reopened as it might need both #9019 and #12912, and maybe other upstream works from rust to make it happen.
weihanglo commentedon Nov 9, 2023
FWIW, there is a
--no-keep-memoryflag for GNU linker. Haven't tried it but might help before we make some progress on this.https://linux.die.net/man/1/ld
weihanglo commentedon Dec 13, 2023
rust-lang/rust#117962 has made into nightly. It could alleviate the pain of linker OOM to some extent.
luser commentedon Dec 13, 2023
I suspect this will make performance much worse in the average case, unfortunately.
soloturn commentedon Oct 19, 2024
i think this can be closed, as it is the linkers business to not run out of memory. e.g. mold does this, see rui314/mold#1319 via its
MOLD_JOBSenvironment variable. to avoid cargo does everything and nothing well ...epage commentedon Oct 21, 2024
For something like that, jobserver support in a linker would be a big help so we can coordinate across rustc and the linker for how many threads are available to use.
That also only focuses on number of parallel threads and not actual memory consumption. Like with threads, likely any solution coordination will be needed between links and rustc / cargo
Don't build/test binaries with cargo test to workaround OOM issues
Don't build/test binaries with cargo test to workaround OOM issues
Auto merge of #140525 - lqd:stabilize-lld, r=<try>
Auto merge of #140525 - lqd:stabilize-lld, r=petrochenkov
Auto merge of #140525 - lqd:stabilize-lld, r=petrochenkov
kornelski commentedon Aug 22, 2025
I don't expect jobserver support in linkers to solve the problem of linking running out of memory:
The main issue isn't the number of threads. Linkers may be keeping too much data in RAM regardless of how many threads are active. I'm running into OOMs with GNU
ld(without LTO), which AFAIK is still mostly single-threaded.It's not easy to change a linker, and may not even be possible to have one with jobserver support. Platforms other than x86 Linux may require use of OS vendor's own linker. It's also tricky to customize the default linker in Cargo in a robust way, because
rustchas a bunch of-Cflags for linkers, which can break when the linker is changed using Cargo's per-targetlinkerpath. It's hard to keep these two in sync and without duplicates when builds may be overriding them individually via env vars. It gets even more complicated when cross-compiling and cmake is involved.Even if there was a jobserver support in linkers, Cargo still needs some setting to configure lower number of jobs for the linkers' jobserver.
A simple concurrency limit for the number of linker invocations would work for me, something like
build.link-jobs. Even a basic limit of just one linker invocation at a time would be helpful (which I find necessary for LTO).weihanglo commentedon Aug 25, 2025
Yeah jobserver may not help control "the number of parallel linker invocations" but only overall parallelism unless Cargo knows what calls are linker calls. Unfortunately Cargo has no control nor visibility over linker invocation now. In #9157 (comment) I linked some relevant issue we'd like to see rustc to support thought it has bug preventing Cargo from integration (#9019 (comment)).
Apart from
-Zno-link/-Zlink-only, another other possible workaround is Cargo to limit the parallelism of the rustc invocation producing binaries (e.g.,--crate-type bin|dylib|…), so we can get an approximiate linker parallelism control. This might be bad to performance because the non-codegen part of the final crate is usually not that memory heavy and can be processed in parallel.weihanglo commentedon Sep 28, 2026
FWIW a zulip discussion: #t-cargo > cargo#9157: Restrict parallel linker executions