Repository navigation
Cargo cross workspace cache #626
Description
| Metadata | |
|---|---|
| Contact | @ranger-ross |
| Team(s) | cargo (@epage) |
| Goal document | 2026/cargo-cross-workspace-cache |
| Funding contact | Jess Izen |
Summary
Work towards a build cache that is shared across workspaces to build times and reduce disk usage.
This issue is intended for status updates only.
For general questions or comments, please contact a goal owner or post in the Zulip topic.
Activity
tomassedovic commentedon May 5, 2026
This issue is intended for status updates only.
For general questions or comments, please contact the owner(s) directly.
ranger-ross commentedon May 28, 2026
Update May 28, 2026
The last week for me was spent at Rust Week and the all hands.
There was a lot of good discussion about the possibilities for how the cache may work and usecases.
We are continuing the effort to stabilize the new
build-dirlayout in Cargo as that is a requirement for the cross workspace cache.The stabilization PR (rust-lang/cargo#16807) has been posted and FCPed, and it is just waiting on the rust-lang/rust PR (rust-lang/rust#155439) to be merged to add support for the new layout to the compiler. I've been working through the issues and I believe we are quiet close to merging, just need to fix the remaining CI failures.
ranger-ross commentedon Jul 28, 2026
Update July 28, 2026
The new
build-dirlayout stabilization was merged in rust-lang/cargo#16807 but later rolled back in rust-lang/cargo#17187 due to build failures discovered in some projects on nightly.The build failure stemmed from increased arguments (primary
-L) being passed to rustc from Cargo hitting OS limits. Also a similar failure mode where the dylib search path (LD_LIBRARY_PATHon Linux andPATHon Windows) would get too long and hit OS limits.We then merged a series of fixes rust-lang/cargo#17191, rust-lang/cargo#17236, rust-lang/cargo#17168 to reduce the paths being passes to rustc and added to the search.
In addition to the fixes in Cargo, we also added arg file support to sccache in mozilla/sccache#2782 which is a work around for OS command length limits that is already supported in Cargo+rustc.
With the increase in likelihood of a project getting large enough to need an arg file this was a preemptive optimize to minimize breakage on stabilization.
With the fixes above, we are feeling better about the new layout and plan to enable it by default on nightly. (rust-lang/cargo#17258)
Instead of stabilizing the new layout immediately, we are opting to enable it on nightly by default to gather feedback and find the places we can improve before committing to stabilization.
If all goes well, you should see the new build-dir layout on nightly in the near future
ranger-ross commentedon Sep 4, 2026
Update Sept 4, 2026
The new
build-dirlayout was restabilized in rust-lang/cargo#17354 and is expected to release in 1.100.0.There were also some more optimizations made to reduce the amount of args passed to rustc when the new layout is enabled (rust-lang/cargo#17410)
At this point, the
build-dirlayout has been enabled on nightly for a few weeks and there hasn't been any new issues reported so I have claimed victory on this!With the layout stablized, I've shifted my attention back to actually implementing the cache. Until the old layout code is removed (which we will do like 1 release after the new layout lands) I have been working on prototyping the cache design. I have a few MVP implementation and design discussions are happening in rust-lang/cargo#5931.
I expect this prototyping loop to continue for another month while we iterate on cache design.
My goal is to get something into nightly in October. :)
ranger-ross commentedon Sep 21, 2026
Update Sept 21, 2026
I have continue experimenting with different cache design over the last 2 weeks.
Most of the discussion has happened in rust-lang/cargo#5931
Currently the design that seems most promising to me is having a shared content addressable storage (CAS) location for the cache, with a cache "entry" that has "pointers" into that CAS storage. (see rust-lang/cargo#5931 (comment))
I built a prototype and ran some benchmarks that do show some promising results (rust-lang/cargo#5931 (comment), rust-lang/cargo#5931 (comment))
During this discussion, the idea of using the CAS for deduplication of artifacts between multiple workspace
build-dirdirectories.Built units output artifacts would be able to be deduplicated regardless of if we can safely cache the build unit. (e.g. its a non-deterministic build script)
Currently, we are considering pursuing this as a step stone towards the build cache.
It's notable to point out that the deduplication across multiple Git worktree's of the same workspace (which has become popular in AI workflows) gets a large reduction in disk space from my testing.
CAS deduplication feature is being tracked in rust-lang/cargo#17453
Over the month, I'll continue building on the existing prototypes and refining the design.
I am still hopeful to get something on nightly in October.
ranger-ross commentedon Oct 1, 2026
Update Oct 1, 2026
Over the last 2 weeks, I've continued to experiment with CAS deduplication. (going forward's I'll refer to it as blob storage)
Implementing the blob storage and deduplication itself is not that difficult.
The tricky part is adding garbage collection in a cross platform + performant way.
Shared blob storage is going to fill up and is unaffected by
cargo clean, it actually makescargo cleanmuch less effective on it's own since deleting a file fromtargetdoes not actually recoever the disk space.This makes having garbage collection for blob storage really important for being able to land this feature.
Initially, I tried leveraging the existing Cargo garbage collection infrastructure which uses sqlite.
I had performance concerns about sqlite on large workspaces, but to my surprise it was not too bad.
But the cost of hashing all of the outputs so that we can make blobs as used was huge and not feasible.
Another approach I tried was using a
<original-file-name>.timestampfile that is hardlinked into each workspace, that serves as a "pointer" that can be updated by setting it'smtime. This approach does not require hashing files which helps performance quiet a bit, but there are concerns that this not going to work well (or at all) for extending blob storage to work remotely.I am currently exploring other approaches for how we can make blob storage garbage collection work.
If you are interested in helping with the design please get in contact with me!
The discussion about the blob storage GC implementation is primarily happening in rust-lang/cargo#17512
ranger-ross commentedon Oct 14, 2026
Update Oct 14, 2026
Over the last 2 weeks, I've continued to iterate on the blob storage and cache design.
Last week, I decided to take a step back from the blob storage PR and created a design document for the larger feature instead of trying to design pieces in isolation.
This document is a living document and is actively being updated as the design changes.
As questions have been answered, more questions have been opened and I am still working through them.
I feel I've made some meaningful progress exploring the design space and I am getting closer to a cache design that could be feasible in accomplishing our goals.
One of the many things to solve is how we handle (reduce/remove) absolute paths from the rustc output so that things are more cachable.
rust-lang/cargo#17573 was a first step, but there still a lot of work to be done here.
Design discussions are happening in #t-cargo > Cargo remote cache / deduplication design
(Feedback is welcome! Hackmd comments, in the Zulip thread, or on the github issue rust-lang/cargo#5931)