Repository navigation
Tracking issue for RFC 1977: public & private dependencies, as it relates to the resolver #6129
Description
Summary
RFC (original, superseded): #1977
RFC: #3516
Compiler tracking issue: rust-lang/rust#44663
Implementation:
- part of the infrastructure for public & private dependencies in the resolver #6653
- Implement the 'frontend' of public-private dependencies #6772
- add public & private prop tests. #6962
- Support
publicdependency configuration with workspace deps #12817 - Limit
exported-private-dependencieslints to libraries #13039 - fix(resolver): Remove unused public-deps error handling #13036
- Add
--publicsupport forcargo add#13037 -
cargo addshould check public transitive dependencies when auto-picking a version #13038 - Switch public/private dependencies to a non-blocking feature gate #13308
- Include public/private dependency status in
cargo metadata#14502 -
Command to update Cargo.lock to minimal versions (Command to update Cargo.lock to minimal versions #4100) -
Makecargo publishuse the minimal versions allowed by Cargo.toml - Ensure quality of error messages is sufficient
-
cargo fixsupport forexported-private-dependencies#13095 - Consider public dependencies when choosing a version in cargo add (#1… #15966
Non-blocking further improvements
- Add
Cargo.tomlcontext toexported-private-dependencieslints #13096 -
Proper flag to display public-private dependencies in(fix(tree): Switch fromcargo-tree(Tracking issue for RFC 1977: public & private dependencies, as it relates to the resolver #6129 (comment))--depth publicto--edges public#16081)
Considerations for stabilization
Unresolved Issues
- How do we handle unifying
publicbetween more than one dependencies table (normal, target A normal, target B normal), bothpublic = <bool>and default andpublic = truewithpublic = false - How to show only public dependencies in
cargo tree, see also Tracking issue for RFC 1977: public & private dependencies, as it relates to the resolver #6129 (comment)
Known Issues
cargo addthat auto-selects a version req from a transitive dep can have confusing output due to extra resolve (Consider public dependencies when choosing a version in cargo add (#1… #15966 (comment))
Future Extensions
About tracking issues
Tracking issues are used to record the overall progress of implementation.
They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions.
A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature.
Instead, open a dedicated issue for the specific matter and add the relevant feature gate label.
Original issue
This is a sub part of the discussion of implementation of RFC 1977, specifically how the resolver could implement this protocol. The main Tracking issue is at rust-lang/rust#44663.
This comment rust-lang/rfcs#1977 (comment) described the properties we want to uphold the way that fits best with my brain.
So to use the termanology, I feel like when we are adding a packedge B we need to walk up the dependency tree to all potential C's that can see the addition to test if there is a conflicting A.
That can be done pretty fast if we have a fast way to walk up dependency tree and a fast way to see all the pub reachable packages for each ancestor. (Neither of which we have at this time.)
But that feels like a lot of state to be cloned for each tick.
Currently we avoid the expensive clones with extensive use of RC.
Is it time to add a im-rs dependency?
Is there a better algorithm?
Is there a small part that would make a good starting PR?\
Activity
alexcrichton commentedon Oct 5, 2018
Adding new dependencies seems fine by me! My historical strategy with the resolver has been "don't add any fancy representation tricks, profile Servo, add tricks as necessary"
Eh2406 commentedon Oct 17, 2018
I have a branch that stubs out getting this working. It involves all the ugly things I can think of. The state is a mess of big nested data structures. The algorithm is a bunch of nested loops in the hottest part of the resolver. It is not even checking for redundant work. The code is mostly repeated between the code that check if a candidate can be added and where the the candidate is added. Several existing optimizations are completely removed in the case that a pub-dep has ben looked at. The error messages are just "Todo", and most of the commit messages are "oops".
But it is working! I modified the passes_validation to check for pub-dep conflicts, then modified the
registry_strategyto make them. It promptly generated a failing case. Copy the result to a standalone test, hand minimize, and add an explanation. Add something, anything to get that test passing. Then rinse and repeat. Lets just say that the fuzzer is very good at finding every optimization that this can react (badly) with. Eventually it passes!Next try the
limited_independence_of_irrelevant_alternativesand if finds a case of a registry that should work, but doesn't. Then rinse and repeat 3 more times. Finding backjumping optimization that need to be handled more correctly, disabled for now. But with the backjumping optimization disabled, it now finds a case that experiences exponential blowup.So my plan for the Impl day at RBR is to attempt to get
backjumpingworking, and clean my history into something submittable.edit: the cleaned up history is at https://github.com/Eh2406/cargo/tree/pub-dep
Eh2406 commentedon Oct 19, 2018
Cc @necaris
Eh2406 commentedon Oct 24, 2018
So at RBR I got the history cleaned up. Alex and I reviewed the progress so far and a plan for how to re enable backjumping without losing correctness.
That backjumping strategy was implemented today. Unfortunately, the fuzz tests are finding kasese of exponential blowup (50_000 ticks). I suspect it is similar to #5213, in which we backtrack to where
ConflictReasonno longer applies without getting closer to the gole. Not that I really understand it well enough to explain / make a test case.On a different note I also ran the other resolver tests, we have several that are hitting the 30 sec timeout on this branch. So we will need to work on per tick performance before this can merge.
Eh2406 commentedon Oct 24, 2018
Good news If you run proptest overnight with
--releaseit does an ok job of minimizing the problem (~100 lines, hand minimized to ~30). Bad news, I no longer think the backjumping strategy that I and Alex came up with is correct. I.E. it will reject cases it should except. In addition to being incomplete, allowing exponential blowup.Eh2406 commentedon Oct 29, 2018
I am (for now) ignoring the fact that the current strategy will fail to resolve when there is a valid solution in rare circumstances. I have cleaned up the case of exponential blow up, so I understand why it is happening. What I don't understand is why It is not happening using normal deps. (it is a lot harder to write, but should be possible.) I am investigating that, hoping that I will either find an optimization I forgot to add to pub/priv deps, or a case that can be reproduced on master.
Eh2406 commentedon Oct 29, 2018
I can now confirm that the
normal depsgets saved by #5213, where thepub depsdo not.Unfortunately, a simple solution did not work. I think doing something special for
PublicDependencyis not going to work. But I worry that doing the obviously correct thing will just move the slowenes around. I guess jump off that bridge when we come to it. here gose.Eh2406 commentedon Nov 13, 2018
I implemented the "obviously correct thing", and the fuzz tests pointed out that it was not backtracking correctly. I assumed I had done something silly, but could not find it. Then I got distracted by #6258. Coming back to it the minimized test was correctly pointing out that the "obviously correct thing" will not work. It is possible to make a pub-dep violation, without changing
Activations, by changing which previously activated version a dependency resolves to. So any iplomantation ofis_conflictingthat just looks atis_activewill not backtrack correctly in the presence of pub-dep. To complicate things that implementation detail ofis_conflictingwas just used for a new data structure in #6283, making it much harder to change.Auto merge of #6336 - Eh2406:im-rs, r=alexcrichton
Eh2406 commentedon Dec 8, 2018
I have a branch where I have implemented both versions simultaneously. Like the "obviously correct thing" it records every PID in the path from the dependency that can see the conflict to both conflicting crates. Like the strategy Alex and I came up with at RBR, it ensures the parents relationship is still active when the Back Jumping occurs. Because it looks at the dependency tree it passes the test the previous strategy failed. Because it will records all the PIDs involved in conflict it can learn like #5213. Unfortunately, because it adds a lot of things to the conflict it is a highly efficient way of creating a test case like #6283. So efficient that, In debug the fuzz test only tell me that it hit the 50k limit, In release the fuzz test only tell me that there are cases that took longer than 90sec to resolve!
I have to say, adding new features to the resolver has gotten a lot harder now that we have fuzz tests making sure that they are correct, complete, and efficient. :-P
Eh2406 commentedon Feb 18, 2019
Copied from #6653 (comment):
So the specific questions are:
How does this interact with renamed dependencies?
There are lots of juicy corner cases here. On one had being able to depend on 2 different versions of the same library is one of the points of renamed dependencies. Fore example so you can publicly derive
serde1andserde2traits on the same type. On the other hand making sure you and your dependencies agree on the same version ofserdeis the hole point of public dependencies.This makes my head hurt, is there something simple I am missing? If not, where is the appropriate place to discuss fundamental problems with an accepted RFC?
How does this interact with
'cfg({})'ing in different versions?Conceptually this one is very simple. Instead of asking "is there a package that can see two different versions of a dependency" we need to ask "is there a set of arguments so that there is a package that can see two different versions of a dependency". But the complexity is in the details. If we are not careful this is a SAT problem that needs to be solved for each tick. Where is the existing
'cfg({})'logic handled?alexcrichton commentedon Feb 19, 2019
For renaming dependencies I think the main question is that it's the first time you've been able to depend on two versions of the same dependency. The name itself doesn't matter so much (it's only relevant when compiling the local crate), but the fact that you can publicly depend on two versions of the same crate does indeed create a weird situation. My best guess of what to do here is that, like we'd probably always need anyway, an escape hatch would be used to say "please don't consider this for public crate graph resolution" or something like that.
For
cfgthat's also a good question, but I think the way to go is to probably take the route of the rest of Cargo's resolver and basically ignorecfg. We resolve a lock file assuming that allcfgclauses are satisfied and we only actually apply the target-specific dependencies in a later pass.25 remaining items
weihanglo commentedon Oct 6, 2025
Not blocking but we meed to consider a way to display public-private dependencies in
cargo-treePosted by weihanglo in #15243 (comment):
Posted by epage in #15243 (comment):
fix(tree): Switch from `--depth public` to `--edges public` (#16081)
epage commentedon Oct 10, 2025
#16081 changed
cargo tree --depth publictocargo tree --edges publicobi1kenobi commentedon May 22, 2026
Stabilization concern: I'd love to explicitly define and document the intended behavior in the case where both of the following are simultaneously true:
foolistsbaras a public dependency, andfoo::Ahas aDerefimpl pointing tobar::B, plusbarlistsquuxas a public dependency, andbar::Bhas aDerefimpl pointing toquux::C.In this case:
appears to be valid public API usage of
quuxinfo that syntactically appears to be direct fromfoo, despite a lack of a pub dependency onquux. This happens becauseDeref / DerefMutare applied recursively.It's possible this is okay! But I think it's important to proactively address any possible misunderstanding that may rhyme with "
foo's public API only includes items fromfooand its direct public deps." That is not the case — in this case, a transitive public dep is also part of thefoopublic API surface, at least from a syntactic perspective.(Raised at Rust All Hands 2026 as a result of in-person discussion 🙌)
epage commentedon May 22, 2026
This would best be opened as an Issue in the rust-lang/rust repo and we can link to it from the rust tracking issue which is for the lint itself while this is inteded for Cargo's handling of the lint.
epage commentedon May 22, 2026
I should add that in creating a reproduction case to file an issue, you will need to be careful about not hitting rust-lang/rust#119428.
epage commentedon May 25, 2026
I talked with @Mark-Simulacrum about this at the all hands.
They would be fine stabilizing the compiler lint as
allowand move it towarnwhen the bugs are fixed, getting this into people's hands sooner to get some insight and so they populate more data for tooling to leverage.We talked a bit about the required MSRV bump. A rename of the field would resolve that. I tried that with
pubbut we preferred not using rustc abbreviatons.privatewould work and would align with the proposal for features but that means people would mostly be writingprivate = falsewhich isn't great. We could supportpublicandprivateand maybe remove support forprivatein the 2024 edition since that has a higher MSRV thanpublic. Or as an undiscussed idea we could havevisibility = "public"which gives features more flexibility, keeps features and deps consistent (except defaults) but is more wordy. It might also be good to diverge from features so users don't get confused on the defaults.If the plan is a deprecated
private = false, I would lean towards a deferral just so we can better focus on polishing what we do have and getting it stable. We can then evaluate how much of a need there is for pre-1.83 msrvs.epage commentedon May 26, 2026
@Mark-Simulacrum would stabilizing the lint as
allowclose out the tracking issue? If so, should we create a separate on for tracking changing it towarn?Mark-Simulacrum commentedon May 27, 2026
I think that would make sense, yes, or just repurposing the existing tracking issue - my guess is starting a new one is generally nice to keep it cleaner though (especially given age of these features).
Eh2406 commentedon Jun 19, 2026
Yes, we need to make sure people understand that your API surface does not stop at your direct dependencies it is transitive. I assumed that most of the people involved in these conversations know that, but that may be a bias on my part based on how long I've been thinking about these problems. We should assume that when talking to other people developing this functionality when they say "public dependencies" they do not mean direct dependencies they mean recursive transitive closure. But also, when we get to writing public communication about this functionality We need to make sure we call that distinction out specifically.
epage commentedon Jun 19, 2026
And
cargo tree --edges publicis a great way for people to see the impact of this.