Repository navigation
Replace submodules with subtrees #70651
Description
cc @rust-lang/clippy @RalfJung
cc @rust-lang/compiler @rust-lang/infra we gate on clippy soon, so if you break clippy you need to fix it immediately in the same PR. This is trivial now though, since we are basically in a monorepo state, so you just edit stuff in src/tools/clippy/ until everything works again.
Synchronizations of the remote repo and the local subtree is explained in CONTRIBUTING.md
The first port (clippy) is being done in #70655
Reacted by Matthias Krüger, Phil Hansch, Ralf Jung, Wesley Wiser, Josh Triplett, Félix Fischer, Geoffry Song, Laurențiu Nicola, Tyler Mandry, TennyZhuang and 1 more
Reacted by kennytm and Changsheng Wu
Activity
RalfJung commentedon Apr 1, 2020
Cc @rust-lang/miri because we have that now :D
eddyb commentedon Apr 1, 2020
Hmm, I was led to believe that
git subrepowas nestinggitrepositories, in a way whichgitnatively handles. But it turns out it's just a monorepo sync tool, same as manually making large commits. I don't think losing commit history is acceptable.At a glance,git subtreelooks like it might do the right thing, i.e. keep the repository nesting of a submodule, but I haven't tried it yet to see how it actually behaves in practice.EDIT: ah, no,git subtreejust seems to create synthetic commit histories. Still, could be better than a single sync mega-commit.EDIT2: somehow it still managed to preserve commit history?! Compare:
https://github.com/rust-lang/rust-clippy/commits/master
https://github.com/rust-lang/rust/commits/c211cea3e99e04c2980a853b6637de22881b72eb
[-]Replace submodules with subrepos[/-][+]Replace submodules with subtrees[/+]oli-obk commentedon Apr 1, 2020
I moved to git subtree.
eddyb commentedon Apr 1, 2020
In case it gets lost on the PR, I said: #70655 (comment)
RalfJung commentedon Apr 1, 2020
Would be good to get the pro's and con's of that decision documented somewhere.
git submoduleadvertises itself as a better alternative togit subtree; why issubtreestill the better choice for us?(Also I have no idea in which thread to post now, there's at least this issue and 2 PRs. It's very confusing.^^)
oli-obk commentedon Apr 1, 2020
I don't know why
git subrepois better thangit subtreeother than what the documentation ofgit subrepostates at https://github.com/ingydotnet/git-subrepo#benefitsgit subtreeis better thangit subrepobecause it preserves all commits in history instead of squishing them in the name of the person who synchronizes it.eddyb commentedon Apr 1, 2020
git subtreeis upstream, which is nice, and it also supports preserving the commits better.I believe the "rebase failures" that
git subrepotalks about are actually an anti-pattern: rebasing a merge commit, it's not meant to work at all (the rebase will include both branches and lose the merge commit - this can already happens if you accidentally rebase the wrong thing inrust-lang/rust).git subreposeems to want to just throw all the changes into single large commits so you "don't have to deal with it", whereasgit subtreeis more history-preserving-oriented, which I prefer.RalfJung commentedon Apr 1, 2020
Is this true in both directions (subtree pull/push)?
Does
git subtreealso have the problem that a "subtree push" needs a commit on the rustc side?oli-obk commentedon Apr 1, 2020
no,
git subtree pushdoes not need a commit on the rustc side.The only downside of
git subtreeis that you need to specify the folder name and target repo on every synchronization (because there's no synchronization metadata file)Manishearth commentedon Apr 1, 2020
I would prefer the history preserving option. I was under the impression that subrepo had a no squash mode but that could be subtree.
RalfJung commentedon Apr 1, 2020
36 remaining items
bjorn3 commentedon Oct 31, 2020
The crate metadata doesn't refer to the repo, so posting it here for others trying to find it: https://github.com/mominul/gitree
RalfJung commentedon Mar 26, 2021
Using subtrees can lead to spuriously re-closing issues that were re-opened. This just happened to #82080:
I am not sure what to do about this, but we have a notable risk here of losing track of issues by accidentally closing them.
Can we make it so that the commit that is synthesized for clippy doesn't just have its non-clippy changes removed, but also has its commit message "sanitized" to ensure that it will not cause any automatic actions by GitHub?
bjorn3 commentedon Mar 26, 2021
I think git subtree would then consider it a different commit and try to pull the original conmit every time you sync.
flip1995 commentedon Mar 26, 2021
There's the option in subtree to squash-sync the commits to rustc. But that way we would lose the Clippy history in rustc.
If we want to keep the history of Clippy and keep using subtree, we just have to live with this, I guess...
RalfJung commentedon Mar 26, 2021
Why that, doesn't it have some kind of bookeeping to match up the commits? Relying on commit messages seems rather fragile?
bjorn3 commentedon Mar 26, 2021
I assumed it required the commit to be identical where possible. Based on a cursory reading of the source it seems to only require the tree to be the same: https://github.com/gitgitgadget/git/blob/fe2e4819b869725f870cd3ce99f1f8150fe17dc1/contrib/subtree/git-subtree.sh#L598
flip1995 commentedon Mar 26, 2021
Copying comments from #82080:
@jplatte wrote:
@flip1995 wrote:
RalfJung commentedon Mar 26, 2021
Good point. Not doing that for the "active" messages could also help. (It'd also cut down on notifications, if bors removed @mentions as well.)
Manishearth commentedon Mar 27, 2021
I feel like that's a pretty rare occurrence and unfortunately the tooling doesn't support doing such a transform currently. We could tweak the bors thing though
RalfJung commentedon Mar 27, 2021
Well, it's also something that has a high chance of flying below the radar when it happens. All we know is it happened at least once.
Auto merge of rust-lang#82208 - jyn514:rustfmt-subtree, r=Mark-Simula…
tmandry commentedon Jan 11, 2022
FYI there's been some discussion of experimenting with miri on Fuchsia, and transitioning miri to an always-green subtree is one of the primary blockers.
oli-obk commentedon Jul 3, 2023
Clippy is now using subtree, miri is using https://github.com/josh-project/josh/ (a Rust thing that is better and more powerful in various ways than subtree)