Replace submodules with subtrees

[oli-obk]

cc @rust-lang/clippy @RalfJung

cc @rust-lang/compiler @rust-lang/infra we gate on clippy soon, so if you break clippy you need to fix it immediately in the same PR. This is trivial now though, since we are basically in a monorepo state, so you just edit stuff in src/tools/clippy/ until everything works again.

Synchronizations of the remote repo and the local subtree is explained in CONTRIBUTING.md

The first port (clippy) is being done in #70655

[RalfJung]

Cc @rust-lang/miri because we have that now :D

[eddyb]

Hmm, I was led to believe that git subrepo was nesting git repositories, in a way which git natively handles. But it turns out it's just a monorepo sync tool, same as manually making large commits. I don't think losing commit history is acceptable.

At a glance, git subtree looks like it might do the right thing, i.e. keep the repository nesting of a submodule, but I haven't tried it yet to see how it actually behaves in practice.
EDIT: ah, no, git subtree just seems to create synthetic commit histories. Still, could be better than a single sync mega-commit.
EDIT2: somehow it still managed to preserve commit history?! Compare:
https://github.com/rust-lang/rust-clippy/commits/master
https://github.com/rust-lang/rust/commits/c211cea3e99e04c2980a853b6637de22881b72eb

[oli-obk]

I moved to git subtree.

[eddyb]

In case it gets lost on the PR, I said: #70655 (comment)

Whoa, clippy commit hashes were kept intact, this makes me really happy!11

[RalfJung]

I moved to git subtree.

Would be good to get the pro's and con's of that decision documented somewhere. git submodule advertises itself as a better alternative to git subtree; why is subtree still the better choice for us?

(Also I have no idea in which thread to post now, there's at least this issue and 2 PRs. It's very confusing.^^)

[oli-obk]

I don't know why git subrepo is better than git subtree other than what the documentation of git subrepo states at https://github.com/ingydotnet/git-subrepo#benefits

Fixes known rebase failures with git-subtree.

git subtree is better than git subrepo because it preserves all commits in history instead of squishing them in the name of the person who synchronizes it.

[eddyb]

git subtree is upstream, which is nice, and it also supports preserving the commits better.
I believe the "rebase failures" that git subrepo talks about are actually an anti-pattern: rebasing a merge commit, it's not meant to work at all (the rebase will include both branches and lose the merge commit - this can already happens if you accidentally rebase the wrong thing in rust-lang/rust).

git subrepo seems to want to just throw all the changes into single large commits so you "don't have to deal with it", whereas git subtree is more history-preserving-oriented, which I prefer.

[RalfJung]

git subtree is better than git subrepo because it preserves all commits in history instead of squishing them in the name of the person who synchronizes it.

Is this true in both directions (subtree pull/push)?

Does git subtree also have the problem that a "subtree push" needs a commit on the rustc side?

[oli-obk]

no, git subtree push does not need a commit on the rustc side.

The only downside of git subtree is that you need to specify the folder name and target repo on every synchronization (because there's no synchronization metadata file)

[Manishearth]

I would prefer the history preserving option. I was under the impression that subrepo had a no squash mode but that could be subtree.

[oli-obk]

I was under the impression that subrepo had a no squash mode but that could be subtree.

subtree doesn't squash by default and has a sqash option.

[flip1995]

@oli-obk IIUC, Clippy can then fix a nightly version and doesn't have to use the rustc master anymore (correct?). If that's the case, I'd go ahead and make these changes to the Clippy CI/infra, so it will be ready once the subtree PRs are ready to get merged.

[oli-obk]

Clippy can then fix a nightly version and doesn't have to use the rustc master anymore (correct?). If that's the case, I'd go ahead and make these changes to the Clippy CI/infra, so it will be ready once the subtree PRs are ready to get merged.

well, only if we make the synchronization at specific commits that work on the current nightly. Between two nightlies there can be many states where the clippy subtree will only compile with a master-rustc.

[flip1995]

IMO we want to do this, since otherwise, we would have to do rustup commits in Clippy just as often as before. It will just be easier, since it will only be a git subtree push.

[oli-obk]

Well, we can do it at random points in time, but yes, we can target those points in time to the commit that caused the most recent nightly to be created.

[Manishearth]

Right, we can have a nightly clippy CI job that notices when clippy stops building with master, and sync then.

[oli-obk]

at that point we can just have a nightly job that synchronizes once a night automatically to ensure that even nonbreaking changes get synchronized as often as possible.

[mati865]

Clippy's CI could perform pull every time it tries to merge PR.
That would avoid most of conflicts (from maintainer PoV) when new lint using the "old code" is being added but Clippy in Rust repo was updated for the "new code" already.

Edit: There would be still Rustups necessary for lints added to Clippy shortly before breaking changes since Clippy in Rust repo won't have them yet.

[eddyb]

@oli-obk @RalfJung What do you think of this logic? Just thought of it, not sure where to leave it.

git subtree pull is an injection, git subtree push is a projection, hence why pulling preserves commits verbatim (hash included) while pushing rewrites them to keep only the relevant parts.
They're both information preserving (wrt file diffs and commit messages) for the subtree directory, and only for that directory.

[Manishearth]

Though I guess this doesn't actually need to be versioned with Rust so a cargo install makes more sense

[oli-obk]

I marked this as "looking for participation". Working on this git subcommand that can be installed via cargo install requires zero work on rustc. Basically what you need to do is to grab the latest version of the subcommand and copy the file with the appropriate license mentioned into a new cargo project. The main function can then include_str! the script and invoke sh to execute the text (maybe piped in via stdin?) by forwarding all command line args to the script. You'll need to add rust-lang-owner to the owners of your git repository and the crates.io owners of your crate.

Ideally you test the crate by talking with the clippy team on zulip and doing a clippy sync with the "new" tool. If all that works, we'll update the documentation on our git subtree usage to reflect the fact that ppl need to install the tool and invoke a slightly different subcommand name.

[mominul]

I'd like to work on this issue!

[oli-obk]

great! If you have any questions, feel free to ping me on zulip or just ask here. There's also a bot, so if you write @rustbot claim it should assign this issue to you.

[mominul]

@rustbot claim

[pietroalbini]

I'm wondering if we need this to actually be on crates.io. We could vendor the script in the repository and call ./x.py subtree.

[oli-obk]

I like having it as an independent thing. Others may want to use it, too. It's not rustc specific

[mominul]

Here's the gitree crate which wraps the git subtree script. I have included the script form the git PR mentioned here. I am not well versed in handling git, so can anyone test it or give me the instructions for doing it? Here's the Zulip topic about this crate.

[bjorn3]

The crate metadata doesn't refer to the repo, so posting it here for others trying to find it: https://github.com/mominul/gitree

[RalfJung]

Using subtrees can lead to spuriously re-closing issues that were re-opened. This just happened to #82080:

the original commit did a lot of things, among them fixing clippy. The commit description contained "Fixes #82080". Each commit that affects clippy is synced to clippy (creating a new commit ID since all the non-clippy changes are removed from it), and then that is synced back (with the same commit ID as what it got in clippy). So now we got a fresh commit that still has "Fixes #82080" in its description.

I am not sure what to do about this, but we have a notable risk here of losing track of issues by accidentally closing them.

Can we make it so that the commit that is synthesized for clippy doesn't just have its non-clippy changes removed, but also has its commit message "sanitized" to ensure that it will not cause any automatic actions by GitHub?

[bjorn3]

Can we make it so that the commit that is synthesized for clippy doesn't just have its non-clippy changes removed, but also has its commit message "sanitized" to ensure that it will not cause any automatic actions by GitHub?

I think git subtree would then consider it a different commit and try to pull the original conmit every time you sync.

[flip1995]

There's the option in subtree to squash-sync the commits to rustc. But that way we would lose the Clippy history in rustc.

If we want to keep the history of Clippy and keep using subtree, we just have to live with this, I guess...

[RalfJung]

I think git subtree would then consider it a different commit and try to pull the original conmit every time you sync.

Why that, doesn't it have some kind of bookeeping to match up the commits? Relying on commit messages seems rather fragile?

[bjorn3]

I assumed it required the commit to be identical where possible. Based on a cursory reading of the source it seems to only require the tree to be the same: https://github.com/gitgitgadget/git/blob/fe2e4819b869725f870cd3ce99f1f8150fe17dc1/contrib/subtree/git-subtree.sh#L598

[flip1995]

Copying comments from #82080:

@jplatte wrote:

This sounds like a general problem with the subtree approach...

IMHO this is more of a problem with GitHub issue / PR references in commit messages. It also happens when merging repos, no matter if you use subtree or sth. else, as long as you keep the history you can have super old commits that suddenly affect (or just reference) new and entirely irrelevant issues. My approach has been to always keep these references out of commits, and only have them in PR descriptions and comments.

@flip1995 wrote:

The problem is, that bors includes the PR body in the commit message. Maybe bors could sanitize the PR body and remove keywords that results in closing issues. This doesn't fix the problem, when normal commit messages have a <keyword> #issue in them, but I think the main problem are bors commits copying PR bodies.

[RalfJung]

The problem is, that bors includes the PR body in the commit message.

Good point. Not doing that for the "active" messages could also help. (It'd also cut down on notifications, if bors removed @mentions as well.)

[Manishearth]

I feel like that's a pretty rare occurrence and unfortunately the tooling doesn't support doing such a transform currently. We could tweak the bors thing though

[RalfJung]

Well, it's also something that has a high chance of flying below the radar when it happens. All we know is it happened at least once.

[tmandry]

FYI there's been some discussion of experimenting with miri on Fuchsia, and transitioning miri to an always-green subtree is one of the primary blockers.

[oli-obk]

Clippy is now using subtree, miri is using https://github.com/josh-project/josh/ (a Rust thing that is better and more powerful in various ways than subtree)