Use git subtree (instead of git submodule) for external dependencies

[oli-obk]

TL;DR

Replace all our git submodules with git subtrees. This allows us to gate CI on tools, since contributors can just fix tools in tree and tool devs synchronize the in-tree state with their repo at their leisure.

Links and Details

Git submodules shallowly add just the commit id of the remote repository to the rustc repository. This means contributors need to use git submodule update whenever they change branches. It also means, that if you make a change to rustc that breaks a submodule, you need to do a second PR just for the submodule and point the submodule at that PR. This can quickly get out of hand with quickly changing projects like rustc and its tools.

Git subtrees on the other hand basically work like a monorepo. All tool sources are in tree, and git subtree is just a synchronization tool to push changes that happened to the in-tree tool sources back to the tool repository and vice versa. Git subtree preserves commits on both sides, so changes to the tool are fully traceable in the rustc repository and rustc changes that touched the tool sources show up in the tool repository.

This creates some synchronization overhead for tool developers, but the current status quo with submodules is worse, because we have tool breakages and synchronization overhead. We don't have much experience with git subtree, so we'll have to see whether this is actually a better experience. The fact that we get rid of tool breakages without slowing down tool development in its own repo seems like a great boon even if the synchronization overhead stays the same.

• PR documenting git subtree usage: Explain how to work with subtree rust#70654
• git subtree tracking issue: Replace submodules with subtrees rust#70651
• Porting clippy to subtree: Make clippy a git subtree instead of a git submodule rust#70655

Mentors or Reviewers

• @Centril

[Centril]

@Mark-Simulacrum is probably the better technical reviewer here though :)

[nikomatsakis]

I second the overall approach -- I'd like to see experimentation here, and it seems clear that the current tooling isn't cutting it!

I understand the plan to be that we will start with one tool (clippy) and experiment until we're happy, and then expand to other tools?

The main thing I would like us to evaluate is how effective we are at doing the two-way sync. In particular, whose responsibility is it? And how do they know when to do it? (Every so often? Monitoring for relevant PRs? Can they get pinged automatically by a bot when a PR lands that touches one of the tools?) Anyway, I don't see a need to block on that, I just think it's a good thing to pay attention to and try to document.

[Centril]

• When do we want to consider whether to move the next tool to a subtree (e.g., Miri)? Maybe give it 6 weeks?

[nikomatsakis]

So @oli-obk announced this in the meeting on 2020-04-02 and I seconded shortly thereafter. We didn't really have the policy all figured out here, but nonetheless I'm going to declare this MCP accepted since I think conversation has been going on for a while and the relevant stakeholders are all aware.

(cc @rust-lang/compiler)