Inline git submodules when ingesting individual repositories

[premun]

Context

Based on the decisions made during 📄 external dependency investigations, the submodules of individual repositories need to be recursively ingested as sources and not kept as git submodules.

Technical goal

This will need some work around how we ingest repositories where we will probably have to:

1. Cloak all git submodule tree entries so that we don't get these in the patches
2. Read the SHA of the commit the git submodules point to from the tree item
3. Create patch for the submodule based on the previous and new SHA
   • We must handle cases when adding/removing submodules
   • We will cloak the files of the submodule based on the parent's repo filters
4. Do this recursively for all submodules inside the currently processed one

Licensing/open-source goal

Subsequent goal is to figure out how to adhere to the open-source playbook so that it's clear why are external repositories copied into "some dotnet VMR".

[premun]

When we ingest new code for submodules, the process now goes like this:

1. Clone the individual repo
2. List all submodules in the individual repo (before and after)
3. Revert all source-build patched files in the VMR to their state from the ingested repo (de-patchify)
4. Create diff for the individual repo while cloaking all submodule paths (no submodules are being brought in)
5. For submodules, check whether there are changes (submodules added/removed, submodule SHA changed)
   • When changes, recursively do steps 1-5. but for the submodule

[premun]

@richlander can we have a discussion about the way we want to treat the inlined git submodules in the VMR?

I expect we should do something like this:

• Have a top-level README for the VMR explaining what the VMR is, why it contains other repositories
  • Should we list all submodules explicitly?
  • Should we maybe list our individual product repos as well?
• Possibly have some auto-generated README side-by-side with the submodule folders containing information about the reference (e.g. remote URL, SHA, link to the top-level document)

Side note: When we already vendorize components in runtime (https://github.com/dotnet/runtime/tree/main/src/native/external), we have put some basic .txt files describing this in the external folder.

[richlander]

That all sounds good to me.

Generalizing this topic, I think it would be useful to have a doc that listed all the repos that are included in the VMR, submodule or otherwise. I see a few options:

• List all the repos in alpha order, possibly with extra decoration for the sudmodules.
• Split repos and submodules into separate lists.
• List the submodules under the repo that referenced them.

[premun]

Feedback from Rich was pulled into #11159

Otherwise, this is completed, we have reached parity with tarball