Allow file fetchers to opt into loading git submodules

[bdragon]

Context

Closes #5975

Repos are always git-cloned with --no-recurse-submodules. This can be problematic if a dependency lives in a git submodule and dependency resolution requires reading from it. For my use case, this is showing up when using go modules.

What's Changing

This PR adds #recurse_submodules_when_cloning? to Dependabot::FileFetchers::Base. If it returns a truthy value, repos are git-cloned with --recurse-submodules and --shallow-submodules; if it returns a falsy value, --no-recurse-submodules is used (the current behavior).

The default implementation of the method returns false, preserving the existing behavior for all file fetchers. Subclasses of Dependabot::FileFetchers::Base may override to opt into the behavior, and this PR does so for the go modules file fetcher.

The change also extends the behavior to the git-fetch and git-reset operations if source.commit is present (i.e., for testing). The relevant options used are git fetch --recurse-submodules=on-demand and git reset --recurse-submodules.

How to Review

I recommend reviewing this PR by commit:

0b1f097: I noticed that Dependabot::FileFetchers::Base declares some methods "private" (ruby private, prefixed with underscore) under a comment heading that says they should not be used by subclasses, but some "protected" (ruby private, no underscore) methods were mixed in with these under the same comment. This commit moves the protected methods above the comment, which makes the overall diff appear larger than what's actually changing.

41bac21: adds #recurse_submodules_when_cloning? and integrates it into #_clone_repo_contents.

2f8a624: overrides #recurse_submodules_when_cloning? for the go_modules file fetcher in order to opt into the behavior.

[jakecoffman]

Makes sense to me, great work!

[thepwagner]

👋 hey friends! This broke our updates 😿

We build a go_modules app that uses a git submodule for some non-code resources. This submodule is irrelevant to our updates.
The submodule repository is owned by a different GitHub organization, so providing a token with appropriate access is inconvenient (e.g. requires a Personal Access Token?)

Before this PR, dependabot worked great.
After this PR was deployed, Dependabot can no longer clone our go_modules repository and therefore can not give us any updates.

Is there any way for us to disable this new functionality via configuration?

[Nishnha]

Something we can try here to mitigate the issue @thepwagner brought up is only install submodules if the initial dependency resolution step failed to run

We do something of that sorts for Terraform modules in

dependabot-core/terraform/lib/dependabot/terraform/file_updater.rb

Lines 242 to 253 in 95df3a4

	rescue SharedHelpers::HelperSubprocessFailed => e
	if @retrying_lock && e.message.match?(MODULE_NOT_INSTALLED_ERROR)
	mod = e.message.match(MODULE_NOT_INSTALLED_ERROR).named_captures.fetch("mod")
	raise Dependabot::DependencyFileNotResolvable, "Attempt to install module #{mod} failed"
	end
	raise if @retrying_lock || !e.message.include?("terraform init")
	# NOTE: Modules need to be installed before terraform can update the lockfile
	@retrying_lock = true
	run_terraform_init
	retry
	end

[jakecoffman]

Another option might be we leave the submodule clone by default, but ignore any errors to attempt the update anyway? That way if it didn't matter then we still do the update.