Prioritize tagged releases over latest commit in git_submodules

[etan-status]

When using package-ecosystem: gitsubmodule, dependencies are always bumped to the latest commit on the branch from .gitmodules, even when releases or tags exist. This is frequently undesirable:

• Git submodules update to latest tag? #1639
• [Git submodules] Bump to the latest release #2192

The logic is updated to assign the following version to commits:

• Only commits on branch are considered
• When semver tags exist (optionally with v prefix), candidates are emitted as indicated by the tags
• For each commit a default candidate is emitted with version 0.0.0-0.#, with non-0 patch version based on recency (same as current behaviour)
• The old checkout is added with version 0.0.0-0.0 as a fallback

Further, the following bugs are fixed:

• bin/dry-run.rb did not work with git_submodules, due to PACKAGE_MANAGER_LOOKUP table in file.rb expecting submodules, but valid_package_managers expecting git_submodules. The working commands are:
  • bin/docker-dev-shell git_submodules
  • bin/dry-run.rb submodules owner/repo
• Git metadata was only fetched for 1 page (100 commits), which can be barely too little for a typical repository with monthly release. Implemented pagination with a limit to 500 commits (6 requests). Might be bugged if a branch name and sha commit conflict, not sure how to handle that (GitHub API limitation, if bugged was also before).
• The latest_version_finder always picked the first result, without ordering by version. This happened to always be the latest commit.
• The pseudo_version assigned by package_details_fetcher was emitted in the wrong order, so that the latest commit had the lowest version. Fixed it to count down from the highest version instead. This did not matter due to the latest_version_finder ignoring version numbers.
• The old checkout was added with version 1.0.0, taking precedence over any newer checkout if a repo only has tags before 1.0.0. This did not matter due to the latest_version_finder ignoring version numbers.

Checklist

• I have run the complete test suite to ensure all tests and linters pass.
• I have thoroughly tested my code changes to ensure they work as expected, including adding additional tests for new functionality.
• I have written clear and descriptive commit messages.
• I have provided a detailed description of the changes in the pull request, including the problem it addresses, how it fixes the problem, and any relevant details about the implementation.
• I have ensured that the code is well-documented and easy to understand.

[etan-status]

Also tested manually against a couple dependencies:

• Large dependency: bin/dry-run.rb submodules status-im/nimbus-eth2 --cache=files --dep=vendor/nim-libp2p
• Dependency without tags: bin/dry-run.rb submodules status-im/nimbus-eth2 --cache=files --dep=vendor/sepolia
• Dependency where the latest tag is not on the configured branch: bin/dry-run.rb submodules status-im/nimbus-eth2 --cache=files --dep=vendor/nim-bearssl --branch=dev/etan/zz-y

[ReenigneArcher]

Thank you for this! I am not a contributor to dependabot, but looking forward to this feature/fix.

Instead of relying strictly on the git tag to determine if it's a pre-release, could you use the github api to check that property? Perhaps this could be a first check, falling back to the git tag regex when the submodule is not hosted on GitHub? I understand if it's out of scope or too complex, but wanted to ask because it would definitely be beneficial for me.

Could using a pre-release be configurable somehow? Maybe per depedency with the @dependabot commands? Or globally in the dependabot.yml config file?

Could you also test another situation?

1. Dependency when current commit is not a tag, but there are tags on the submodule. This should continue using the latest commit. There are many submodules I use that used tags in the past, but maybe changed their methodology over the years.

[etan-status]

I don't think that using the GitHub tags API has an advantage over just using the existing tag functionality. In any case, the branch from .gitmodules is honored, and any tags not part of that branch are ignored.

Thanks for the additional situation, it makes sense that if one explicitly moved away to an untagged commit, that the latest commit keeps getting followed (until there is another tag). I've fixed that now by only proposing a forward motion (or sideward motion, in case of a rebase) on the branch, and never a backward motion.

[ReenigneArcher]

I don't think that using the GitHub tags API has an advantage over just using the existing tag functionality.

I think you're assuming everyone uses additional information like -rc or something else appended to the version number to indicate a pre-release. Even though that's the standard for semver, that's not always the case though as many are creating a pre-release with a normal semver tag, then later convert the pre-release to a normal release. Here's an example https://github.com/orgs/community/discussions/4993#discussioncomment-8406394

There's also other versioning standards which are not semver, so I'm not sure how that would be handled. e.g. calver or other custom ones. And, .NET uses a four-part version number format (Major.Minor.Build.Revision), I'm not sure how often a .NET project is used as a submodule though.

[etan-status]

How does the GitHub tags API differ in information from the tags that are part of the Git repository itself?

• https://docs.github.com/en/rest/repos/repos?apiVersion=2022-11-28#list-repository-tags

As for filtering out pre-releases, that seems to be a separate feature. What this PR does is that it exposes candidates of potential updates with correct version numbers, as in, properly assign a synthetic version number to commits based on recency (currently it does the opposite), and use the semver information from tags, if available (currently tags are ignored). That's all in package_details_fetcher.rb. There is the additional fix of actually using the highest version instead of the first result. But any additional filtering would be out of scope and a separate PR against latest_version_finder.rb (which could further filter pre-releases and so on).

[etan-status]

The .NET version scheme is a good point, however, I'd also exclude it for now, as the rest of Dependabot seems to be centered around Semver (the ignoring mechanism surrounding version.rb etc). However, a future PR could obviously further extend support to also integrate the .NET scheme.

[etan-status]

The effective change introduced from this PR is, that if there is a semver tag that's more recent than current submodule checkout, then Dependabot would update to the latest semver tag (on the configured branch), instead of to the latest commit.

If one is already on a commit after the latest semver tag, behaviour does not change and updates continue to be to the latest commit on the configured branch

[v-kbukum1]

@thavaahariharangit , can you check the status of this PR. If everything is alright, we can consider of shipping it, if not we can discuss what we can do?

[evenh]

I'm also eagerly awaiting the merging of this PR 🤞🏻

[v-thavaahariharangit]

Other than renaming the package manager name, this PR is good to be deployed and test

[v-thavaahariharangit]

@etan-status Could you please avoid renaming git_submodules to submodules so we can maintain consistency across the other modules.

[etan-status]

Sure, have renamed it back. Note that bin/dry-run.rb does not work with git_submodules, due to PACKAGE_MANAGER_LOOKUP table in file.rb expecting submodules, but valid_package_managers expecting git_submodules. So, with the rename gone, this is broken again. However, it's also fine to address separately, this PR is not held back by this.

[v-thavaahariharangit]

Following the discussion here: #13534,

we have agreed to proceed with these changes.

[ReenigneArcher]

I don't know if this is an edge case, but I have this submodule at the latest tagged release and it's trying to update to a non tag.

LizardByte/Moonlight-XboxOG#142

[etan-status]

You are currently on a tag that doesn't reflect a semver version (v1.17.x). The dependabot functionality only covers semver based versions, semver doesn't allow "x", so dependabot treats it as any other random tag.

[ReenigneArcher]

@etan-status thank you for the fast response, but I am on the tag v1.17.0, not the v1.17.x branch.

google/googletest@main...v1.17.0

I guess they point to the same commit, but it is a tagged commit that uses semver.

[etan-status]

Ah, I got it wrong, yeah, the branch name is the v1.17.x, not the tag name.

Could it be the branch configuration in .gitmodules?
https://github.com/LizardByte/Moonlight-XboxOG/blob/master/.gitmodules
You currently track main, is the tag actually on the main branch?

[ReenigneArcher]

You currently track main, is the tag actually on the main branch?

It appears not.

[etan-status]

OK, so if you update .gitmodules to point to the correct branch (v1.17.x) it should correctly track that branch going forward. If you point it to main, it will track commits on main only.

[OrKoN]

It looks like this change makes it impossible to update to the latest commit when there are tags in the repo?

dependabot-core/git_submodules/lib/dependabot/git_submodules/update_checker/latest_version_finder.rb

Lines 25 to 33 in 2ef9fd2

	def latest_tag(language_version: nil) # rubocop:disable Lint/UnusedMethodArgument
	releases = version_list
	releases = filter_by_cooldown(T.must(releases))
	releases = filter_ignored_versions(releases)
	# if there are no releases after applying filters, we fallback to the current tag to avoid empty results
	releases = apply_post_fetch_latest_versions_filter(releases)
	releases.max_by(&:version)&.tag

+

dependabot-core/git_submodules/lib/dependabot/git_submodules/update_checker.rb

Lines 60 to 75 in 2ef9fd2

	sig { returns(T.nilable(String)) }
	def fetch_latest_version
	T.let(
	LatestVersionFinder.new(
	dependency: dependency,
	dependency_files: dependency_files,
	credentials: credentials,
	ignored_versions: ignored_versions,
	security_advisories: security_advisories,
	cooldown_options: update_cooldown,
	raise_on_ignored: raise_on_ignored
	).latest_tag,
	T.nilable(String)
	)
	end
	end

seems to be only considering tags now. Is it intended? is there a workaround to force dependabot to pick the latest commit on the branch?

[etan-status]

Yes, just ignore the tags:

version: 2
updates:
  - package-ecosystem: gitsubmodule
    directory: /
    ignore:
      - dependency-name: vendor/some-submodule
        versions: ["> 0.0.0"]
    schedule:
      interval: daily
    target-branch: unstable

each commit still gets an implicit v0.0.0 tag (on top of any actual tags), so by filtering those >0.0.0, you'll automatically track the latest HEAD, whether that is tagged or untagged.

[tooomm]

We would love to use this for VCPKG, a common package manager for C++ projects. It's the recommended way to use it as submodule in their own docs at Microsoft and the repo.

Tracking latest commits is just too busy for us and sticking with their releases would be perfect.
However, they do not use SemVer but a structured date format "YYYY.MM.DD" as tags. While it's separated by dots, month and day are filled with 0's if it's single digit values. As I understood it, that's the reason it'll not work.

Is there any addition planned to provide support?

[etan-status]

I'd recommend raising a feature request for non-semver based tags. It's unlikely that commenting here has an effect.