fetchgit: fetch tags reproducibly and allow unstable version package to use git describe

[ShamrockLee]

Description

This PR addresses the unsolved question of #465497 about the reproducible git describe for packages with unstable versions.

• This PR allows specifying fetchTags as an attribute set, and changes its default to { }, allowing reproducible fetching of additional tags from the main module and submodules.
  • Unlike fetchTags = true, specifying fetchTags as an attribute set does not imply leaveDotGit = true. This encourages the use of postCheckout for better hash stability.
  • The corresponding command-line arguments for nix-prefetch-git is --fetch-tag TAG, which can be specified multiple times.
• This PR allows fetchgit to take both rev and tag simultaneously. In such scenario, the rev will be observed as the revision to fetch, and the tag will be added to the fetchTags list.

Example use case:

{
  lib,
  stdenv,
  fetchgit,
}:
let
  getPreviousStableVersion =
    v:
    let
      matched = lib.match "(.*)-unstable-[0-9]{4}-[0-9]{2}-[0-9]{2}" v;
    in
    if matched == null then v else lib.head matched;
in
stdenv.mkDerivation {
  pname = "my-package";
  version = "0.1.0-unstable-2024-12-31";
  src = fetchgit {
    url = "https://example.com/source.git";
    rev = "9d9dbe6ed05854e03811c361a3380e09183f4f4a"
    tag = "v${getPreviousStableVersion finalAttrs.version}";
    hash = "sha256-7DszvbCNTjpzGRmpIVAWXk20P0/XTrWZ79KSOGLrUWY=";
    postCheckout = ''
      git describe | tee git_describe_output.txt
    '';
  };
}

This PR depends on the following PRs:

• fetchgit: give leaveDotGit, nonConeMode, and sparseCheckout a default value null #462032
• fetchFromGitHub: converge arguments that determines useFetchGit #456226
• fetchurl, fetchgit: use __structuredAttrs = true and pass curlOptsList and sparseCheckout as lists #464475
• fetchgit: take postCheckout to allow gathering revision and commit metadata without leaving .git #465497

The diff will be shorter once the depending PR is merged.

Things done

• Built on platform:
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• Tested, as applicable:
  • NixOS tests in nixos/tests.
  • Package tests at passthru.tests.
  • Tests in lib/tests or pkgs/test for functions and "core" functionality.
• Ran nixpkgs-review on this PR. See nixpkgs-review usage.
• Tested basic functionality of all binary files, usually in ./result/bin/.
• Nixpkgs Release Notes
  • Package update: when the change is major or breaking.
• NixOS Release Notes
  • Module addition: when adding a new NixOS module.
  • Module update: when the change is significant.
• Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other READMEs.

---

Add a 👍 reaction to pull requests you find important.

[ony]

What about sub-modules?
From example I see that it is inferred easily from version of package. But sub-modules may have own version and not always listed in derivation explicitly.

When considering interface for tags in sub-modules, worth to think about how they are going to be updated since I was not able to find update scripts that do something for sub-modules.

Asking because faced this problem in #461391 where root package vends bundled in sub-module having own version.

[ShamrockLee]

Rebased after merging PR #462032.

[ShamrockLee]

@ony I changed the new fetchTags interface to an attribute set that looks like

fetchgit {
  fetchTags = {
    # For the main repo
    "" = [ "tag1" ];
    # For submodule at path/to/nlohmann_json
    "path/to/nlohmann_json" = [ "tag2" "tag3" ];
  };
}

Update:
The subpaths no longer have a slash (/) prefiix. The subpath for the main repository is now an empty string ("").

[ony]

@ony I changed the new fetchTags interface to an attribute set that looks like

Thank you. What about inferring those tags?
And how auto-update scripts¹ will integrate with this?

As I mentioned before they are not necessary in relation with package version as root source likely is. And thus they have to be hard-coded if they specified explicitly.

---

¹ From what I saw it seems that automatic updates focused on version and hash updates. See "Automatic package updates" section in README.md, "Nixpkgs/Update Scripts" wiki, and nixpkgs-update documentation.

[ShamrockLee]

What about inferring those tags?\nAnd how auto-update scripts will integrate with this?

nix-prefetch-git now takes --fetch-submodule-tags subpath to fetch all tags for a submodule. The corresponding fetchTags represintation is { "path/to/submodule" = true; }.

The drawback is NixOS/nix#14675 preventing us from specifying { fetchTags = { "" = true; }.

Update:

• I dropped the fetchgit (fetchTags) side changes. The all-tag fetching should be limited to nix-prefetch-git (or perhaps some adjustment at fetchgit/builder.sh).
• I split this part of change to PR [WIP] nix-prefetch-git: facilitate submodule tag discovery for update scripts #469678 as the requirements and use cases are quite different from the current PR.