fixVersioneerSourcesHook: init

[YorikSar]

Add a hook that would fix a common issue with projects using Versioneer on GitHub: when GitHub runs analog of git archive to generate an archive from a tag, it subsitutes $Format:%d$ with (HEAD -> main, tag: the_tag) when the tag points to the latest commit, but when more commits are added, it becomes (tag: the_tag), which changes source hash.

Packages that are either automatically or manually updated early, get source with this HEAD -> main part and so their sources are not reproducible more often.

Adding this hook to projects that use Versioneer will remove this HEAD -> main part of the description string and make the source reproducible. Here's how it would look, for example:

src = fetchFromGitHub {
  owner = "ANCPLabOldenburg";
  repo = "ancp-bids";
  tag = version;
  hash = "sha256-vmw8SAikvbaHnPOthBQxTbyvDwnnZwCOV97aUogIgxw=";
  nativeBuildInputs = [ fixVersioneerSourcesHook ];
};

I ran into this issue when preparing #416464, the general case of this issue is described in #84312.

To see which Python packages are using Versioneer, I ran following script:

find-versioneer.sh

#!/usr/bin/env bash
e='
with import ./. {
  system = "x86_64-linux";
  overlays = [ ];
  config = { };
};
python3Packages
|> lib.mapAttrsToList (
  name: p:
  if
    lib.isAttrs p
    && lib.isAttrs p.src or null
    && lib.hasPrefix "https://github.com" p.src.url or ""
    && p.src ? owner
    && p.src ? repo
  then
    {
      inherit name;
      inherit (p.src) url owner repo;
    }
  else
    null
)
|> lib.filter (p: p != null |> builtins.tryEval |> (r: r.success && r.value))
|> lib.concatMapStringsSep "\n" (p: "${p.name} ${p.owner} ${p.repo}")
'

eval() {
  nix eval --extra-experimental-features pipe-operators --raw --impure --expr "$e"
}

while read name owner repo; do
  echo -n "$name"
  if gh api --cache 48h "/repos/${owner}/${repo}/contents/versioneer.py" > /dev/null 2>&1; then
    echo
  fi
  echo -ne "\r${name//?/ }\r"
done < <(eval)

And it produced the following list of packages that most likely have unstable source hashes after each release:

Packages

ancp-bids
bayespy
birch
cons
coredis
crochet
dask-jobqueue
dask-yarn
datalad
datalad-gooey
datashape
debugpy
ed25519
eliot
etelemetry
etuples
fipy
flask-limiter
gcsfs
geometric
geopandas
great-expectations
hieroglyph
intake-parquet
junos-eznc
jupyter-repo2docker
kotsu
libusb1
limits
llvmlite
logical-unification
magic-wormhole
matchpy
minikanren
monai
monai-deploy
msgspec
nbconflux
ncclient
ndindex
numba
numbaWithCuda
parallel-ssh
pims
propka
pydata-google-auth
pylatex
pyprecice
pyrevolve
pytest-mpi
python-jsonrpc-server
rembg
s3fs
salmon-mail
shapely
spake2
ssh-python
ssh2-python
strenum
trackpy

Trying to rebuild all of them, I got following packages with broken hashes:

Packages with broken sources

cons
datalad
debugpy
etuples
fipy
intake-parquet
logical-unification
monai
monai-deploy
msgspec
pims
propka
pydata-google-auth
pylatex
rembg
salmon-mail
trackpy

I've added the hook to some packages and fixed hashes if they were broken to show some examples. I picked a small number of packages with no dependants to avoid mass rebuild.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other contributing documentation in corresponding paths.

---

Add a 👍 reaction to pull requests you find important.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/prs-ready-for-review/3032/5635

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/fetchfromgithub-and-the-versioneer-fixing-source-reproducibility/66539/1