treewide: migrate rev = "refs/tags/..." to tag

[pbsds]

Redo of #368177, part of #346453.
I've modernized the script with what i've learned during other treewides in preparation for adapting it for non "refs/tags/..." migrations of rev.

Done with:

#!/usr/bin/env nix-shell
#!nix-shell -I nixpkgs=. --pure -i bash -p ripgrep sd jq git git-wait lix util-linux diffutils delta
set -euo pipefail

export NIX_PATH= # no nixpkgs-overlays plz
export NIXPKGS_CONFIG= # no surprises plz

export NIXPKGS_ALLOW_UNFREE=1
export NIXPKGS_ALLOW_INSECURE=1
export NIXPKGS_ALLOW_BROKEN=1

#  === helpers ===

verbose() (
  set -x
  "$@"
)

mk_packages_json() (
  set -x
  time nix-env  \
    --extra-experimental-features no-url-literals  \
    --option system x86_64-linux  \
    -f ./.  \
    -qaP  \
    --json \
    --meta \
    --drv-path \
    --out-path \
    --show-trace \
    --no-allow-import-from-derivation  \
    --arg config '{ allowAliases = false; }' \
    | sd '.nix:[0-9]+"' '.nix:XX"' \
    | sd '"line":[0-9]+,' '"line":"XX",' \
    | sd '"line":[0-9]+}' '"line":"XX"}'
)

list_attrpath_fname_column() {
    jq <packages.json 'to_entries[] | select(.value.meta.position==null|not) | "\(.key)\t\(.value.meta.position)"' -r |
        sed -e "s#\t$(realpath .)/#\t#" |
        sed -E 's#:[^:]+$##' |
        grep . |
        grep -iv haskell |
        grep -iv /top-level/ |
        grep -iv chicken |
        grep -iv build |
        grep -E '/(package|default)\.nix'
}

_eval_json_with_fixup() {
  local expression="$1"
  nix eval \
    --extra-experimental-features nix-command \
    --log-format raw \
    --impure \
    --json \
    --expr '
      let
        pkgs = import ./. { };
        inherit (pkgs) lib;
        fixup = attrs: lib.attrsets.mapAttrs'"'"' (k: v: lib.nameValuePair (if k != "outPath" then k else "outPath_") v) attrs;
      in
        lib.attrsets.filterAttrsRecursive (n: v: !lib.isFunction v) ('"$expression"')
      ' \
    | sd '.nix:[0-9]+"' '.nix:XX"' \
    | sd '"line":[0-9]+,' '"line":"XX",' \
    | sd '"line":[0-9]+}' '"line":"XX"}'
}

eval_package_out_attributes() {
  local attrpath="$1"
  _eval_json_with_fixup '
    fixup {
      inherit (pkgs.'"${attrpath}"') pname version drvPath outPath passthru meta;
      src = fixup { inherit (pkgs.'"${attrpath}"'.src) name owner repo drvPath outPath; };
    }
  ' | sd \
    '"changelog":"https://github.com/([^/]+)/([^/]+)/blob/refs/tags/' \
    '"changelog":"https://github.com/$1/$2/blob/'
}
eval_package_rev_tag() {
  local attrpath="$1"
  _eval_json_with_fixup '
    fixup {
      inherit (pkgs.'"${attrpath}"'.src) owner repo rev tag;
    }
  '
}


# === start ===

git-wait restore .

test -s packages.json || mk_packages_json >packages.json
touch bad-paths.txt

set +e

while IFS=$'\t' read attrpath fname; do
    [[ -n "$attrpath" ]] || continue
    [[ -s "$fname" ]]    || continue
    grep -qE "(fetchgit|fetchFromGitHub)" "$fname" || continue
    grep -qE '\brev = "refs/tags/'        "$fname" || continue
    grep -qE '\btag = "'                  "$fname" && continue ||:

    # skip previous failures
    grep -qxF "$fname" bad-paths.txt && continue ||:

    (
        set -euo pipefail
        echo "$attrpath"

        # === read ===

        pkgdata1a="$(eval_package_rev_tag "$attrpath")"; [[ -n "$pkgdata1a" ]]

        owner="$(jq <<<"$pkgdata1a" .owner -r)"; [[ -n "$owner" ]]
        repo="$( jq <<<"$pkgdata1a" .repo  -r)"; [[ -n "$repo"  ]]
        rev="$(  jq <<<"$pkgdata1a" .rev   -r)"; [[ -n "$rev"   ]]

        grep -qE <<<"$rev" "^refs/tags/"

        pkgdata1b="$(
          eval_package_out_attributes "$attrpath" \
            | sd \
              '"changelog":"https://github.com/([^/]+)/([^/]+)/releases/tag/refs/tags/' \
              '"changelog":"https://github.com/$1/$2/releases/tag/'
        )";
        [[ -n "$pkgdata1b" ]]

        # the nix expression for rev
        tag_expr="$(grep -E '\brev = "refs/tags/(.*)";' "$fname" -o | cut -d'"' -f2 | cut -d/ -f3-)"
        # we have no guarantee of correctness if there are multiple matches
        [[ -n "$tag_expr" ]]
        [[ "$(wc -l <<<"$tag_expr" ||:)" -eq 1 ]]


        # === patch ===

        # convert rev -> tag
        verbose sd '\brev = "refs/tags/(.*)";'  'tag = "$1";'  "$fname"
        verbose sd '\btag = "\$\{(.*)\}";'      'tag = $1;'    "$fname"

        # fix meta changelog (YAGNI)
        prefix="https://github.com/$owner/$repo/releases/tag"
        verbose sd -F "$prefix"'/${self.src.rev}'        "$prefix/$tag_expr"  "$fname"
        verbose sd -F "$prefix"'/${src.rev}'             "$prefix/$tag_expr"  "$fname"
        verbose sd -F "$prefix"'/${finalAttrs.src.rev}'  "$prefix/$tag_expr"  "$fname"
        # undo https://github.com/NixOS/nixpkgs/pull/338301
        verbose sd -F "$prefix"'/${lib.removePrefix "refs/tags/" src.rev}'             "$prefix/$tag_expr"  "$fname"
        verbose sd -F "$prefix"'/${lib.removePrefix "refs/tags/" finalAttrs.src.rev}'  "$prefix/$tag_expr"  "$fname"
        verbose sd -F "$prefix"'/${lib.removePrefix "refs/tags/" self.src.rev}'        "$prefix/$tag_expr"  "$fname"


        # === read and verify ===

        pkgdata2a="$(eval_package_rev_tag "$attrpath")"; [[ -n "$pkgdata2a" ]]

        if [[ "$pkgdata1a" = "$pkgdata2a" ]]; then
            echo >&2 "ERROR: src.tag didn't change"
            git-wait diff "$fname" | delta --paging=never
            false
        fi

        pkgdata2b="$(eval_package_out_attributes "$attrpath")"; [[ -n "$pkgdata2b" ]]

        if [[ "$pkgdata1b" != "$pkgdata2b" ]]; then
            echo >&2 "ERROR: derivations, outputs, passthru or meta has changed"
            git-wait diff "$fname" | delta --paging=never
            diff -u <(cat <<<"$pkgdata1b") <(cat <<<"$pkgdata2b") | delta --paging=never
            false
        fi

        # Test if src builds, this checks the FOD hash.
        # This is not actually needed since all drvPath are unchanged (FODs only fix the outPath)
        #if ! nix-build . -A "$attrpath".src --check ; then
        #    >&2 echo "src doesn't build"
        #    git-wait diff "$fname" | delta --paging=never
        #    false
        #fi


        #  === commit ===

        verbose git-wait add "$fname"

    ) </dev/null

    #  === rollback ===

    if [[ "$?" -ne 0 ]]; then
      verbose git-wait restore "$fname" || verbose git-wait restore .
      # track failures, to faster resume if aborting the script
      echo "$fname" >> bad-paths.txt
    fi

done < <(list_attrpath_fname_column)

# git-wait restore .
mk_packages_json >packages2.json

and filtered with GIT_DIFF_OPTS='--unified=20' git -c interactive.singleKey=true add --patch.
I the checked diff -u packages{,2}.json which only shows changed meta.changelogs and the nixos-manual, hence no eval failures (which CI does not report on) are introduced.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• Nixpkgs 25.11 Release Notes (or backporting 24.11 and 25.05 Nixpkgs Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
• NixOS 25.11 Release Notes (or backporting 24.11 and 25.05 NixOS Release notes)
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[dotlambda]

Suggested change

	changelog = "https://github.com/KeisukeYamashita/commitlint-rs/releases/tag/v${version}";
	changelog = "https://github.com/KeisukeYamashita/commitlint-rs/releases/tag/${src.tag}";

or src.rev if you want. Otherwise this will 404 if the tag format changes.

[pbsds]

• see Using finalAttrs.src.tag breaks if src is overriden to use rev #367739
• and treewide: remove refs/tags/ from github release meta.changelog urls #338301

In #368177 we ended up going with this approach for being simpler and avoiding most pitfalls.

[dotlambda]

If you're worried about that use src.rev which is always available.
But the biggest pitfall is the tag format changing.

[dotlambda]

Note that this expression doesn't even use finalAttrs.

[pbsds]

I treat all src.* and pname as if they have finalAttrs, since any derivation is one PR away from migrating to it.

[dotlambda]

Sure, but then the changelog URL is most likely broken anyway and it's actually good that it breaks loudly.

[pbsds]

to my understanding nix-update migrates to src.tag, which is a slow mass-migration in the other direction of this and the prior PR. I think we should come to a treewide preference on this one.

[dotlambda]

I agree and I have strong preference for something that doesn't require changing the changelog URL whenever the tag format changes, especially since @r-ryantm does the latter automatically.

[dotlambda]

Anyway, since there is no consensus this PR has no right to make a treewide change.
EDIT: I didn't realize the PR only fixes instances where the changelog is already broken.

[philiptaron]

I agree and I have strong preference for something that doesn't require changing the changelog URL whenever the tag format changes, especially since @r-ryantm does the latter automatically.

Unsurprisingly I also agree and want a treewide preference. The tag format changing happens far less than I expected.

Maybe the point of leverage is the update script and a test inside of that, at the time that the script was run, the change log did not 404.

[philiptaron]

Appreciate you, Peder. Get that degree!