Skip to content

treewide: fetchCargoVendor: inherit pname+version#412224

Merged
alyssais merged 1 commit intoNixOS:masterfrom
pbsds:migrate-cargodeps-pname-version-1748558652
May 30, 2025
Merged

treewide: fetchCargoVendor: inherit pname+version#412224
alyssais merged 1 commit intoNixOS:masterfrom
pbsds:migrate-cargodeps-pname-version-1748558652

Conversation

@pbsds
Copy link
Member

@pbsds pbsds commented May 30, 2025

candidate for #346453, but may not need repeating.

Should be zero rebuilds. Made with:

#!/usr/bin/env nix-shell
#!nix-shell -I nixpkgs=. --pure -i bash -p ripgrep sd jq git git-wait lix util-linux diffutils delta

export NIX_PATH= # no nixpkgs-overlays plz
export NIXPKGS_CONFIG= # no surprises plz

export NIXPKGS_ALLOW_UNFREE=1
export NIXPKGS_ALLOW_INSECURE=1
export NIXPKGS_ALLOW_BROKEN=1

git-wait restore .

mk_packages_json() (
  set -x
  time nix-env  \
    --extra-experimental-features no-url-literals  \
    --option system x86_64-linux  \
    -f ./.  \
    -qaP  \
    --json \
    --meta \
    --drv-path \
    --out-path \
    --show-trace \
    --no-allow-import-from-derivation  \
    --arg config '{ allowAliases = false; }'
)

test -s packages.json || {
   mk_packages_json > packages.json
}

list_attrpath_fname_column() {
    jq <packages.json 'to_entries[] | select(.value.meta.position==null|not) | "\(.key)\t\(.value.meta.position)"' -r |
        sed -e "s#\t$(realpath .)/#\t#" |
        sed -e 's#:\([0-9]*\)$#\t\1#' |
        grep . |
        grep -iv haskell |
        grep -iv /top-level/ |
        grep -iv chicken |
        grep -iv build |
        grep -E '/(package|default)\.nix'
}

eval_package_outputs() {
  local attrpath="$1"
  nix eval \
    --log-format raw \
    --impure \
    --json \
    --expr '
      with import ./. {};
      let
        fixup = attrs: lib.attrsets.mapAttrs'"'"' (k: v: lib.nameValuePair (if k != "outPath" then k else "outPath_") v) attrs;
      in
        lib.attrsets.filterAttrsRecursive (n: v: !lib.isFunction v) (fixup {
          inherit ('"${attrpath}"') pname version drvPath outPath passthru meta;
          src = fixup { inherit ('"${attrpath}"'.src) name drvPath outPath; };
          cargoDeps = fixup { inherit ('"${attrpath}"'.cargoDeps) name drvPath outPath; };
        })
      ' |
    sd '.nix:[0-9]+"' '.nix:XX"' |
    sd '"line":[0-9]+,' '"line":"XX",' |
    sd '"line":[0-9]+}' '"line":"XX"}'
}

while read attrpath fname column; do
    grep -qF 'name = "${pname}-${version}";' "$fname" || continue
    grep -qF 'cargoDeps = '                  "$fname" || continue

    echo | (
        echo "$attrpath"

        pkgdata="$(eval_package_outputs "$attrpath")" || exit
        test -n "$pkgdata" || exit

        (set -x
            sd -F 'name = "${pname}-${version}";' 'inherit pname version;' "$fname"
            sd 'inherit src;'$'\n'' *inherit pname version;' 'inherit pname version src;' "$fname"
            sd 'inherit pname version;'$'\n'' *inherit src;' 'inherit pname version src;' "$fname"
        )

        pkgdata2="$(eval_package_outputs "$attrpath")"
        if [[ "$pkgdata" = "$pkgdata2" ]]; then
            (set -x; git-wait add "$fname")
        else
            diff -u <(cat <<<"$pkgdata") <(cat <<<"$pkgdata2") | delta
            (set -x; git-wait restore "$fname")
        fi

    )

done < <( list_attrpath_fname_column )

git restore .

mk_packages_json > packages2.json

With manual modifications to join the remaining inherit statements that the script could not.

The script checks that the drvPath of cargoDeps is unchanged (FODs only lock outPath), which ensures that this code change makes no difference to the cargo crate fetchers.

I compared packages.json and packages2.json to ensure there are no eval errors introduced.

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • Nixpkgs 25.11 Release Notes (or backporting 24.11 and 25.05 Nixpkgs Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
  • NixOS 25.11 Release Notes (or backporting 24.11 and 25.05 NixOS Release notes)
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@github-actions github-actions bot added the 6.topic: python Python is a high-level, general-purpose programming language. label May 30, 2025
@pbsds pbsds force-pushed the migrate-cargodeps-pname-version-1748558652 branch from 6ece740 to 9127f21 Compare May 30, 2025 02:44
@github-actions github-actions bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux. labels May 30, 2025
@JohnRTitor
Copy link
Member

At this point, we should make fetchCargoVendor the default and remove all the:

useFetchCargoVendor = true;

across tens of thousands of packages.

Non fetchCargoVendor method FODs are tied to Rust version sometimes, and they are prone to breakages and hash mismatch when formats change.

CC @TomaSajt @alyssais

@JohnRTitor JohnRTitor requested review from TomaSajt and alyssais May 30, 2025 05:42
@TomaSajt
Copy link
Contributor

TomaSajt commented May 30, 2025

@JohnRTitor useFetchCargoVendor = true is already the default, even in 25.05
We've decided to keep the useFetchCargoVendor = true occurrences around until 24.11 goes out of support (for backporting reasons).
After that, we may do a treewide both in 25.05 and master that removes the usages.

(The old fetcher has been deleted, so it is not possible to use it anymore)

@TomaSajt
Copy link
Contributor

Related treewide: #373541

@alyssais alyssais merged commit 5b38d1d into NixOS:master May 30, 2025
15 of 17 checks passed
@TomaSajt
Copy link
Contributor

Opened followup PR: #412358

@pbsds
Copy link
Member Author

pbsds commented Jun 2, 2025

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: python Python is a high-level, general-purpose programming language. 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants