Skip to content

treewide: substitute pname for strings#387769

Merged
pbsds merged 1 commit intoNixOS:masterfrom
pbsds:migrate-pname-1741296491
Mar 11, 2025
Merged

treewide: substitute pname for strings#387769
pbsds merged 1 commit intoNixOS:masterfrom
pbsds:migrate-pname-1741296491

Conversation

@pbsds
Copy link
Member

@pbsds pbsds commented Mar 7, 2025

Inspired by #387725 (comment), script is based on #336172 using what i learned in #386865, this time limited to pkgs/by-name, part of #346453

Should be zero rebuilds.

All candidates were made using:

#!/usr/bin/env nix-shell
#!nix-shell -I nixpkgs=. --pure -i bash -p ripgrep sd jq git git-wait lix util-linux

export NIXPKGS_ALLOW_UNFREE=1
export NIXPKGS_ALLOW_INSECURE=1
export NIXPKGS_ALLOW_BROKEN=1

git-wait restore .

test -s packages.json || ( set -x;
  time nix-env --extra-experimental-features no-url-literals --option system x86_64-linux -f ./. -qaP --json --meta --drv-path --out-path --show-trace --no-allow-import-from-derivation --arg config '{ allowAliases = false; }' > packages.json
)

list_attrpath_fname_col() {
    jq <packages.json 'to_entries[] | select(.value.meta.position==null|not) | "\(.key)\t\(.value.meta.position)"' -r |
        sed -e "s#\t$(realpath .)/#\t#" |
        sed -e 's#:\([0-9]*\)$#\t\1#' |
        grep . |
        grep -iv haskell |
        grep -iv /top-level/ |
        grep -iv chicken |
        grep pkgs/by-name/ |
        grep -iv build |
        grep -E '/(package|default)\.nix'
}

FLOCKDIR="$(mktemp -d)"
N_WORKERS=4
while read attrpath fname col; do
    grep -qE 'repo *= *("\$\{pname\}"|pname);' "$fname" || continue

    echo | (
        # mutex on fname
        flock --nonblock 200 || {
            >&2 echo "failed to aquire lock for $fname"
            exit 1
        }

        echo "$attrpath"
        data="$(nix eval --impure  --expr 'with import ./. {}; { inherit ('"$attrpath"') pname drvPath passthru meta; drvPath2='"$attrpath"'.src.drvPath; }' --json)" || exit
        test -n "$data" || exit
        pname="$(jq <<<"$data" .pname -r)"
        test -n "$pname" || exit

        (set -x
            sd -F '${pname}'  "$pname"         "$fname"
            sd -F ' = pname;' " = \"$pname\";" "$fname"
        )

        data2="$(nix eval --impure  --expr 'with import ./. {}; { inherit ('"$attrpath"') pname drvPath passthru meta; drvPath2='"$attrpath"'.src.drvPath; }' --json)"
        if [[ "$data" = "$data2" ]]; then
            (set -x; git-wait add "$fname")
        else
            (set -x; git-wait restore "$fname")
            exit
        fi

        (set -x
            sd -F ' rec {' ' {' "$fname"
        )

        data3="$(nix eval --impure  --expr 'with import ./. {}; { inherit ('"$attrpath"') pname drvPath passthru meta; drvPath2='"$attrpath"'.src.drvPath; }' --json 2>/dev/nul)"

        if [[ "$data" = "$data3" ]]; then
            (set -x; git-wait add "$fname")
        else
            (set -x; git-wait restore "$fname")
        fi

    ) 200>"$FLOCKDIR"/"$(sha256sum - <<<"$fname" | cut -d' ' -f1)".lock &

    while [[ $(jobs -p | wc -l) -ge $N_WORKERS ]]; do
        wait -n < <(jobs -p) || true
    done

done < <(list_attrpath_fname_col)

wait

git restore .

time nix-env --extra-experimental-features no-url-literals --option system x86_64-linux -f ./. -qaP --json --meta --drv-path --out-path --show-trace --no-allow-import-from-derivation --arg config '{ allowAliases = false; }' > packages2.json

diff packages{,2}.json is empty, indicating that no package nor src derivation has changed, nor have i introduced any eval failures.
I checked and cherry-picked the changes using GIT_DIFF_OPTS='-u15' git -c interactive.singleKey=true add --patch while watching the planet nix stream

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@github-actions github-actions bot added 6.topic: cinnamon Desktop environment 6.topic: xen-project Issues and PRs related to the Xen Project Hypervisor. labels Mar 7, 2025
@ofborg ofborg bot added the 2.status: merge conflict This PR has merge conflicts with the target branch label Mar 7, 2025
@nix-owners nix-owners bot requested a review from mkg20001 March 7, 2025 00:45
@pbsds pbsds force-pushed the migrate-pname-1741296491 branch from 53d0161 to d1c0914 Compare March 7, 2025 00:46
@ofborg ofborg bot removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Mar 7, 2025
@github-actions github-actions bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux. labels Mar 7, 2025
Copy link
Contributor

@philiptaron philiptaron left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this looks good. It's quite hard to review in the web interface, but I read through the diff on my machine with git and I trust you Peder (@pbsds). The lack of rebuilds means it's good too.

@pbsds pbsds force-pushed the migrate-pname-1741296491 branch from d1c0914 to f946332 Compare March 11, 2025 22:19
Inspired by NixOS#387725 (comment), script is based on NixOS#336172 using what i learned in NixOS#386865, part of NixOS#346453

Should be zero rebuilds.

All candidates were made using:

```shell

export NIXPKGS_ALLOW_UNFREE=1
export NIXPKGS_ALLOW_INSECURE=1
export NIXPKGS_ALLOW_BROKEN=1

git-wait restore .

test -s packages.json || ( set -x;
  time nix-env --extra-experimental-features no-url-literals --option system x86_64-linux -f ./. -qaP --json --meta --drv-path --out-path --show-trace --no-allow-import-from-derivation --arg config '{ allowAliases = false; }' > packages.json
)

list_attrpath_fname_col() {
    jq <packages.json 'to_entries[] | select(.value.meta.position==null|not) | "\(.key)\t\(.value.meta.position)"' -r |
        sed -e "s#\t$(realpath .)/#\t#" |
        sed -e 's#:\([0-9]*\)$#\t\1#' |
        grep . |
        grep -iv haskell |
        grep -iv /top-level/ |
        grep -iv chicken |
        grep pkgs/by-name/ |
        grep -iv build |
        grep -E '/(package|default)\.nix'
}

FLOCKDIR="$(mktemp -d)"
N_WORKERS=4
while read attrpath fname col; do
    grep -qE 'repo *= *("\$\{pname\}"|pname);' "$fname" || continue

    echo | (
        # mutex on fname
        flock --nonblock 200 || {
            >&2 echo "failed to aquire lock for $fname"
            exit 1
        }

        echo "$attrpath"
        data="$(nix eval --impure  --expr 'with import ./. {}; { inherit ('"$attrpath"') pname drvPath passthru meta; drvPath2='"$attrpath"'.src.drvPath; }' --json)" || exit
        test -n "$data" || exit
        pname="$(jq <<<"$data" .pname -r)"
        test -n "$pname" || exit

        (set -x
            sd -F '${pname}'  "$pname"         "$fname"
            sd -F ' = pname;' " = \"$pname\";" "$fname"
        )

        data2="$(nix eval --impure  --expr 'with import ./. {}; { inherit ('"$attrpath"') pname drvPath passthru meta; drvPath2='"$attrpath"'.src.drvPath; }' --json)"
        if [[ "$data" = "$data2" ]]; then
            (set -x; git-wait add "$fname")
        else
            (set -x; git-wait restore "$fname")
            exit
        fi

        (set -x
            sd -F ' rec {' ' {' "$fname"
        )

        data3="$(nix eval --impure  --expr 'with import ./. {}; { inherit ('"$attrpath"') pname drvPath passthru meta; drvPath2='"$attrpath"'.src.drvPath; }' --json 2>/dev/nul)"

        if [[ "$data" = "$data3" ]]; then
            (set -x; git-wait add "$fname")
        else
            (set -x; git-wait restore "$fname")
        fi

    ) 200>"$FLOCKDIR"/"$(sha256sum - <<<"$fname" | cut -d' ' -f1)".lock &

    while [[ $(jobs -p | wc -l) -ge $N_WORKERS ]]; do
        wait -n < <(jobs -p) || true
    done

done < <(list_attrpath_fname_col)

wait

git restore .

time nix-env --extra-experimental-features no-url-literals --option system x86_64-linux -f ./. -qaP --json --meta --drv-path --out-path --show-trace --no-allow-import-from-derivation --arg config '{ allowAliases = false; }' > packages2.json
```

`diff packages{,2}.json` is empty, indicating that no package nor src derivation has changed.
I checked and cherry-picked the changes using `GIT_DIFF_OPTS='-u15' git -c interactive.singleKey=true add --patch`
@pbsds pbsds force-pushed the migrate-pname-1741296491 branch from f946332 to 357d253 Compare March 11, 2025 22:55
@pbsds pbsds merged commit e607ef1 into NixOS:master Mar 11, 2025
23 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: cinnamon Desktop environment 6.topic: xen-project Issues and PRs related to the Xen Project Hypervisor. 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants