Skip to content

support .originalSrc attr as a fallback#324

Merged
ryantm merged 1 commit intonix-community:mainfrom
abathur:original_src_fallback
Sep 17, 2022
Merged

support .originalSrc attr as a fallback#324
ryantm merged 1 commit intonix-community:mainfrom
abathur:original_src_fallback

Conversation

@abathur
Copy link
Copy Markdown
Contributor

@abathur abathur commented Sep 17, 2022

If a builder creates a sequence of derivations on the way to building a package, nixpkgs-update won't be able to find the
correct src attr to query. We can technically use passthru to lie about the src, but src is load-bearing and this may
mislead or confuse humans (or other tools?) later.

This change checks an originalSrc fallback attr that isn't load-bearing within builds or nixpkgs.

before
../nixpkgs-update/result/bin/nixpkgs-update update  --attrpath "bats 1.7.0 1.8.0 repology.org"
Configured Nixpkgs-Update Options:
----------------------------------
GitHub User:                   abathur
Send pull request on success:  NO
Batch update:                  NO
Calculate Outpaths:            NO
CVE Security Report:           NO
Run nixpkgs-review:            NO
Nixpkgs Dir:                   "/home/abathur/work/nixpkgs"
update info uses attrpath:     YES
----------------------------------
For attrpath bats, using log file: /run/user/1000/nixpkgs-update/bats/2022-09-17.log
bats 1.7.0 -> 1.8.0 repology.org
attrpath: bats
[version] 
Received ExitFailure 1 when running
Raw command: /nix/store/6fr58pqy02hmgw1h7b9nqmv7vlp3v346-nix-2.3.15/bin/nix eval -f . --raw pkgs.bats.src.drvAttrs.outputHash
Standard error:

error: attribute 'outputHash' in selection path 'pkgs.bats.src.drvAttrs.outputHash' not found
Received ExitFailure 1 when running
Raw command: /nix/store/6fr58pqy02hmgw1h7b9nqmv7vlp3v346-nix-2.3.15/bin/nix eval -f . --raw pkgs.bats.drvAttrs.outputHash
Standard error:

error: attribute 'outputHash' in selection path 'pkgs.bats.drvAttrs.outputHash' not found

Failed to update
after (still fails, but for package reasons!)
../nixpkgs-update/result/bin/nixpkgs-update update  --attrpath "bats 1.7.0 1.8.0 repology.org"
Configured Nixpkgs-Update Options:
----------------------------------
GitHub User:                   abathur
Send pull request on success:  NO
Batch update:                  NO
Calculate Outpaths:            NO
CVE Security Report:           NO
Run nixpkgs-review:            NO
Nixpkgs Dir:                   "/home/abathur/work/nixpkgs"
update info uses attrpath:     YES
----------------------------------
For attrpath bats, using log file: /run/user/1000/nixpkgs-update/bats/2022-09-17.log
bats 1.7.0 -> 1.8.0 repology.org
attrpath: bats
[version] 
[version] updated version and sha256
[rustCrateVersion] 
[rustCrateVersion] No cargoSha256 found
[golangModuleVersion] 
[golangModuleVersion] Not a buildGoModule package with vendorSha256
[updateScript] 
[updateScript] skipping because derivation has no updateScript

[quotedUrls]
[quotedUrls] nothing found to replace
Diff after rewrites:
diff --git a/pkgs/development/interpreters/bats/default.nix b/pkgs/development/interpreters/bats/default.nix
index 89539850163..f37e70e8eae 100644
--- a/pkgs/development/interpreters/bats/default.nix
+++ b/pkgs/development/interpreters/bats/default.nix
@@ -22,13 +22,13 @@
 
 resholve.mkDerivation rec {
   pname = "bats";
-  version = "1.7.0";
+  version = "1.8.0";
 
   src = fetchFromGitHub {
     owner = "bats-core";
     repo = "bats-core";
     rev = "v${version}";
-    sha256 = "sha256-joNne/dDVCNtzdTQ64rK8GimT+DOWUa7f410hml2s8Q=";
+    sha256 = "sha256:1v4czrhryqh5sbmh07dm5qi8r1ig9gbszxv1rrqqzgy4dgrl2wvn";
   };
 
   patchPhase = ''

 
Received ExitFailure 100 when running
Raw command: /nix/store/6fr58pqy02hmgw1h7b9nqmv7vlp3v346-nix-2.3.15/bin/nix-build --option sandbox true --arg config "{ allowBroken = true; allowUnfree = true; allowAliases = false; }" --arg overlays "[ ]" -A bats
nix build failed.
@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking sources
unpacking source archive /nix/store/5mbr2bfn9kil7lin389ycx1b4qch774w-bats-unresholved-1.8.0
source root is bats-unresholved-1.8.0
@nix { "action": "setPhase", "phase": "patchPhase" }
patching sources
@nix { "action": "setPhase", "phase": "installPhase" }
installing
@nix { "action": "setPhase", "phase": "fixupPhase" }
post-installation fixup
[resholve context] : invoking resholve with PWD=/nix/store/xfnkqnb97zblybdplqj1wccja48jfr28-bats-1.8.0
[resholve context] RESHOLVE_LORE=/nix/store/qrdv29qd8vqyqw97mqbb45c2jl32i15w-more-binlore
[resholve context] RESHOLVE_EXECER='cannot:/nix/store/jqgflyv7j2z5qzbc0rpjdjwfssps7am7-parallel-20220322/bin/parallel cannot:/nix/store/cc6wq3fkhgwsad89cdyjb0krq1hgnf7i-flock-0.2.3/bin/flock cannot:libexec/bats-core/bats-preprocess cannot:libexec/bats-core/bats-exec-file cannot:libexec/bats-core/bats-exec-suite'
[resholve context] RESHOLVE_FAKE='external:'\''greadlink'\'';'\''shlock'\'''
[resholve context] RESHOLVE_FIX='$BATS_LIBEXEC:'\''/nix/store/xfnkqnb97zblybdplqj1wccja48jfr28-bats-1.8.0/libexec/bats-core'\'' $BATS_ROOT:'\''/nix/store/xfnkqnb97zblybdplqj1wccja48jfr28-bats-1.8.0'\'''
[resholve context] RESHOLVE_INPUTS=/nix/store/p7bpdnxqd3i5hwm92mrscf7mvxk66404-bash-5.1-p16/bin:/nix/store/n95s7s6ilkjc7xwqml93acxzj6k0hsfn-coreutils-9.1/bin:/nix/store/ix6nn9lwp2w04agkfb1wzwvcnis0mmxw-gnugrep-3.7/bin:/nix/store/9yj3jdmm5g0jfbcmhxg0bqd75yjhlx9g-ncurses-6.3-p20220507/bin:/nix/store/k0kpf3r2k1d8p9h0gmx23msw3qrybkfk-findutils-4.9.0/bin:/nix/store/fswqx8lra27sg4lj6f5pswviwzdm5alk-hostname-net-tools-2.10/bin:/nix/store/jqgflyv7j2z5qzbc0rpjdjwfssps7am7-parallel-20220322/bin:/nix/store/cc6wq3fkhgwsad89cdyjb0krq1hgnf7i-flock-0.2.3/bin:lib/bats-core:libexec/bats-core
[resholve context] RESHOLVE_INTERPRETER=/nix/store/p7bpdnxqd3i5hwm92mrscf7mvxk66404-bash-5.1-p16/bin/bash
[resholve context] RESHOLVE_KEEP='$BATS_TEST_NAME $formatter $pre_command $report_formatter /nix/store/xfnkqnb97zblybdplqj1wccja48jfr28-bats-1.8.0/libexec/bats-core/bats /nix/store/xfnkqnb97zblybdplqj1wccja48jfr28-bats-1.8.0/libexec/bats-core/bats-exec-test source:'\''/nix/store/xfnkqnb97zblybdplqj1wccja48jfr28-bats-1.8.0/lib/bats-core/validator.bash'\'';'\''/nix/store/xfnkqnb97zblybdplqj1wccja48jfr28-bats-1.8.0/lib/bats-core/preprocessing.bash'\'';'\''$BATS_TEST_SOURCE'\'';'\''/nix/store/xfnkqnb97zblybdplqj1wccja48jfr28-bats-1.8.0/lib/bats-core/tracing.bash'\'';'\''/nix/store/xfnkqnb97zblybdplqj1wccja48jfr28-bats-1.8.0/lib/bats-core/test_functions.bash'\'';'\''$library_load_path'\'';'\''/nix/store/xfnkqnb97zblybdplqj1wccja48jfr28-bats-1.8.0/lib/bats-core/common.bash'\'';'\''/nix/store/xfnkqnb97zblybdplqj1wccja48jfr28-bats-1.8.0/lib/bats-core/semaphore.bash'\'';'\''/nix/store/xfnkqnb97zblybdplqj1wccja48jfr28-bats-1.8.0/lib/bats-core/formatter.bash'\'';'\''/nix/store/xfnkqnb97zblybdplqj1wccja48jfr28-bats-1.8.0/lib/bats-core/warnings.bash'\'';'\''$setup_suite_file'\'''
[resholve context] /nix/store/0li62dp5srswb21acyvxqkvw9kkr067n-resholve-0.8.1/bin/resholve --overwrite bin/bats libexec/bats-core/bats libexec/bats-core/bats-exec-file libexec/bats-core/bats-exec-suite libexec/bats-core/bats-exec-test libexec/bats-core/bats-format-cat libexec/bats-core/bats-format-junit libexec/bats-core/bats-format-pretty libexec/bats-core/bats-format-tap libexec/bats-core/bats-format-tap13 libexec/bats-core/bats-preprocess lib/bats-core/common.bash lib/bats-core/formatter.bash lib/bats-core/preprocessing.bash lib/bats-core/semaphore.bash lib/bats-core/test_functions.bash lib/bats-core/tracing.bash lib/bats-core/validator.bash lib/bats-core/warnings.bash
      tee >("$interpolated_report_formatter" "${report_formatter_flags[@]}" >"${BATS_REPORT_OUTPUT_DIR}/${BATS_REPORT_FILENAME}") | \
            ^
/nix/store/xfnkqnb97zblybdplqj1wccja48jfr28-bats-1.8.0/libexec/bats-core/bats:457: Can't resolve dynamic command
 
Failed to update

If I can get a little guidance on where they should be documented and whether you'd like them in one or multiple PRs, I'd also like to follow up with one or more PRs to at least partially document:

  1. Which derivation attrs nixpkgs-update uses (including this new one).
  2. Document the macOS blocker in running update yields 'openFile: does not exist (No such file or directory)' error on macOS #323 if I can't find someone else who has it working there.
  3. Better document filesystem preconditions:
    • setting up the github/hub auth token
    • if we use an existing local nixpkgs checkout, it looks like it needs to have an upstream remote that has been fetched

If a builder creates a sequence of derivations on the way to
building a package, nixpkgs-update won't be able to find the
correct src attr to query. We can technically use passthru
to lie about the src, but src is load-bearing and this may
mislead or confuse humans (or other tools?) later. This also
checks an originalSrc fallback attr that isn't load-bearing.
Copy link
Copy Markdown
Collaborator

@ryantm ryantm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fine. Thanks!

@ryantm ryantm merged commit 473d33d into nix-community:main Sep 17, 2022
@ryantm
Copy link
Copy Markdown
Collaborator

ryantm commented Sep 17, 2022

@abathur This is deployed now.

abathur added a commit to abathur/nixpkgs that referenced this pull request Sep 17, 2022
Effort to fix automatic nixpkgs-update updates for resholved
packages in 9f6310d did help the bot get further, but it
then failed to find the source outputHash (the outer
derivation's source is the inner derivation; bot looks for
outer.src.outputHash; ours is at outer.src.src.outputHash).

This change uses `originalSrc` to indicate the source of the
inner derivation. Along with nix-community/nixpkgs-update#324, this
enables the bot to fall back on an attr that Nix/nixpkgs are
not directly depending on, supporting automatic updates for
packages built with `resholve.mkDerivation`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants