fetchFromGitHub: re-fetch upon version changes by ShamrockLee · Pull Request #294068 · NixOS/nixpkgs

ShamrockLee · 2024-03-07T15:21:12Z

Description of changes

This PR mitigates the development issue of irrelevent FOD hashes after version bump/version overriding, by making rev part of the store path through the default value of the name attribute.

Depends on #294329 and #294334. Once they're merged, there will be only one commit and two changed files (including the release note) in this PR.

Things done

Add a 👍 reaction to pull requests you find important.

ShamrockLee · 2024-03-07T20:29:02Z

@ofborg build archivebox python311Packages.jwt python312Packages.jwt

ShamrockLee · 2024-03-07T22:43:04Z

@ofborg build cowsay

ShamrockLee · 2024-03-07T22:53:31Z

@ofborg build nixdoc

ShamrockLee · 2024-03-08T17:39:52Z

Let's limit the scope of this PR and avoid changing the interface too much.

ShamrockLee · 2024-03-10T10:57:44Z

nixpkgs-review shows no evaluation error. The ofborg-eval-lib-tests is due to failure to bootstrap its own tool chain.

I couldn't find a way to reproduce the build failure as well. Even llvm builds alright locally.

ghost · 2024-03-10T11:06:16Z

nixpkgs-review shows no evaluation error. The ofborg-eval-lib-tests is due to failure to bootstrap its own tool chain.

I couldn't find a way to reproduce the build failure as well. Even llvm builds alright locally.

ofborg-eval-lib-tests failure

   ran test tests/add.sh... [FAIL]
    ++ nix-store --add ./dummy
    + path1=/build/nix-test/add/store/n7knm3rma6fckrfwkx2vkiql3d1rgs1i-dummy
    + echo /build/nix-test/add/store/n7knm3rma6fckrfwkx2vkiql3d1rgs1i-dummy
    /build/nix-test/add/store/n7knm3rma6fckrfwkx2vkiql3d1rgs1i-dummy
    ++ nix-store --add-fixed sha256 --recursive ./dummy
    + path2=/build/nix-test/add/store/n7knm3rma6fckrfwkx2vkiql3d1rgs1i-dummy
    + echo /build/nix-test/add/store/n7knm3rma6fckrfwkx2vkiql3d1rgs1i-dummy
    /build/nix-test/add/store/n7knm3rma6fckrfwkx2vkiql3d1rgs1i-dummy
    + test /build/nix-test/add/store/n7knm3rma6fckrfwkx2vkiql3d1rgs1i-dummy '!=' /build/nix-test/add/store/n7knm3rma6fckrfwkx2vkiql3d1rgs1i-dummy
    ++ nix-store --add-fixed sha256 ./dummy
    + path3=/build/nix-test/add/store/1b0chpd74drqysiwsskw53zwlg18rcjl-dummy
    + echo /build/nix-test/add/store/1b0chpd74drqysiwsskw53zwlg18rcjl-dummy
    /build/nix-test/add/store/1b0chpd74drqysiwsskw53zwlg18rcjl-dummy
    + test /build/nix-test/add/store/n7knm3rma6fckrfwkx2vkiql3d1rgs1i-dummy '!=' /build/nix-test/add/store/1b0chpd74drqysiwsskw53zwlg18rcjl-dummy
    ++ nix-store --add-fixed sha1 --recursive ./dummy
    + path4=/build/nix-test/add/store/dgvdyhjxyg2ychmw8nm1vb0cdszqqnyr-dummy
    + echo /build/nix-test/add/store/dgvdyhjxyg2ychmw8nm1vb0cdszqqnyr-dummy
    /build/nix-test/add/store/dgvdyhjxyg2ychmw8nm1vb0cdszqqnyr-dummy
    + test /build/nix-test/add/store/n7knm3rma6fckrfwkx2vkiql3d1rgs1i-dummy '!=' /build/nix-test/add/store/dgvdyhjxyg2ychmw8nm1vb0cdszqqnyr-dummy
    ++ nix-store -q --hash /build/nix-test/add/store/n7knm3rma6fckrfwkx2vkiql3d1rgs1i-dummy
    + hash1=sha256:0lc8c8k1yc8m563wxg9ikalz4q9f56gc667qnnsjiwgiv7ya8xbw
    + echo sha256:0lc8c8k1yc8m563wxg9ikalz4q9f56gc667qnnsjiwgiv7ya8xbw
    sha256:0lc8c8k1yc8m563wxg9ikalz4q9f56gc667qnnsjiwgiv7ya8xbw
    ++ nix-hash --type sha256 --base32 ./dummy
    + hash2=0ip26j2h11n1kgkz36rl4akv694yz65hr72q4kv4b3lxcbi65b3p
    + echo 0ip26j2h11n1kgkz36rl4akv694yz65hr72q4kv4b3lxcbi65b3p
    0ip26j2h11n1kgkz36rl4akv694yz65hr72q4kv4b3lxcbi65b3p
    + test sha256:0lc8c8k1yc8m563wxg9ikalz4q9f56gc667qnnsjiwgiv7ya8xbw = sha256:0ip26j2h11n1kgkz36rl4akv694yz65hr72q4kv4b3lxcbi65b3p
make: *** [mk/lib.mk:128: tests/add.sh.test] Error 1
make: *** Waiting for unfinished jobs....

i've seen this failure before and filed a bug: #294433

full log: https://gist.githubusercontent.com/GrahamcOfBorg/a54aaa359754668dc899a98c49f6db7f/raw/6d8ed13eda6981f76fcffded3cadc6ff43f06795/ofborg-eval-lib-tests

re-runing the eval usually clears it:
@ofborg eval

ShamrockLee · 2024-03-10T18:05:48Z

I'm running nix-build --arg pkgs import ./. {} ./lib/tests/release.nix locally on my laptop. There are chances of failure caused by network issue during source fetching, and this PR changes a bunch of FOD out path, implying a whole lot of re-fetches. It takes multiple re-running to fetch all of them when the network connection is less than ideal.

ShamrockLee · 2024-03-10T20:50:38Z

Update: Manual run on my x86_64-linux NixOS laptop succeeded.

Command: nix-build --arg pkgs "import ./. {}" ./lib/tests/release.nix --no-out-link

ShamrockLee · 2024-03-10T22:52:06Z

@ofborg eval

Take optional argument `passthru` for custom passthru attribute set. Update the `meta` attribute via `<pkg>.overrideAttrs` instead of attribute set update (`//`). Attach attributes `owner`, `repo` and `rev` via `passthru` instead of attribute set update.

Prefix the default value of `name` with `rev` to force re-fetch everytime `rev` changes.

ShamrockLee · 2024-03-14T07:44:23Z

Bump as #294334 is merged and has headed staging.

ShamrockLee · 2024-03-14T16:33:44Z

@ofborg eval

lheckemann · 2024-03-23T11:22:34Z

Similar changes were rejected previously: #59858 (review)

ShamrockLee · 2024-03-23T14:44:20Z

Similar changes were rejected previously: #59858 (review)

@lheckemann Thanks a lot for this information!

Things have changed a lot this five years. We don't even have stdenv.mkDerivation and <pkg>.overrideAttrs taking finalAttrs back then. As Nixpkgs Manual explicitly specifies that sourceRoot must be formed using ${src.name} for fetchgit-based src and deprecates the src.name == "source" assumption, the rejection is no longer valid.

eclairevoyant · 2024-03-23T15:53:24Z

Actually the concern we have now was the same concern from 7 years ago 🙃

At least for me descriptive names in /nix/store are important (for example if I want to look at the sources of a package that I've recently built I just ls -d /nix/store/*name*), and independence of the hash from the fetch method is not

And now every nix user gets to trip on FOD moments, so badly that we have to warn them in the manual about it.

Anyway maybe the new RFC will help sort it.

nixos-discourse · 2024-07-12T03:52:13Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/puzzling-behavior-of-fetchfromgithub-with-submodules/48911/3

nixos-discourse · 2025-10-30T23:04:51Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/fix-your-fods-fods-and-security/71531/3

github-actions bot added 8.has: documentation This PR adds or changes documentation 6.topic: fetch Fetchers (e.g. fetchgit, fetchsvn, ...) labels Mar 7, 2024

ShamrockLee changed the base branch from master to staging March 7, 2024 15:21

ShamrockLee mentioned this pull request Mar 7, 2024

Automatic finalAttrs pattern migration #293452

Open

ShamrockLee force-pushed the fetchgithub-refetch branch from c2f6d70 to 65a152b Compare March 7, 2024 15:47

ShamrockLee changed the base branch from staging to master March 7, 2024 16:01

ShamrockLee force-pushed the fetchgithub-refetch branch from 65a152b to c0ae33f Compare March 7, 2024 17:16

github-actions bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: changelog This PR adds or changes release notes labels Mar 7, 2024

ofborg bot added 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. labels Mar 7, 2024

ShamrockLee changed the title ~~fetchFromGitHub: support version and re-fetch upon version/rev changes~~ fetchFromGitHub: support version and re-fetch upon version changes Mar 7, 2024

ShamrockLee force-pushed the fetchgithub-refetch branch from c0ae33f to 70bc532 Compare March 7, 2024 19:15

ShamrockLee changed the base branch from master to staging March 7, 2024 21:28

ShamrockLee force-pushed the fetchgithub-refetch branch from 70bc532 to 56cc63c Compare March 7, 2024 21:42

github-actions bot added the 6.topic: python Python is a high-level, general-purpose programming language. label Mar 8, 2024

ShamrockLee force-pushed the fetchgithub-refetch branch 2 times, most recently from 08f6449 to 3b8b903 Compare March 8, 2024 17:37

github-actions bot added the 6.topic: vim Advanced text editor label Mar 8, 2024

ShamrockLee changed the title ~~fetchFromGitHub: support version and re-fetch upon version changes~~ fetchFromGitHub: re-fetch upon version changes Mar 8, 2024

ShamrockLee force-pushed the fetchgithub-refetch branch 3 times, most recently from 408e863 to f893a82 Compare March 8, 2024 18:27

ShamrockLee mentioned this pull request Mar 8, 2024

treewide: fix hard-coded sourceRoot prefix for fetchgit-based src #294334

Merged

13 tasks

ofborg bot added 10.rebuild-darwin-stdenv This PR causes stdenv to rebuild on Darwin and must target a staging branch. 10.rebuild-darwin: 501+ This PR causes many rebuilds on Darwin and should normally target the staging branches. labels Mar 8, 2024

ofborg bot removed the 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. label Mar 8, 2024

ShamrockLee force-pushed the fetchgithub-refetch branch from f893a82 to f1db6e9 Compare March 9, 2024 00:02

ghost mentioned this pull request Mar 10, 2024

Build failure: nix_2_3 fails in test tests/add.sh sometimes #294433

Closed

ShamrockLee mentioned this pull request Mar 10, 2024

treewide: mostly noop: refer to src.name or similar in sourceRoot where appropriate #245388

Merged

ShamrockLee added 3 commits March 14, 2024 15:39

fetchgit: takes passthru argument

49b921b

fetchFromGitHub: force re-fetch when rev changes

b6f434c

Prefix the default value of `name` with `rev` to force re-fetch everytime `rev` changes.

ShamrockLee force-pushed the fetchgithub-refetch branch from f1db6e9 to b6f434c Compare March 14, 2024 07:40

github-actions bot removed 6.topic: python Python is a high-level, general-purpose programming language. 6.topic: vim Advanced text editor labels Mar 14, 2024

eclairevoyant mentioned this pull request Mar 20, 2024

[RFC 0171] Default name of fetchFromGithub FOD to include revision NixOS/rfcs#171

Closed

ShamrockLee mentioned this pull request Mar 23, 2024

fetchFromGitHub: takes passthru and enhance overriding #294329

Closed

13 tasks

wegank added the 2.status: merge conflict This PR has merge conflicts with the target branch label Apr 5, 2024

bbjubjub2494 mentioned this pull request May 14, 2024

fetchFromGitHub: force re-fetch when rev changes auxolotl/core#11

Merged

eclairevoyant mentioned this pull request Jul 5, 2024

Superfluous use of finalAttrs.{pname,version} introducing overriding traps #310373

Open

eclairevoyant mentioned this pull request Jul 16, 2024

docker_27 update (27.0.2 -> 27.0.3) only includes cli hash, not moby package #327692

Closed

wegank added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jan 2, 2025

nixpkgs-ci bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Oct 30, 2025

Uh oh!

Comments

Conversation

ShamrockLee commented Mar 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of changes

Things done

Uh oh!

ShamrockLee commented Mar 7, 2024

Uh oh!

ShamrockLee commented Mar 7, 2024

Uh oh!

ShamrockLee commented Mar 7, 2024

Uh oh!

ShamrockLee commented Mar 8, 2024

Uh oh!

ShamrockLee commented Mar 10, 2024

Uh oh!

ghost commented Mar 10, 2024

Uh oh!

ShamrockLee commented Mar 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ShamrockLee commented Mar 10, 2024

Uh oh!

ShamrockLee commented Mar 10, 2024

Uh oh!

ShamrockLee commented Mar 14, 2024

Uh oh!

ShamrockLee commented Mar 14, 2024

Uh oh!

lheckemann commented Mar 23, 2024

Uh oh!

ShamrockLee commented Mar 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eclairevoyant commented Mar 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nixos-discourse commented Jul 12, 2024

Uh oh!

nixos-discourse commented Oct 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ShamrockLee commented Mar 7, 2024 •

edited

Loading

ShamrockLee commented Mar 10, 2024 •

edited

Loading

ShamrockLee commented Mar 23, 2024 •

edited

Loading

eclairevoyant commented Mar 23, 2024 •

edited

Loading