fetchFromGitHub: pass `owner`, `repo`, `rev`, `tag` to `mkDerivation` and make `rev` and `tag` overridable with `<pkg>.overrideAttrs` by ShamrockLee · Pull Request #456751 · NixOS/nixpkgs

ShamrockLee · 2025-10-29T15:30:43Z

This PR adds the derivationArgs argument to fetchers like fetchurl, fetchzip, and fetchgit for derived fetchers to pass down default arguments to stdenvNoCC.mkDerivation.

Contrasting to previous works to preserve those attributes after <pkg>.overrideAttrs using passtru, this PR passes them directly down stdenv.mkDerivation and make them overridable as, e.g., tag instead of passthru.tag. Even with a larger diff, it could better address #444503 and #370418 from the overriding perspective.

This PR surpasses:

Detail:

fetchurl, fetchzip, and fetchgit: Add argument derivationArgs
It takes derivationArgs from extended build helpers and pass them all the way down stdenvNoCC.mkDerivation.
Their precedence are lower than any other arguments specified by the current build helper or by the user.
fetchzip: Pass arguments extension and stripRoot down mkDerivation via derivationArgs, and make them overridable with <pkg>.overrideAttrs.
fetchgit: Make tag and rev overridable with <pkg>.overrideAttrs as tag and revCustom.
- Pass tag directly to mkDerivation instead of via passthru.tag.
- Pass rev to mkDerivation as revCustom (since revWithHash has been passed as rev).
- Make other arguments observe the tag and revCustom overridden via <pkg>.overrideAttrs, i.e., finalAttrs.tag and finalAttrs.revCustom.
fetchgit: Abstract, simplify, and expose getRevWithHash
- Abstract the revWithTag construction as getRevWithHash.
- Simplify the conditional expression. The number of branch condition stays the same when either one of tag and rev is specified, and only increase when both unspecified (which is a rare edge-case).
- Expose the helper function getRevWithTag as fetchgit.getRevWithTag so that other fetchers can form this value the same way.
Fix how fetchFromGitHub handle its owner, repo, rev, tag, and meta arguments, making them equally overridable whether the backend fetcher is fetchzip or fetchgit.
- Pass meta properly via fetcherArgs.
- Add finalAttrs to fetcherArgs to utilize fetchurl and fetchgit's fixed-point arguments support, as a workaround before fetchFromGitHub itself are reconstructed with lib.extendMkDerivation.
- Pass owner and repo to mkDerivation via derivationArgs. We can start fixing their overriding after lib.extendMkDerivation reconstruction.
- Make tag and rev pass the same way as fetchzip does when using the fetchzip backend, with the help of derivationArgs:
```
finalAttrs:
let
  revWithTag = finalAttrs.rev;
in
{
  derivationArgs = {
    inherit tag;
    revCustom = rev;
    rev = fetchgit.getRevWithTag {
      inherit (finalAttrs) tag;
      rev = finalAttrs.revCustom;
    };
  };
  # Other fetcher arguments using `revWithTag`
}
```
Move name specification temporarily down the fetcherArg so as to reference from finalAttrs. We can move it back during reconstruction with lib.extendMkDerivation.
Pass githubBase with derivationArgs to prevent rebuilds when we make it overridable in the future.

Things done

Add a 👍 reaction to pull requests you find important.

ShamrockLee · 2025-10-29T15:51:53Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 456751
Commit: 27c7b8f6c286dabb62f86bda4f9ec5bde45781e7

`x86_64-linux`

❌ 1 package failed to build:

tests.fetchurl.header

✅ 48 packages built:

tests.fetchFirefoxAddon.simple (tests.fetchFirefoxAddon.overridden-source)
tests.fetchFromBitbucket.withEncodedWhitespace
tests.fetchFromBitbucket.withEncodedWhitespaceGit
tests.fetchFromBitbucket.withoutWhitespace
tests.fetchFromGitHub.dumb-http-signed-tag
tests.fetchFromGitHub.fetchTags
tests.fetchFromGitHub.leave-git
tests.fetchFromGitHub.rootDir
tests.fetchFromGitHub.simple
tests.fetchFromGitHub.sparseCheckout
tests.fetchFromGitHub.sparseCheckoutNonConeMode
tests.fetchFromGitHub.submodule-deep
tests.fetchFromGitHub.submodule-leave-git
tests.fetchFromGitHub.submodule-leave-git-deep
tests.fetchFromGitHub.submodule-simple
tests.fetchPypiLegacy.fetchSimple
tests.fetchgit.dumb-http-signed-tag
tests.fetchgit.fetchTags
tests.fetchgit.leave-git
tests.fetchgit.prefetch-git-no-add-path
tests.fetchgit.rootDir
tests.fetchgit.simple
tests.fetchgit.sparseCheckout
tests.fetchgit.sparseCheckoutNonConeMode
tests.fetchgit.submodule-deep
tests.fetchgit.submodule-leave-git
tests.fetchgit.submodule-leave-git-deep
tests.fetchgit.submodule-simple
tests.fetchgit.withGitConfig
tests.fetchpatch2.decode
tests.fetchpatch2.fileWithApostrophe
tests.fetchpatch2.fileWithSpace
tests.fetchpatch2.full
tests.fetchpatch2.hunks
tests.fetchpatch2.relative
tests.fetchpatch2.simple
tests.fetchtorrent.http-link (tests.fetchtorrent.http-link-transmission)
tests.fetchtorrent.http-link-rqbit (tests.fetchtorrent.http-link-rqbit-flattened)
tests.fetchtorrent.http-link-rqbit-unflattened
tests.fetchtorrent.magnet-link (tests.fetchtorrent.magnet-link-transmission)
tests.fetchtorrent.magnet-link-rqbit (tests.fetchtorrent.magnet-link-rqbit-flattened)
tests.fetchtorrent.magnet-link-rqbit-unflattened
tests.fetchzip.hiddenDir
tests.fetchzip.postFetch
tests.fetchzip.simple
tests.haskell.cabalSdist.assumptionLocalHasDirectReference
tests.haskell.cabalSdist.localHasNoDirectReference
tests.testers.runCommand.bork

Error logs: `x86_64-linux`

tests.fetchurl.header

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (22) The requested URL returned error: 503 Warning: Problem : HTTP error. Will retry in 1 second. 3 retries left.

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (22) The requested URL returned error: 503
Warning: Problem : HTTP error. Will retry in 1 second. 2 retries left.

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (22) The requested URL returned error: 503
Warning: Problem : HTTP error. Will retry in 1 second. 1 retry left.

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (22) The requested URL returned error: 503
error: cannot download source-salted-5zj6fci3fdrs from any mirror

ShamrockLee · 2025-10-29T15:55:57Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 456751
Commit: 27c7b8f6c286dabb62f86bda4f9ec5bde45781e7

`x86_64-linux`

✅ 49 packages built:

tests.fetchFirefoxAddon.simple (tests.fetchFirefoxAddon.overridden-source)
tests.fetchFromBitbucket.withEncodedWhitespace
tests.fetchFromBitbucket.withEncodedWhitespaceGit
tests.fetchFromBitbucket.withoutWhitespace
tests.fetchFromGitHub.dumb-http-signed-tag
tests.fetchFromGitHub.fetchTags
tests.fetchFromGitHub.leave-git
tests.fetchFromGitHub.rootDir
tests.fetchFromGitHub.simple
tests.fetchFromGitHub.sparseCheckout
tests.fetchFromGitHub.sparseCheckoutNonConeMode
tests.fetchFromGitHub.submodule-deep
tests.fetchFromGitHub.submodule-leave-git
tests.fetchFromGitHub.submodule-leave-git-deep
tests.fetchFromGitHub.submodule-simple
tests.fetchPypiLegacy.fetchSimple
tests.fetchgit.dumb-http-signed-tag
tests.fetchgit.fetchTags
tests.fetchgit.leave-git
tests.fetchgit.prefetch-git-no-add-path
tests.fetchgit.rootDir
tests.fetchgit.simple
tests.fetchgit.sparseCheckout
tests.fetchgit.sparseCheckoutNonConeMode
tests.fetchgit.submodule-deep
tests.fetchgit.submodule-leave-git
tests.fetchgit.submodule-leave-git-deep
tests.fetchgit.submodule-simple
tests.fetchgit.withGitConfig
tests.fetchpatch2.decode
tests.fetchpatch2.fileWithApostrophe
tests.fetchpatch2.fileWithSpace
tests.fetchpatch2.full
tests.fetchpatch2.hunks
tests.fetchpatch2.relative
tests.fetchpatch2.simple
tests.fetchtorrent.http-link (tests.fetchtorrent.http-link-transmission)
tests.fetchtorrent.http-link-rqbit (tests.fetchtorrent.http-link-rqbit-flattened)
tests.fetchtorrent.http-link-rqbit-unflattened
tests.fetchtorrent.magnet-link (tests.fetchtorrent.magnet-link-transmission)
tests.fetchtorrent.magnet-link-rqbit (tests.fetchtorrent.magnet-link-rqbit-flattened)
tests.fetchtorrent.magnet-link-rqbit-unflattened
tests.fetchurl.header
tests.fetchzip.hiddenDir
tests.fetchzip.postFetch
tests.fetchzip.simple
tests.haskell.cabalSdist.assumptionLocalHasDirectReference
tests.haskell.cabalSdist.localHasNoDirectReference
tests.testers.runCommand.bork

ShamrockLee · 2025-10-29T21:15:42Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 456751
Commit: c5f90acef939551aa220ff2b2f9d0f575286b1e7

`x86_64-linux`

✅ 49 packages built:

tests.fetchFirefoxAddon.simple (tests.fetchFirefoxAddon.overridden-source)
tests.fetchFromBitbucket.withEncodedWhitespace
tests.fetchFromBitbucket.withEncodedWhitespaceGit
tests.fetchFromBitbucket.withoutWhitespace
tests.fetchFromGitHub.dumb-http-signed-tag
tests.fetchFromGitHub.fetchTags
tests.fetchFromGitHub.leave-git
tests.fetchFromGitHub.rootDir
tests.fetchFromGitHub.simple
tests.fetchFromGitHub.sparseCheckout
tests.fetchFromGitHub.sparseCheckoutNonConeMode
tests.fetchFromGitHub.submodule-deep
tests.fetchFromGitHub.submodule-leave-git
tests.fetchFromGitHub.submodule-leave-git-deep
tests.fetchFromGitHub.submodule-simple
tests.fetchPypiLegacy.fetchSimple
tests.fetchgit.dumb-http-signed-tag
tests.fetchgit.fetchTags
tests.fetchgit.leave-git
tests.fetchgit.prefetch-git-no-add-path
tests.fetchgit.rootDir
tests.fetchgit.simple
tests.fetchgit.sparseCheckout
tests.fetchgit.sparseCheckoutNonConeMode
tests.fetchgit.submodule-deep
tests.fetchgit.submodule-leave-git
tests.fetchgit.submodule-leave-git-deep
tests.fetchgit.submodule-simple
tests.fetchgit.withGitConfig
tests.fetchpatch2.decode
tests.fetchpatch2.fileWithApostrophe
tests.fetchpatch2.fileWithSpace
tests.fetchpatch2.full
tests.fetchpatch2.hunks
tests.fetchpatch2.relative
tests.fetchpatch2.simple
tests.fetchtorrent.http-link (tests.fetchtorrent.http-link-transmission)
tests.fetchtorrent.http-link-rqbit (tests.fetchtorrent.http-link-rqbit-flattened)
tests.fetchtorrent.http-link-rqbit-unflattened
tests.fetchtorrent.magnet-link (tests.fetchtorrent.magnet-link-transmission)
tests.fetchtorrent.magnet-link-rqbit (tests.fetchtorrent.magnet-link-rqbit-flattened)
tests.fetchtorrent.magnet-link-rqbit-unflattened
tests.fetchurl.header
tests.fetchzip.hiddenDir
tests.fetchzip.postFetch
tests.fetchzip.simple
tests.haskell.cabalSdist.assumptionLocalHasDirectReference
tests.haskell.cabalSdist.localHasNoDirectReference
tests.testers.runCommand.bork

adeci · 2025-10-30T00:28:47Z

This derivationArgs approach is much cleaner. Attributes are directly accessible as finalAttrs.owner and override naturally now. Much better than nesting in passthru, which felt really hacky.
Tested successfully with my test case from #455637, and I'll be closing that with this succeeding it.

Reduce future test rebuilds when making githubBase overridable.

ShamrockLee · 2025-11-03T13:32:27Z

Rebased for CI fix. See issue #457845

MattSturgeon

Diff LGTM, and the rebuilds seem reasonable (exclusively tests). Others have tested, and CI is green.

SuperSandro2000 · 2025-11-07T16:52:21Z

This broke fetchgit with prefer-remote-fetch but that will also be fixed in #457313 most likely.

nixpkgs-ci bot added 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux. 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. 6.topic: fetch Fetchers (e.g. fetchgit, fetchsvn, ...) labels Oct 29, 2025

ShamrockLee force-pushed the fetchgithub-derivationArgs branch from ab42386 to 27c7b8f Compare October 29, 2025 15:38

ShamrockLee mentioned this pull request Oct 30, 2025

fetch{url,zip,git}: restructure with lib.extendMkDerivation #455994

Merged

13 tasks

ShamrockLee marked this pull request as ready for review November 3, 2025 13:27

ShamrockLee added 14 commits November 3, 2025 21:31

fetchurl: take derivationArgs

4ab0918

fetchzip: make overridable extension and stripRoot with derivationArgs

b29af41

fetchzip: take derivationArgs

0fc9eec

fetchgit: take derivationArgs

38f61c3

fetchgit: abstract and simplify revWithTag

076076e

fetchgit: pass tag directly and make overridable

dbf76b7

fetchgit: pass rev as revCustom and make overridable

21677a6

fetchgit: expose getRevWithTag as fetchgit.getRevWithTag

2946555

fetchFromGitHub: pass meta via fetcherArgs

e0ba367

fetchFromGitHub: pass owner and repo via derivationArgs

dd9af80

fetchFromGitHub: pass tag to mkDerivation uniformly

3754cf5

fetchFromGitHub: pass rev uniformly

aede4d2

fetchFromGitHub: make name respect overridden arguments

4e996b3

fetchFromGitHub: pass githubBase to mkDerivation via derivationArgs

8bb3bbb

Reduce future test rebuilds when making githubBase overridable.

ShamrockLee force-pushed the fetchgithub-derivationArgs branch from c5f90ac to 8bb3bbb Compare November 3, 2025 13:31

nix-owners bot requested a review from philiptaron November 3, 2025 13:36

ShamrockLee requested review from YorikSar, infinisil and roberth November 3, 2025 13:41

ShamrockLee requested a review from MattSturgeon November 3, 2025 13:46

MattSturgeon approved these changes Nov 3, 2025

View reviewed changes

nixpkgs-ci bot added the 12.approvals: 1 This PR was reviewed and approved by one person. label Nov 3, 2025

MattSturgeon added this pull request to the merge queue Nov 5, 2025

Merged via the queue into NixOS:master with commit 37a08ba Nov 5, 2025
32 of 34 checks passed

adeci mentioned this pull request Nov 9, 2025

fetchgit, fetchFromGitHub: preserve attributes through overrideAttrs #455637

Closed

13 tasks

ShamrockLee mentioned this pull request Nov 12, 2025

prefer-remote-fetch: use override instead of knowledge of build helper internals #457313

Closed

1 task

This was referenced Nov 27, 2025

fetchFromGitHub: takes passthru and enhance overriding #294329

Closed

fetchFromGitHub: use passthru instead of // to support overrideAttrs #370418

Closed

TomaSajt mentioned this pull request Nov 28, 2025

fetchFromGitHub: use passthru instead of // to expose rev and friends; fetchgit: support adding extra passthru values #370432

Closed

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fetchFromGitHub: pass `owner`, `repo`, `rev`, `tag` to `mkDerivation` and make `rev` and `tag` overridable with `<pkg>.overrideAttrs`#456751

fetchFromGitHub: pass `owner`, `repo`, `rev`, `tag` to `mkDerivation` and make `rev` and `tag` overridable with `<pkg>.overrideAttrs`#456751
MattSturgeon merged 14 commits intoNixOS:masterfrom
ShamrockLee:fetchgithub-derivationArgs

ShamrockLee commented Oct 29, 2025 •

edited

Loading

Uh oh!

ShamrockLee commented Oct 29, 2025

Uh oh!

ShamrockLee commented Oct 29, 2025

Uh oh!

ShamrockLee commented Oct 29, 2025

Uh oh!

adeci commented Oct 30, 2025

Uh oh!

ShamrockLee commented Nov 3, 2025

Uh oh!

MattSturgeon left a comment

Uh oh!

Uh oh!

SuperSandro2000 commented Nov 7, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

ShamrockLee commented Oct 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Things done

Uh oh!

ShamrockLee commented Oct 29, 2025

nixpkgs-review result

x86_64-linux

Uh oh!

ShamrockLee commented Oct 29, 2025

nixpkgs-review result

x86_64-linux

Uh oh!

ShamrockLee commented Oct 29, 2025

nixpkgs-review result

x86_64-linux

Uh oh!

adeci commented Oct 30, 2025

Uh oh!

ShamrockLee commented Nov 3, 2025

Uh oh!

MattSturgeon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

SuperSandro2000 commented Nov 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ShamrockLee commented Oct 29, 2025 •

edited

Loading

`nixpkgs-review` result

`x86_64-linux`

`nixpkgs-review` result

`x86_64-linux`

`nixpkgs-review` result

`x86_64-linux`

SuperSandro2000 commented Nov 7, 2025 •

edited

Loading