Skip to content

Allow customizing fetchurl hashedMirrors #409010

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
philiptaron merged 2 commits into from
Oct 6, 2025
Merged

Allow customizing fetchurl hashedMirrors #409010

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c0004e6
stdenv: Pass rewriteURL to freebsd and native stdenvs
infinisil Oct 6, 2025
56f680d
fetchurl: Allow hashed mirror overriding with config.hashedMirrors
infinisil Oct 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/release-notes/rl-2511.section.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -225,6 +225,7 @@
<!-- To avoid merge conflicts, consider adding your item at an arbitrary place in the list instead. -->

- Added `rewriteURL` attribute to the nixpkgs `config`, to allow for rewriting the URLs downloaded by `fetchurl`.
- Added `hashedMirrors` attribute to the nixpkgs `config`, to allow for customization of the hashed mirrors used by `fetchurl`.

- Added `gitConfig` and `gitConfigFile` option to the nixpkgs `config`, to allow for setting a default `gitConfigFile` for all `fetchgit` invocations.

Expand Down
5 changes: 4 additions & 1 deletion pkgs/build-support/fetchurl/default.nix
View file
Open in desktop

@philiptaron philiptaron Oct 6, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My memory from #414769 is that there's an equivalent change needed in pkgs/build-support/fetchurl/booter.nix and minimal-bootstrap for any change in pkgs/build-support/fetchurl/default.nix.

@infinisil infinisil Oct 6, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rewriteURL made sense to apply to fetchurlBoot because it's an eval-time feature and fetchurlBoot supported mirror:// too. hashedMirrors however is a build-time feature that fetchurlBoot doesn't support, so can't be applied here. I tested the command in the PR, works without issues.

Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,14 @@
curl, # Note that `curl' may be `null', in case of the native stdenvNoCC.
cacert ? null,
rewriteURL,
hashedMirrors,
}:

let

mirrors = import ./mirrors.nix;
mirrors = import ./mirrors.nix // {
inherit hashedMirrors;
};

# Write the list of mirrors to a file that we can reuse between
# fetchurl instantiations, instead of passing the mirrors to
Expand Down
5 changes: 1 addition & 4 deletions pkgs/build-support/fetchurl/mirrors.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,6 @@
{

# Content-addressable Nix mirrors
hashedMirrors = [
"https://tarballs.nixos.org"
];
hashedMirrors = throw "Use config.hashedMirrors instead of (import ./pkgs/build-support/fetchurl/mirrors.nix).hashedMirrors";

# Mirrors for mirror://site/filename URIs, where "site" is
# "sourceforge", "gnu", etc.
Expand Down
16 changes: 15 additions & 1 deletion pkgs/build-support/fetchurl/tests.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,21 @@
${jq}/bin/jq -r '.headers.Hello' $out | ${moreutils}/bin/sponge $out
'';
};

# Tests that hashedMirrors works
hashedMirrors = testers.invalidateFetcherByDrvHash fetchurl {
# Make sure that we can only download from hashed mirrors
url = "http://broken";
# A file with this hash is definitely on tarballs.nixos.org
sha256 = "1j1y3cq6ys30m734axc0brdm2q9n2as4h32jws15r7w5fwr991km";

# No chance
curlOptsList = [
"--retry"
"0"
];
};

# Tests that downloadToTemp works with hashedMirrors
no-skipPostFetch = testers.invalidateFetcherByDrvHash fetchurl {
# Make sure that we can only download from hashed mirrors
Expand All @@ -40,6 +55,5 @@
# $downloadedFile, but here we know that because the URL is broken, it will
# have to fallback to fetching the previously-built derivation from
# tarballs.nixos.org, which provides pre-built derivation outputs.

};
}
2 changes: 1 addition & 1 deletion pkgs/stdenv/darwin/default.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,7 @@ let
inherit lib;
stdenvNoCC = prevStage.ccWrapperStdenv or thisStdenv;
curl = bootstrapTools;
inherit (config) rewriteURL;
inherit (config) hashedMirrors rewriteURL;
};

inherit cc;
Expand Down
4 changes: 2 additions & 2 deletions pkgs/stdenv/freebsd/default.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -404,7 +404,7 @@ let
fetchurlBoot = import ../../build-support/fetchurl {
inherit lib stdenvNoCC;
inherit (prevStage) curl;
inherit (config) rewriteURL;
inherit (config) hashedMirrors rewriteURL;
};
stdenv = import ../generic {
inherit
Expand Down Expand Up @@ -502,7 +502,7 @@ in
inherit lib;
inherit (self) stdenvNoCC;
inherit (prevStage) curl;
inherit (config) rewriteURL;
inherit (config) hashedMirrors rewriteURL;
};
gettext = super.gettext.overrideAttrs {
NIX_CFLAGS_COMPILE = "-DHAVE_ICONV=1"; # we clearly have iconv. what do you want?
Expand Down
1 change: 1 addition & 0 deletions pkgs/stdenv/native/default.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,7 @@ in
inherit lib stdenvNoCC;
# Curl should be in /usr/bin or so.
curl = null;
inherit (config) hashedMirrors rewriteURL;
};

}
Expand Down
2 changes: 1 addition & 1 deletion pkgs/top-level/all-packages.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -591,7 +591,7 @@ with pkgs;
makeOverridable (import ../build-support/fetchurl) {
inherit lib stdenvNoCC buildPackages;
inherit cacert;
inherit (config) rewriteURL;
inherit (config) hashedMirrors rewriteURL;
curl = buildPackages.curlMinimal.override (old: rec {
# break dependency cycles
fetchurl = stdenv.fetchurlBoot;
Expand Down
13 changes: 13 additions & 0 deletions pkgs/top-level/config.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -288,6 +288,19 @@ let
'';
};

hashedMirrors = mkOption {
type = types.listOf types.str;
default = [ "https://tarballs.nixos.org" ];
description = ''
The set of content-addressed/hashed mirror URLs used by [`pkgs.fetchurl`](#sec-pkgs-fetchers-fetchurl).
In case `pkgs.fetchurl` can't download from the given URLs,
it will try the hashed mirrors based on the expected output hash.

See [`copy-tarballs.pl`](https://github.com/NixOS/nixpkgs/blob/a2d829eaa7a455eaa3013c45f6431e705702dd46/maintainers/scripts/copy-tarballs.pl)
for more details on how hashed mirrors are constructed.
'';
};

rewriteURL = mkOption {
type = types.functionTo (types.nullOr types.str);
description = ''
Expand Down
Loading