fetchurl: move netrcPhase invokation into build.sh and stringify `curlOpts` early by ShamrockLee · Pull Request #471172 · NixOS/nixpkgs

ShamrockLee · 2025-12-15T22:50:45Z

This PR cleans up the leftover of PR #464475 regarding curlOpts and netrcPhase.

Don't rely on the postPhase in $stdenv/setup.sh provided by stdenv/generic/make-derivation.nix's realBuilder hack.
If curlOpts is passed as a Bash array from a Nix Language list (which we warned against at eval time), set it as a plain Bash variable after sourcing $NIX_ATTRS_SH_FILE, so that we don't have to handle such edge cases.

Things done

Add a 👍 reaction to pull requests you find important.

pkgs/build-support/fetchurl/builder.sh

ShamrockLee · 2025-12-15T23:06:28Z

`nixpkgs-review` result

Generated using nixpkgs-review-gha

Command: nixpkgs-review pr 471172
Commit: f8225c5c8214b29628b60bb41393b3ab4fb78f39 (subsequent changes)
Merge: b969f7d0e752dd91d3a2e21dc0fbe9ca72452f18

Logs: https://github.com/ShamrockLee/nixpkgs-review-gha/actions/runs/20250213341

`x86_64-linux`

✅ 57 packages built:

tests.fetchDebianPatch.libPackage
tests.fetchDebianPatch.simple
tests.fetchFirefoxAddon.simple (tests.fetchFirefoxAddon.overridden-source)
tests.fetchFromBitbucket.withEncodedWhitespace
tests.fetchFromBitbucket.withEncodedWhitespaceGit
tests.fetchFromBitbucket.withoutWhitespace
tests.fetchFromGitHub.dumb-http-signed-tag
tests.fetchFromGitHub.fetchTags
tests.fetchFromGitHub.leave-git
tests.fetchFromGitHub.rootDir
tests.fetchFromGitHub.simple
tests.fetchFromGitHub.sparseCheckout
tests.fetchFromGitHub.sparseCheckoutNonConeMode
tests.fetchFromGitHub.submodule-deep
tests.fetchFromGitHub.submodule-leave-git
tests.fetchFromGitHub.submodule-leave-git-deep
tests.fetchFromGitHub.submodule-simple
tests.fetchNextcloudApp.simple-sha256
tests.fetchNextcloudApp.simple-sha512
tests.fetchPypiLegacy.fetchSimple
tests.fetchgit.dumb-http-signed-tag
tests.fetchgit.fetchTags
tests.fetchgit.leave-git
tests.fetchgit.prefetch-git-no-add-path
tests.fetchgit.rootDir
tests.fetchgit.simple
tests.fetchgit.sparseCheckout
tests.fetchgit.sparseCheckoutNonConeMode
tests.fetchgit.submodule-deep
tests.fetchgit.submodule-leave-git
tests.fetchgit.submodule-leave-git-deep
tests.fetchgit.submodule-simple
tests.fetchgit.withGitConfig
tests.fetchpatch.decode
tests.fetchpatch.fileWithApostrophe
tests.fetchpatch.fileWithSpace
tests.fetchpatch.full
tests.fetchpatch.hunks
tests.fetchpatch.relative
tests.fetchpatch.simple
tests.fetchpatch2.decode
tests.fetchpatch2.fileWithApostrophe
tests.fetchpatch2.fileWithSpace
tests.fetchpatch2.full
tests.fetchpatch2.hunks
tests.fetchpatch2.relative
tests.fetchpatch2.simple
tests.fetchurl.hashedMirrors
tests.fetchurl.header
tests.fetchurl.no-skipPostFetch
tests.fetchzip.hiddenDir
tests.fetchzip.postFetch
tests.fetchzip.simple
tests.haskell.cabalSdist.assumptionLocalHasDirectReference
tests.haskell.cabalSdist.localHasNoDirectReference
tests.testers.runCommand.bork
tests.testers.runCommand.dns-resolution

`aarch64-linux`

✅ 57 packages built:

tests.fetchDebianPatch.libPackage
tests.fetchDebianPatch.simple
tests.fetchFirefoxAddon.simple (tests.fetchFirefoxAddon.overridden-source)
tests.fetchFromBitbucket.withEncodedWhitespace
tests.fetchFromBitbucket.withEncodedWhitespaceGit
tests.fetchFromBitbucket.withoutWhitespace
tests.fetchFromGitHub.dumb-http-signed-tag
tests.fetchFromGitHub.fetchTags
tests.fetchFromGitHub.leave-git
tests.fetchFromGitHub.rootDir
tests.fetchFromGitHub.simple
tests.fetchFromGitHub.sparseCheckout
tests.fetchFromGitHub.sparseCheckoutNonConeMode
tests.fetchFromGitHub.submodule-deep
tests.fetchFromGitHub.submodule-leave-git
tests.fetchFromGitHub.submodule-leave-git-deep
tests.fetchFromGitHub.submodule-simple
tests.fetchNextcloudApp.simple-sha256
tests.fetchNextcloudApp.simple-sha512
tests.fetchPypiLegacy.fetchSimple
tests.fetchgit.dumb-http-signed-tag
tests.fetchgit.fetchTags
tests.fetchgit.leave-git
tests.fetchgit.prefetch-git-no-add-path
tests.fetchgit.rootDir
tests.fetchgit.simple
tests.fetchgit.sparseCheckout
tests.fetchgit.sparseCheckoutNonConeMode
tests.fetchgit.submodule-deep
tests.fetchgit.submodule-leave-git
tests.fetchgit.submodule-leave-git-deep
tests.fetchgit.submodule-simple
tests.fetchgit.withGitConfig
tests.fetchpatch.decode
tests.fetchpatch.fileWithApostrophe
tests.fetchpatch.fileWithSpace
tests.fetchpatch.full
tests.fetchpatch.hunks
tests.fetchpatch.relative
tests.fetchpatch.simple
tests.fetchpatch2.decode
tests.fetchpatch2.fileWithApostrophe
tests.fetchpatch2.fileWithSpace
tests.fetchpatch2.full
tests.fetchpatch2.hunks
tests.fetchpatch2.relative
tests.fetchpatch2.simple
tests.fetchurl.hashedMirrors
tests.fetchurl.header
tests.fetchurl.no-skipPostFetch
tests.fetchzip.hiddenDir
tests.fetchzip.postFetch
tests.fetchzip.simple
tests.haskell.cabalSdist.assumptionLocalHasDirectReference
tests.haskell.cabalSdist.localHasNoDirectReference
tests.testers.runCommand.bork
tests.testers.runCommand.dns-resolution

`x86_64-darwin` (sandbox = relaxed)

❌ 1 package failed to build:

tests.testers.runCommand.bork

✅ 57 packages built:

tests.fetchDebianPatch.libPackage
tests.fetchDebianPatch.simple
tests.fetchFirefoxAddon.simple (tests.fetchFirefoxAddon.overridden-source)
tests.fetchFromBitbucket.withEncodedWhitespace
tests.fetchFromBitbucket.withEncodedWhitespaceGit
tests.fetchFromBitbucket.withoutWhitespace
tests.fetchFromGitHub.dumb-http-signed-tag
tests.fetchFromGitHub.fetchTags
tests.fetchFromGitHub.leave-git
tests.fetchFromGitHub.rootDir
tests.fetchFromGitHub.simple
tests.fetchFromGitHub.sparseCheckout
tests.fetchFromGitHub.sparseCheckoutNonConeMode
tests.fetchFromGitHub.submodule-deep
tests.fetchFromGitHub.submodule-leave-git
tests.fetchFromGitHub.submodule-leave-git-deep
tests.fetchFromGitHub.submodule-simple
tests.fetchNextcloudApp.simple-sha256
tests.fetchNextcloudApp.simple-sha512
tests.fetchPypiLegacy.fetchSimple
tests.fetchgit.dumb-http-signed-tag
tests.fetchgit.fetchTags
tests.fetchgit.leave-git
tests.fetchgit.prefetch-git-no-add-path
tests.fetchgit.rootDir
tests.fetchgit.simple
tests.fetchgit.sparseCheckout
tests.fetchgit.sparseCheckoutNonConeMode
tests.fetchgit.submodule-deep
tests.fetchgit.submodule-leave-git
tests.fetchgit.submodule-leave-git-deep
tests.fetchgit.submodule-simple
tests.fetchgit.withGitConfig
tests.fetchpatch.decode
tests.fetchpatch.fileWithApostrophe
tests.fetchpatch.fileWithSpace
tests.fetchpatch.full
tests.fetchpatch.hunks
tests.fetchpatch.relative
tests.fetchpatch.simple
tests.fetchpatch2.decode
tests.fetchpatch2.fileWithApostrophe
tests.fetchpatch2.fileWithSpace
tests.fetchpatch2.full
tests.fetchpatch2.hunks
tests.fetchpatch2.relative
tests.fetchpatch2.simple
tests.fetchurl.hashedMirrors
tests.fetchurl.header
tests.fetchurl.no-skipPostFetch
tests.fetchzip.hiddenDir
tests.fetchzip.postFetch
tests.fetchzip.simple
tests.haskell.cabalSdist.assumptionLocalHasDirectReference
tests.haskell.cabalSdist.localHasNoDirectReference
tests.testers.runCommand.dns-resolution
tests.testers.runCommand.nonDefault-hash

Error logs: `x86_64-darwin`

tests.testers.runCommand.bork

                    exc.stdout, exc.stderr = process.communicate()
                else:
                    # POSIX _communicate already populated the output so
                    # far into the TimeoutExpired exception.
                    process.wait()
                raise
            except:  # Including KeyboardInterrupt, communicate handled that.
                process.kill()
                # We don't call process.wait() as .__exit__ does that for us.
                raise
            retcode = process.poll()
            if check and retcode:
>               raise CalledProcessError(retcode, process.args,
                                         output=stdout, stderr=stderr)
E               subprocess.CalledProcessError: Command '['/nix/store/84s01mzl74vh69aphpl3rf07pp49pkz0-python3-3.13.9/bin/python3.13', '-m', 'bork', 'download', 'gh:duckinator/emanate', 'v7.0.0']' returned non-zero exit status 1.
/nix/store/84s01mzl74vh69aphpl3rf07pp49pkz0-python3-3.13.9/lib/python3.13/subprocess.py:577: CalledProcessError

=========================== short test summary info ============================

FAILED bork/tests/test_cmd_download.py::test_download - subprocess.CalledProcessError: Command '['/nix/store/84s01mzl74vh69aphpl3rf...

================== 1 failed, 3 passed, 5 deselected in 52.01s ==================

`aarch64-darwin` (sandbox = relaxed)

❌ 1 package failed to build:

tests.fetchurl.header

✅ 57 packages built:

tests.fetchDebianPatch.libPackage
tests.fetchDebianPatch.simple
tests.fetchFirefoxAddon.simple (tests.fetchFirefoxAddon.overridden-source)
tests.fetchFromBitbucket.withEncodedWhitespace
tests.fetchFromBitbucket.withEncodedWhitespaceGit
tests.fetchFromBitbucket.withoutWhitespace
tests.fetchFromGitHub.dumb-http-signed-tag
tests.fetchFromGitHub.fetchTags
tests.fetchFromGitHub.leave-git
tests.fetchFromGitHub.rootDir
tests.fetchFromGitHub.simple
tests.fetchFromGitHub.sparseCheckout
tests.fetchFromGitHub.sparseCheckoutNonConeMode
tests.fetchFromGitHub.submodule-deep
tests.fetchFromGitHub.submodule-leave-git
tests.fetchFromGitHub.submodule-leave-git-deep
tests.fetchFromGitHub.submodule-simple
tests.fetchNextcloudApp.simple-sha256
tests.fetchNextcloudApp.simple-sha512
tests.fetchPypiLegacy.fetchSimple
tests.fetchgit.dumb-http-signed-tag
tests.fetchgit.fetchTags
tests.fetchgit.leave-git
tests.fetchgit.prefetch-git-no-add-path
tests.fetchgit.rootDir
tests.fetchgit.simple
tests.fetchgit.sparseCheckout
tests.fetchgit.sparseCheckoutNonConeMode
tests.fetchgit.submodule-deep
tests.fetchgit.submodule-leave-git
tests.fetchgit.submodule-leave-git-deep
tests.fetchgit.submodule-simple
tests.fetchgit.withGitConfig
tests.fetchpatch.decode
tests.fetchpatch.fileWithApostrophe
tests.fetchpatch.fileWithSpace
tests.fetchpatch.full
tests.fetchpatch.hunks
tests.fetchpatch.relative
tests.fetchpatch.simple
tests.fetchpatch2.decode
tests.fetchpatch2.fileWithApostrophe
tests.fetchpatch2.fileWithSpace
tests.fetchpatch2.full
tests.fetchpatch2.hunks
tests.fetchpatch2.relative
tests.fetchpatch2.simple
tests.fetchurl.hashedMirrors
tests.fetchurl.no-skipPostFetch
tests.fetchzip.hiddenDir
tests.fetchzip.postFetch
tests.fetchzip.simple
tests.haskell.cabalSdist.assumptionLocalHasDirectReference
tests.haskell.cabalSdist.localHasNoDirectReference
tests.testers.runCommand.bork
tests.testers.runCommand.dns-resolution
tests.testers.runCommand.nonDefault-hash

Error logs: `aarch64-darwin`

tests.fetchurl.header

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (22) SSL certificate OpenSSL verify result: unable to get local issuer certificate (20) Warning: Problem : HTTP error. Will retry in 1 second. 3 retries left.

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (22) The requested URL returned error: 503
Warning: Problem : HTTP error. Will retry in 2 seconds. 2 retries left.

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (22) The requested URL returned error: 503
Warning: Problem : HTTP error. Will retry in 4 seconds. 1 retry left.

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (22) The requested URL returned error: 503
error: cannot download source-salted-lr2mzh5n379v from any mirror

Don't rely on the `postPhase` in `$stdenv/setup.sh` provided by `stdenv/generic/make-derivation.nix`'s realBuilder hack.

pkgs/build-support/fetchurl/builder.sh

Co-authored-by: Matt Sturgeon <matt@sturgeon.me.uk>

pkgs/build-support/fetchurl/tests.nix

ShamrockLee · 2025-12-17T22:13:21Z

Added name for test cases for friendlier build logs.

ShamrockLee mentioned this pull request Dec 15, 2025

Revert "fetchurl, fetchgitlab: fix curlOpts extension" #470831

Merged

ShamrockLee requested review from MattSturgeon, panicgh and philiptaron and removed request for philiptaron December 15, 2025 22:55

nixpkgs-ci bot requested a review from philiptaron December 15, 2025 22:57

nixpkgs-ci bot added 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux. 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. 6.topic: fetch Fetchers (e.g. fetchgit, fetchsvn, ...) labels Dec 15, 2025

panicgh reviewed Dec 15, 2025

View reviewed changes

pkgs/build-support/fetchurl/builder.sh Outdated Show resolved Hide resolved

fetchurl: move netrcPhase invokation into build.sh

915b484

Don't rely on the `postPhase` in `$stdenv/setup.sh` provided by `stdenv/generic/make-derivation.nix`'s realBuilder hack.

ShamrockLee force-pushed the fetchurl-netrcPhase-move branch 3 times, most recently from 09ddf0d to 21311af Compare December 17, 2025 13:52

ShamrockLee requested a review from panicgh December 17, 2025 13:52

ShamrockLee force-pushed the fetchurl-netrcPhase-move branch from 21311af to e469228 Compare December 17, 2025 13:54

MattSturgeon reviewed Dec 17, 2025

View reviewed changes

pkgs/build-support/fetchurl/builder.sh Outdated Show resolved Hide resolved

ShamrockLee force-pushed the fetchurl-netrcPhase-move branch 2 times, most recently from 0827ece to a9fdc3b Compare December 17, 2025 18:51

ShamrockLee and others added 2 commits December 18, 2025 02:52

fetchurl: builder.sh: convert curlOpts to string if array encountered

b9f1e05

Co-authored-by: Matt Sturgeon <matt@sturgeon.me.uk>

fetchurl: builder.sh: simplify curlOpts/curlOptsList appending

04419e1

ShamrockLee force-pushed the fetchurl-netrcPhase-move branch from a9fdc3b to 901c16a Compare December 17, 2025 18:53

MattSturgeon approved these changes Dec 17, 2025

View reviewed changes

pkgs/build-support/fetchurl/tests.nix Show resolved Hide resolved

nixpkgs-ci bot added the 12.approvals: 1 This PR was reviewed and approved by one person. label Dec 17, 2025

ShamrockLee added 2 commits December 18, 2025 06:10

tests.fetchurl: add flag-appending tests

dd7414b

tests.fetchurl: add test flag-appending-netrcPhase-curlOptsList

d0f0e55

ShamrockLee force-pushed the fetchurl-netrcPhase-move branch from 901c16a to d0f0e55 Compare December 17, 2025 22:12

panicgh approved these changes Dec 17, 2025

View reviewed changes

nixpkgs-ci bot added 12.approvals: 2 This PR was reviewed and approved by two persons. and removed 12.approvals: 1 This PR was reviewed and approved by one person. labels Dec 17, 2025

MattSturgeon added this pull request to the merge queue Dec 17, 2025

Merged via the queue into NixOS:master with commit c2884c4 Dec 17, 2025
28 of 32 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fetchurl: move netrcPhase invokation into build.sh and stringify `curlOpts` early#471172

fetchurl: move netrcPhase invokation into build.sh and stringify `curlOpts` early#471172
MattSturgeon merged 5 commits intoNixOS:masterfrom
ShamrockLee:fetchurl-netrcPhase-move

ShamrockLee commented Dec 15, 2025 •

edited

Loading

Uh oh!

Uh oh!

ShamrockLee commented Dec 15, 2025

Uh oh!

Uh oh!

Uh oh!

ShamrockLee commented Dec 17, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

ShamrockLee commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Things done

Uh oh!

Uh oh!

ShamrockLee commented Dec 15, 2025

nixpkgs-review result

x86_64-linux

aarch64-linux

x86_64-darwin (sandbox = relaxed)

aarch64-darwin (sandbox = relaxed)

Uh oh!

Uh oh!

Uh oh!

ShamrockLee commented Dec 17, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ShamrockLee commented Dec 15, 2025 •

edited

Loading

`nixpkgs-review` result

`x86_64-linux`

`aarch64-linux`

`x86_64-darwin` (sandbox = relaxed)

`aarch64-darwin` (sandbox = relaxed)