unable to build behind SSL proxy #201189

zoranbosnjak · 2022-11-11T12:06:01Z

nix-build does not work behing SSL proxy (nix on ubuntu-22.04).

I was given a company.crt file which was installed to /etc/ssl/certs, followed by sudo update-ca-certificates. Internet connectivity is working correctly, for example:

curl https://nixos.org
git clone https://git.savannah.gnu.org/git/hello.git/
...

However, it looks like the nix-build can not fetch from https. For example:

# myhello.nix
with (import <nixpkgs> {});

stdenv.mkDerivation rec {
  name = "test";
  src = fetchgit {
    url = "https://git.savannah.gnu.org/git/hello.git";
    rev = "fab5dca74c099b1663dce30d9e11c2d83ecf3078";
    sha256 = "1p0pnwcbgf5aks0zd52k9xkkjn8pfmk475aq640cyd05f06zzlgi";
    fetchLFS = false;
    fetchSubmodules = false;
    deepClone = false;
    leaveDotGit = false;
  };

  installPhase = ''
    mkdir -p $out
    cat ${src}/README > $out/README
  '';
}

nix-build myhello.nix works as expected in the environment without SSL proxy. But in the case of a proxy, I am getting this error:

$ nix-build myhello.nix 
these 2 derivations will be built:
  /nix/store/9jdmvzsy5s9xz7j4cazi9rjckagr6brm-hello-fab5dca.drv
  /nix/store/466hiq5vj1ha9xms8n2nc886r9751nz1-test.drv
building '/nix/store/9jdmvzsy5s9xz7j4cazi9rjckagr6brm-hello-fab5dca.drv'...
exporting https://git.savannah.gnu.org/git/hello.git (rev fab5dca74c099b1663dce30d9e11c2d83ecf3078) into /nix/store/5x5qhmxly7vnpr5ldc0q3wz5z3f9h3q5-hello-fab5dca
Initialized empty Git repository in /nix/store/5x5qhmxly7vnpr5ldc0q3wz5z3f9h3q5-hello-fab5dca/.git/
fatal: unable to access 'https://git.savannah.gnu.org/git/hello.git/': SSL certificate problem: unable to get local issuer certificate
fatal: unable to access 'https://git.savannah.gnu.org/git/hello.git/': SSL certificate problem: unable to get local issuer certificate
fatal: unable to access 'https://git.savannah.gnu.org/git/hello.git/': SSL certificate problem: unable to get local issuer certificate
Unable to checkout fab5dca74c099b1663dce30d9e11c2d83ecf3078 from https://git.savannah.gnu.org/git/hello.git.
error: builder for '/nix/store/9jdmvzsy5s9xz7j4cazi9rjckagr6brm-hello-fab5dca.drv' failed with exit code 1
error: 1 dependencies of derivation '/nix/store/466hiq5vj1ha9xms8n2nc886r9751nz1-test.drv' failed to build

The certificates are fine with git clone from the shell:

$ git clone https://git.savannah.gnu.org/git/hello.git/
Cloning into 'hello'...
remote: Counting objects: 4610, done.
remote: Compressing objects: 100% (1147/1147), done.
remote: Total 4610 (delta 3431), reused 4610 (delta 3431)
Receiving objects: 100% (4610/4610), 7.63 MiB | 6.82 MiB/s, done.
Resolving deltas: 100% (3431/3431), done.

Steps To Reproduce

Need to be behind SSL proxy.
Install ubuntu-22.04, adjust certificates for internet access.
Install nix, following https://nixos.org/manual/nix/unstable/installation/env-variables.html#nix_ssl_cert_file
Create simple myhello.nix which is fetching something from https (see above).
run nix-build myhello.nix

Expected behavior

A result link is expected after running nix-build, even if running behind proxy.

nix-env --version output

$ nix-env --version
nix-env (Nix) 2.11.1

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS"

Additional context

There are already a lot of issues reported on this topic, some are 6 or 7 years old, a lot of them are closed (sorry if it's a duplicate). But I was not able to find any workaround to the problem, despite trying hard with many proposed workarounds.

The text was updated successfully, but these errors were encountered:

thufschmitt · 2022-11-14T13:31:28Z

Does setting NIX_SSL_CERT_FILE fixes your issue?

zoranbosnjak · 2022-11-14T14:34:21Z

@thufschmitt No, it does not. I have followed the instructions from the link exactly, but the nix-build does not work.

thufschmitt · 2022-11-14T14:43:57Z

Aha, indeed, you're using nixpkgs's fetchgit 🤦‍♂️. That NIX_SSL_CERT_FILE environment variable probably only concerns the fetches that Nix itself is doing. But in your case, the fetch is done within a derivation defined in Nixpkgs. That's mostly out of my knowledge then, but looking at the code, it looks like fetchgit is picking-up the NIX_GIT_SSL_CAINFO environment variable.

I'm moving your issue to https://github.com/NixOS/nixpkgs as it's more likely to get a proper answer there if that doesn't work.

zoranbosnjak · 2022-11-15T08:10:20Z

Thanks for the tip. The NIX_GIT_SSL_CAINFO environment variable has influence on the build process. It still fails, but with different error. It looks like the fetchgit does not see the certificate file. I have tested with both, absolute path on the system and with the path inside nix store. The result is the same.

$ cat /etc/bash.bashrc | tail -n 5
# export NIX_GIT_SSL_CAINFO=/etc/ssl/certs/ca-certificates.crt
export NIX_GIT_SSL_CAINFO=/nix/store/sm1ksdlp0zlkxw5kb1nynrfws1k5d1yx-test/certificates.crt
export NIX_SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt

$ nix-build myhello.nix 
these 2 derivations will be built:
  /nix/store/9jdmvzsy5s9xz7j4cazi9rjckagr6brm-hello-fab5dca.drv
  /nix/store/466hiq5vj1ha9xms8n2nc886r9751nz1-test.drv
building '/nix/store/9jdmvzsy5s9xz7j4cazi9rjckagr6brm-hello-fab5dca.drv'...
exporting https://git.savannah.gnu.org/git/hello.git (rev fab5dca74c099b1663dce30d9e11c2d83ecf3078) into /nix/store/5x5qhmxly7vnpr5ldc0q3wz5z3f9h3q5-hello-fab5dca
Initialized empty Git repository in /nix/store/5x5qhmxly7vnpr5ldc0q3wz5z3f9h3q5-hello-fab5dca/.git/
fatal: unable to access 'https://git.savannah.gnu.org/git/hello.git/': error setting certificate file: /nix/store/sm1ksdlp0zlkxw5kb1nynrfws1k5d1yx-test/certificates.crt
fatal: unable to access 'https://git.savannah.gnu.org/git/hello.git/': error setting certificate file: /nix/store/sm1ksdlp0zlkxw5kb1nynrfws1k5d1yx-test/certificates.crt
fatal: unable to access 'https://git.savannah.gnu.org/git/hello.git/': error setting certificate file: /nix/store/sm1ksdlp0zlkxw5kb1nynrfws1k5d1yx-test/certificates.crt
Unable to checkout fab5dca74c099b1663dce30d9e11c2d83ecf3078 from https://git.savannah.gnu.org/git/hello.git.
error: builder for '/nix/store/9jdmvzsy5s9xz7j4cazi9rjckagr6brm-hello-fab5dca.drv' failed with exit code 1
error: 1 dependencies of derivation '/nix/store/466hiq5vj1ha9xms8n2nc886r9751nz1-test.drv' failed to build

So, how do I use the NIX_GIT_SSL_CAINFO variable correctly?

layus · 2022-11-15T10:34:05Z

This works, in the sense that it makes the build fail for a lack of a valid certificate.
If you pass the correct certificate bundle, it should work behind the proxy.

# myhello.nix
with (import <nixpkgs> {});

stdenv.mkDerivation rec {
  name = "test";
  src = (fetchgit {
    url = "https://git.savannah.gnu.org/git/hello.git";
    rev = "fab5dca74c099b1663dce30d9e11c2d83ecf3078";
    sha256 = "1p0pnwcbgf5aks0zd52k9xkkjn8pfmk475aq640cyd05f06zzlgi";
    fetchLFS = false;
    fetchSubmodules = false;
    deepClone = false;
    leaveDotGit = false;
  }).overrideAttrs (oldAttrs: { GIT_SSL_CAINFO = ./hello.nix; })
  ;

  installPhase = ''
    mkdir -p $out
    cat ${src}/README > $out/README
  '';
}

zoranbosnjak · 2022-11-15T11:48:10Z

It still does not see the cert file...

$ nix-build hello.nix 
/nix/store/sm1ksdlp0zlkxw5kb1nynrfws1k5d1yx-test

$ nix-build myhello.nix
these 2 derivations will be built:
  /nix/store/61r3xfdmh7zzxk73bvqfgf0jfw273n6h-hello-fab5dca.drv
  /nix/store/60n1a4l3asissyccracihi1bx5dllf1s-test.drv
building '/nix/store/61r3xfdmh7zzxk73bvqfgf0jfw273n6h-hello-fab5dca.drv'...
exporting https://git.savannah.gnu.org/git/hello.git (rev fab5dca74c099b1663dce30d9e11c2d83ecf3078) into /nix/store/5x5qhmxly7vnpr5ldc0q3wz5z3f9h3q5-hello-fab5dca
Initialized empty Git repository in /nix/store/5x5qhmxly7vnpr5ldc0q3wz5z3f9h3q5-hello-fab5dca/.git/
fatal: unable to access 'https://git.savannah.gnu.org/git/hello.git/': error setting certificate file: /nix/store/sm1ksdlp0zlkxw5kb1nynrfws1k5d1yx-test/certificates.crt
fatal: unable to access 'https://git.savannah.gnu.org/git/hello.git/': error setting certificate file: /nix/store/sm1ksdlp0zlkxw5kb1nynrfws1k5d1yx-test/certificates.crt
fatal: unable to access 'https://git.savannah.gnu.org/git/hello.git/': error setting certificate file: /nix/store/sm1ksdlp0zlkxw5kb1nynrfws1k5d1yx-test/certificates.crt
Unable to checkout fab5dca74c099b1663dce30d9e11c2d83ecf3078 from https://git.savannah.gnu.org/git/hello.git.
error: builder for '/nix/store/61r3xfdmh7zzxk73bvqfgf0jfw273n6h-hello-fab5dca.drv' failed with exit code 1
error: 1 dependencies of derivation '/nix/store/60n1a4l3asissyccracihi1bx5dllf1s-test.drv' failed to build

... but the file is there

$ ls /nix/store/sm1ksdlp0zlkxw5kb1nynrfws1k5d1yx-test/certificates.crt
/nix/store/sm1ksdlp0zlkxw5kb1nynrfws1k5d1yx-test/certificates.crt

Another problem is that a myhello.nix is now polluted with hello.nix cert file, which would be a problem in the environment without SSL proxy. Is there a way to override a fetchGit or something else, such that myhello.nix remains independent of the proxy/non-proxy setting?

layus · 2022-11-15T14:30:36Z

That's what I meant by a quick and dirty solution. A proper solution would add the system certificate bundle inside the build sandbox, and use it when available. But that requires changes to nix. That is also not a problem on other platforms if you use an impure path to specify the bundle. Lots of things to think about, but first let's ensure that we can provide a proper GIT_SSL_CAINFO file. Maybe the format is different ? I wonder why it refuses to use yours 🤔

zoranbosnjak · 2022-11-15T15:40:57Z

I don't quite understand the part about other platforms and the way to specify the impure path. What platforms do you have in mind? How do you use an impure path on those platforms?

The file format seems correct. There is no difference between that file and the file which is used to access the internet outside of nix.

$ diff /etc/ssl/certs/ca-certificates.crt /nix/store/sm1ksdlp0zlkxw5kb1nynrfws1k5d1yx-test/certificates.crt
<no diff output>

layus · 2022-11-16T13:17:46Z

I think the issue was that nix copies the symlink to the store, instead of the content of the file. Here is a version that works:

# hello.nix
with (import <nixpkgs> {});

stdenv.mkDerivation rec {
  name = "test";
  src = (fetchgit {
    url = "https://git.savannah.gnu.org/git/hello.git";
    rev = "fab5dca74c099b1663dce30d9e11c2d83ecf3078";
    sha256 = "1p0pnwcbgf5aks0zd52k9xkkjn8pfmk475aq640cyd05f06zzlgi";
    fetchLFS = false;
    fetchSubmodules = false;
    deepClone = false;
    leaveDotGit = false;
  }).overrideAttrs (oldAttrs: { GIT_SSL_CAINFO = writeText "certificates.pem" (builtins.readFile "/etc/ssl/certs/ca-bundle.crt"); })
  ;

  installPhase = ''
    mkdir -p $out
    cat ${src}/README > $out/README
  '';
}

The problem with the platforms is that the system-wide certificates are not always in the same location. So hard-coding the path is not okay.

zoranbosnjak · 2022-11-16T14:42:51Z

Thanks for your help. The observation in my previous comments was wrong. The content of the cert file in the nix store was actually not correct. I was testing with NIX_GIT_SSL_CAINFO and NIX_SSL_CERT_FILE variables set in the /etc/bash.bashrc, which made the difference. So, it did not work, but for another reason. Sorry about the confusion.

Anyway, your last suggestion works, with minor filename change, to match the actual cert filename /etc/ssl/certs/ca-certificates.crt, instead of ca-bundle.crt. But it only works if the environment variable NIX_GIT_SSL_CAINFO is not set. It looks like this variable has priority, but the path is not visible during nix-build.

Question: When nix-build reports something like

fatal: unable to access 'https://git.savannah.gnu.org/git/hello.git/': SSL certificate problem: unable to get local issuer certificate

... where does it look for the certificate files?

Another confusion is that the source code actually contains impureEnvVars. This makes me belive that one should actually be able to override it. It just doesn't work correctly.

nixpkgs/pkgs/build-support/fetchgit/default.nix

Lines 91 to 95 in a110f08

    
           GIT_SSL_CAINFO = "${cacert}/etc/ssl/certs/ca-bundle.crt"; 
        
           impureEnvVars = lib.fetchers.proxyImpureEnvVars ++ netrcImpureEnvVars ++ [ 
        
             "GIT_PROXY_COMMAND" "NIX_GIT_SSL_CAINFO" "SOCKS_SERVER" 
        
           ];

zoranbosnjak · 2023-02-14T11:52:49Z

Duplicate with issue NixOS/nix#4173
Closing.

AriFordsham · 2023-07-23T16:31:44Z

@zoranbosnjak It seems the working workaround is to add these lines:

Environment=NIX_GIT_SSL_CAINFO=/etc/ssl/certs/ca-certificates.crt
Environment=NIX_SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt

To the [Service] section of /etc/systemd/system/nix-daemon.service and then restarting the machine.

This ensures that derivations that are built under the Nix daemon process get the correct environment variables.

Source: #3382 (comment)

zoranbosnjak added the bug label Nov 11, 2022

thufschmitt transferred this issue from NixOS/nix Nov 14, 2022

zoranbosnjak mentioned this issue Nov 15, 2022

Not all fetches use system ssl certs NixOS/nix#4173

Open

zoranbosnjak closed this as completed Feb 14, 2023

Artturin added 0.kind: bug Something is broken and removed bug labels Apr 19, 2023

Mastermjr mentioned this issue Apr 20, 2024

How to make the lock building process for python proxy aware? nix-community/dream2nix#933

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

unable to build behind SSL proxy #201189

unable to build behind SSL proxy #201189

zoranbosnjak commented Nov 11, 2022

thufschmitt commented Nov 14, 2022

zoranbosnjak commented Nov 14, 2022

thufschmitt commented Nov 14, 2022

zoranbosnjak commented Nov 15, 2022

layus commented Nov 15, 2022

zoranbosnjak commented Nov 15, 2022

layus commented Nov 15, 2022

zoranbosnjak commented Nov 15, 2022

layus commented Nov 16, 2022

zoranbosnjak commented Nov 16, 2022

zoranbosnjak commented Feb 14, 2023

AriFordsham commented Jul 23, 2023

unable to build behind SSL proxy #201189

unable to build behind SSL proxy #201189

Comments

zoranbosnjak commented Nov 11, 2022

thufschmitt commented Nov 14, 2022

zoranbosnjak commented Nov 14, 2022

thufschmitt commented Nov 14, 2022

zoranbosnjak commented Nov 15, 2022

layus commented Nov 15, 2022

zoranbosnjak commented Nov 15, 2022

layus commented Nov 15, 2022

zoranbosnjak commented Nov 15, 2022

layus commented Nov 16, 2022

zoranbosnjak commented Nov 16, 2022

zoranbosnjak commented Feb 14, 2023

AriFordsham commented Jul 23, 2023