http_proxy, https_proxy not honored by Dockerfile ADD instruction?

[dancysoft]

At Wikimedia Foundation we build buildkit releases locally to publish to our own registry. We build using buildkitd (currently v0.15.2). The node that buildkitd runs on does not have direct Internet access and must use a proxy. We pass --opt build-arg:http_proxy="$http_proxy" --opt build-arg:https_proxy="$https_proxy" --opt build-arg:no_proxy="$no_proxy" (with variable values coming from the environment) to buildctl build which generally works well. However, buildkit v0.16's Dockerfile started using instructions like ADD --keep-git-dir=true https://github.com/containers/dnsname.git#v1.3.1 . These instructions are failing when we attempt to build buildkit v0.16:

> [dnsname 2/3] ADD --keep-git-dir=true https://github.com/containers/dnsname.git#v1.3.1 .:
#33 129.4 fatal: unable to access 'https://github.com/containers/dnsname.git/': Failed to connect to github.com port 443 after 129401 ms: Couldn't connect to server

Other instructions which use the Internet work fine, for example:

#19 [cni-plugins 5/7] RUN curl -Ls https://github.com/containernetworking/plugins/releases/download/v1.5.1/cni-plugins-linux-amd64-v1.5.1.tgz | tar xzv

It seems like whatever implements ADD (dockerfile-frontend, presumably) isn't honoring the proxy settings.

The transcript of the failing job is available here: https://gitlab.wikimedia.org/repos/releng/buildkit/-/jobs/361358

[dancysoft]

This may be the same thing as #4643.

[tonistiigi]

@thaJeztah Any memory if this might have been intentional? I think we should use them if set with build-arg.

Note that proxy settings do not apply to FROM as well, but I think there it is more expected as registry config/mirrors is in daemon config.

Atm. you should be able to work around it by setting docker buildx create --driver-opt env.HTTP_PROXY=foo

[dancysoft]

Atm. you should be able to work around it by setting docker buildx create --driver-opt env.HTTP_PROXY=foo

In the original description of this ticket I said we run docker build, but we actually use buildctl build. What would be the corresponding workaround for that client?

[tonistiigi]

In the original description of this ticket I said we run docker build, but we actually use buildctl build. What would be the corresponding workaround for that client?

You need to define the proxy env when you are starting the buildkitd daemon, so depends on how you are doing it. The previous buildx command runs buildkitd in container and --driver-opt defines env for that container.

[thaJeztah]

Any memory if this might have been intentional? I think we should use them if set with build-arg.

Hmm.. no, I don't think this was ever discussed, or a conscious decision.

Note that proxy settings do not apply to FROM as well, but I think there it is more expected as registry config/mirrors is in daemon config.

So, that's actually an interesting point; I'm guessing COPY --from=<some image reference> will also take the daemon proxy settings.

I'm considering what's most logical here, and "least surprise";

• Proxy as configured per-build (through --build-arg)
• Proxy as configured for the daemon environment (networking connections made by the daemon itself)

Out of curiousity, I wondered what the classic builder used, and it looks like the classic builder uses the daemon proxy settings;

Start the daemon with some proxy (I used example.com which will fail when used as proxy);

HTTP_PROXY=http://example.com HTTPS_PROXY=http://example.com dockerd --debug

Try a build with a ADD <github URL>;

echo -e 'FROM scratch\nADD https://github.com/containers/dnsname.git#v1.3.1 .' | DOCKER_BUILDKIT=0 docker build -
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM scratch
 --->
Step 2/2 : ADD https://github.com/containers/dnsname.git#v1.3.1 .
ADD failed: Get "https://github.com/containers/dnsname.git#v1.3.1": Bad Request

Starting the daemon without proxy configured, and trying again;

echo -e 'FROM scratch\nADD https://github.com/containers/dnsname.git#v1.3.1 .' | DOCKER_BUILDKIT=0 docker build -
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
            environment-variable.

Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM scratch
 --->
Step 2/2 : ADD https://github.com/containers/dnsname.git#v1.3.1 .
Downloading  353.2kB

 ---> a014e8d6b763
Successfully built a014e8d6b763

[thaJeztah]

So, from the above, at least it looks like BuildKit behaves different than the classic builder, which could be considered a regression.

Whether the "build-arg" proxy, or "daemon" proxy should be used is still an interesting topic. The daemon may require a specific proxy (if the daemon is running in a different environment than the client).

But a specific build requiring a specific proxy can be a very valid use-case as well. I'm still considering if we need explicit options to allow the user to decide, because either could be considered valid. (e.g. proxy=host or whatever.

If "per build" proxies are considered for this; wondering if allowing different proxies for git would (should?) impact whether those can be shared between builds? i.e. should a docker build --build-arg <proxy-X> share the same git checkout as a build cloning the same repository but without (or with a different) proxy? I think we always considered proxy to not be affecting build-cache, so maybe that's ok; not sure if this specific situation (git clone) is different in that respect.

[dancysoft]

In the original description of this ticket I said we run docker build, but we actually use buildctl build. What would be the corresponding workaround for that client?

You need to define the proxy env when you are starting the buildkitd daemon, so depends on how you are doing it. The previous buildx command runs buildkitd in container and --driver-opt defines env for that container.

Thank you. We run buildkitd using docker. We pass --env-file /etc/default/buildkitd to docker run and /etc/default/buildkitd has all of the proxy variables set (lower and uppercase forms). I can confirm that the variables are set in the buildkitd container using docker exec <containerid> env:

# docker exec 2c9c1f0d9696 env | egrep -i https?_proxy
http_proxy=http://webproxy:8080
https_proxy=http://webproxy:8080
HTTP_PROXY=http://webproxy:8080
HTTPS_PROXY=http://webproxy:8080