Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Proxy configuration does not seem to be respected with HTTPS Git build contexts #4643

Open
sigwinch28 opened this issue Feb 13, 2024 · 3 comments
Open

Proxy configuration does not seem to be respected with HTTPS Git build contexts #4643

sigwinch28 opened this issue Feb 13, 2024 · 3 comments
Labels
status/needs-investigation

Comments

@sigwinch28
Copy link

sigwinch28 commented Feb 13, 2024 •
edited
Loading

I can't seem to use an HTTPS git source from behind a corporate proxy, e.g. http://proxy.contoso.com:8080.

Buildx version: github.com/docker/buildx v0.12.1 30feaa1a915b869ebc2eea6328624b49facd4bfb
Buildkit runner creation command:

docker buildx create --use --bootstrap --driver docker-container --driver-opt env.http_proxy="$http_proxy" --driver-opt env.https_proxy="$https_proxy" --driver-opt image=moby/buildkit:v0.12.5 --config buildkitd.toml

buildkit.toml:

debug = true

Build command:

docker buildx build \
    --build-arg HTTP_PROXY=$http_proxy \
    --build-arg HTTPS_PROXY=$https_proxy \
    --build-arg http_proxy=$http_proxy \
    --build-arg https_proxy=$https_proxy \
    --build-arg NO_PROXY=$no_proxy \
    --build-arg no_proxy=$no_proxy \
    --secret id=GIT_AUTH_TOKEN,src=.token \
    https://github.com/something/foo.git

I have veirifed that github.com is not in the $no_proxy env var.

Build output:

[+] Building 129.5s (2/2) FINISHED                                                                                                                                                                         docker-container:vigorous_maxwell
 => [internal] connecting to local controller                                                                                                                                                                                           0.0s
 => ERROR [internal] load git source https://github.com/something/foo.git                                                                                                                                                             129.4s
------
 > [internal] load git source https://github.com/something/foo.git:
0.028 Initialized empty Git repository in /var/lib/buildkit/runc-overlayfs/snapshots/snapshots/1/fs/
129.4 fatal: unable to access 'https://github.com/something/foo.git/': Failed to connect to github.com port 443 after 129328 ms: Couldn't connect to server
------
WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
ERROR: failed to read dockerfile: failed to load cache key: error fetching default branch for repository https://github.com/something/foo.git: exit status 128

logs from build container:

time="2024-02-13T11:48:22Z" level=info msg="auto snapshotter: using overlayfs"
time="2024-02-13T11:48:22Z" level=warning msg="using host network as the default"
time="2024-02-13T11:48:22Z" level=info msg="found worker \"8dv9gbdy9fwylk06nyymgrlic\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:119ae4dff89f org.mobyproject.buildkit.worker.network:host org.mobyproject.buildkit.worker.oci.process-mode:sandbox org.mobyproject.buildkit.worker.selinux.enabled:false org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/amd64/v2 linux/amd64/v3 linux/amd64/v4 linux/386]"
time="2024-02-13T11:48:22Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
time="2024-02-13T11:48:22Z" level=info msg="found 1 workers, default=\"8dv9gbdy9fwylk06nyymgrlic\""
time="2024-02-13T11:48:22Z" level=warning msg="currently, only the default worker can be used."
time="2024-02-13T11:48:22Z" level=info msg="running server on /run/buildkit/buildkitd.sock"
time="2024-02-13T11:50:35Z" level=debug msg="session started" spanID=6733343121a26110 traceID=1e33d2941065bddc3290db2a64b66f61
time="2024-02-13T11:50:35Z" level=debug msg="session finished: <nil>" spanID=6733343121a26110 traceID=1e33d2941065bddc3290db2a64b66f61
time="2024-02-13T11:50:35Z" level=debug msg="session started" spanID=79dfd300b2c04502 traceID=b03e9542c28dd957985c99ae53dc0dc8
time="2024-02-13T11:52:44Z" level=error msg="/moby.buildkit.v1.frontend.LLBBridge/Solve returned error: rpc error: code = Unknown desc = failed to read dockerfile: failed to load cache key: error fetching default branch for repository https://github.com/something/foo.git: exit status 128"
failed to read dockerfile: failed to load cache key: error fetching default branch for repository https://github.com/something/foo.git: exit status 128
7 v0.12.5 buildkitd --config /etc/buildkit/buildkitd.toml
github.com/moby/buildkit/source/git.getDefaultBranch
        /src/source/git/gitsource.go:719
github.com/moby/buildkit/source/git.(*gitSourceHandler).CacheKey
        /src/source/git/gitsource.go:340
github.com/moby/buildkit/solver/llbsolver/ops.(*SourceOp).CacheMap
        /src/solver/llbsolver/ops/source.go:82
github.com/moby/buildkit/solver.(*sharedOp).CacheMap.func2
        /src/solver/jobs.go:832
github.com/moby/buildkit/util/flightcontrol.(*call[...]).run
        /src/util/flightcontrol/flightcontrol.go:121
sync.(*Once).doSlow
        /usr/local/go/src/sync/once.go:74
sync.(*Once).Do
        /usr/local/go/src/sync/once.go:65
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1598

7 v0.12.5 buildkitd --config /etc/buildkit/buildkitd.toml
github.com/moby/buildkit/solver.(*edge).unpark.func1
        /src/solver/edge.go:350
github.com/moby/buildkit/solver/internal/pipe.NewWithFunction.func2
        /src/solver/internal/pipe/pipe.go:82
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1598

7 v0.12.5 buildkitd --config /etc/buildkit/buildkitd.toml
github.com/moby/buildkit/frontend/dockerui.(*Client).ReadEntrypoint
        /src/frontend/dockerui/config.go:367
github.com/moby/buildkit/frontend/dockerfile/builder.Build
        /src/frontend/dockerfile/builder/build.go:44
github.com/moby/buildkit/frontend/gateway/forwarder.(*GatewayForwarder).Solve
        /src/frontend/gateway/forwarder/frontend.go:36
github.com/moby/buildkit/solver/llbsolver.(*provenanceBridge).Solve
        /src/solver/llbsolver/provenance.go:168
github.com/moby/buildkit/frontend/gateway.(*llbBridgeForwarder).Solve
        /src/frontend/gateway/gateway.go:662
github.com/moby/buildkit/control/gateway.(*GatewayForwarder).Solve
        /src/control/gateway/gateway.go:103
github.com/moby/buildkit/frontend/gateway/pb._LLBBridge_Solve_Handler.func1
        /src/frontend/gateway/pb/gateway.pb.go:3108
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1
        /src/vendor/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go:342
main.unaryInterceptor.func1
        /src/cmd/buildkitd/main.go:603
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
github.com/moby/buildkit/frontend/gateway/pb._LLBBridge_Solve_Handler
        /src/frontend/gateway/pb/gateway.pb.go:3110
google.golang.org/grpc.(*Server).processUnaryRPC
        /src/vendor/google.golang.org/grpc/server.go:1336
google.golang.org/grpc.(*Server).handleStream
        /src/vendor/google.golang.org/grpc/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
        /src/vendor/google.golang.org/grpc/server.go:965
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1598

7 v0.12.5 buildkitd --config /etc/buildkit/buildkitd.toml
main.unaryInterceptor.func1
        /src/cmd/buildkitd/main.go:607
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
github.com/moby/buildkit/frontend/gateway/pb._LLBBridge_Solve_Handler
        /src/frontend/gateway/pb/gateway.pb.go:3110
google.golang.org/grpc.(*Server).processUnaryRPC
        /src/vendor/google.golang.org/grpc/server.go:1336
google.golang.org/grpc.(*Server).handleStream
        /src/vendor/google.golang.org/grpc/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
        /src/vendor/google.golang.org/grpc/server.go:965
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1598

time="2024-02-13T11:52:44Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = failed to read dockerfile: failed to load cache key: error fetching default branch for repository https://github.com/something/foo.git: exit status 128"
failed to read dockerfile: failed to load cache key: error fetching default branch for repository https://github.com/something/foo.git: exit status 128
7 v0.12.5 buildkitd --config /etc/buildkit/buildkitd.toml
github.com/moby/buildkit/source/git.getDefaultBranch
        /src/source/git/gitsource.go:719
github.com/moby/buildkit/source/git.(*gitSourceHandler).CacheKey
        /src/source/git/gitsource.go:340
github.com/moby/buildkit/solver/llbsolver/ops.(*SourceOp).CacheMap
        /src/solver/llbsolver/ops/source.go:82
github.com/moby/buildkit/solver.(*sharedOp).CacheMap.func2
        /src/solver/jobs.go:832
github.com/moby/buildkit/util/flightcontrol.(*call[...]).run
        /src/util/flightcontrol/flightcontrol.go:121
sync.(*Once).doSlow
        /usr/local/go/src/sync/once.go:74
sync.(*Once).Do
        /usr/local/go/src/sync/once.go:65
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1598

7 v0.12.5 buildkitd --config /etc/buildkit/buildkitd.toml
github.com/moby/buildkit/solver.(*edge).unpark.func1
        /src/solver/edge.go:350
github.com/moby/buildkit/solver/internal/pipe.NewWithFunction.func2
        /src/solver/internal/pipe/pipe.go:82
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1598

7 v0.12.5 buildkitd --config /etc/buildkit/buildkitd.toml
github.com/moby/buildkit/frontend/dockerui.(*Client).ReadEntrypoint
        /src/frontend/dockerui/config.go:367
github.com/moby/buildkit/frontend/dockerfile/builder.Build
        /src/frontend/dockerfile/builder/build.go:44
github.com/moby/buildkit/frontend/gateway/forwarder.(*GatewayForwarder).Solve
        /src/frontend/gateway/forwarder/frontend.go:36
github.com/moby/buildkit/solver/llbsolver.(*provenanceBridge).Solve
        /src/solver/llbsolver/provenance.go:168
github.com/moby/buildkit/frontend/gateway.(*llbBridgeForwarder).Solve
        /src/frontend/gateway/gateway.go:662
github.com/moby/buildkit/control/gateway.(*GatewayForwarder).Solve
        /src/control/gateway/gateway.go:103
github.com/moby/buildkit/frontend/gateway/pb._LLBBridge_Solve_Handler.func1
        /src/frontend/gateway/pb/gateway.pb.go:3108
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1
        /src/vendor/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go:342
main.unaryInterceptor.func1
        /src/cmd/buildkitd/main.go:603
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
github.com/moby/buildkit/frontend/gateway/pb._LLBBridge_Solve_Handler
        /src/frontend/gateway/pb/gateway.pb.go:3110
google.golang.org/grpc.(*Server).processUnaryRPC
        /src/vendor/google.golang.org/grpc/server.go:1336
google.golang.org/grpc.(*Server).handleStream
        /src/vendor/google.golang.org/grpc/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
        /src/vendor/google.golang.org/grpc/server.go:965
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1598

3641819 v0.12.1 /home/joe/.docker/cli-plugins/docker-buildx buildx build --build-arg HTTP_PROXY=http://proxy.contoso.com:8080 --build-arg HTTPS_PROXY=http://proxy.contoso.com:8080 --build-arg http_proxy=http://proxy.contoso.com:8080 --build-arg https_proxy=http://proxy.contoso.com:8080 --build-arg NO_PROXY=169.254.169.254,.s3.eu-west-1.amazonaws.com,10.0.0.0/8,192.168.0.0/16,172.16.0.0/12,127.0.0.0/8,169.254.0.0/16,localhost,contoso.com --build-arg no_proxy=169.254.169.254,.s3.eu-west-1.amazonaws.com,10.0.0.0/8,192.168.0.0/16,172.16.0.0/12,127.0.0.0/8,169.254.0.0/16,localhost,contoso.com --secret id=GIT_AUTH_TOKEN,src=.token https://github.com/something/foo.git
google.golang.org/grpc.getChainUnaryInvoker.func1
        google.golang.org/[email protected]/clientconn.go:361
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryClientInterceptor.func1
        go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/[email protected]/interceptor.go:99
github.com/moby/buildkit/client.New.filterInterceptor.func6
        github.com/moby/[email protected]/client/client.go:398
google.golang.org/grpc.DialContext.chainUnaryClientInterceptors.func3
        google.golang.org/[email protected]/clientconn.go:349
google.golang.org/grpc.(*ClientConn).Invoke
        google.golang.org/[email protected]/call.go:35
github.com/moby/buildkit/frontend/gateway/pb.(*lLBBridgeClient).Solve
        github.com/moby/[email protected]/frontend/gateway/pb/gateway.pb.go:2875
github.com/moby/buildkit/client.(*gatewayClientForBuild).Solve
        github.com/moby/[email protected]/client/build.go:89
github.com/moby/buildkit/frontend/gateway/grpcclient.(*grpcClient).Solve
        github.com/moby/[email protected]/frontend/gateway/grpcclient/client.go:411
github.com/docker/buildx/build.BuildWithResultHandler.func3.1.2
        github.com/docker/buildx/build/build.go:914
github.com/docker/buildx/build.NewResultHandle.func3.1
        github.com/docker/buildx/build/result.go:91
github.com/moby/buildkit/frontend/gateway/grpcclient.(*grpcClient).Run
        github.com/moby/[email protected]/frontend/gateway/grpcclient/client.go:214
github.com/moby/buildkit/client.(*Client).Build.func2
        github.com/moby/[email protected]/client/build.go:59
github.com/moby/buildkit/client.(*Client).solve.func3
        github.com/moby/[email protected]/client/solve.go:283
golang.org/x/sync/errgroup.(*Group).Go.func1
        golang.org/x/[email protected]/errgroup/errgroup.go:75
runtime.goexit
        runtime/asm_amd64.s:1650

7 v0.12.5 buildkitd --config /etc/buildkit/buildkitd.toml
github.com/moby/buildkit/frontend/gateway.(*llbBridgeForwarder).Return
        /src/frontend/gateway/gateway.go:923
github.com/moby/buildkit/control/gateway.(*GatewayForwarder).Return
        /src/control/gateway/gateway.go:135
github.com/moby/buildkit/frontend/gateway/pb._LLBBridge_Return_Handler.func1
        /src/frontend/gateway/pb/gateway.pb.go:3216
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1
        /src/vendor/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go:342
main.unaryInterceptor.func1
        /src/cmd/buildkitd/main.go:603
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
github.com/moby/buildkit/frontend/gateway/pb._LLBBridge_Return_Handler
        /src/frontend/gateway/pb/gateway.pb.go:3218
google.golang.org/grpc.(*Server).processUnaryRPC
        /src/vendor/google.golang.org/grpc/server.go:1336
google.golang.org/grpc.(*Server).handleStream
        /src/vendor/google.golang.org/grpc/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
        /src/vendor/google.golang.org/grpc/server.go:965
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1598

7 v0.12.5 buildkitd --config /etc/buildkit/buildkitd.toml
main.unaryInterceptor.func1
        /src/cmd/buildkitd/main.go:607
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
github.com/moby/buildkit/api/services/control._Control_Solve_Handler
        /src/api/services/control/control.pb.go:2440
google.golang.org/grpc.(*Server).processUnaryRPC
        /src/vendor/google.golang.org/grpc/server.go:1336
google.golang.org/grpc.(*Server).handleStream
        /src/vendor/google.golang.org/grpc/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
        /src/vendor/google.golang.org/grpc/server.go:965
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1598

time="2024-02-13T11:52:44Z" level=debug msg="session finished: <nil>" spanID=79dfd300b2c04502 traceID=b03e9542c28dd957985c99ae53dc0dc8

Also, if I provide a branch, it fails too:

docker buildx build \
    --build-arg HTTP_PROXY=$http_proxy \
    --build-arg HTTPS_PROXY=$https_proxy \
    --build-arg http_proxy=$http_proxy \
    --build-arg https_proxy=$https_proxy \
    --build-arg NO_PROXY=$no_proxy \
    --build-arg no_proxy=$no_proxy \
    --secret id=GIT_AUTH_TOKEN,src=.token \
    https://github.com/something/foo.git#master

The logs show:

time="2024-02-13T12:00:23Z" level=error msg="/moby.buildkit.v1.frontend.LLBBridge/Solve returned error: rpc error: code = Unknown desc = failed to read dockerfile: failed to load cache key: failed to fetch remote https://github.com/something/foo.git: exit status 128"
failed to read dockerfile: failed to load cache key: failed to fetch remote https://github.com/something/foo.git: exit status 128
7 v0.12.5 buildkitd --config /etc/buildkit/buildkitd.toml
github.com/moby/buildkit/source/git.(*gitSourceHandler).CacheKey
        /src/source/git/gitsource.go:350
github.com/moby/buildkit/solver/llbsolver/ops.(*SourceOp).CacheMap
        /src/solver/llbsolver/ops/source.go:82
github.com/moby/buildkit/solver.(*sharedOp).CacheMap.func2
        /src/solver/jobs.go:832
github.com/moby/buildkit/util/flightcontrol.(*call[...]).run
        /src/util/flightcontrol/flightcontrol.go:121
sync.(*Once).doSlow
        /usr/local/go/src/sync/once.go:74
sync.(*Once).Do
        /usr/local/go/src/sync/once.go:65
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1598

7 v0.12.5 buildkitd --config /etc/buildkit/buildkitd.toml
github.com/moby/buildkit/solver.(*edge).unpark.func1
        /src/solver/edge.go:350
github.com/moby/buildkit/solver/internal/pipe.NewWithFunction.func2
        /src/solver/internal/pipe/pipe.go:82
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1598

7 v0.12.5 buildkitd --config /etc/buildkit/buildkitd.toml
github.com/moby/buildkit/frontend/dockerui.(*Client).ReadEntrypoint
        /src/frontend/dockerui/config.go:367
github.com/moby/buildkit/frontend/dockerfile/builder.Build
        /src/frontend/dockerfile/builder/build.go:44
github.com/moby/buildkit/frontend/gateway/forwarder.(*GatewayForwarder).Solve
        /src/frontend/gateway/forwarder/frontend.go:36
github.com/moby/buildkit/solver/llbsolver.(*provenanceBridge).Solve
        /src/solver/llbsolver/provenance.go:168
github.com/moby/buildkit/frontend/gateway.(*llbBridgeForwarder).Solve
        /src/frontend/gateway/gateway.go:662
github.com/moby/buildkit/control/gateway.(*GatewayForwarder).Solve
        /src/control/gateway/gateway.go:103
github.com/moby/buildkit/frontend/gateway/pb._LLBBridge_Solve_Handler.func1
        /src/frontend/gateway/pb/gateway.pb.go:3108
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1
        /src/vendor/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go:342
main.unaryInterceptor.func1
        /src/cmd/buildkitd/main.go:603
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
github.com/moby/buildkit/frontend/gateway/pb._LLBBridge_Solve_Handler
        /src/frontend/gateway/pb/gateway.pb.go:3110
google.golang.org/grpc.(*Server).processUnaryRPC
        /src/vendor/google.golang.org/grpc/server.go:1336
google.golang.org/grpc.(*Server).handleStream
        /src/vendor/google.golang.org/grpc/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
        /src/vendor/google.golang.org/grpc/server.go:965
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1598

7 v0.12.5 buildkitd --config /etc/buildkit/buildkitd.toml
main.unaryInterceptor.func1
        /src/cmd/buildkitd/main.go:607
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
github.com/moby/buildkit/frontend/gateway/pb._LLBBridge_Solve_Handler
        /src/frontend/gateway/pb/gateway.pb.go:3110
google.golang.org/grpc.(*Server).processUnaryRPC
        /src/vendor/google.golang.org/grpc/server.go:1336
google.golang.org/grpc.(*Server).handleStream
        /src/vendor/google.golang.org/grpc/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
        /src/vendor/google.golang.org/grpc/server.go:965
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1598

time="2024-02-13T12:00:23Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = failed to read dockerfile: failed to load cache key: failed to fetch remote https://github.com/something/foo.git: exit status 128"
failed to read dockerfile: failed to load cache key: failed to fetch remote https://github.com/something/foo.git: exit status 128
7 v0.12.5 buildkitd --config /etc/buildkit/buildkitd.toml
github.com/moby/buildkit/source/git.(*gitSourceHandler).CacheKey
        /src/source/git/gitsource.go:350
github.com/moby/buildkit/solver/llbsolver/ops.(*SourceOp).CacheMap
        /src/solver/llbsolver/ops/source.go:82
github.com/moby/buildkit/solver.(*sharedOp).CacheMap.func2
        /src/solver/jobs.go:832
github.com/moby/buildkit/util/flightcontrol.(*call[...]).run
        /src/util/flightcontrol/flightcontrol.go:121
sync.(*Once).doSlow
        /usr/local/go/src/sync/once.go:74
sync.(*Once).Do
        /usr/local/go/src/sync/once.go:65
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1598

7 v0.12.5 buildkitd --config /etc/buildkit/buildkitd.toml
github.com/moby/buildkit/solver.(*edge).unpark.func1
        /src/solver/edge.go:350
github.com/moby/buildkit/solver/internal/pipe.NewWithFunction.func2
        /src/solver/internal/pipe/pipe.go:82
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1598

7 v0.12.5 buildkitd --config /etc/buildkit/buildkitd.toml
github.com/moby/buildkit/frontend/dockerui.(*Client).ReadEntrypoint
        /src/frontend/dockerui/config.go:367
github.com/moby/buildkit/frontend/dockerfile/builder.Build
        /src/frontend/dockerfile/builder/build.go:44
github.com/moby/buildkit/frontend/gateway/forwarder.(*GatewayForwarder).Solve
        /src/frontend/gateway/forwarder/frontend.go:36
github.com/moby/buildkit/solver/llbsolver.(*provenanceBridge).Solve
        /src/solver/llbsolver/provenance.go:168
github.com/moby/buildkit/frontend/gateway.(*llbBridgeForwarder).Solve
        /src/frontend/gateway/gateway.go:662
github.com/moby/buildkit/control/gateway.(*GatewayForwarder).Solve
        /src/control/gateway/gateway.go:103
github.com/moby/buildkit/frontend/gateway/pb._LLBBridge_Solve_Handler.func1
        /src/frontend/gateway/pb/gateway.pb.go:3108
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1
        /src/vendor/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go:342
main.unaryInterceptor.func1
        /src/cmd/buildkitd/main.go:603
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
github.com/moby/buildkit/frontend/gateway/pb._LLBBridge_Solve_Handler
        /src/frontend/gateway/pb/gateway.pb.go:3110
google.golang.org/grpc.(*Server).processUnaryRPC
        /src/vendor/google.golang.org/grpc/server.go:1336
google.golang.org/grpc.(*Server).handleStream
        /src/vendor/google.golang.org/grpc/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
        /src/vendor/google.golang.org/grpc/server.go:965
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1598

4116759 v0.12.1 /home/joe/.docker/cli-plugins/docker-buildx buildx build --build-arg HTTP_PROXY=http://proxy.contoso.com:8080 --build-arg HTTPS_PROXY=http://proxy.contoso.com:8080 --build-arg http_proxy=http://proxy.contoso.com:8080 --build-arg https_proxy=http://proxy.contoso.com:8080 --build-arg NO_PROXY=169.254.169.254,.s3.eu-west-1.amazonaws.com,10.0.0.0/8,192.168.0.0/16,172.16.0.0/12,127.0.0.0/8,169.254.0.0/16,localhost,contoso.com --build-arg no_proxy=169.254.169.254,.s3.eu-west-1.amazonaws.com,10.0.0.0/8,192.168.0.0/16,172.16.0.0/12,127.0.0.0/8,169.254.0.0/16,localhost,contoso.com --secret id=GIT_AUTH_TOKEN,src=.token https://github.com/something/foo.git#master
google.golang.org/grpc.getChainUnaryInvoker.func1
        google.golang.org/[email protected]/clientconn.go:361
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryClientInterceptor.func1
        go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/[email protected]/interceptor.go:99
github.com/moby/buildkit/client.New.filterInterceptor.func6
        github.com/moby/[email protected]/client/client.go:398
google.golang.org/grpc.DialContext.chainUnaryClientInterceptors.func3
        google.golang.org/[email protected]/clientconn.go:349
google.golang.org/grpc.(*ClientConn).Invoke
        google.golang.org/[email protected]/call.go:35
github.com/moby/buildkit/frontend/gateway/pb.(*lLBBridgeClient).Solve
        github.com/moby/[email protected]/frontend/gateway/pb/gateway.pb.go:2875
github.com/moby/buildkit/client.(*gatewayClientForBuild).Solve
        github.com/moby/[email protected]/client/build.go:89
github.com/moby/buildkit/frontend/gateway/grpcclient.(*grpcClient).Solve
        github.com/moby/[email protected]/frontend/gateway/grpcclient/client.go:411
github.com/docker/buildx/build.BuildWithResultHandler.func3.1.2
        github.com/docker/buildx/build/build.go:914
github.com/docker/buildx/build.NewResultHandle.func3.1
        github.com/docker/buildx/build/result.go:91
github.com/moby/buildkit/frontend/gateway/grpcclient.(*grpcClient).Run
        github.com/moby/[email protected]/frontend/gateway/grpcclient/client.go:214
github.com/moby/buildkit/client.(*Client).Build.func2
        github.com/moby/[email protected]/client/build.go:59
github.com/moby/buildkit/client.(*Client).solve.func3
        github.com/moby/[email protected]/client/solve.go:283
golang.org/x/sync/errgroup.(*Group).Go.func1
        golang.org/x/[email protected]/errgroup/errgroup.go:75
runtime.goexit
        runtime/asm_amd64.s:1650

7 v0.12.5 buildkitd --config /etc/buildkit/buildkitd.toml
github.com/moby/buildkit/frontend/gateway.(*llbBridgeForwarder).Return
        /src/frontend/gateway/gateway.go:923
github.com/moby/buildkit/control/gateway.(*GatewayForwarder).Return
        /src/control/gateway/gateway.go:135
github.com/moby/buildkit/frontend/gateway/pb._LLBBridge_Return_Handler.func1
        /src/frontend/gateway/pb/gateway.pb.go:3216
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1
        /src/vendor/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc/interceptor.go:342
main.unaryInterceptor.func1
        /src/cmd/buildkitd/main.go:603
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
github.com/moby/buildkit/frontend/gateway/pb._LLBBridge_Return_Handler
        /src/frontend/gateway/pb/gateway.pb.go:3218
google.golang.org/grpc.(*Server).processUnaryRPC
        /src/vendor/google.golang.org/grpc/server.go:1336
google.golang.org/grpc.(*Server).handleStream
        /src/vendor/google.golang.org/grpc/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
        /src/vendor/google.golang.org/grpc/server.go:965
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1598

7 v0.12.5 buildkitd --config /etc/buildkit/buildkitd.toml
main.unaryInterceptor.func1
        /src/cmd/buildkitd/main.go:607
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:25
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1
        /src/vendor/github.com/grpc-ecosystem/go-grpc-middleware/chain.go:34
github.com/moby/buildkit/api/services/control._Control_Solve_Handler
        /src/api/services/control/control.pb.go:2440
google.golang.org/grpc.(*Server).processUnaryRPC
        /src/vendor/google.golang.org/grpc/server.go:1336
google.golang.org/grpc.(*Server).handleStream
        /src/vendor/google.golang.org/grpc/server.go:1704
google.golang.org/grpc.(*Server).serveStreams.func1.2
        /src/vendor/google.golang.org/grpc/server.go:965
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1598

time="2024-02-13T12:00:23Z" level=debug msg="session finished: <nil>" spanID=7d4752973653c612 traceID=16a27479102dfac5684516898af85aec

however, running a git clone in the build container via docker exec works fine.

It seems that either:

  1. The proxy config is not respected when fetching a git source over HTTPS; or
  2. I have misconfigured my environment to some degree, despite best efforts to follow existing documentation and reading many existing issues.

Is this a known issue or an unsupported use case? or maybe some nonobvious missing config?
@profnandaa profnandaa mentioned this issue Mar 4, 2024
Open
@TBBle
Copy link
Collaborator

TBBle commented Mar 6, 2024 •
edited
Loading

Hmm. At a quick guess, the BuildKit-internal HTTP code is not seeing the proxy config; it should be pulling it from the environment from a brief skim of the code, and I expect the env.https_proxy driver option to have configured the environment correctly.

It works when cloned locally because the image pull is not done by BuildKit directly, but by a request to containerd, so clearly it is seeing the right proxy config, and the issue lies on the BuildKit side somewhere.

#4725 is a similar issue, as it hits the same failure when trying to talk to auth.docker.io from BuildKit code, but not when underlying containerd pulls an image from a different registry (that doesn't require auth), which aligns with this hypothesis.

Edit: On closer examination, we shell out to git, so it might actually be unrelated to #4725, but just a flat-out bug that we don't pass HTTP proxy info into that shell-out's environment, and didn't in 0.12.5 either.

@13steinj
Copy link

13steinj commented Jun 5, 2024

but just a flat-out bug that we don't pass HTTP proxy info into that shell-out's environment, and didn't in 0.12.5 either.

Preface: I know nothing about docker / moby internals.

Is it valid to pass the proxy information through in the place you have linked?

If so I'll probably try to test it locally and make a PR; this isn't a hard-requirement for my org's use of docker; but it would be nice to gain the git-related benefits across our images (less information to download across multiple versions since only the git deltas/packs need to download).

@TBBle
Copy link
Collaborator

TBBle commented Jun 7, 2024 •
edited
Loading

Note: I don't have the repo locally in front of me, so this is based on browsing in GitHub. So I may have mistaken linkages here.

The source I linked to should be the right place to expose the HTTP proxy information to Git; it should be handled somewhat like the HTTP auth config, I think, e.g.,

buildkit/client/llb/source.go

Lines 290 to 295 in efcde2f

if gi.AuthTokenSecret != "" {
attrs[pb.AttrAuthTokenSecret] = gi.AuthTokenSecret
if gi.addAuthCap {
addCap(&gi.Constraints, pb.CapSourceGitHTTPAuth)
}
}
, provided by a new llb.GitOption passed into llb.Git, something like WithGitHTTPProxy(http_proxy, https_proxy, all_proxy, no_proxy).

For git sources, the call to llb.Git is in DetectGitContext, in this case it's going to be similar to how BUILDKIT_CONTEXT_KEEP_GIT_DIR is handled, i.e. with a bit of luck the settings are naturally in opts and can be easily passed into DetectGitContext. And adjacently, DetectHTTPContext doesn't appear to get any HTTP proxy settings either, so it might make sense to handle this consistently here. (I suspect that if http sources already work, it's because llb.HTTP inherits from the proxy settings in the environment rather than from the build-args.)

For copy/add git sources, we should get the values from the proxyEnv extracted from the build-args, which is currently only provided to the "Run" operation, but for this case will need to be provided to the "Copy" operation. That will take care of Copy, but not the actual use-case here, which is for a Git source for the build. (I haven't checked for copy/add http sources, they might also currently work from execution environment instead of build-args.)

A fallback hack for all of the above would be to make llb.Git work like llb.HTTP appears to, and grab the http_proxy etc. env-vars directly in that function, ignoring the build-args passed to the client, and relying on the buildkit server being configured with the correct proxy environment. (Assuming I'm correct that this is how llb.HTTP works now...)

@dancysoft dancysoft mentioned this issue Sep 12, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
status/needs-investigation
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@TBBle @sigwinch28 @13steinj @thompson-shaun