Build --secret with buildkit #1288

tiborvass · 2018-08-14T01:54:11Z

This patch implements docker build --secret id=mysecret,src=/secret/file
for buildkit frontends that request the mysecret secret.

It is currently implemented in the tonistiigi/dockerfile:secrets20180808
frontend via RUN --mount=type=secret,id=mysecret

Signed-off-by: Tibor Vass [email protected]

Also revendors buildkit and docker/docker

Full diff for docker/docker: moby/moby@562df8c...a7ff19d. Relevant changes:
- Bump API version to 1.39 moby/moby#37640 Bump API version to 1.39
- Fix #28814: use emulation for ConEmu and ConsoleZ moby/moby#37600 Fix #28814: use emulation for ConEmu and ConsoleZ
- Do not return "<unknown>" in /info response moby/moby#37472 Do not return "<unknown>" in /info response

tiborvass · 2018-08-14T01:55:31Z

@vdemeester @cpuguy83 :) this is pretty dope. This is just wiring stuff together, the heavy lifting was done in buildkit repo.

vdemeester

LGTM 🐯
Small lint failure to fix though 😉

cli/command/image/build_buildkit.go:1::warning: file is not gofmted with -s (gofmt)
cli/command/image/build_buildkit.go:1::warning: file is not goimported (goimports)

codecov-io · 2018-08-14T17:37:09Z

Codecov Report

Merging #1288 into master will increase coverage by 0.01%.
The diff coverage is 60.46%.

@@            Coverage Diff             @@
##           master    #1288      +/-   ##
==========================================
+ Coverage   54.03%   54.05%   +0.01%     
==========================================
  Files         272      272              
  Lines       18072    18114      +42     
==========================================
+ Hits         9766     9792      +26     
- Misses       7690     7706      +16     
  Partials      616      616

tiborvass · 2018-08-14T17:37:10Z

@vdemeester fixed

cpuguy83

LGTM

Weird that it leaves an empty file in the container image where you mount a secret. Is this an issue with the frontend or with buildkit?

cpuguy83 · 2018-08-14T18:05:24Z

cli/command/image/build_buildkit.go

+	return secretsprovider.NewSecretProvider(store), nil
+}
+
+func parseSecret(value string) (*secretsprovider.FileSource, error) {


Maybe we can add some test for these two new functions?

thaJeztah · 2018-08-15T13:27:13Z

Trying to get this to work, but I'm probably doing it wrong (running against Docker 18.06)

printf "hello secret" > ./mysecret.txt

export DOCKER_BUILDKIT=1

docker build --no-cache --console=false --secret id=mysecret,src=$(pwd)/mysecret.txt -f - . <<EOF
# syntax = tonistiigi/dockerfile:runmount20180618
FROM busybox
RUN echo "hello world"
RUN --mount=type=secret,id=mysecret echo "anything here"
RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobar
EOF

Whatever I try to do with --mount=type=secret.... seems to give me an exit code 2

cpuguy83 · 2018-08-15T14:32:02Z

It doesn't work on 18.06, missing some daemon stuff.

AkihiroSuda · 2018-08-16T11:45:28Z

Is # syntax = still needed?

tiborvass · 2018-08-17T13:59:45Z

@thaJeztah you're using the runmount flavored dockefile frontend, instead of secrets.

Note that --secret is now guarded by API version 1.39.

@cpuguy83

Weird that it leaves an empty file in the container image where you mount a secret. Is this an issue with the frontend or with buildkit?

Agreed it's weird, will debug it but shouldn't be a blocker for this PR. I added a couple of tests.

@AkihiroSuda

Is # syntax = still needed?

Yes, this is not part of the stable compiled-in default frontend.

PTAL :)

This patch implements `docker build --secret id=mysecret,src=/secret/file` for buildkit frontends that request the mysecret secret. It is currently implemented in the tonistiigi/dockerfile:secrets20180808 frontend via RUN --mount=type=secret,id=mysecret Signed-off-by: Tibor Vass <[email protected]>

vendors github.com/docker/docker to a7ff19d69a90dfe152abd146221c8b9b46a0903d Signed-off-by: Tibor Vass <[email protected]>

thaJeztah · 2018-08-17T16:15:41Z

Tried this again, and looks good :)

printf "hello secret" > ./mysecret.txt

export DOCKER_BUILDKIT=1

docker build --no-cache --progress=plain --secret id=mysecret,src=$(pwd)/mysecret.txt -f - . <<EOF
# syntax = tonistiigi/dockerfile:secrets20180808
FROM busybox
RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret
RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobar
EOF

#8 [2/3] RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret
#8       digest: sha256:ab7659590ba7fcf4ec338708fcf824ea76ee7b48b13dce4b9c01faf19395c39d
#8         name: "[2/3] RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret"
#8      started: 2018-08-17 16:08:28.0910424 +0000 UTC
#8 0.303 hello secret#8    completed: 2018-08-17 16:08:28.6508443 +0000 UTC
#8     duration: 559.8019ms
#8 0.303 hello secret

#9 [3/3] RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobar
#9       digest: sha256:67234c5c8ff0bbd7d6c342980f66f9e5ac75405d2860bbd15747e44291b5cb1d
#9         name: "[3/3] RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobar"
#9      started: 2018-08-17 16:08:28.6597585 +0000 UTC
#9 0.343 hello secret#9    completed: 2018-08-17 16:08:29.2602578 +0000 UTC
#9     duration: 600.4993ms
#9 0.343 hello secret

thaJeztah

LGTM

thaJeztah · 2018-08-17T16:19:27Z

/cc @albers for bash completion 😅

tiborvass · 2018-08-17T16:21:33Z

Big thanks to y'all for the review and to Tonis for the implementation!

It is inaccurate to say that build args will not persist in the final image. They are visible with `docker history` and `docker inspect`. As soon as possible, we should upgrade to docker 18.09 to support `--secret` - a way to securely pass credentials into the build context. docker/cli#1288

haizaar · 2018-11-04T02:36:36Z

@thaJeztah
Can you please elaborate on export DOCKER_BUILDKIT=1?
Will we have to define this env var when this feature becomes available as part of docker-ce 18.09?

GordonTheTurtle added the status/0-triage label Aug 14, 2018

vdemeester added status/1-design-review and removed status/0-triage labels Aug 14, 2018

vdemeester requested review from vdemeester, tonistiigi, cpuguy83 and thaJeztah August 14, 2018 07:26

vdemeester approved these changes Aug 14, 2018

View reviewed changes

tiborvass force-pushed the build-secrets branch from e42eb79 to 77827e4 Compare August 14, 2018 17:37

cpuguy83 approved these changes Aug 14, 2018

View reviewed changes

cpuguy83 reviewed Aug 14, 2018

View reviewed changes

tiborvass force-pushed the build-secrets branch from 77827e4 to 56cf398 Compare August 17, 2018 03:47

tiborvass force-pushed the build-secrets branch from 56cf398 to c4c4825 Compare August 17, 2018 14:01

vendor: Bump default API version to 1.39

b4057f0

vendors github.com/docker/docker to a7ff19d69a90dfe152abd146221c8b9b46a0903d Signed-off-by: Tibor Vass <[email protected]>

thaJeztah approved these changes Aug 17, 2018

View reviewed changes

thaJeztah added impact/changelog kind/experimental status/2-code-review area/builder and removed status/1-design-review labels Aug 17, 2018

thaJeztah merged commit cb142fa into docker:master Aug 17, 2018

GordonTheTurtle added this to the 18.09.0 milestone Aug 17, 2018

nphmuller mentioned this pull request Aug 17, 2018

Docker use private NuGet Feed microsoft/azure-pipelines-tasks#6135

Closed

bradrydzewski mentioned this pull request Aug 17, 2018

Support for --secret flag drone-plugins/drone-docker#189

Open

chrishiestand mentioned this pull request Aug 22, 2018

Improve build_args description concourse/docker-image-resource#236

Merged

thaJeztah added the impact/documentation label Aug 31, 2018

gunziptarball mentioned this pull request Dec 17, 2022

Pass extra index url with build secrets apache/airflow#28430

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build --secret with buildkit #1288

Build --secret with buildkit #1288

tiborvass commented Aug 14, 2018 •

edited by thaJeztah

Loading

tiborvass commented Aug 14, 2018

vdemeester left a comment

codecov-io commented Aug 14, 2018 •

edited

Loading

tiborvass commented Aug 14, 2018

cpuguy83 left a comment

cpuguy83 Aug 14, 2018

tiborvass Aug 17, 2018

thaJeztah commented Aug 15, 2018

cpuguy83 commented Aug 15, 2018

AkihiroSuda commented Aug 16, 2018

tiborvass commented Aug 17, 2018 •

edited

Loading

thaJeztah commented Aug 17, 2018

thaJeztah left a comment

thaJeztah commented Aug 17, 2018

tiborvass commented Aug 17, 2018

haizaar commented Nov 4, 2018

Build --secret with buildkit #1288

Build --secret with buildkit #1288

Conversation

tiborvass commented Aug 14, 2018 • edited by thaJeztah Loading

tiborvass commented Aug 14, 2018

vdemeester left a comment

Choose a reason for hiding this comment

codecov-io commented Aug 14, 2018 • edited Loading

Codecov Report

tiborvass commented Aug 14, 2018

cpuguy83 left a comment

Choose a reason for hiding this comment

cpuguy83 Aug 14, 2018

Choose a reason for hiding this comment

tiborvass Aug 17, 2018

Choose a reason for hiding this comment

thaJeztah commented Aug 15, 2018

cpuguy83 commented Aug 15, 2018

AkihiroSuda commented Aug 16, 2018

tiborvass commented Aug 17, 2018 • edited Loading

thaJeztah commented Aug 17, 2018

thaJeztah left a comment

Choose a reason for hiding this comment

thaJeztah commented Aug 17, 2018

tiborvass commented Aug 17, 2018

haizaar commented Nov 4, 2018

tiborvass commented Aug 14, 2018 •

edited by thaJeztah

Loading

codecov-io commented Aug 14, 2018 •

edited

Loading

tiborvass commented Aug 17, 2018 •

edited

Loading