Proposal: Hermetic and Reproducible Docker files #111

jeremyje · 2019-03-15T16:47:46Z

Currently the Open Match Dockerfiles are pulling code from multiple sources.
Let's see backendapi as an example.

# Pulls code from internal and config from 040wip from upstream.
RUN svn export https://github.com/GoogleCloudPlatform/open-match/branches/040wip/internal
RUN svn export https://github.com/GoogleCloudPlatform/open-match/branches/040wip/config
WORKDIR /go/src/github.com/GoogleCloudPlatform/open-match/cmd/backendapi
# Copies the main file from the local repository.
COPY . .
# Pulls dependencies from head.
RUN go get -d -v

The problem with this is the code for Open Match is pulled from multiple places, a bit from the local checkout and most from the upstream branch of Open Match. This build is problematic since it makes development more difficult and the build output is not consistent.

I'd like to propose 2 ways to build docker images.

Build within Docker

Example: https://github.com/GoogleCloudPlatform/open-match/pull/98/files
With this approach all the building happens within a docker image. The current approach in the example would be to do the following:

FROM golang:1.12
ENV GO111MODULE=on

WORKDIR /go/src/github.com/GoogleCloudPlatform/open-match
COPY . .
RUN go mod init github.com/GoogleCloudPlatform/open-match
RUN go mod tidy
RUN go mod edit -require k8s.io/[email protected]
RUN go mod vendor

Baseline image that has the codebase and dependencies.

FROM open-match-base-build as builder

WORKDIR /go/src/github.com/GoogleCloudPlatform/open-match/cmd/backendapi/
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo .

FROM gcr.io/distroless/static  
COPY --from=builder /go/src/github.com/GoogleCloudPlatform/open-match/cmd/backendapi/backendapi .

ENTRYPOINT ["/backendapi"]

Creates a minimal image that just has the output binary. Everything is built within docker.

Advantages

Hermetic building, builds are isolated since the developer box might have customizations that can creep in.
Easier for new engineers to build since a development environment only requires docker to be installed.

Build Locally Copy to Docker Image

An alternative approach is to build outside of docker and simply copy the binary inside.
Example: https://github.com/jeremyje/open-match/blob/buildsys/cmd/backendapi/Dockerfile

# Golang application builder steps
FROM alpine:3.8

ARG APP_ROOT=/go/src/github.com/GoogleCloudPlatform/open-match
ARG APP_NAME=backendapi

RUN apk --update add ca-certificates \
  && adduser -D openmatch \
  && mkdir -p ${APP_ROOT}/config \
  && mkdir -p ${APP_ROOT}/cmd/${APP_NAME} \
  && chown -R openmatch:openmatch /go

COPY --chown=openmatch:root ./cmd/${APP_NAME}/${APP_NAME} ${APP_ROOT}/cmd/${APP_NAME}/${APP_NAME}

USER openmatch
WORKDIR ${APP_ROOT}
ENTRYPOINT ["/go/src/github.com/GoogleCloudPlatform/open-match/cmd/backendapi/backendapi"]

This example does 2 things, create a user which we should do anyways but assumes the binary is built outside already.

Advantages

Simpler Dockerfiles.
Reduces dependency downloads, go mod tidy/download is very expensive. This also significantly speeds up CI builds.
More development friendly, works with build on save workflows.

ihrankouski · 2019-03-18T13:24:37Z

Is it possible to combine both approaches? Similar to what Agones does at the moment (except the go modules), using Makefile.

Have a "base builder" ending with RUN go mod vendor
Have the binaries being built inside the Docker, but written to local FS:

docker run --rm \
    -v /Local-abs-path-to-OM-repo-root/cmd/backendapi/bin:/go/src/github.com/GoogleCloudPlatform/open-match/cmd/backendapi/bin \
    open-match-base-build \
    go build  \
        -o /go/src/github.com/GoogleCloudPlatform/open-match/cmd/backendapi/bin/backendapi \
        github.com/GoogleCloudPlatform/open-match/cmd/backendapi

Have Open-match components' Dockerfiles copying the local binaries to the image.

And then to speed up the building of images during developement we can modify Docker mounts at step 2 adding the local changes.

-v /Local-abs-path-to-OM-repo-root/cmd/backendapi:/go/src/github.com/GoogleCloudPlatform/open-match/cmd/backendapi \
-v /Local-abs-path-to-OM-repo-root/internal:/go/src/github.com/GoogleCloudPlatform/open-match/internal \
-v /Local-abs-path-to-OM-repo-root/vendor:/go/src/github.com/GoogleCloudPlatform/open-match/vendor \

Or just build everything outside of Docker with local Go and proceed directly to step 3.

However I'm only guessing, didn't tested that yet.

jeremyje · 2019-03-18T14:55:31Z

Looking at https://docs.docker.com/v17.09/engine/userguide/eng-image/baseimages/ and https://docs.docker.com/v17.09/engine/userguide/eng-image/multistage-build/ docker is definitely advocating for Build within Docker.

Based on https://groups.google.com/forum/#!searchin/golang-nuts/go$20mod%7Csort:date/golang-nuts/FWHKA3ZGSg4/R-RaONB6BwAJ, they mention using vendor mode for go mod will work.

I think for either approach we are not going to be able to avoid base image because of all the tooling that's involved. I tried to abstract a lot of it via make install-toolchain but there's some system level things that need to be installed, golang and python. Also .net core for linux for certain images complicates things to.

jeremyje · 2019-03-18T15:20:24Z

Is it possible to combine both approaches? Similar to what Agones does at the moment (except the go modules), using Makefile.

Have a "base builder" ending with RUN go mod vendor

Have the binaries being built inside the Docker, but written to local FS:
docker run --rm \
    -v /Local-abs-path-to-OM-repo-root/cmd/backendapi/bin:/go/src/github.com/GoogleCloudPlatform/open-match/cmd/backendapi/bin \
    open-match-base-build \
    go build  \
        -o /go/src/github.com/GoogleCloudPlatform/open-match/cmd/backendapi/bin/backendapi \
        github.com/GoogleCloudPlatform/open-match/cmd/backendapi
Have Open-match components' Dockerfiles copying the local binaries to the image.

And then to speed up the building of images during developement we can modify Docker mounts at step 2 adding the local changes.
-v /Local-abs-path-to-OM-repo-root/cmd/backendapi:/go/src/github.com/GoogleCloudPlatform/open-match/cmd/backendapi \
-v /Local-abs-path-to-OM-repo-root/internal:/go/src/github.com/GoogleCloudPlatform/open-match/internal \
-v /Local-abs-path-to-OM-repo-root/vendor:/go/src/github.com/GoogleCloudPlatform/open-match/vendor \
Or just build everything outside of Docker with local Go and proceed directly to step 3.

However I'm only guessing, didn't tested that yet.

Agones has a bug to move to go modules. googleforgames/agones#625. There's a pull request to add go mod support for us, #106. That's what the state is.

If we want to do build within docker approach I think we'd still need to support local builds as a first class citizen. That way developers can do things like debug unit tests (more coming later, #115).

I worry that build in docker and then use those binaries for local will break that flow. Golang creates artifacts in pkg/ which are probably useful debug symbols that will be lost. For example, I use vscode to debug the unit tests that I've created.

A simple fix to this is to just use the Makefile to build the docker images and local binaries for automation. That way both flows are kept in sync.

make install-toolchain
make all-protos
make all
make test

Lastly, I'll need to follow up on vendor/ ing. My worry is that it'll be killed off and we'll be left stalling on upgrading to a later go version to readdress this problem. vendor/ does solve the download dependency issue very nicely though.

ihrankouski · 2019-03-18T22:25:45Z

If we want to do build within docker approach I think we'd still need to support local builds as a first class citizen.

BTW there’s a switch in Agones Makefile to to do this: export LOCAL_GO=1

jeremyje · 2019-03-18T22:59:28Z

My bias is to prefer what the best practices are and start with that. I don't see other projects doing what agones has done where the build process is done within docker driven by make. It's usually the inverse. To me we should split the concerns where the make assumes a reasonable build environment and the dockerfile provides one.

That way the dockerfile can build open match the same way it's done locally.

jeremyje · 2019-03-19T04:03:38Z

Golang team has this to say about vendor/ing. It's not going away and you can use it with go mod. go mod caching is very painful right now in the CI builds. It's the main source of failures because if any of the dependencies time out the whole build fails and we pull deps twice.

https://github.com/golang/go/wiki/Modules#how-do-i-use-vendoring-with-modules-is-vendoring-going-away

More context:
https://www.reddit.com/r/golang/comments/9qdfp5/using_go_mod_download_to_speed_up_golang_docker/

ihrankouski · 2019-03-19T12:14:23Z

Some notes.

Golang creates artifacts in pkg/ which are probably useful debug symbols that will be lost. For example, I use vscode to debug the unit tests that I've created.

It looks like there's a movement to eliminate $GOPATH/pkg:

Discussion at cmd/go: drop $GOPATH/pkg golang/go#4719
$GOCACHE env var was introduced in 1.10 to store build cache in separate directory (~/.gocache by default)
https://golang.org/doc/go1.11#gocache
Starting from 1.12 it's not possible to disable $GOCACHE any more
https://tip.golang.org/doc/go1.12#gocache
When I do go build with 1.12 nothing appears under pkg/ on my machine...

I'm not sure however how this all works with VS Code go debugger as didn't use it.

In Agones the contents of .gocache is persisted across docker builds: https://github.com/GoogleCloudPlatform/agones/blob/master/build/Makefile#L106

Also looking at https://docs.docker.com/v17.09/engine/userguide/eng-image/baseimages/#create-a-simple-parent-image-using-scratch: they compile the code of 'hello' inside the Docker there but write it out to mounted directory. And then the binary is copied as-is into scratch container.

BTW 'go mod' PR in Agones was merged, and go mod vendor removed 7M LOC under vendor/ there 😮

jeremyje · 2019-03-19T21:23:25Z

The killing pkg/ is interesting I'll definitely look more into that.
The challenge I see is windows and osx native build which doesn't work well if you funnel everything thru docker. I know about GOOS but there's more languages that can be involved like c#.

jeremyje · 2019-03-27T18:46:38Z

I'm going to say this is done with:
#129
#98

jeremyje mentioned this issue Mar 15, 2019

Update the Dockerfiles to build completely from local source and to CI. #98

Merged

jeremyje self-assigned this Mar 27, 2019

jeremyje closed this as completed Mar 27, 2019

Laremere added this to the v0.4.0 milestone Apr 23, 2019

jeremyje mentioned this issue May 29, 2019

open-match-base-build shouldn't download all dependencies every time. #439

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Proposal: Hermetic and Reproducible Docker files #111

Proposal: Hermetic and Reproducible Docker files #111

jeremyje commented Mar 15, 2019 •

edited

Loading

ihrankouski commented Mar 18, 2019 •

edited

Loading

jeremyje commented Mar 18, 2019

jeremyje commented Mar 18, 2019 •

edited

Loading

ihrankouski commented Mar 18, 2019 •

edited

Loading

jeremyje commented Mar 18, 2019

jeremyje commented Mar 19, 2019

ihrankouski commented Mar 19, 2019 •

edited

Loading

jeremyje commented Mar 19, 2019

jeremyje commented Mar 27, 2019

Proposal: Hermetic and Reproducible Docker files #111

Proposal: Hermetic and Reproducible Docker files #111

Comments

jeremyje commented Mar 15, 2019 • edited Loading

Build within Docker

Build Locally Copy to Docker Image

ihrankouski commented Mar 18, 2019 • edited Loading

jeremyje commented Mar 18, 2019

jeremyje commented Mar 18, 2019 • edited Loading

ihrankouski commented Mar 18, 2019 • edited Loading

jeremyje commented Mar 18, 2019

jeremyje commented Mar 19, 2019

ihrankouski commented Mar 19, 2019 • edited Loading

jeremyje commented Mar 19, 2019

jeremyje commented Mar 27, 2019

jeremyje commented Mar 15, 2019 •

edited

Loading

ihrankouski commented Mar 18, 2019 •

edited

Loading

jeremyje commented Mar 18, 2019 •

edited

Loading

ihrankouski commented Mar 18, 2019 •

edited

Loading

ihrankouski commented Mar 19, 2019 •

edited

Loading