Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dockerfree build and ecs-init make integration #3149

Merged
merged 2 commits into from
Mar 13, 2022
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Resolve go modules/vendor conflicts and add dockerfree-all make target
fierlion committed Mar 11, 2022
commit ed39a1d376249657497b81443fd96ee7054a7798
2 changes: 2 additions & 0 deletions .github/workflows/linux.yml
Original file line number Diff line number Diff line change
@@ -28,3 +28,5 @@ jobs:
cd $GITHUB_WORKSPACE/src/github.com/aws/amazon-ecs-agent
make test-silent
make analyze-cover-profile
make test-init
make analyze-cover-profile-init
26 changes: 26 additions & 0 deletions .github/workflows/static.yml
Original file line number Diff line number Diff line change
@@ -29,6 +29,32 @@ jobs:
make get-deps
make static-check
init-check:
Realmonia marked this conversation as resolved.
Show resolved Hide resolved
name: Static Analysis Init
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
with:
path: src/github.com/aws/amazon-ecs-agent
- name: get GO_VERSION
run: |
cd $GITHUB_WORKSPACE/src/github.com/aws/amazon-ecs-agent
echo "GO_VERSION=$(cat GO_VERSION)" >> $GITHUB_ENV
- uses: actions/setup-go@v2
with:
go-version: ${{ env.GO_VERSION }}
- uses: actions/checkout@v2
with:
path: src/github.com/aws/amazon-ecs-agent
- name: run static checks
run: |
export GOPATH=$GITHUB_WORKSPACE
export PATH=$PATH:$(go env GOPATH)/bin
export GO111MODULE=auto
cd $GITHUB_WORKSPACE/src/github.com/aws/amazon-ecs-agent
make get-deps-init
make static-check-init
x-platform-build:
name: Cross platform build
runs-on: ubuntu-latest
109 changes: 104 additions & 5 deletions Makefile
Original file line number Diff line number Diff line change
@@ -39,7 +39,6 @@ all: docker
gobuild:
./scripts/build false


# create output directories
.out-stamp:
mkdir -p ./out/test-artifacts ./out/cni-plugins ./out/amazon-ecs-cni-plugins ./out/amazon-vpc-cni-plugins
@@ -49,6 +48,9 @@ gobuild:
static:
./scripts/build

static-with-pause:
./scripts/build true "" false true

# Cross-platform build target for static checks
xplatform-build:
GOOS=linux GOARCH=arm64 ./scripts/build true "" false
@@ -119,6 +121,10 @@ gogenerate:
go generate -x ./agent/...
$(MAKE) goimports

gogenerate-init:
PATH=$(PATH):$(shell pwd)/scripts go generate -x ./ecs-init/...
$(MAKE) goimports

# 'go' may not be on the $PATH for sudo tests
GO_EXECUTABLE=$(shell command -v go 2> /dev/null)

@@ -140,6 +146,10 @@ test:
${GOTEST} -tags unit -coverprofile cover.out -timeout=60s ./agent/...
go tool cover -func cover.out > coverprofile.out

test-init:
go test -count=1 -short -v -coverprofile cover.out ./ecs-init/...
go tool cover -func cover.out > coverprofile-init.out

test-silent:
$(eval VERBOSE=)
${GOTEST} -tags unit -coverprofile cover.out -timeout=60s ./agent/...
@@ -149,6 +159,10 @@ test-silent:
analyze-cover-profile: coverprofile.out
./scripts/analyze-cover-profile

.PHONY: analyze-cover-profile-init
analyze-cover-profile-init: coverprofile-init.out
./scripts/analyze-cover-profile-init

run-integ-tests: test-registry gremlin container-health-check-image run-sudo-tests
ECS_LOGLEVEL=debug ${GOTEST} -tags integration -timeout=30m ./agent/...

@@ -158,7 +172,6 @@ run-sudo-tests:
benchmark-test:
go test -run=XX -bench=. ./agent/...


.PHONY: build-image-for-ecr upload-images replicate-images

build-image-for-ecr: netkitten volumes-test image-cleanup-test-images fluentd exec-command-agent-test
@@ -223,6 +236,21 @@ cni-plugins: get-cni-sources .out-stamp build-ecs-cni-plugins build-vpc-cni-plug
mv $(PWD)/out/amazon-vpc-cni-plugins/* $(PWD)/out/cni-plugins
@echo "Built all cni plugins successfully."

# dockerfree build process will build the agent container image from scratch
# and with minimal dependencies
# requires glibc-static

dockerfree-pause:
GOOS=linux GOARCH=amd64 ./scripts/build-pause
Copy link
Contributor

@lydiafilipe lydiafilipe Mar 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you clarify why you are setting GOOS/GOARCH here? What if this is being built on arm?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there's still more work to build arm -- I'll update the description to say this is focused on amd linux.


dockerfree-certs:
GOOS=linux GOARCH=amd64 ./scripts/get-host-certs

dockerfree-cni-plugins: get-cni-sources
GOOS=linux GOARCH=amd64 ./scripts/build-cni-plugins

dockerfree-agent-image: dockerfree-pause dockerfree-certs dockerfree-cni-plugins static-with-pause
GOOS=linux GOARCH=amd64 ./scripts/build-agent-image

.PHONY: codebuild
codebuild: .out-stamp
@@ -259,7 +287,6 @@ image-cleanup-test-images:
container-health-check-image:
$(MAKE) -C misc/container-health $(MFLAGS)


# all .go files in the agent, excluding vendor/, model/ and testutils/ directories, and all *_test.go and *_mocks.go files
GOFILES:=$(shell go list -f '{{$$p := .}}{{range $$f := .GoFiles}}{{$$p.Dir}}/{{$$f}} {{end}}' ./agent/... \
| grep -v /testutils/ | grep -v _test\.go$ | grep -v _mocks\.go$ | grep -v /model)
@@ -286,12 +313,23 @@ gogenerate-check: gogenerate
# check that gogenerate does not generate a diff.
git diff --exit-code

.PHONY: gogenerate-check-init
gogenerate-check-init: gogenerate-init
# check that gogenerate does not generate a diff.
git diff --exit-code

.PHONY: static-check
static-check: gocyclo govet importcheck gogenerate-check
# use default checks of staticcheck tool, except style checks (-ST*) and depracation checks (-SA1019)
# depracation checks have been left out for now; removing their warnings requires error handling for newer suggested APIs, changes in function signatures and their usages.
# https://github.com/dominikh/go-tools/tree/master/cmd/staticcheck
staticcheck -tests=false -checks "inherit,-ST*,-SA1019,-SA9002" ./agent/...
staticcheck -tests=false -checks "inherit,-ST*,-SA1019,-SA9002,-SA4006" ./agent/...

.PHONY: static-check-init
static-check-init: gocyclo govet importcheck gogenerate-check-init
# use default checks of staticcheck tool, except style checks (-ST*)
# https://github.com/dominikh/go-tools/tree/master/cmd/staticcheck
staticcheck -tests=false -checks "inherit,-ST*" ./ecs-init/...

.PHONY: goimports
goimports:
@@ -309,13 +347,53 @@ GOPATH=$(shell go env GOPATH)

get-deps: .get-deps-stamp

get-deps-init:
go get golang.org/x/tools/cover
go get golang.org/x/tools/cmd/cover
go get github.com/golang/mock/mockgen
cd "${GOPATH}/src/github.com/golang/mock/mockgen" && git checkout 1.3.1 && go get ./... && go install ./... && cd -
GO111MODULE=on go get github.com/fzipp/gocyclo/cmd/gocyclo@v0.3.1
go get golang.org/x/tools/cmd/goimports
go get honnef.co/go/tools/cmd/staticcheck

.generic-rpm-done:
./scripts/update-version.sh
cp packaging/generic-rpm/amazon-ecs-init.spec amazon-ecs-init.spec
cp packaging/generic-rpm/ecs.service ecs.service
cp packaging/generic-rpm/amazon-ecs-volume-plugin.service amazon-ecs-volume-plugin.service
cp packaging/generic-rpm/amazon-ecs-volume-plugin.socket amazon-ecs-volume-plugin.socket
tar -czf ./sources.tgz ecs-init scripts
test -e SOURCES || ln -s . SOURCES
rpmbuild --define "%_topdir $(PWD)" -bb amazon-ecs-init.spec
find RPMS/ -type f -exec cp {} . \;
touch .rpm-done

generic-rpm: .generic-rpm-done

dockerfree-all: dockerfree-agent-image generic-rpm

.deb-done: BUILDROOT/ecs-agent.tar
./scripts/update-version.sh
tar -czf ./amazon-ecs-init_${VERSION}.orig.tar.gz ecs-init scripts README.md
cp -r packaging/generic-deb/debian ecs-init scripts README.md BUILDROOT
cd BUILDROOT && debuild -uc -us --lintian-opts --suppress-tags bad-distribution-in-changes-file,file-in-unusual-dir
touch .deb-done

deb: .deb-done

clean:
# ensure docker is running and we can talk to it, abort if not:
docker ps > /dev/null
-docker rmi $(BUILDER_IMAGE) "amazon/amazon-ecs-agent-cleanbuild:make"
-docker rmi $(BUILDER_IMAGE) "amazon/amazon-ecs-agent-cleanbuild-windows:make"
rm -f misc/certs/ca-certificates.crt &> /dev/null
rm -f misc/certs/ca-bundle.crt &> /dev/null
rm -rf misc/pause-container/image/
rm -rf misc/pause-container/rootfs/
rm -rf misc/plugins/
rm -f misc/pause-container/amazon-ecs-pause.tar
rm -rf out/
rm -rf rootfs/
-$(MAKE) -C $(ECS_CNI_REPOSITORY_SRC_DIR) clean
-$(MAKE) -C misc/netkitten $(MFLAGS) clean
-$(MAKE) -C misc/volumes-test $(MFLAGS) clean
@@ -329,4 +407,25 @@ clean:
-rm -rf $(PWD)/bin
-rm -rf cover.out
-rm -rf coverprofile.out

-rm -rf coverprofile-init.out
# ecs-init & rpm cleanup
-rm -f ecs-init.spec
-rm -f amazon-ecs-init.spec
-rm -f ecs.conf
-rm -f ecs.service
-rm -f amazon-ecs-volume-plugin.conf
-rm -f amazon-ecs-volume-plugin.service
-rm -f amazon-ecs-volume-plugin.socket
-rm -rf ./bin
-rm -f ./sources.tgz
-rm -f ./amazon-ecs-init
-rm -f ./amazon-ecs-init-*.rpm
-rm -f ./ecs-agent-*.tar
-rm -f ./ecs-init-*.src.rpm
-rm -rf ./ecs-init-*
-rm -rf ./BUILDROOT BUILD RPMS SRPMS SOURCES SPECS
-rm -rf ./x86_64
-rm -f ./amazon-ecs-init_${VERSION}*
-rm -f .srpm-done .rpm-done .generic-rpm-done
-rm -f .deb-done
-rm -f amazon-ecs-volume-plugin
2 changes: 2 additions & 0 deletions agent-container/agent-config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"author":"Amazon Web Services, Inc.","config":{"Cmd":["/agent"],"ArgsEscaped":true},"created":"2014-12-12T01:12:53.332832423Z","history":[{"created":"2014-12-12T01:12:53.332832423Z","author":"Amazon Web Services, Inc.","created_by":"[] + [] === \"\"","empty_layer":true}],"os":"linux","rootfs":{"type":"layers","diff_ids":["sha256:~~digest~~"]}}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are these static files? shouldn't they be generated at build-time?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The diff_ids field is being generated at build-time. Otherwise I agree maybe we can update the history.created with an actual timestamp.
These are generated by Docker otherwise -- if we don't have Docker, we need to create/provide these config files manually.

Bottlerocket uses static configs to build its pause container: https://github.com/bottlerocket-os/bottlerocket/blob/develop/packages/ecs-agent/pause-config.json

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So do we have to ensure these files get rebuilt/committed each time then from current agent changes? Might need some sort of check to reinforce that

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated to string replace and use the date util to match expected formatting: sed -i "s/~~timestamp~~/$(date +"%FT%T.%NZ")/g" image/config.json
Otherwise we'll use https://github.com/aws/amazon-ecs-agent/blob/master/VERSION to fill in our version.


1 change: 1 addition & 0 deletions agent-container/agent-image-VERSION
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
1.0
2 changes: 2 additions & 0 deletions agent-container/agent-manifest.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
[{"Config":"config.json","RepoTags":["amazon/amazon-ecs-agent:1.60.0"],"Layers":["rootfs/layer.tar"]}]
fierlion marked this conversation as resolved.
Show resolved Hide resolved

1 change: 1 addition & 0 deletions agent-container/agent-repositories
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"amazon/amazon-ecs-agent":{"amazon-ecs":"rootfs"}}
15 changes: 15 additions & 0 deletions dependencies_mocks.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
// Copyright 2015-2022 Amazon.com, Inc. or its affiliates. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License"). You may
// not use this file except in compliance with the License. A copy of the
// License is located at
//
// http://aws.amazon.com/apache2.0/
//
// or in the "license" file accompanying this file. This file is distributed
// on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
// express or implied. See the License for the specific language governing
// permissions and limitations under the License.
//
// Source: dependencies.go in package cache

1 change: 1 addition & 0 deletions ecs-init/cache/cache_test.go
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
// +build test
// Copyright 2014-2015 Amazon.com, Inc. or its affiliates. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License"). You may
2 changes: 1 addition & 1 deletion ecs-init/cache/dependencies.go
Original file line number Diff line number Diff line change
@@ -18,7 +18,7 @@ package cache
// package-level functions. These interfaces are then used to create mocks
// for the unit tests.

//go:generate mockgen.sh $GOPACKAGE $GOFILE
//go:generate mockgen.sh cache $GOFILE

import (
"io"
2 changes: 1 addition & 1 deletion ecs-init/cache/dependencies_mocks.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions ecs-init/config/config_al2.go
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
//go:build al2
// +build al2

// Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
1 change: 1 addition & 0 deletions ecs-init/config/config_generic_rpm.go
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
//go:build generic_rpm
// +build generic_rpm

// Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
1 change: 1 addition & 0 deletions ecs-init/config/config_suse_ubuntu_debian.go
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
//go:build suse || ubuntu || debian
// +build suse ubuntu debian

// Copyright 2017-2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
1 change: 1 addition & 0 deletions ecs-init/config/config_unspecified.go
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
//go:build !suse && !ubuntu && !al2 && !debian && !generic_rpm
// +build !suse,!ubuntu,!al2,!debian,!generic_rpm

// Copyright 2017-2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
1 change: 1 addition & 0 deletions ecs-init/config/development.go
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
//go:build development
// +build development

// Copyright 2015 Amazon.com, Inc. or its affiliates. All Rights Reserved.
2 changes: 1 addition & 1 deletion ecs-init/config/logger.go
Original file line number Diff line number Diff line change
@@ -34,7 +34,7 @@ and limitations under the License.
<seelog type="asyncloop">
<outputs formatid="main">
<console formatid="console" />
<rollingfile filename="` + initLogFile() + `" type="date"
<rollingfile filename="`+initLogFile()+`" type="date"
datepattern="2006-01-02-15" archivetype="zip" maxrolls="5" />
</outputs>
<formats>
1 change: 1 addition & 0 deletions ecs-init/config/release.go
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
//go:build !development
// +build !development

// Copyright 2015 Amazon.com, Inc. or its affiliates. All Rights Reserved.
2 changes: 1 addition & 1 deletion ecs-init/docker/backoff_mocks.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion ecs-init/docker/dependencies_mocks.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions ecs-init/docker/dependencies_test.go
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
// +build test
// Copyright 2015-2017 Amazon.com, Inc. or its affiliates. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License"). You may
1 change: 1 addition & 0 deletions ecs-init/docker/docker_test.go
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
// +build test
// Copyright 2015-2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License"). You may
Binary file added ecs-init/ecs-init
Binary file not shown.
2 changes: 1 addition & 1 deletion ecs-init/engine/dependencies_mocks.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions ecs-init/engine/engine_test.go
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
// +build test
// Copyright 2015 Amazon.com, Inc. or its affiliates. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License"). You may
2 changes: 1 addition & 1 deletion ecs-init/exec/iptables/cmd_mocks.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion ecs-init/exec/iptables/exec_mocks.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion ecs-init/exec/sysctl/cmd_mocks.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion ecs-init/exec/sysctl/exec_mocks.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion ecs-init/gpu/nvidia_gpu_manager_mocks.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions ecs-init/version/version.go
Original file line number Diff line number Diff line change
@@ -18,10 +18,10 @@
package version

// Version is the version of the ecs-init
var Version string = "unknown"
var Version string = "1.60.0"
fierlion marked this conversation as resolved.
Show resolved Hide resolved

// GitDirty indicates the cleanliness of the git repo when this ecs-init was built
var GitDirty string = "true"

// GitShortHash is the short hash of this ecs-init build
var GitShortHash string = "unknown"
var GitShortHash string = "11173cad"
2 changes: 2 additions & 0 deletions misc/pause-container/pause-config.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"author":"Amazon Web Services, Inc.","config":{"Cmd":["/pause"],"ArgsEscaped":true},"created":"2014-12-12T01:12:53.332832423Z","history":[{"created":"2014-12-12T01:12:53.332832423Z","author":"Amazon Web Services, Inc.","created_by":"[] + [] === \"\"","empty_layer":true}],"os":"linux","rootfs":{"type":"layers","diff_ids":["sha256:~~digest~~"]}}

1 change: 1 addition & 0 deletions misc/pause-container/pause-image-VERSION
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
1.0
2 changes: 2 additions & 0 deletions misc/pause-container/pause-manifest.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
[{"Config":"config.json","RepoTags":["amazon/amazon-ecs-pause:0.1.0"],"Layers":["rootfs/layer.tar"]}]

1 change: 1 addition & 0 deletions misc/pause-container/pause-repositories
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"amazon/amazon-ecs-pause":{"amazon-ecs":"rootfs"}}
1 change: 0 additions & 1 deletion packaging/generic-rpm/amazon-ecs-init.spec
Original file line number Diff line number Diff line change
@@ -42,7 +42,6 @@ Source4: https://s3.amazonaws.com/amazon-ecs-agent/ecs-agent-arm64-v%{bun
Source5: amazon-ecs-volume-plugin.service
Source6: amazon-ecs-volume-plugin.socket

BuildRequires: golang >= 1.7
Realmonia marked this conversation as resolved.
Show resolved Hide resolved
BuildRequires: systemd
Requires: systemd
Requires: iptables
27 changes: 27 additions & 0 deletions scripts/analyze-cover-profile-init
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
#!/usr/bin/env python

MINIMUM_TEST_COVERAGE = 73.7

# coverprofile.out (see Makefile for how this is generated) will look something like this:
#
# github.com/aws/amazon-ecs-agent/agent/wsclient/error.go:66: Retry 0.0%
# github.com/aws/amazon-ecs-agent/agent/wsclient/errors.go:23: Error 0.0%
# github.com/aws/amazon-ecs-agent/agent/wsclient/errors.go:28: Retry 0.0%
# github.com/aws/amazon-ecs-agent/agent/wsclient/errors.go:41: Retry 0.0%
# github.com/aws/amazon-ecs-agent/agent/wsclient/errors.go:46: Error 0.0%
# github.com/aws/amazon-ecs-agent/agent/wsclient/errors.go:59: Error 0.0%
# github.com/aws/amazon-ecs-agent/agent/wsclient/types.go:40: BuildTypeDecoder 100.0%
# github.com/aws/amazon-ecs-agent/agent/wsclient/types.go:49: NewOfType 0.0%
# github.com/aws/amazon-ecs-agent/agent/wsclient/types.go:57: GetRecognizedTypes 100.0%
# total: (statements) 73.8%

with open("coverprofile-init.out") as f:
for line in f:
if line.startswith("total:"):
splitline = line.split()
print("Total unit test coverage: " + splitline[2])
coverage = float(splitline[2].rstrip("%"))
if coverage < MINIMUM_TEST_COVERAGE:
raise BaseException(
"Unit test coverage ({0}%) is below the minimum ({1}%)".
format(coverage, MINIMUM_TEST_COVERAGE))
9 changes: 9 additions & 0 deletions scripts/build
Original file line number Diff line number Diff line change
@@ -25,6 +25,11 @@ set -ex
static=${1:-true}
output_directory=${2:-}
version_gen=${3:-true}
with_pause=${4:-false}

PAUSE_CONTAINER_IMAGE="amazon/amazon-ecs-pause"
PAUSE_CONTAINER_TAG="0.1.0"
PAUSE_CONTAINER_TARBALL="amazon-ecs-pause.tar"

# Normalize to working directory being build root (up one level from ./scripts)
ROOT=$( cd "$( dirname "${BASH_SOURCE[0]}" )/.." && pwd )
@@ -48,6 +53,10 @@ if [[ "${version_gen}" == "true" ]]; then
GO111MODULE=off go run gen/version-gen.go
fi

if [[ "${with_pause}" == "true" ]]; then
LDFLAGS="-X github.com/aws/amazon-ecs-agent/agent/config.DefaultPauseContainerTag=$PAUSE_CONTAINER_TAG -X github.com/aws/amazon-ecs-agent/agent/config.DefaultPauseContainerImageName=$PAUSE_CONTAINER_IMAGE"
fi

if [ "${TARGET_OS}" == "windows" ]; then
unset static
build_exe="out/amazon-ecs-agent.exe"
54 changes: 54 additions & 0 deletions scripts/build-agent-image
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
#!/bin/bash
# Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You may
# not use this file except in compliance with the License. A copy of the
# License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is distributed
# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied. See the License for the specific language governing
# permissions and limitations under the License.

# This script builds a copy of the agent.
# It exists to wrap go build and properly make a static binary, as well as to
# correctly setup versioning before creating the binary

set -ex

ROOT=$( cd "$( dirname "${BASH_SOURCE[0]}" )/.." && pwd )
cd "${ROOT}"
AGENT_VERSION=$(cat VERSION)

# add cni-plugins
mkdir -p rootfs/amazon-ecs-cni-plugins/
cp ./misc/plugins/aws-appmesh rootfs/amazon-ecs-cni-plugins/aws-appmesh
cp ./misc/plugins/ecs-bridge rootfs/amazon-ecs-cni-plugins/ecs-bridge
cp ./misc/plugins/ecs-eni rootfs/amazon-ecs-cni-plugins/ecs-eni
cp ./misc/plugins/ecs-ipam rootfs/amazon-ecs-cni-plugins/ecs-ipam
cp ./misc/plugins/vpc-branch-eni rootfs/amazon-ecs-cni-plugins/vpc-branch-eni

# add certs
mkdir -p rootfs/etc/ssl/certs/
cp ./misc/certs/ca-bundle.crt rootfs/etc/ssl/certs/ca-certificates.crt
fierlion marked this conversation as resolved.
Show resolved Hide resolved

# add pause container
mkdir -p rootfs/images/
cp ./misc/pause-container/amazon-ecs-pause.tar rootfs/images/amazon-ecs-pause.tar

# add agent
cp ./out/amazon-ecs-agent rootfs/agent

# build container
mkdir -p image/rootfs
tar --mtime="@1492525740" --owner=0 --group=0 --numeric-owner -cf image/rootfs/layer.tar -C rootfs .
DIGEST=$(sha256sum image/rootfs/layer.tar | sed -e 's/ .*//')
install -m 0644 ./agent-container/agent-image-VERSION image/rootfs/VERSION
install -m 0644 ./agent-container/agent-config.json image/config.json
sed -i "s/~~digest~~/${DIGEST}/" image/config.json
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it possible for us to use jq inside of the koji build? if it is we could maybe turn these into jq templates? That way the json is always guaranteed to be valid. It looks something like this:

template.jq


{
    "descriptionContentType": "text/plain",
    "title": $title,
    "description": "$description"
}

then pass the parameters via jq like:

jq -n --arg description "My Description" --arg title "My Title" -f template.jq

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't want to add any dependencies unless it's absolutely necessary. I updated to use sed to replace the tokens.

install -m 0644 ./agent-container/agent-manifest.json image/manifest.json
install -m 0644 ./agent-container/agent-repositories image/repositories
tar --mtime="@1492525740" --owner=0 --group=0 --numeric-owner -cf ./ecs-agent-v${AGENT_VERSION}.tar -C image .
rm -rf image/
32 changes: 32 additions & 0 deletions scripts/build-cni-plugins
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
#!/bin/bash
# Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You may
# not use this file except in compliance with the License. A copy of the
# License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is distributed
# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied. See the License for the specific language governing
# permissions and limitations under the License.

# This script builds a copy of the agent.
fierlion marked this conversation as resolved.
Show resolved Hide resolved
# It exists to wrap go build and properly make a static binary, as well as to
# correctly setup versioning before creating the binary

set -ex

# this script assumes we've run the get-cni-sources make target to update the cni submodules
ROOT=$( cd "$( dirname "${BASH_SOURCE[0]}" )/.." && pwd )
cd "${ROOT}"
# copy submodules to expected go build path github.com/aws/amazon-ecs-cni-plugins
cp -r amazon-ecs-cni-plugins ../amazon-ecs-cni-plugins
cp -r amazon-vpc-cni-plugins ../amazon-vpc-cni-plugins
cd ../amazon-ecs-cni-plugins && GO111MODULE=auto make plugins
mkdir -p ../amazon-ecs-agent/misc/plugins && cp -a ./bin/plugins/. ../amazon-ecs-agent/misc/plugins/
make clean
cd ../amazon-vpc-cni-plugins && GO111MODULE=auto make build
cp -a ./build/linux_amd64/. ../amazon-ecs-agent/misc/plugins/
make clean
33 changes: 33 additions & 0 deletions scripts/build-pause
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
#!/bin/bash
# Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You may
# not use this file except in compliance with the License. A copy of the
# License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is distributed
# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied. See the License for the specific language governing
# permissions and limitations under the License.

# This script builds a copy of the agent.
# It exists to wrap go build and properly make a static binary, as well as to
# correctly setup versioning before creating the binary

set -ex

ROOT=$( cd "$( dirname "${BASH_SOURCE[0]}" )/.." && pwd )
cd "${ROOT}/misc/pause-container"
mkdir -p rootfs/
gcc -static pause.c -o rootfs/pause
mkdir -p image/rootfs
tar --mtime="@1492525740" --owner=0 --group=0 --numeric-owner -cf image/rootfs/layer.tar -C rootfs .
DIGEST=$(sha256sum image/rootfs/layer.tar | sed -e 's/ .*//')
install -m 0644 pause-image-VERSION image/rootfs/VERSION
install -m 0644 pause-config.json image/config.json
sed -i "s/~~digest~~/${DIGEST}/" image/config.json
install -m 0644 pause-manifest.json image/manifest.json
install -m 0644 pause-repositories image/repositories
tar --mtime="@1492525740" --owner=0 --group=0 --numeric-owner -cf ./amazon-ecs-pause.tar -C image .
41 changes: 41 additions & 0 deletions scripts/get-host-certs
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
#!/bin/bash
# Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You may
# not use this file except in compliance with the License. A copy of the
# License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is distributed
# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied. See the License for the specific language governing
# permissions and limitations under the License.

# This script builds a copy of the agent.
fierlion marked this conversation as resolved.
Show resolved Hide resolved
# It exists to wrap go build and properly make a static binary, as well as to
# correctly setup versioning before creating the binary

set -ex

ROOT=$( cd "$( dirname "${BASH_SOURCE[0]}" )/.." && pwd )
cd "${ROOT}/misc/certs"

# Possible certificate files; stop after finding one
certFiles=("/etc/ssl/certs/ca-certificates.crt" "/etc/pki/tls/certs/ca-bundle.crt" "/etc/ssl/ca-bundle.pem" "/etc/pki/tls/cacert.pem" "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem" "/etc/ssl/cert.pem")

thisCertFile=""
for file in ${certFiles[@]}; do
if test -f $file; then
thisCertFile=$file
break
fi
done

# if we found a cert file, we'll copy else return 0
if [ ! -z "$thisCertFile" ]
then
cp $thisCertFile .
else
exit 1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe echo some sort of error message to help debug in case this happens

fi