Skip to content

Trap errors and remove ec2 instance #854

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tsachiherman merged 34 commits into from
Feb 27, 2020
Merged

Trap errors and remove ec2 instance #854

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
e403aaa
Added "prod" pipeline to `release/`
Feb 5, 2020
047ee2e
Added snapshot.sh to be invoked manually
Feb 6, 2020
5cfa9ab
Restructuring
Feb 6, 2020
08ee95e
test stage
Feb 7, 2020
856eb15
add common build params
Feb 7, 2020
53577f1
Merge branch 'master' into deploy_to_prod_ben-branch
Feb 8, 2020
2cc681f
fix build
Feb 11, 2020
5315272
test
Feb 12, 2020
e1229fc
Remove build parameters
Feb 14, 2020
4a6a929
wip
Feb 14, 2020
97a5006
remove test dir
Feb 14, 2020
bf99d0a
still trying to fix random build errors
Feb 14, 2020
230c5f9
updating test phase
Feb 14, 2020
60a67cd
Merge branch 'master' into deploy_to_prod_ben-branch_polling
Feb 14, 2020
2840963
extract build_env values
Feb 14, 2020
e3cc634
add trigger for test phase
Feb 15, 2020
ccfb096
test
Feb 15, 2020
f4415b6
fix build
Feb 17, 2020
85c95ec
improve compute_branch_channel.sh
Feb 17, 2020
e0ce7a3
fix build
Feb 17, 2020
2296157
Improving dockerfile
Feb 20, 2020
4aa0da1
move centos{-build,}.Dockerfile
Feb 22, 2020
1d13349
try to fix apt-get fail (again)
Feb 22, 2020
ff2852f
Add new util/check_remote.sh
Feb 22, 2020
c2d3ac2
test
Feb 22, 2020
f72f9a7
try to fix seek() 3 args error
Feb 22, 2020
708ba1b
Merge branch 'master' into improve_dockerfile2
Feb 25, 2020
8e572f4
Trap any non-zero exit codes and cleanup ec2 instance
Feb 25, 2020
16ba15b
Merge branch 'improve_dockerfile2' into trap_errors
Feb 26, 2020
7130093
Remove temporary location
Feb 26, 2020
9b489d7
Merge branch 'master' into trap_errors
Feb 26, 2020
47388c3
Implement reviewer suggestions
Feb 26, 2020
3ed9316
Merge branch 'master' into trap_errors
Feb 27, 2020
5ad0517
Merge branch 'master' into trap_errors
Feb 27, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions scripts/release/build/stage/build/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

set -ex

trap 'bash ./scripts/release/common/ec2/shutdown.sh' ERR

# Path(s) are relative to the root of the Jenkins workspace.
ssh -i ReleaseBuildInstanceKey.pem -A ubuntu@"$(cat scripts/release/common/ec2/tmp/instance)" bash go/src/github.com/algorand/go-algorand/scripts/release/build/stage/build/task.sh

2 changes: 1 addition & 1 deletion scripts/release/build/stage/build/task.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ fi
# Run RPM build in Centos7 Docker container
sg docker "docker build -t algocentosbuild - < $HOME/go/src/github.com/algorand/go-algorand/scripts/release/common/docker/centos.Dockerfile"

sg docker "docker run --rm --mount type=bind,src=${HOME},dst=/root/subhome algocentosbuild /root/subhome/go/src/github.com/algorand/go-algorand/scripts/release/build/rpm/build.sh"
sg docker "docker run --rm --env-file ${HOME}/build_env_docker --mount type=bind,src=${HOME},dst=/root/subhome algocentosbuild /root/subhome/go/src/github.com/algorand/go-algorand/scripts/release/build/rpm/build.sh"

echo
date "+build_release end BUILD stage %Y%m%d_%H%M%S"
Expand Down
2 changes: 2 additions & 0 deletions scripts/release/build/stage/package/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

set -ex

trap 'bash ./scripts/release/common/ec2/shutdown.sh' ERR

# Path(s) are relative to the root of the Jenkins workspace.
ssh -i ReleaseBuildInstanceKey.pem -A ubuntu@"$(cat scripts/release/common/ec2/tmp/instance)" bash go/src/github.com/algorand/go-algorand/scripts/release/build/stage/package/task.sh

2 changes: 2 additions & 0 deletions scripts/release/build/stage/setup/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@

set -ex

trap 'bash ./scripts/release/common/ec2/shutdown.sh' ERR

# Path(s) are relative to the root of the Jenkins workspace.
BRANCH=$(./scripts/release/util/check_remote.sh "$1")
INSTANCE=$(cat ./scripts/release/common/ec2/tmp/instance)
Expand Down
2 changes: 2 additions & 0 deletions scripts/release/build/stage/sign/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

set -ex

trap 'bash ./scripts/release/common/ec2/shutdown.sh' ERR

# Path(s) are relative to the root of the Jenkins workspace.
ssh -i ReleaseBuildInstanceKey.pem -A ubuntu@"$(cat scripts/release/common/ec2/tmp/instance)" bash go/src/github.com/algorand/go-algorand/scripts/release/build/stage/sign/task.sh

4 changes: 2 additions & 2 deletions scripts/release/build/stage/upload/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

set -ex

trap 'bash ./scripts/release/common/ec2/shutdown.sh' ERR

# Path(s) are relative to the root of the Jenkins workspace.

#BUCKET_LOCATION="$2"
Expand All @@ -15,8 +17,6 @@ rm -rf pkg && mkdir -p pkg/"$FULLVERSION"
ssh -i ReleaseBuildInstanceKey.pem -A ubuntu@"$INSTANCE" bash go/src/github.com/algorand/go-algorand/scripts/release/build/stage/upload/task.sh
scp -i ReleaseBuildInstanceKey.pem -o StrictHostKeyChecking=no -r ubuntu@"$INSTANCE":~/node_pkg/* pkg/"$FULLVERSION"/

# Create the buildlog file.
scp -i ReleaseBuildInstanceKey.pem -o StrictHostKeyChecking=no ubuntu@"$INSTANCE":~/build_status_"$CHANNEL"_*.asc.gz pkg/"$FULLVERSION"
#aws s3 sync --exclude dev* --exclude master* --exclude nightly* --exclude stable* --acl public-read pkg/"$FULLVERSION" s3://"$BUCKET_LOCATION"/"$CHANNEL"/"$FULLVERSION"/
aws s3 sync --exclude dev* --exclude master* --exclude nightly* --exclude stable* --acl public-read pkg/"$FULLVERSION" s3://ben-test-2.0.3/"$CHANNEL"/"$FULLVERSION"/

8 changes: 1 addition & 7 deletions scripts/release/common/ec2/shutdown.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,12 @@
#!/usr/bin/env bash
# shellcheck disable=2164

AWS_REGION="$1"
AWS_REGION="${1:-us-west-1}"
GREEN_FG=$(echo -en "\e[32m")
YELLOW_FG=$(echo -en "\e[33m")
END_FG_COLOR=$(echo -en "\e[39m")
REPO_ROOT="$( cd "$(dirname "$0")" ; pwd -P )"

if [ "$AWS_REGION" = "" ]
then
echo "Missing AWS_REGION argument"
exit 1
fi

pushd "$REPO_ROOT"/tmp > /dev/null
SGID=$(cat sgid)
INSTANCE_ID=$(cat instance-id)
Expand Down
14 changes: 11 additions & 3 deletions scripts/release/common/setup.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,17 +7,25 @@ if [ -z "${BUILDTIMESTAMP}" ]; then
BUILDTIMESTAMP=$(cat "${HOME}/buildtimestamp")
export BUILDTIMESTAMP
echo run "${0}" with output to "${HOME}/buildlog_${BUILDTIMESTAMP}"
(bash "${0}" "${1}" 2>&1) | tee "${HOME}/buildlog_${BUILDTIMESTAMP}"
exit 0
bash "${0}" "${1}" 2>&1 | tee "${HOME}/buildlog_${BUILDTIMESTAMP}"
# http://tldp.org/LDP/abs/html/internalvariables.html#PIPESTATUSREF
exit "${PIPESTATUS[0]}"
Copy link
Copy Markdown
Author

@btoll btoll Feb 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will get the exit code of the first command in the pipeline, which is what we want.

Copy link
Copy Markdown
Contributor

@tsachiherman tsachiherman Feb 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice, but that's not what you want. You want to use set -o pipefail which would set the error code you get from the pipe to the first failuire.

Copy link
Copy Markdown
Author

@btoll btoll Feb 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, will change and test.

Copy link
Copy Markdown
Author

@btoll btoll Feb 26, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I remember now why I used PIPESTATUS instead. I agree that in most cases setting pipefail is the better choice, but we do need a return value here or the script will execute one more time than we need.

if [ -z "${BUILDTIMESTAMP}" ]; then
    date "+%Y%m%d_%H%M%S" > "${HOME}/buildtimestamp"
    BUILDTIMESTAMP=$(cat "${HOME}/buildtimestamp")
    export BUILDTIMESTAMP
    echo run "${0}" with output to "${HOME}/buildlog_${BUILDTIMESTAMP}"
    bash "${0}" "${1}" 2>&1 | tee "${HOME}/buildlog_${BUILDTIMESTAMP}"
    # http://tldp.org/LDP/abs/html/internalvariables.html#PIPESTATUSREF
    exit "${PIPESTATUS[0]}"
fi
  1. When the script first runs, BUILDTIMESTAMP isn't set so we set it and then start capturing the logs.
  2. We call the script again: bash "${0}" "${1}" 2>&1
  3. Since BUILDTIMESTAMP is set skip this portion and parse the file.
  4. Then, we need the return value and exit the script.

Copy link
Copy Markdown
Contributor

@tsachiherman tsachiherman Feb 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, setting it to pipefail would set the error flag once the bash "${0}" "${1}" 2>&1 | tee "${HOME}/buildlog_${BUILDTIMESTAMP}" command is complete. Then, given that you already have a set -ex, it will exit right away.

Copy link
Copy Markdown
Contributor

@tsachiherman tsachiherman Feb 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note that pipefail would not propagate to the child bash invocation since SHELLOPTS is not exported.

Copy link
Copy Markdown
Author

@btoll btoll Feb 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but the happy path is not to have an error. So, if we don't explicitly return here, the script will be parsed again, which is not what we want.

fi

echo
date "+build_release begin SETUP stage %Y%m%d_%H%M%S"
echo

# `apt-get` fails randomly when downloading package, this is a hack that "works" reasonably well.
echo -e "deb http://us.archive.ubuntu.com/ubuntu/ bionic main universe multiverse\ndeb http://archive.ubuntu.com/ubuntu/ bionic main universe multiverse" | sudo tee /etc/apt/sources.list.d/ubuntu

Copy link
Copy Markdown
Author

@btoll btoll Feb 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a hack to ensure that apt-get will download all the packages will need.

sudo apt-get update
sudo apt-get upgrade -y
sudo apt-get install -y build-essential automake autoconf awscli docker.io git gpg nfs-common python3 rpm sqlite3 python3-boto3 g++ libtool rng-tools

# `apt-get` fails randomly when downloading package, this is a hack that "works" reasonably well.
sudo apt-get update

Copy link
Copy Markdown
Author

@btoll btoll Feb 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like the above comment, this is a needed hack so we don't get stuck. I really think we should leave these in for now.

Copy link
Copy Markdown
Contributor

@tsachiherman tsachiherman Feb 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm quite skeptical that the upgrade -y is needed. ( i.e. if you need specific upgrades for specific components, you should explicitly ask for that.

Copy link
Copy Markdown
Author

@btoll btoll Feb 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I'll try removing it.

sudo apt-get install -y build-essential automake autoconf awscli docker.io git gpg nfs-common python python3 rpm sqlite3 python3-boto3 g++ libtool rng-tools
sudo rngd -r /dev/urandom

#umask 0077
Expand Down
4 changes: 4 additions & 0 deletions scripts/release/prod/stage/setup/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
#!/usr/bin/env bash
# shellcheck disable=2029

set -ex

trap 'bash ./scripts/release/common/ec2/shutdown.sh' ERR

# Path(s) are relative to the root of the Jenkins workspace.
INSTANCE=$(cat scripts/release/common/ec2/tmp/instance)
#BUCKET="$1"
Expand Down
4 changes: 4 additions & 0 deletions scripts/release/prod/stage/sync/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
#!/usr/bin/env bash
# shellcheck disable=2029

set -ex

trap 'bash ./scripts/release/common/ec2/shutdown.sh' ERR

# Path(s) are relative to the root of the Jenkins workspace.
STAGING="$1"
PROD="$2"
Expand Down
2 changes: 0 additions & 2 deletions scripts/release/test/deb/run_ubuntu.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
#!/usr/bin/env bash
#
# This script exists to give a trap atexit context for killing the httpd so that we're not waiting on that

set -ex

Expand Down
2 changes: 2 additions & 0 deletions scripts/release/test/stage/setup/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@

set -ex

trap 'bash ./scripts/release/common/ec2/shutdown.sh' ERR

# Path(s) are relative to the root of the Jenkins workspace.
INSTANCE=$(cat scripts/release/common/ec2/tmp/instance)
BRANCH="$1"
Expand Down
2 changes: 2 additions & 0 deletions scripts/release/test/stage/test/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

set -ex

trap 'bash ./scripts/release/common/ec2/shutdown.sh' ERR

# Path(s) are relative to the root of the Jenkins workspace.

INSTANCE=$(cat scripts/release/common/ec2/tmp/instance)
Expand Down