Skip to content

build refinements 20190617 #44

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
algobolson merged 3 commits into from
Jun 18, 2019
Merged

build refinements 20190617 #44

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2a3eea6
stop making copies to obsolete -telem package names
algobolson Jun 14, 2019
4abf9a0
only upload when s3 destination vars are set
algobolson Jun 17, 2019
cc88d9e
fixed aptly line like jecassis branch
algobolson Jun 17, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 0 additions & 8 deletions scripts/build_packages.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,13 +131,5 @@ for var in "${VARIATION_ARRAY[@]}"; do
cp -p *.deb ${PKG_ROOT}/${GATE_PREFIX}algorand_${CHANNEL}_${PKG_NAME}_${FULLVERSION}.deb
popd
fi

# For now, we only deploy packages with telemetry (so we can transition smoothly).
# Copy xxx_channel_yyy -> xxx_channel-telem_yyy
if [[ ("${var}" = "") && (! -z ${TRANSITION_TELEMETRY_BUILDS}) ]]; then
Copy link
Copy Markdown
Contributor

@winder winder Jun 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have we transitioned? Can we remove this flag entirely?

cp ${PKG_ROOT}/${GATE_PREFIX}node_${CHANNEL}_${PKG_NAME}_${FULLVERSION}.tar.gz ${PKG_ROOT}/${GATE_PREFIX}node_${CHANNEL}-telem_${PKG_NAME}_${FULLVERSION}.tar.gz
cp ${PKG_ROOT}/${GATE_PREFIX}install_${CHANNEL}_${PKG_NAME}_${FULLVERSION}.tar.gz ${PKG_ROOT}/${GATE_PREFIX}install_${CHANNEL}-telem_${PKG_NAME}_${FULLVERSION}.tar.gz
cp ${PKG_ROOT}/${GATE_PREFIX}tools_${CHANNEL}_${PKG_NAME}_${FULLVERSION}.tar.gz ${PKG_ROOT}/${GATE_PREFIX}tools_${CHANNEL}-telem_${PKG_NAME}_${FULLVERSION}.tar.gz
fi
done
done
23 changes: 16 additions & 7 deletions scripts/build_release.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@
# be prompted for GPG key password at a couple points.
#
# Externally settable env vars:
# S3_PREFIX= where to upload build artifacts
# S3_PREFIX= where to upload build artifacts (no trailing /)
# S3_PREFIX_BUILDLOG= where upload build log (no trailing /)
# AWS_EFS_MOUNT= NFS to mount for `aptly` persistent state and scratch storage
# SIGNING_KEY_ADDR= dev@algorand.com or similar for GPG key
# RSTAMP= `scripts/reverse_hex_timestamp`
Expand All @@ -16,10 +17,6 @@ date "+build_release start %Y%m%d_%H%M%S"
set -e
set -x

if [ -z "${S3_PREFIX}" ]; then
S3_PREFIX=s3://algorand-builds
fi

# persistent storage of repo manager scratch space is on EFS
if [ ! -z "${AWS_EFS_MOUNT}" ]; then
if mount|grep -q /data; then
Expand All @@ -37,6 +34,14 @@ fi
export GOPATH=${HOME}/go
export PATH=${HOME}/gpgbin:${GOPATH}/bin:/usr/local/go/bin:${PATH}

# a previous docker centos build can leave junk owned by root. chown and clean
sudo chown -R ${USER} ${GOPATH}
if [ -f ${GOPATH}/src/github.com/algorand/go-algorand/crypto/libsodium-fork/Makefile ]; then
(cd ${GOPATH}/src/github.com/algorand/go-algorand/crypto/libsodium-fork && make distclean)
fi
rm -rf ${GOPATH}/src/github.com/algorand/go-algorand/crypto/lib


cd ${GOPATH}/src/github.com/algorand/go-algorand
export RELEASE_GENESIS_PROCESS=true
export TRANSITION_TELEMETRY_BUILDS=true
Expand Down Expand Up @@ -136,7 +141,9 @@ gpg --detach-sign "${HASHFILE}"
gpg --clearsign "${HASHFILE}"

echo RSTAMP=${RSTAMP} > "${HOME}/rstamp"
aws s3 sync --quiet --exclude dev\* --exclude master\* --exclude nightly\* --exclude stable\* --acl public-read ./ ${S3_PREFIX}/${CHANNEL}/${RSTAMP}_${FULLVERSION}/
if [ ! -z "${S3_PREFIX}" ]; then
aws s3 sync --quiet --exclude dev\* --exclude master\* --exclude nightly\* --exclude stable\* --acl public-read ./ ${S3_PREFIX}/${CHANNEL}/${RSTAMP}_${FULLVERSION}/
fi

# copy .rpm file to intermediate yum repo scratch space, actual publish manually later
if [ ! -d /data/yumrepo ]; then
Expand Down Expand Up @@ -172,7 +179,9 @@ EOF
dpkg -l >>"${STATUSFILE}"
gpg --clearsign "${STATUSFILE}"
gzip "${STATUSFILE}.asc"
aws s3 cp --quiet "${STATUSFILE}.asc.gz" "s3://algorand-devops-misc/buildlog/${RSTAMP}/${STATUSFILE}.asc.gz"
if [ ! -z "${S3_PREFIX_BUILDLOG}" ]; then
aws s3 cp --quiet "${STATUSFILE}.asc.gz" "${S3_PREFIX_BUILDLOG}/${RSTAMP}/${STATUSFILE}.asc.gz"
Copy link
Copy Markdown
Contributor

@Karmastic Karmastic Jun 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any reason this is silent but buildlog copy below (in build_release_local.sh) is not?

Copy link
Copy Markdown
Contributor Author

@algobolson algobolson Jun 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this script is expected to run as a script and be recorded to a log file. aws s3 cp without --quiet prints out status a couple times a second and floods that log file. build_release_local.sh is at this time really more of a copy&paste source of things to do to run the build that get run in the build engineer's terminal.

fi

# use aptly to push .deb to its serving repo
# Leave .deb publishing to manual step after we do more checks on the release artifacts.
Expand Down
20 changes: 17 additions & 3 deletions scripts/build_release_local.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
#!/bin/bash
#
# This is a file of commands to copy and paste to run build_release.sh on an AWS EC2 instance.
# Should work on Ubuntu 16.04 ro 18.04
#
# Externally settable env vars:
# S3_PREFIX_BUILDLOG= where upload build log (no trailing /)

echo "this is a file of commands to copy and paste to run build_release.sh on an AWS EC2 instance"
exit 1
Expand Down Expand Up @@ -40,7 +46,7 @@ umask 0002
# this will require your key password, and export a private key file protected by the same password

# warm up your local gpg-agent
gpg --clearsign
gpg -u dev@algorand.com --clearsign
type some stuff
^D

Expand All @@ -52,6 +58,12 @@ REMOTE_GPG_SOCKET=$(ssh ubuntu@${TARGET} gpgbin/remote_gpg_socket)
LOCAL_GPG_SOCKET=$(gpgconf --list-dir agent-extra-socket)
ssh -A -R "${REMOTE_GPG_SOCKET}:${LOCAL_GPG_SOCKET}" ubuntu@${TARGET}

# check gpg agent connection
gpg -u dev@algorand.com --clearsign
blah blah
^D


# set AWS credentials so we can upload to S3 and connect to EFS
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
Expand All @@ -74,5 +86,7 @@ if [ -z "${RSTAMP}" ]; then
echo "could not figure out RSTAMP, script must have failed early"
exit 1
fi
gzip buildlog_*
aws s3 cp buildlog_*.gz s3://algorand-devops-misc/buildlog/${RSTAMP}/
gzip "buildlog_${BUILDTIMESTAMP}"
if [ ! -z "${S3_PREFIX_BUILDLOG}" ]; then
aws s3 cp "buildlog_${BUILDTIMESTAMP}.gz" "${S3_PREFIX_BUILDLOG}/${RSTAMP}/buildlog_${BUILDTIMESTAMP}.gz"
fi
2 changes: 1 addition & 1 deletion scripts/release_deb.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ SNAPSHOT=algorand-$(date +%Y%m%d_%H%M%S)
aptly snapshot create ${SNAPSHOT} from repo algorand
if [ ! -z "${FIRSTTIME}" ]; then
echo "first publish"
aptly publish snapshot ${SNAPSHOT} "s3:${APTLY_S3_NAME}:"
aptly publish snapshot -origin=Algorand -label=Algorand ${SNAPSHOT} "s3:${APTLY_S3_NAME}:"
else
echo "publish snapshot ${SNAPSHOT}"
aptly publish switch stable "s3:${APTLY_S3_NAME}:" ${SNAPSHOT}
Expand Down