Skip to content

Build tests installers; reorg #95

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
algobolson merged 13 commits into from
Jul 1, 2019
Merged

Build tests installers; reorg #95

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
b1029e9
build tweaks
algobolson Jun 20, 2019
1d057c3
Merge remote-tracking branch 'origin/master' into build_tweaks_20190720
algobolson Jun 20, 2019
90df9a3
more gpg -u __
algobolson Jun 24, 2019
349e57d
test installers inside docker environments
algobolson Jun 24, 2019
dc4ced9
fixes
algobolson Jun 25, 2019
2212ce7
fixes
algobolson Jun 25, 2019
c665a35
fixes
algobolson Jun 25, 2019
eb21bf5
fix centos rpm install test
algobolson Jun 25, 2019
f515008
docker cleanup. yum install
algobolson Jun 25, 2019
7764909
more explicit paths
algobolson Jun 25, 2019
14767d7
Merge remote-tracking branch 'origin/master' into build_tweaks_20190720
algobolson Jun 25, 2019
b99505d
comment, remove real endpoint from dummy aptly
algobolson Jun 25, 2019
d150408
some PR feedback
algobolson Jun 26, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
179 changes: 89 additions & 90 deletions scripts/build_release.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,23 +17,6 @@ date "+build_release start %Y%m%d_%H%M%S"
set -e
set -x

# persistent storage of repo manager scratch space is on EFS
if [ ! -z "${AWS_EFS_MOUNT}" ]; then
if mount|grep -q /data; then
echo /data already mounted
else
sudo mkdir -p /data
sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport "${AWS_EFS_MOUNT}":/ /data
# make environment for release_deb.sh
sudo mkdir -p /data/_aptly
sudo chown -R ${USER} /data/_aptly
export APTLY_DIR=/data/_aptly
fi
fi

export GOPATH=${HOME}/go
export PATH=${HOME}/gpgbin:${GOPATH}/bin:/usr/local/go/bin:${PATH}

# a previous docker centos build can leave junk owned by root. chown and clean
sudo chown -R ${USER} ${GOPATH}
if [ -f ${GOPATH}/src/github.com/algorand/go-algorand/crypto/libsodium-fork/Makefile ]; then
Expand All @@ -58,6 +41,17 @@ export VARIATIONS="base"
export NO_BUILD=true
if [ -z "${RSTAMP}" ]; then
RSTAMP=$(scripts/reverse_hex_timestamp)
echo RSTAMP=${RSTAMP} > "${HOME}/rstamp"
fi
# What's my default IP address?
# get the datacenter IP address for this EC2 host.
# this might equivalently be gotten from `netstat -rn` and `ifconfig -a`
if [ -z "${DC_IP}" ]; then
DC_IP=$(curl --silent http://169.254.169.254/latest/meta-data/local-ipv4)
fi
if [ -z "${DC_IP}" ]; then
echo "ERROR: need DC_IP to be set to your local (but not localhost) IP"
exit 1
fi

# Update version file for this build
Expand All @@ -71,7 +65,6 @@ fi
echo ${BUILD_NUMBER} > ./buildnumber.dat
git add -A
git commit -m "Build ${BUILD_NUMBER}"
git push
Comment thread
Karmastic marked this conversation as resolved.
export FULLVERSION=$(./scripts/compute_build_number.sh -f)

# a bash user might `source build_env` to manually continue a broken build
Expand All @@ -89,6 +82,7 @@ export VARIATIONS=${VARIATIONS}
RSTAMP=${RSTAMP}
BUILD_NUMBER=${BUILD_NUMBER}
export FULLVERSION=${FULLVERSION}
DC_IP=${DC_IP}
EOF
# strip leading 'export ' for docker --env-file
sed 's/^export //g' < ${HOME}/build_env > ${HOME}/build_env_docker
Expand All @@ -102,91 +96,96 @@ make build

scripts/build_packages.sh "${PLATFORM}"

# Run RPM bulid in Centos7 Docker container
sg docker "docker build -t algocentosbuild - < scripts/centos-build.Dockerfile"

# cleanup our libsodium build
if [ -f ${GOPATH}/src/github.com/algorand/go-algorand/crypto/libsodium-fork/Makefile ]; then
(cd ${GOPATH}/src/github.com/algorand/go-algorand/crypto/libsodium-fork && make distclean)
fi
rm -rf ${GOPATH}/src/github.com/algorand/go-algorand/crypto/lib

# do the RPM build
sg docker "docker run --env-file ${HOME}/build_env_docker --mount type=bind,src=${GOPATH}/src,dst=/root/go/src --mount type=bind,src=${HOME},dst=/root/subhome --mount type=bind,src=/usr/local/go,dst=/usr/local/go -a stdout -a stderr algocentosbuild /root/go/src/github.com/algorand/go-algorand/scripts/build_release_centos_docker.sh"
# Test .deb installer

# Tag Source
mkdir -p ${HOME}/docker_test_resources
Copy link
Copy Markdown
Contributor

@Karmastic Karmastic Jun 26, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't want these tests to be part of the build - we need to do more comprehensive tests of the deb and rpm; the build should just generate the binaries / packages and not also do tests.
Please extract the testing process to separate script(s) that can be run manually against the pushed but not promoted packages.

if [ ! -f "${HOME}/docker_test_resources/gnupg2.2.9_centos7_amd64.tar.bz2" ]; then
aws s3 cp s3://algorand-devops-misc/tools/gnupg2.2.9_centos7_amd64.tar.bz2 ${HOME}/docker_test_resources
fi
cp -p "${HOME}/key.gpg" "${HOME}/docker_test_resources/key.pub"

TAG=${BRANCH}-${FULLVERSION}
if [ ! -z "${SIGNING_KEY_ADDR}" ]; then
git tag -s -u "${SIGNING_KEY_ADDR}" ${TAG} -m "Genesis Timestamp: $(cat ./genesistimestamp.dat)"
# copy previous installers into ~/docker_test_resources
cd "${HOME}/docker_test_resources"
if [ "${TEST_UPGRADE}" == "no" ]; then
echo "upgrade test disabled"
else
git tag -s ${TAG} -m "Genesis Timestamp: $(cat ./genesistimestamp.dat)"
fi
git push origin ${TAG}

git archive --prefix=algorand-${FULLVERSION}/ "${TAG}" | gzip > ${PKG_ROOT}/algorand_${CHANNEL}_source_${FULLVERSION}.tar.gz

# create *.sig gpg signatures
cd ${PKG_ROOT}
for i in *.tar.gz *.deb *.rpm; do
gpg --detach-sign "${i}"
done
HASHFILE=hashes_${CHANNEL}_${OS}_${ARCH}_${FULLVERSION}
rm -f "${HASHFILE}"
touch "${HASHFILE}"
md5sum *.tar.gz *.deb *.rpm >> "${HASHFILE}"
shasum -a 256 *.tar.gz *.deb *.rpm >> "${HASHFILE}"
shasum -a 512 *.tar.gz *.deb *.rpm >> "${HASHFILE}"
gpg --detach-sign "${HASHFILE}"
gpg --clearsign "${HASHFILE}"

echo RSTAMP=${RSTAMP} > "${HOME}/rstamp"
if [ ! -z "${S3_PREFIX}" ]; then
aws s3 sync --quiet --exclude dev\* --exclude master\* --exclude nightly\* --exclude stable\* --acl public-read ./ ${S3_PREFIX}/${CHANNEL}/${RSTAMP}_${FULLVERSION}/
python3 ${GOPATH}/src/github.com/algorand/go-algorand/scripts/get_current_installers.py "${S3_PREFIX}/${CHANNEL}"
fi

# copy .rpm file to intermediate yum repo scratch space, actual publish manually later
if [ ! -d /data/yumrepo ]; then
sudo mkdir -p /data/yumrepo
sudo chown ${USER} /data/yumrepo
fi
cp -p -n *.rpm *.rpm.sig /data/yumrepo
echo "TEST_UPGRADE=${TEST_UPGRADE}" >> ${HOME}/build_env_docker

rm -rf ${HOME}/dummyaptly
mkdir -p ${HOME}/dummyaptly
cat <<EOF>${HOME}/dummyaptly.conf
{
"rootDir": "${HOME}/dummyaptly",
"downloadConcurrency": 4,
"downloadSpeedLimit": 0,
"architectures": [],
"dependencyFollowSuggests": false,
"dependencyFollowRecommends": false,
"dependencyFollowAllVariants": false,
"dependencyFollowSource": false,
"dependencyVerboseResolve": false,
"gpgDisableSign": false,
"gpgDisableVerify": false,
"gpgProvider": "gpg",
"downloadSourcePackages": false,
"skipLegacyPool": true,
"ppaDistributorID": "ubuntu",
"ppaCodename": "",
"skipContentsPublishing": false,
"FileSystemPublishEndpoints": {},
"S3PublishEndpoints": {},
"SwiftPublishEndpoints": {}
}
EOF
aptly -config=${HOME}/dummyaptly.conf repo create -distribution=stable -component=main algodummy
aptly -config=${HOME}/dummyaptly.conf repo add algodummy ${HOME}/node_pkg/*.deb
SNAPSHOT=algodummy-$(date +%Y%m%d_%H%M%S)
aptly -config=${HOME}/dummyaptly.conf snapshot create ${SNAPSHOT} from repo algodummy
aptly -config=${HOME}/dummyaptly.conf publish snapshot -origin=Algorand -label=Algorand ${SNAPSHOT}

cd ${HOME}
STATUSFILE=build_status_${CHANNEL}_${FULLVERSION}
echo "ami-id:" > "${STATUSFILE}"
curl --silent http://169.254.169.254/latest/meta-data/ami-id >> "${STATUSFILE}"
cat <<EOF>>"${STATUSFILE}"
(cd ${HOME}/dummyaptly/public && python3 ${GOPATH}/src/github.com/algorand/go-algorand/scripts/httpd.py --pid ${HOME}/phttpd.pid) &


go version:
EOF
go version >>"${STATUSFILE}"
cat <<EOF>>"${STATUSFILE}"
sg docker "docker run --rm --env-file ${HOME}/build_env_docker --mount type=bind,src=${HOME}/docker_test_resources,dst=/stuff --mount type=bind,src=${GOPATH}/src,dst=/root/go/src --mount type=bind,src=/usr/local/go,dst=/usr/local/go ubuntu:16.04 bash /root/go/src/github.com/algorand/go-algorand/scripts/build_release_ubuntu_test_docker.sh"
sg docker "docker run --rm --env-file ${HOME}/build_env_docker --mount type=bind,src=${HOME}/docker_test_resources,dst=/stuff --mount type=bind,src=${GOPATH}/src,dst=/root/go/src --mount type=bind,src=/usr/local/go,dst=/usr/local/go ubuntu:18.04 bash /root/go/src/github.com/algorand/go-algorand/scripts/build_release_ubuntu_test_docker.sh"

go env:
EOF
go env >>"${STATUSFILE}"
cat <<EOF>>"${STATUSFILE}"
kill $(cat ${HOME}/phttpd.pid)

build_env:
EOF
cat <${HOME}/build_env>>"${STATUSFILE}"
cat <<EOF>>"${STATUSFILE}"
date "+build_release done building ubuntu %Y%m%d_%H%M%S"

dpkg-l:
EOF
dpkg -l >>"${STATUSFILE}"
gpg --clearsign "${STATUSFILE}"
gzip "${STATUSFILE}.asc"
if [ ! -z "${S3_PREFIX_BUILDLOG}" ]; then
aws s3 cp --quiet "${STATUSFILE}.asc.gz" "${S3_PREFIX_BUILDLOG}/${RSTAMP}/${STATUSFILE}.asc.gz"
# Run RPM bulid in Centos7 Docker container
sg docker "docker build -t algocentosbuild - < ${GOPATH}/src/github.com/algorand/go-algorand/scripts/centos-build.Dockerfile"

# cleanup our libsodium build
if [ -f ${GOPATH}/src/github.com/algorand/go-algorand/crypto/libsodium-fork/Makefile ]; then
(cd ${GOPATH}/src/github.com/algorand/go-algorand/crypto/libsodium-fork && make distclean)
fi
rm -rf ${GOPATH}/src/github.com/algorand/go-algorand/crypto/lib

# do the RPM build, sign and validate it

sudo rm -rf ${HOME}/dummyrepo
mkdir -p ${HOME}/dummyrepo

cat <<EOF>${HOME}/dummyrepo/algodummy.repo
[algodummy]
name=Algorand
baseurl=http://${DC_IP}:8111/
enabled=1
gpgcheck=1
gpgkey=https://releases.algorand.com/rpm/rpm_algorand.pub
EOF
(cd ${HOME}/dummyrepo && python3 ${GOPATH}/src/github.com/algorand/go-algorand/scripts/httpd.py --pid ${HOME}/phttpd.pid) &

sg docker "docker run --rm --env-file ${HOME}/build_env_docker --mount type=bind,src=${HOME}/.gnupg/S.gpg-agent,dst=/S.gpg-agent --mount type=bind,src=${HOME}/dummyrepo,dst=/dummyrepo --mount type=bind,src=${HOME}/docker_test_resources,dst=/stuff --mount type=bind,src=${GOPATH}/src,dst=/root/go/src --mount type=bind,src=${HOME},dst=/root/subhome --mount type=bind,src=/usr/local/go,dst=/usr/local/go algocentosbuild /root/go/src/github.com/algorand/go-algorand/scripts/build_release_centos_docker.sh"

kill $(cat ${HOME}/phttpd.pid)

# use aptly to push .deb to its serving repo
# Leave .deb publishing to manual step after we do more checks on the release artifacts.
# ${GOPATH}/src/github.com/algorand/go-algorand/scripts/release_deb.sh ${PKG_ROOT}/*deb
date "+build_release done building centos %Y%m%d_%H%M%S"

# TODO: manually post rpm to repo
# NEXT: build_release_sign.sh

date "+build_release finish %Y%m%d_%H%M%S"
77 changes: 77 additions & 0 deletions scripts/build_release_centos_docker.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,3 +42,80 @@ RPMTMP=$(mktemp -d 2>/dev/null || mktemp -d -t "rpmtmp")
trap "rm -rf ${RPMTMP}" 0
scripts/build_rpm.sh ${RPMTMP}
cp -p ${RPMTMP}/*/*.rpm /root/subhome/node_pkg

(cd ${HOME} && tar jxf /stuff/gnupg*.tar.bz2)
export PATH="${HOME}/gnupg2/bin:${PATH}"
export LD_LIBRARY_PATH=${HOME}/gnupg2/lib

umask 0077
mkdir -p ~/.gnupg
umask 0022

touch "${HOME}/.gnupg/gpg.conf"
if grep -q no-autostart "${HOME}/.gnupg/gpg.conf"; then
echo ""
else
echo "no-autostart" >> "${HOME}/.gnupg/gpg.conf"
fi
rm -f ${HOME}/.gnupg/S.gpg-agent
(cd ~/.gnupg && ln -s /S.gpg-agent S.gpg-agent)

gpg --import /stuff/key.pub
gpg --import ${GOPATH}/src/github.com/algorand/go-algorand/installer/rpm/RPM-GPG-KEY-Algorand

cat <<EOF>"${HOME}/.rpmmacros"
%_gpg_name Algorand RPM <rpm@algorand.com>
%__gpg ${HOME}/gnupg2/bin/gpg
%__gpg_check_password_cmd true
EOF

cat <<EOF>"${HOME}/rpmsign.py"
import rpm
import sys
rpm.addSign(sys.argv[1], '')
EOF

NEWEST_RPM=$(ls -t /root/subhome/node_pkg/*rpm|head -1)
python2 "${HOME}/rpmsign.py" "${NEWEST_RPM}"

cp -p "${NEWEST_RPM}" /dummyrepo
createrepo --database /dummyrepo
rm -f /dummyrepo/repodata/repomd.xml.asc
gpg -u rpm@algorand.com --detach-sign --armor /dummyrepo/repodata/repomd.xml

OLDRPM=$(ls -t /stuff/*.rpm|head -1)
Copy link
Copy Markdown
Contributor

@Karmastic Karmastic Jun 26, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Likewise, please extract the tests so they can be run independently and as part of a larger suite of verification tests against packages.

if [ -f "${OLDRPM}" ]; then
yum install -y "${OLDRPM}"
algod -v
if algod -v | grep -q ${FULLVERSION}; then
echo "already installed current version. wat?"
false
fi

mkdir -p /root/testnode
cp -p /var/lib/algorand/genesis/testnet/genesis.json /root/testnode

goal node start -d /root/testnode
goal node wait -d /root/testnode -w 60
goal node stop -d /root/testnode
fi


yum-config-manager --add-repo http://${DC_IP}:8111/algodummy.repo

yum install -y algorand
algod -v
# check that the installed version is now the current version
algod -v | grep -q ${FULLVERSION}

if [ ! -d /root/testnode ]; then
mkdir -p /root/testnode
cp -p /var/lib/algorand/genesis/testnet/genesis.json /root/testnode
fi

goal node start -d /root/testnode
goal node wait -d /root/testnode -w 60
goal node stop -d /root/testnode


echo CENTOS_DOCKER_TEST_OK
10 changes: 7 additions & 3 deletions scripts/build_release_local.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,8 @@ gpg -u dev@algorand.com --clearsign
type some stuff
^D

gpg -u rpm@algorand.com --clearsign


# TODO: use simpler expression when we can rely on gpg 2.2 on ubuntu >= 18.04
#REMOTE_GPG_SOCKET=$(ssh ubuntu@${TARGET} gpgconf --list-dir agent-socket)
Expand All @@ -75,7 +77,9 @@ export AWS_EFS_MOUNT=
# to be prompted for GPG key password at a couple points.
# It can still steal the outer terminal from within piping the output to tee. Nifty, huh?
BUILDTIMESTAMP=$(cat "${HOME}/buildtimestamp")
(bash "${HOME}/go/src/github.com/algorand/go-algorand/scripts/build_release.sh" 2>&1)|tee -a "buildlog_${BUILDTIMESTAMP}"
(bash "${HOME}/go/src/github.com/algorand/go-algorand/scripts/build_release.sh" 2>&1)|tee -a "${HOME}/buildlog_${BUILDTIMESTAMP}"
(bash "${HOME}/go/src/github.com/algorand/go-algorand/scripts/build_release_sign.sh" 2>&1)|tee -a "${HOME}/buildlog_${BUILDTIMESTAMP}"
(bash "${HOME}/go/src/github.com/algorand/go-algorand/scripts/build_release_upload.sh" 2>&1)|tee -a "${HOME}/buildlog_${BUILDTIMESTAMP}"
if [ -f "${HOME}/rstamp" ]; then
. "${HOME}/rstamp"
fi
Expand All @@ -86,7 +90,7 @@ if [ -z "${RSTAMP}" ]; then
echo "could not figure out RSTAMP, script must have failed early"
exit 1
fi
gzip "buildlog_${BUILDTIMESTAMP}"
gzip "${HOME}/buildlog_${BUILDTIMESTAMP}"
if [ ! -z "${S3_PREFIX_BUILDLOG}" ]; then
aws s3 cp "buildlog_${BUILDTIMESTAMP}.gz" "${S3_PREFIX_BUILDLOG}/${RSTAMP}/buildlog_${BUILDTIMESTAMP}.gz"
aws s3 cp "${HOME}/buildlog_${BUILDTIMESTAMP}.gz" "${S3_PREFIX_BUILDLOG}/${RSTAMP}/buildlog_${BUILDTIMESTAMP}.gz"
fi
8 changes: 6 additions & 2 deletions scripts/build_release_setup.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ chmod +x ${HOME}/gpgbin/remote_gpg_socket
if [ "${DISTRIB_ID}" = "Ubuntu" ]; then
if [ "${DISTRIB_RELEASE}" = "16.04" ]; then
echo "WARNING: Ubuntu 16.04 is DEPRECATED"
sudo apt-get install -y autoconf awscli docker.io g++ fakeroot git gnupg2 gpgv2 make nfs-common python3 rpm sqlite3
sudo apt-get install -y autoconf awscli docker.io g++ fakeroot git gnupg2 gpgv2 make nfs-common python3 rpm sqlite3 python3-boto3
cat <<EOF>${HOME}/gpgbin/gpg
#!/bin/bash
exec /usr/bin/gpg2 "\$@"
Expand All @@ -58,7 +58,7 @@ exec /usr/bin/gpgv2 "\$@"
EOF
chmod +x ${HOME}/gpgbin/*
elif [ "${DISTRIB_RELEASE}" = "18.04" ]; then
sudo apt-get install -y autoconf awscli docker.io git gpg nfs-common python3 rpm sqlite3
sudo apt-get install -y autoconf awscli docker.io git gpg nfs-common python3 rpm sqlite3 python3-boto3
else
echo "don't know how to build on Ubuntu ${DISTRIB_RELEASE}"
exit 1
Expand Down Expand Up @@ -103,6 +103,8 @@ fi

sudo usermod -a -G docker ubuntu
sg docker "docker pull centos:7"
sg docker "docker pull ubuntu:18.04"
sg docker "docker pull ubuntu:16.04"

# Check out
mkdir -p ${GOPATH}/src/github.com/algorand
Expand All @@ -112,6 +114,8 @@ fi
cd ${GOPATH}/src/github.com/algorand/go-algorand
git checkout "${GIT_CHECKOUT_LABEL}"

gpg --import ${GOPATH}/src/github.com/algorand/go-algorand/installer/rpm/RPM-GPG-KEY-Algorand

# Install latest Go
cd $HOME
# TODO: make a config file in root of repo with single source of truth for Go major-minor version
Expand Down
Loading