Skip to content
Merged
Changes from all commits
Commits
Show all changes
139 commits
Select commit Hold shift + click to select a range
3d9321c
shellchecked `build_deb.sh`
Mar 3, 2020
be9248c
Test pre-packaged executable on variety of linux platforms (#651)
tsachiherman Dec 19, 2019
e3dcb96
Reduce e2e_go_tests execution time twice (#645)
algorandskiy Dec 20, 2019
5e81a11
Promote test_release.sh so that it won't conflict with release testin…
tsachiherman Dec 20, 2019
081054b
Fix concurrent access to wallet handles cache in goal (#654)
algorandskiy Dec 20, 2019
39793d6
Allow upgrades to specify the delay before their execution. (#650)
derbear Dec 20, 2019
cd170f3
Set explicit 30 sec timeout for AlgorandGoal::RawSend in expect test …
algorandskiy Dec 20, 2019
f513ba3
Support variable-delay protocol upgrades in ConsensusFuture. (#659)
derbear Dec 21, 2019
b00444e
Shant/catchup stop on unapproved (#660)
algonautshant Dec 24, 2019
aa5ddea
Compile linux/amd64 binaries with static linking (#625)
tsachiherman Dec 24, 2019
c752c3a
Add logging for the telemetry server connections (#661)
tsachiherman Dec 26, 2019
73e2852
add bool support to algocfg (#667)
algobolson Dec 27, 2019
8dd075f
Reduce execution time of expect tests (#665)
algorandskiy Dec 28, 2019
46e88c6
Avoid upgrading boost on travis Mac builds (#669)
tsachiherman Dec 30, 2019
de82005
Abort algod startup if logging.config file has bad permissions (#662)
algorandskiy Dec 30, 2019
9003e63
Add error logging for getting a cached wallet handle (#663)
algorandskiy Dec 30, 2019
fb779d9
Update license date 2019 -> 2020 (#674)
tsachiherman Jan 2, 2020
54f1e88
Update copyright to use date range. (#676)
tsachiherman Jan 2, 2020
2e1d806
Tee existing tests so we can review output before piping it forward. …
tsachiherman Jan 2, 2020
72f82c5
Make gracefull exit of a node that is waiting for WaitForBlock call (…
tsachiherman Jan 3, 2020
e730482
Remove tput where not supported by terminal (#682)
tsachiherman Jan 3, 2020
aeeba0f
Avoid waiting for block that won't be reached due to unsupported prot…
tsachiherman Jan 4, 2020
349d819
Fix - Indexer now shows received transactions (#684)
algorotem Jan 4, 2020
a651681
Undo teeing to dev/tty as it doesn't work well in terminal free envir…
tsachiherman Jan 6, 2020
ef84eac
Improve lockFile error handling (#687)
tsachiherman Jan 6, 2020
8b83774
Catchup service stop on unsupported and e2e test (#685)
algonautshant Jan 6, 2020
2faa68d
Make ARM64 build mandatory. (#694)
tsachiherman Jan 7, 2020
7edd64d
Updates to the goal node status (#686)
algonautshant Jan 8, 2020
67ec670
telemetry recorded locally as info log (#666)
algobolson Jan 8, 2020
6176ee2
Relax StartNetwork regex (#696)
tsachiherman Jan 8, 2020
19d58e1
Two fixes to basicCatchup_test: cloned node not stopped and env var …
algonautshant Jan 8, 2020
0f8e64e
Make scripts executable. (#702)
winder Jan 9, 2020
3aec0b2
More reliable fetcher unit tests. (#708)
tsachiherman Jan 9, 2020
245f63a
Avoid starting the Telemetry service when logging is disabled (#703)
algobolson Jan 9, 2020
d7a0864
Shutdown kmd when test fixture is going down. (#709)
tsachiherman Jan 10, 2020
0b4f9ff
Fix unit test. (#711)
tsachiherman Jan 10, 2020
6c2801d
Execute e2e tests one at a time on arm64 (#701)
tsachiherman Jan 13, 2020
4b8b686
Disable pprof endpoints by default (#693)
justicz Jan 13, 2020
f265464
Update the makefile to skip the static linking when compiling on cent…
tsachiherman Jan 15, 2020
0435e97
Fail e2e-go tests when node panics (#699)
tsachiherman Jan 15, 2020
ddd97e3
Build release job (#698)
Jan 15, 2020
815de9f
Use service-wide logger instead of logging.Base() in agreement (#714)
algorandskiy Jan 15, 2020
a0b2d24
Mark some of the slow e2e tests as such (#719)
tsachiherman Jan 16, 2020
ec5e373
Wait test less restrictive. (#718)
tsachiherman Jan 16, 2020
7e68060
Move slow test to get executed on nightly builds (#721)
tsachiherman Jan 17, 2020
f54ec74
Faster upgrade tests. (#722)
tsachiherman Jan 18, 2020
23ef9a4
Disable failing test. (#724)
tsachiherman Jan 20, 2020
f2729f7
Generate docs for algokey.
winder Jan 21, 2020
d86ad61
s/goal/algokey
winder Jan 21, 2020
eddeb88
Improve algons error logging (#733)
tsachiherman Jan 22, 2020
bb40775
ledger/eval refactor (#700)
algobolson Jan 22, 2020
c526927
Fix a bug in Credential.lowestOutput caused by improper domain separa…
algoradam Jan 22, 2020
fd576a2
Goal docs tweaks (#731)
EvanJRichard Jan 22, 2020
29e2ce8
test all `goal ... -h` (#730)
algobolson Jan 22, 2020
956fcab
Move EnsureDigest logic into the catchup service (#726)
tsachiherman Jan 23, 2020
d0bbb2e
Faster stringer implementation for Address (#736)
tsachiherman Jan 23, 2020
eb09a5d
Interconnect relays on a locally deployed network (#742)
tsachiherman Jan 24, 2020
9ae1984
static codegen for msgpack encode/decode (#578)
zeldovich Jan 27, 2020
01dd2d2
Use cobra for the kmd command to allow for documentation automation.
winder Jan 27, 2020
a02a92d
Limit client side connection rate, part 1
algonautshant Jan 28, 2020
12e843c
Fix checkdep message. (#745)
tsachiherman Jan 28, 2020
5aeb278
Fix equal stake distribution in generated networks (#749)
algorandskiy Jan 28, 2020
920167b
Update with PR feedback.
winder Jan 28, 2020
214dd72
Change a player test to use either old buggy behavior or new correct …
algoradam Jan 28, 2020
f14e4bb
Bugfix: Fix last relevant proposal period in agreement protocol. (#746)
derbear Jan 28, 2020
99618d8
Format numbers using number specifier (#735)
tsachiherman Jan 29, 2020
da0a1e5
Implement local net template generation with netgoal (#762)
algorandskiy Jan 30, 2020
a677ad4
Improve net templates support (#766)
algorandskiy Jan 31, 2020
6f7c566
Release build pipeline step 1: Build, package, sign, deploy to stagin…
Jan 31, 2020
1255095
Implement automation for release notes generator (#761)
bricerisingalgorand Jan 31, 2020
a4d0aef
back out locking added in c78ada09f230a3c66cd934860700f93ff31a93eb (#…
algobolson Jan 31, 2020
096c17e
Implement dummy telemetry hook to safely perform operations on it whe…
algorandskiy Jan 31, 2020
2fa6981
Remove instances of tagging in our build process (#770)
bricerisingalgorand Feb 1, 2020
66966ee
Configurable consensus protocol (#750)
tsachiherman Feb 1, 2020
f064631
Release build pipeline step 2: Test (#773)
Feb 2, 2020
b1632f3
When locally installing, take the binaries from the first-gopath-bin …
tsachiherman Feb 3, 2020
d893a93
Remove temporary build test location (#777)
Feb 3, 2020
c8e27bf
Make sure to default to Consensus if consensus.json is missing. (#779)
tsachiherman Feb 3, 2020
76eb233
Make util.ExecAndCaptureOutput able to process large output (#771)
algorandskiy Feb 4, 2020
582a95c
Make goal node state change commands systemd aware (#769)
bricerisingalgorand Feb 4, 2020
102d5c4
Restructure release/ dir (#782)
Feb 5, 2020
7716ec6
Allow asset creation transactions to be created while catching up. (#…
winder Feb 6, 2020
56176fb
Tunnel outgoing connection via a rate limiting dialer (#780)
algonautshant Feb 7, 2020
c2c9f1a
Release build pipeline step 3: Added "prod" pipeline to `release/` (#…
Feb 7, 2020
1c15b1c
better algons error messages. (#794)
tsachiherman Feb 8, 2020
e56c76d
Create a rate limiting transport (#795)
tsachiherman Feb 8, 2020
c326822
Some release abstraction (#796)
Feb 8, 2020
8a1a733
Change agreement message encoder to msgp. (#786)
derbear Feb 10, 2020
886a1db
Move fetcher client into catchup (#774)
tsachiherman Feb 10, 2020
1e2a0c0
Fix gpg keygrip code and remove old code (#797)
Feb 10, 2020
86ee853
bugfix : compile correctly teal program that includes a base64 string…
tsachiherman Feb 10, 2020
1ab4aa9
Add support for multiple network protocol versions (#799)
tsachiherman Feb 11, 2020
304a7a9
Include comment about something that looks like a vulnerability, but …
winder Feb 12, 2020
c40879e
Skip logging and telemetry when not needed. (#737)
tsachiherman Feb 12, 2020
4e90281
Added utils for testing release packages (#819)
Feb 12, 2020
c877172
Update docker build script to be more flexible with its naming (#822)
Feb 12, 2020
41a2fc8
Deleting out-of-date wallet folder in go-algorand. (#821)
algonautshant Feb 13, 2020
ec30796
Some build fixes (#818)
Feb 13, 2020
73801fc
Move `dsign` functionality to goal (#800)
justicz Feb 13, 2020
34cc45f
Deferred persistent crash data validation (#823)
tsachiherman Feb 13, 2020
499d8b0
Update Dockerfile for our official docker image (#826)
Feb 14, 2020
46daf07
fix incorrect comments (#825)
stechu Feb 14, 2020
2ee0920
Reduce the log verbosity on scenario 3 deployed network (#828)
tsachiherman Feb 15, 2020
85bd613
Trigger test build (#831)
Feb 15, 2020
aef6241
More release build fixes (#836)
Feb 18, 2020
a6c13b0
Split consensus from config (#832)
tsachiherman Feb 18, 2020
b1ff3ca
netgoal: create accounts in parallel (#827)
tsachiherman Feb 18, 2020
5d2bfe3
Updated job name to match on the Jenkins server (#837)
Feb 18, 2020
20cb28e
Brice/refactor make (#835)
bricerisingalgorand Feb 20, 2020
ea21ac3
Run build and test jobs in a docker container (#840)
Feb 20, 2020
90db97b
Brice/fix deploy linux (#767)
bricerisingalgorand Feb 20, 2020
eea59da
Limit msgp tool warning message scope (#834)
tsachiherman Feb 21, 2020
64fa93d
Refactoring peer unicast implementation (#841)
algonautshant Feb 24, 2020
d531f42
Actively scan for ledgers, normalize names cross platform (#842)
justicz Feb 24, 2020
c1ce127
require Encode() and Decode() to take msgp'ed types (#798)
zeldovich Feb 24, 2020
3ad12bf
Change EnsureDigest to be asynchronous. (#754)
derbear Feb 24, 2020
4eb7a3b
Strip any defined remote repo from branch name when building (#850)
Feb 25, 2020
b5ec1f0
Implement DNSSEC resolving library (#830)
algorandskiy Feb 26, 2020
d5f1ab1
disable the concurrent wallet generation. (#848)
tsachiherman Feb 26, 2020
81456ac
Force docker to use `root` as the user when running the instance (#849)
Feb 26, 2020
d356c93
Improve some error checking and logging for build process (#851)
Feb 26, 2020
14fe2ef
Fix comment in agreement. (#856)
derbear Feb 26, 2020
97ce752
Add MoI to network (#853)
tsachiherman Feb 26, 2020
3c79477
Better error case handling in database utils (#857)
tsachiherman Feb 27, 2020
ccd64ff
Explicitly curl go.1.12.9 and archive `get_latest_go.py` (#855)
Feb 27, 2020
15fbd95
Trap errors and remove ec2 instance (#854)
Feb 27, 2020
90ae489
Update the update script. (#670)
tsachiherman Feb 28, 2020
7511b85
Faster external_build_printlog by using curl instead of aws cli (#847)
tsachiherman Feb 28, 2020
afb1c7e
Fix concurrent SQLite initialization (#872)
algorandskiy Feb 28, 2020
b44510a
change _tx_lock -> _txlock (#871)
justicz Feb 28, 2020
2a9ec4e
Redirect stdout of build log file to build release upload directory (…
Feb 28, 2020
067fb2b
Install boto3 as a build dependency for docker (#875)
Feb 29, 2020
8be2877
Enable some skipped test on MacOS (#876)
algorandskiy Feb 29, 2020
7469d0d
Set root as explicit docker user for test phase (#874)
Mar 1, 2020
1e4ab24
Refactor are combine the phonebook implementations (#870)
algonautshant Mar 2, 2020
c721b39
Adding a verifying signatures step to the build release pipeline (#878)
Mar 3, 2020
23a0a68
Merge branch 'master' into shellcheck
Mar 3, 2020
ef2871b
Merge branch 'master' into shellcheck
Mar 3, 2020
e3b93e6
Wrap entire arguments in quotes
Mar 11, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 38 additions & 35 deletions scripts/build_deb.sh
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
#!/usr/bin/env bash
# shellcheck disable=2038,2064

# build_deb.sh - Build a .deb package for one platform.
#
Expand All @@ -15,20 +16,21 @@ fi
## Need to run inside fakeroot to make sure the files in
## the Debian package are owned by root.
if [ "$EUID" != "0" ]; then
exec fakeroot $0 "$@"
exec fakeroot "$0" "$@"
fi

OS=linux
ARCH=$1
OUTDIR="$2"

export GOPATH=$(go env GOPATH)
GOPATH=$(go env GOPATH)
export GOPATH
REPO_DIR=$(pwd)

echo "Building debian package for '${OS} - ${ARCH}'"

if [ -z "${NO_BUILD}" ]; then
env GOOS=${OS} GOARCH=${ARCH} scripts/build_prod.sh
env GOOS="${OS}" GOARCH="${ARCH}" scripts/build_prod.sh
else
echo "already built"
true
Expand All @@ -44,45 +46,45 @@ DEFAULT_RELEASE_NETWORK=$(./scripts/compute_branch_release_network.sh "${DEFAULT
PKG_ROOT=$(mktemp -d)
trap "rm -rf $PKG_ROOT" 0

mkdir -p ${PKG_ROOT}/usr/bin
mkdir -p "${PKG_ROOT}/usr/bin"

if [ "${VARIATION}" = "" ]; then
# NOTE: keep in sync with installer/rpm/algorand.spec
bin_files=("algocfg" "algod" "algoh" "algokey" "carpenter" "catchupsrv" "ddconfig.sh" "diagcfg" "goal" "kmd" "msgpacktool" "node_exporter")
fi

for bin in "${bin_files[@]}"; do
cp ${GOPATH}/bin/${bin} ${PKG_ROOT}/usr/bin
chmod 755 ${PKG_ROOT}/usr/bin/${bin}
cp "${GOPATH}/bin/${bin}" "${PKG_ROOT}"/usr/bin
chmod 755 "${PKG_ROOT}/usr/bin/${bin}"
done

mkdir -p ${PKG_ROOT}/usr/lib/algorand
mkdir -p "${PKG_ROOT}/usr/lib/algorand"
lib_files=("updater" "find-nodes.sh")
for lib in "${lib_files[@]}"; do
cp ${GOPATH}/bin/${lib} ${PKG_ROOT}/usr/lib/algorand
chmod g-w ${PKG_ROOT}/usr/lib/algorand/${lib}
cp "${GOPATH}/bin/${lib}" "${PKG_ROOT}/usr/lib/algorand"
chmod g-w "${PKG_ROOT}/usr/lib/algorand/${lib}"
done

data_files=("config.json.example" "system.json")
mkdir -p ${PKG_ROOT}/var/lib/algorand
mkdir -p "${PKG_ROOT}/var/lib/algorand"
for data in "${data_files[@]}"; do
cp installer/${data} ${PKG_ROOT}/var/lib/algorand
cp "installer/${data}" "${PKG_ROOT}/var/lib/algorand"
done

if [ ! -z "${RELEASE_GENESIS_PROCESS}" ]; then
genesis_dirs=("devnet" "testnet" "mainnet" "betanet")
for dir in "${genesis_dirs[@]}"; do
mkdir -p ${PKG_ROOT}/var/lib/algorand/genesis/${dir}
cp ${REPO_DIR}/installer/genesis/${dir}/genesis.json ${PKG_ROOT}/var/lib/algorand/genesis/${dir}/genesis.json
mkdir -p "${PKG_ROOT}/var/lib/algorand/genesis/${dir}"
cp "${REPO_DIR}/installer/genesis/${dir}/genesis.json" "${PKG_ROOT}/var/lib/algorand/genesis/${dir}/genesis.json"
#${GOPATH}/bin/buildtools genesis ensure -n ${dir} --source ${REPO_DIR}/gen/${dir}/genesis.json --target ${PKG_ROOT}/var/lib/algorand/genesis/${dir}/genesis.json --releasedir ${REPO_DIR}/installer/genesis
done
# Copy the appropriate network genesis.json for our default (in root ./genesis folder)
cp ${PKG_ROOT}/var/lib/algorand/genesis/${DEFAULT_RELEASE_NETWORK}/genesis.json ${PKG_ROOT}/var/lib/algorand
cp "${PKG_ROOT}/var/lib/algorand/genesis/${DEFAULT_RELEASE_NETWORK}/genesis.json" "${PKG_ROOT}/var/lib/algorand"
elif [[ "${CHANNEL}" == "dev" || "${CHANNEL}" == "stable" || "${CHANNEL}" == "nightly" || "${CHANNEL}" == "beta" ]]; then
cp ${REPO_DIR}/installer/genesis/${DEFAULTNETWORK}/genesis.json ${PKG_ROOT}/var/lib/algorand/genesis.json
cp "${REPO_DIR}/installer/genesis/${DEFAULTNETWORK}/genesis.json" "${PKG_ROOT}/var/lib/algorand/genesis.json"
#${GOPATH}/bin/buildtools genesis ensure -n ${DEFAULTNETWORK} --source ${REPO_DIR}/gen/${DEFAULTNETWORK}/genesis.json --target ${PKG_ROOT}/var/lib/algorand/genesis.json --releasedir ${REPO_DIR}/installer/genesis
else
cp ${REPO_DIR}/installer/genesis/${DEFAULTNETWORK}/genesis.json ${PKG_ROOT}/var/lib/algorand
cp "${REPO_DIR}/installer/genesis/${DEFAULTNETWORK}/genesis.json" "${PKG_ROOT}/var/lib/algorand"
# Disabled because we have static genesis files now
#cp gen/${DEFAULTNETWORK}/genesis.json ${PKG_ROOT}/var/lib/algorand
#if [ -z "${TIMESTAMP}" ]; then
Expand All @@ -92,34 +94,34 @@ else
fi

systemd_files=("algorand.service" "algorand@.service")
mkdir -p ${PKG_ROOT}/lib/systemd/system
mkdir -p "${PKG_ROOT}/lib/systemd/system"
for svc in "${systemd_files[@]}"; do
cp installer/${svc} ${PKG_ROOT}/lib/systemd/system
chmod 644 ${PKG_ROOT}/lib/systemd/system/${svc}
cp "installer/${svc}" "${PKG_ROOT}/lib/systemd/system"
chmod 644 "${PKG_ROOT}/lib/systemd/system/${svc}"
done

unattended_upgrades_files=("51algorand-upgrades")
mkdir -p ${PKG_ROOT}/etc/apt/apt.conf.d
mkdir -p "${PKG_ROOT}/etc/apt/apt.conf.d"
for f in "${unattended_upgrades_files[@]}"; do
cp installer/${f} ${PKG_ROOT}/etc/apt/apt.conf.d
cp "installer/${f}" "${PKG_ROOT}/etc/apt/apt.conf.d"
done

# files should not be group writable but directories should be
chmod -R g-w ${PKG_ROOT}/var/lib/algorand
find ${PKG_ROOT}/var/lib/algorand -type d | xargs chmod g+w
chmod -R g-w "${PKG_ROOT}/var/lib/algorand"
find "${PKG_ROOT}/var/lib/algorand" -type d | xargs chmod g+w

mkdir -p ${PKG_ROOT}/DEBIAN
mkdir -p "${PKG_ROOT}/DEBIAN"
debian_files=("control" "postinst" "prerm" "postrm" "conffiles")
for ctl in "${debian_files[@]}"; do
# Copy first, to preserve permissions, then overwrite to fill in template.
cp -a installer/debian/${ctl} ${PKG_ROOT}/DEBIAN/${ctl}
cat installer/debian/${ctl} \
| sed -e s,@ARCH@,${ARCH}, \
-e s,@VER@,${VER}, \
> ${PKG_ROOT}/DEBIAN/${ctl}
cp -a "installer/debian/${ctl}" "${PKG_ROOT}/DEBIAN/${ctl}"
< installer/debian/"${ctl}" \
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cat version was obvious, this one is tricky

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wow, never seen < in cmd > out usually I've seen cmd < in > out. huh. Today I Learned about a new weird bashism.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it looks very cumbersome, and I'm not a big fan of that.. but it's better cross-platforming between Mac and ubuntu.

Copy link
Copy Markdown
Author

@btoll btoll Mar 3, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was flagged as a UUOC:
https://github.com/koalaman/shellcheck/wiki/SC2002
https://en.wikipedia.org/wiki/Cat_%28Unix%29#Unix_culture

If everyone would rather opt for readability, I can disable this rule at the top of the file.

sed -e s,@ARCH@,"${ARCH}", \
-e s,@VER@,"${VER}", \
> "${PKG_ROOT}/DEBIAN/${ctl}"
done
# TODO: make `Files:` segments for vendor/... and crypto/libsodium-fork, but reasonably this should be understood to cover all _our_ files and copied in packages continue to be licenced under their own terms
cat <<EOF>${PKG_ROOT}/DEBIAN/copyright
cat <<EOF> "${PKG_ROOT}/DEBIAN/copyright"
Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Upstream-Name: Algorand
Upstream-Contact: Algorand developers <dev@algorand.com>
Expand All @@ -129,9 +131,10 @@ Files: *
Copyright: Algorand developers <dev@algorand.com>
License: AGPL-3+
EOF
sed 's/^$/./g' < COPYING | sed 's/^/ /g' >> ${PKG_ROOT}/DEBIAN/copyright
mkdir -p ${PKG_ROOT}/usr/share/doc/algorand
cp -p ${PKG_ROOT}/DEBIAN/copyright ${PKG_ROOT}/usr/share/doc/algorand/copyright
sed 's/^$/./g' < COPYING | sed 's/^/ /g' >> "${PKG_ROOT}/DEBIAN/copyright"
mkdir -p "${PKG_ROOT}/usr/share/doc/algorand"
cp -p "${PKG_ROOT}/DEBIAN/copyright" "${PKG_ROOT}/usr/share/doc/algorand/copyright"

OUTPUT="$OUTDIR/algorand_${VER}_${ARCH}.deb"
dpkg-deb --build "${PKG_ROOT}" "${OUTPUT}"

OUTPUT="$OUTDIR"/algorand_${VER}_${ARCH}.deb
dpkg-deb --build ${PKG_ROOT} ${OUTPUT}