Skip to content
This repository was archived by the owner on Jan 22, 2025. It is now read-only.

testnet-automation: More knobs for rolling upgrades #12337

Closed
t-nelson wants to merge 11 commits into from
Closed

testnet-automation: More knobs for rolling upgrades #12337

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
f083495
testnet-automation: Plumb genesis options
t-nelson Sep 18, 2020
a4460b5
forgive me
t-nelson Jun 10, 2020
f3175a7
Let cargo build all bins
t-nelson Jun 10, 2020
d0fdee0
Fix bootstrap validator stake threshold wait
t-nelson Sep 23, 2020
7f9556b
Don't require tower
t-nelson Oct 9, 2020
e284c40
This variable holds a number not a list
t-nelson Oct 9, 2020
38a6267
Debug
t-nelson Oct 9, 2020
ba68a03
New shellcheck image
t-nelson Oct 9, 2020
d15f87d
Really skip setup
t-nelson Oct 9, 2020
284dc86
Revert "Really skip setup"
t-nelson Oct 10, 2020
f44ad58
hack
t-nelson Oct 10, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions ci/forgive-me.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
#!/usr/bin/env bash

set -x

here="$(realpath "$(dirname "$0")")"
source_dir="${here:?}/.."

committish="${1:?}"
tarballBaseName="${2:-solana-release}"
remote=https://github.com/solana-labs/solana.git

cleanup_tmp_source_dir() {
rm -rf "$tmp_source_dir"
Copy link
Copy Markdown
Contributor

@ryoqun ryoqun Sep 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/solana-labs/solana/pull/12337/files#r493804567 and do this?:

git worktree remove --force "$tmp_source_dir"

}

set -e
tmp_source_dir="$(mktemp --directory --tmpdir="$source_dir")"

trap 'cleanup_tmp_source_dir' EXIT

# Clone the specified committish
git clone "$remote" "$tmp_source_dir"
Comment on lines +21 to +22
Copy link
Copy Markdown
Contributor

@ryoqun ryoqun Sep 23, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any reason we have to clone all new? I think we can add another worktree like this:

# Create temporal source dir to build
git fetch origin "$branch"
if [[ -n $commitish ]]; then
  git worktree add "$tmp_source_dir" $commitish
else
  git worktree add "$tmp_source_dir" FETCH_HEAD
fi

Copy link
Copy Markdown
Contributor Author

@t-nelson t-nelson Sep 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope! This was just a quick and dirty script to see if I could get what I wanted working (see script file name 😅). Plenty of optimizations to be had here

Copy link
Copy Markdown
Contributor

@ryoqun ryoqun Sep 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what a quick reply. :) I updated my comment by picking something from my attempt #12390. let's unify our work for the better. :)

Copy link
Copy Markdown
Contributor

@ryoqun ryoqun Sep 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this way we can build off at arbitrary commit, might be useful for building with fast-changing release branches.


if pushd "$tmp_source_dir"; then
(
set +e

git fetch origin "$committish":branch_to_build
git checkout branch_to_build
Comment on lines +28 to +29
Copy link
Copy Markdown
Contributor

@ryoqun ryoqun Sep 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this can be merged into https://github.com/solana-labs/solana/pull/12337/files#r493804567


# Use our scripts
cp -ar "${source_dir}"/{fetch-perf-libs.sh,scripts,net,multinode-demo,ci} "$tmp_source_dir"

export CHANNEL=edge
export DO_NOT_PUBLISH_TAR=true # Never publish
export TARBALL_BASENAME="$tarballBaseName"
ci/publish-tarball.sh

cp "${tarballBaseName}"*.tar.bz2 "${source_dir}"
)
popd
fi
1 change: 0 additions & 1 deletion multinode-demo/bootstrap-validator.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,6 @@ ledger_dir="$SOLANA_CONFIG_DIR"/bootstrap-validator
args+=(
--enable-rpc-exit
--enable-rpc-set-log-filter
--require-tower
--ledger "$ledger_dir"
--rpc-port 8899
--snapshot-interval-slots 200
Expand Down
1 change: 0 additions & 1 deletion multinode-demo/validator.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -228,7 +228,6 @@ default_arg --ledger "$ledger_dir"
default_arg --log -
default_arg --enable-rpc-exit
default_arg --enable-rpc-set-log-filter
default_arg --require-tower

if [[ -n $SOLANA_CUDA ]]; then
program=$solana_validator_cuda
Expand Down
3 changes: 2 additions & 1 deletion net/remote/remote-node.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -392,6 +392,7 @@ EOF
fi

cat >> ~/solana/on-reboot <<EOF
# $nodeIndex $extraPrimordialStakes
$maybeSkipAccountsCreation
nohup multinode-demo/validator.sh ${args[@]} > validator.log.\$now 2>&1 &
pid=\$!
Expand Down Expand Up @@ -419,7 +420,7 @@ EOF
args+=(--keypair config/validator-identity.json)
fi

if [[ ${#extraPrimordialStakes} -eq 0 ]]; then
if [[ ${extraPrimordialStakes} -eq 0 ]]; then
multinode-demo/delegate-stake.sh "${args[@]}" "$internalNodesStakeLamports"
fi
fi
Expand Down
1 change: 0 additions & 1 deletion run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,6 @@ args=(
--enable-rpc-exit
--enable-rpc-transaction-history
--init-complete-file "$dataDir"/init-completed
--require-tower
)
solana-validator "${args[@]}" &
validator=$!
Expand Down
8 changes: 5 additions & 3 deletions scripts/cargo-install-all.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,12 +109,14 @@ done
(
set -x
# shellcheck disable=SC2086 # Don't want to double quote $rust_version
$cargo $maybeRustVersion build $maybeReleaseFlag "${binArgs[@]}"
$cargo $maybeRustVersion build $maybeReleaseFlag --bins
Copy link
Copy Markdown
Contributor

@ryoqun ryoqun Sep 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

qq: why is this needed? because of shellcheck?

Copy link
Copy Markdown
Contributor Author

@t-nelson t-nelson Sep 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC some bins were missing without

)

mkdir -p "$installDir/bin"
for bin in "${BINS[@]}"; do
cp -fv "target/$buildVariant/$bin" "$installDir"/bin
for bin in "target/$buildVariant/"*; do
Copy link
Copy Markdown
Contributor

@ryoqun ryoqun Sep 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as https://github.com/solana-labs/solana/pull/12337/files#r493815963

Copy link
Copy Markdown
Contributor Author

@t-nelson t-nelson Sep 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Idea here was basically to get rid of the manually specified bin list, so we don't have to maintain it

if [[ "$bin" =~ /solana(-[^./]+)*$ ]]; then
cp -fv "$bin" "$installDir"/bin
fi
done

if [[ -d target/perf-libs ]]; then
Expand Down
3 changes: 2 additions & 1 deletion system-test/automation_utils.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,8 @@ function wait_for_bootstrap_validator_stake_drop {
while true; do
# shellcheck disable=SC2154
bootstrap_validator_validator_info="$(ssh "${sshOptions[@]}" "${validatorIpList[0]}" '$HOME/.cargo/bin/solana --url http://127.0.0.1:8899 validators | grep "$($HOME/.cargo/bin/solana-keygen pubkey ~/solana/config/bootstrap-validator/identity.json)"')"
bootstrap_validator_stake_percentage="$(echo "$bootstrap_validator_validator_info" | awk '{gsub(/[\(,\),\%]/,""); print $9}')"
temp="${bootstrap_validator_validator_info##[^(]*\(}"
bootstrap_validator_stake_percentage="${temp%\%)}"

if [[ $(echo "$bootstrap_validator_stake_percentage < $max_stake" | bc) -ne 0 ]]; then
echo "Bootstrap validator stake has fallen below $max_stake to $bootstrap_validator_stake_percentage"
Expand Down
7 changes: 6 additions & 1 deletion system-test/rolling-upgrade/rolling_upgrade.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,11 @@ if [[ "$UPGRADE_INITIAL_DELAY" -gt 0 ]]; then
"$NET_SH" sanity
fi

declare maybeExtraPrimordialStakes
if [[ -n "$EXTRA_PRIMORDIAL_STAKES" ]]; then
maybeExtraPrimordialStakes="--extra-primordial-stakes $EXTRA_PRIMORDIAL_STAKES"
fi

# Restart validators one by one
# shellcheck disable=SC2154 # sourced from config above
for i in "${!validatorIpList[@]}"; do
Expand All @@ -47,7 +52,7 @@ for i in "${!validatorIpList[@]}"; do
declare ipAddress="${validatorIpList[$i]}"

"$NET_SH" stopnode -i "$ipAddress"
"$NET_SH" startnode -r -i "$ipAddress"
"$NET_SH" startnode -r -i "$ipAddress" ${maybeExtraPrimordialStakes}

# This could be replaced with something based on `solana catchup`
sleep_if_positive "$UPGRADE_INTERVALIDATOR_DELAY"
Expand Down
4 changes: 3 additions & 1 deletion system-test/testnet-automation.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,8 @@ function launch_testnet() {
# shellcheck disable=SC2086
"${REPO_ROOT}"/net/net.sh start $version_args \
-c idle=$NUMBER_OF_CLIENT_NODES $maybeStartAllowBootFailures \
--gpu-mode $startGpuMode $maybeWarpSlot $maybeAsyncNodeInit $maybeExtraPrimordialStakes
--gpu-mode $startGpuMode $maybeWarpSlot $maybeAsyncNodeInit $maybeExtraPrimordialStakes \
$GENESIS_OPTIONS

execution_step "Waiting for bootstrap validator's stake to fall below ${BOOTSTRAP_VALIDATOR_MAX_STAKE_THRESHOLD}%"
wait_for_bootstrap_validator_stake_drop "$BOOTSTRAP_VALIDATOR_MAX_STAKE_THRESHOLD"
Expand Down Expand Up @@ -326,6 +327,7 @@ TEST_PARAMS_TO_DISPLAY=(CLOUD_PROVIDER \
PARTITION_ITERATION_COUNT \
TEST_TYPE \
CUSTOM_SCRIPT \
GENESIS_OPTIONS \
)

TEST_CONFIGURATION=
Expand Down