Skip to content

[HUDI-3340] Fix deploy_staging_jars for different profiles #5240

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
xushiyan merged 3 commits into from
Apr 6, 2022
Merged

[HUDI-3340] Fix deploy_staging_jars for different profiles #5240

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
16b77c3
[HUDI-3340] Fix deploy_staging_jars for different profiles
xushiyan Apr 6, 2022
9863f75
allow run single option
xushiyan Apr 6, 2022
6f719fe
minor rename
xushiyan Apr 6, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 43 additions & 25 deletions scripts/release/deploy_staging_jars.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,40 +21,58 @@
## Variables with defaults (if not overwritten by environment)
##
MVN=${MVN:-mvn}
SPARK_VERSION=2
# fail immediately
set -o errexit
set -o nounset
# print command before executing
set -o xtrace

CURR_DIR=`pwd`
if [[ `basename $CURR_DIR` != "scripts" ]] ; then
echo "You have to call the script from the scripts/ dir"
CURR_DIR=$(pwd)
if [ ! -d "$CURR_DIR/packaging" ] ; then
echo "You have to call the script from the repository root dir that contains 'packaging/'"
exit 1
fi

if [[ $# -lt 1 ]]; then
echo "This script will deploy artifacts to staging repositories"
echo "There is one param required:"
echo "--scala_version=\${SCALA_VERSION}"
exit
else
for param in "$@"
do
if [[ $param =~ --scala_version\=(2\.1[1-2]) ]]; then
SCALA_VERSION=${BASH_REMATCH[1]}
elif [[ $param =~ --spark_version\=([2-3]) ]]; then
SPARK_VERSION=${BASH_REMATCH[1]}
fi
done
if [ "$#" -gt "1" ]; then
echo "Only accept 0 or 1 argument. Use -h to see examples."
exit 1
fi

###########################
declare -a ALL_VERSION_OPTS=(
"-Dscala-2.11 -Dspark2.4 -Dflink1.13"
"-Dscala-2.12 -Dspark2.4 -Dflink1.13"
"-Dscala-2.12 -Dspark3.1 -Dflink1.14"
"-Dscala-2.12 -Dspark3.2 -Dflink1.14"
)
printf -v joined "'%s'\n" "${ALL_VERSION_OPTS[@]}"

if [ "${1:-}" == "-h" ]; then
echo "
Usage: $(basename "$0") [OPTIONS]

Options:
<version option> One of the version options below
${joined}
-h, --help
"
exit 0
fi

cd ..
VERSION_OPT=${1:-}
valid_version_opt=false
for v in "${ALL_VERSION_OPTS[@]}"; do
[[ $VERSION_OPT == "$v" ]] && valid_version_opt=true
done

echo "Deploying to repository.apache.org with scala version ${SCALA_VERSION}"
if [ "$valid_version_opt" = true ]; then
# run deploy for only specified version option
ALL_VERSION_OPTS=("$VERSION_OPT")
elif [ "$#" == "1" ]; then
echo "Version option $VERSION_OPT is invalid. Use -h to see examples."
exit 1
fi

COMMON_OPTIONS="-Dscala-${SCALA_VERSION} -Dspark${SPARK_VERSION} -Prelease -DskipTests -DretryFailedDeploymentCount=10 -DdeployArtifacts=true"
$MVN clean deploy $COMMON_OPTIONS
for v in "${ALL_VERSION_OPTS[@]}"
do
Copy link
Contributor

@nsivabalan nsivabalan Apr 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if for some reason, network issue or service unavailable, and if RM needs to retry, we can't go w/ for loop here. Can you de-couple that. or add an option to do everything. or give an option for user to specify on set of option.

echo "Deploying to repository.apache.org with version option ${v}"
COMMON_OPTIONS="${v} -Prelease -DskipTests -DretryFailedDeploymentCount=10 -DdeployArtifacts=true"
echo $COMMON_OPTIONS
done