Skip to content

Add script to upload cloud gcp jars to appropriate bucket. #203

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Jan 14, 2025
Merged

Add script to upload cloud gcp jars to appropriate bucket. #203

Changes from 3 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
880026d
Add script to upload cloud gcp jars to appropriate bucket.
david-zlai Jan 11, 2025
bbf78a3
don't run sbt assembly for everything just the two we want for now
david-zlai Jan 11, 2025
a391eac
changes
david-zlai Jan 11, 2025
1bd2356
add wheel stuff and upload to all customer ids
david-zlai Jan 14, 2025
42ddfb7
exit 0 when selecting no:
david-zlai Jan 14, 2025
1275391
check python version
david-zlai Jan 14, 2025
74b432a
update tool versions
david-zlai Jan 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions distribution/build_and_upload_gcp_artifacts.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
#!/bin/bash

SCRIPT_DIRECTORY=$(dirname -- "$(realpath -- "$0")")
CHRONON_ROOT_DIR=$(dirname "$SCRIPT_DIRECTORY")

if [ -z "$1" ]; then
echo "Error: Please provide customer id argument (ex: canary)"
exit 1
fi

This comment was marked as resolved.

Sign in to view



CUSTOMER_ID=$1

echo "Working in $CHRONON_ROOT_DIR"
cd $CHRONON_ROOT_DIR

This comment was marked as outdated.

Sign in to view

echo "Building jars"
sbt cloud_gcp/assembly
sbt cloud_gcp_submitter/assembly

Comment on lines +36 to +39

This comment was marked as resolved.

Sign in to view

pwd
CLOUD_GCP_JAR="$CHRONON_ROOT_DIR/cloud_gcp/target/scala-2.12/cloud_gcp-assembly-0.1.0-SNAPSHOT.jar"
CLOUD_GCP_SUBMITTER_JAR="$CHRONON_ROOT_DIR/cloud_gcp_submitter/target/scala-2.12/cloud_gcp_submitter-assembly-0.1.0-SNAPSHOT.jar"
Copy link
Contributor

@coderabbitai coderabbitai bot Jan 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Use dynamic version detection for JAR paths.

-CLOUD_GCP_JAR="$CHRONON_ROOT_DIR/cloud_gcp/target/scala-2.12/cloud_gcp-assembly-0.1.0-SNAPSHOT.jar"
-CLOUD_GCP_SUBMITTER_JAR="$CHRONON_ROOT_DIR/cloud_gcp_submitter/target/scala-2.12/cloud_gcp_submitter-assembly-0.1.0-SNAPSHOT.jar"
+VERSION=$(cat version.sbt | cut -d " " -f3 | tr -d '"')
+CLOUD_GCP_JAR="$CHRONON_ROOT_DIR/cloud_gcp/target/scala-2.12/cloud_gcp-assembly-$VERSION.jar"
+CLOUD_GCP_SUBMITTER_JAR="$CHRONON_ROOT_DIR/cloud_gcp_submitter/target/scala-2.12/cloud_gcp_submitter-assembly-$VERSION.jar"

Committable suggestion skipped: line range outside the PR's diff.


if [ ! -f /$CLOUD_GCP_JAR ]; then
echo "$CLOUD_GCP_JAR not found"
exit 1
fi

if [ ! -f /$CLOUD_GCP_SUBMITTER_JAR ]; then
echo "$CLOUD_GCP_SUBMITTER_JAR not found"
exit 1
fi
Copy link
Contributor

@coderabbitai coderabbitai bot Jan 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Fix incorrect file path checks.

-if [ ! -f /$CLOUD_GCP_JAR ]; then
+if [ ! -f "$CLOUD_GCP_JAR" ]; then

-if [ ! -f /$CLOUD_GCP_SUBMITTER_JAR ]; then
+if [ ! -f "$CLOUD_GCP_SUBMITTER_JAR" ]; then
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if [ ! -f /$CLOUD_GCP_JAR ]; then
echo "$CLOUD_GCP_JAR not found"
exit 1
fi
if [ ! -f /$CLOUD_GCP_SUBMITTER_JAR ]; then
echo "$CLOUD_GCP_SUBMITTER_JAR not found"
exit 1
fi
if [ ! -f "$CLOUD_GCP_JAR" ]; then
echo "$CLOUD_GCP_JAR not found"
exit 1
fi
if [ ! -f "$CLOUD_GCP_SUBMITTER_JAR" ]; then
echo "$CLOUD_GCP_SUBMITTER_JAR not found"
exit 1
fi


BUCKET_JAR_PATH=gs://zipline-artifacts-$CUSTOMER_ID/jars
Copy link
Collaborator

@tchow-zlai tchow-zlai Jan 11, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we make it so if nothing is specified, we just upload everything:

  • jars (including dataproc submitter AND cloud_gcp assembly) to BOTH etsy and canary buckets
  • wheels to BOTH etsy and canary buckets
  • confs to BOTH etsy and canary buckets

Copy link
Contributor Author

@david-zlai david-zlai Jan 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done


echo "Do you wish to upload to $BUCKET_JAR_PATH?"
select yn in "Yes" "No"; do
case $yn in
Yes )
set -euxo pipefail
gcloud storage cp $CLOUD_GCP_JAR $BUCKET_JAR_PATH;
gcloud storage cp $CLOUD_GCP_SUBMITTER_JAR $BUCKET_JAR_PATH;
echo "Succeeded"
break;;
No ) exit 1;;
esac
done

This comment was marked as resolved.

Sign in to view