Skip to content

chore: ebs attach robustness #6108

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ludamad merged 3 commits into from
Apr 30, 2024
Merged

chore: ebs attach robustness #6108

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
9972088
chore: ebs attach robustness
ludamad Apr 30, 2024
0186cc0
Improve message
ludamad Apr 30, 2024
8a39e2c
Merge branch 'master' into chore-ebs-robustness
ludamad Apr 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/setup-runner.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,11 +78,11 @@ jobs:
run: |
# Compare the checked-out CI configuration files with the reference files
if ! git diff --no-index .github/workflows/ci.yml merge-commit-pipeline-files/.github/workflows/ci.yml; then
echo "Error: ci.yml changes in master (or PR base). Please merge these changes."
echo "Error: ci.yml changes in master (or PR base). Please merge these changes. This is to prevent surprises from Github Action's merge behavior."
exit 1
fi
if ! git diff --no-index .github/workflows/setup-runner.yml merge-commit-pipeline-files/.github/workflows/setup-runner.yml; then
echo "Error: setup-runner.yml changes in master (or PR base). Please merge these changes."
echo "Error: setup-runner.yml changes in master (or PR base). Please merge these changes. This is to prevent surprises from Github Action's merge behavior."
exit 1
fi

Expand Down
14 changes: 13 additions & 1 deletion scripts/attach_ebs_cache.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,20 @@ if [ -f /run/.ebs-cache-mounted ] ; then
WAIT_INTERVAL=10 # Interval between checks in seconds
elapsed_time=0
# Check for existing mount, assume we can continue if existing

while ! mount | grep -q "/var/lib/docker type ext4"; do
echo "Someone already marked as mounting, waiting for them..."
echo "Someone already marked as mounting, terminating any stopped instances and waiting..."
# Identify and terminate instances in 'STOPPED' state that are using this volume
STOPPED_INSTANCES=$(aws ec2 describe-instances \
--region $REGION \
--filters "Name=instance-state-name,Values=stopped" "Name=block-device-mapping.volume-id,Values=$VOLUME_ID" \
--query "Reservations[*].Instances[*].InstanceId" \
--output text)

for instance in $STOPPED_INSTANCES; do
echo "Terminating instance $instance"
aws ec2 terminate-instances --instance-ids $instance
done
if [ $elapsed_time -ge $MAX_WAIT_TIME ]; then
echo "Cache mount did not become available within $MAX_WAIT_TIME seconds... race condition?"
exit 1
Expand Down