Add ability to upload mongo dump directly to S3 without saving to local disk.

Author: istepanov

README.md

@@ -7,26 +7,48 @@ Docker image with `mongodump`, `cron` and AWS CLI to upload backups to AWS S3.
 
 | Env var               | Description | Default                 |
 |-----------------------|-------------|-------------------------|
-| MONGO_URI             | Mongo URI.  | `mongodb://mongo:27017` |
-| CRON_SCHEDULE         | Cron schedule. Leave empty to disable cron job. | `''` |
-| TARGET_S3_FOLDER      | Folder to upload backups. Leave it empty to disable upload to S3. **If enabled, backups are deleted from the local folder after a successful upload.** | `''` |
-| AWS_ACCESS_KEY_ID     | AWS Access Key ID. Leave empty if you want to use AWS IAM Role instead. | `''` |
-| AWS_SECRET_ACCESS_KEY | AWS Access Key ID. Leave empty if you want to use AWS IAM Role instead. | `''` |
-
-### Example
-
-Run container with cron job (once a day at 1am), upload backups to AWS S3 folder:
-
-      docker run -d \
-        -v /path/to/target/folder:/backup \
-        -e 'MONGO_URI=mongodb://mongo:27017' \
-        -e 'CRON_SCHEDULE=0 1 * * *' \
-        -e 'TARGET_S3_FOLDER=s3://my_bucket/backup/' \
-        -e 'AWS_ACCESS_KEY_ID=my_aws_key' \
-        -e 'AWS_SECRET_ACCESS_KEY=my_aws_secret' \
-        istepanov/mongodump:4.2
-
-Docker Compose example (no S3 upload, keep backups in `mongo-backup` Docker volume):
+| `MONGO_URI`             | Mongo URI.  | `mongodb://mongo:27017` |
+| `CRON_SCHEDULE`         | Cron schedule. Leave empty to disable cron job. | `''` |
+| `TARGET_FOLDER`         | Local folder (inside the container) to save backups. Mount volume to this folder. Set it to null (empty string) to disable local backups (this will make `TARGET_S3_FOLDER` a required parameter). | `'/backup'` |
+| `TARGET_S3_FOLDER`      | Folder to upload backups. Leave it empty to disable upload to S3. | `''` |
+| `AWS_ACCESS_KEY_ID`     | AWS Access Key ID. Leave empty if you want to use AWS IAM Role instead. | `''` |
+| `AWS_SECRET_ACCESS_KEY` | AWS Access Key ID. Leave empty if you want to use AWS IAM Role instead. | `''` |
+
+### Examples
+
+Run container with cron job (once a day at 1am), save backup to `/path/to/target/folder`, upload backups to AWS S3 folder:
+
+    docker run -d \
+      -v /path/to/target/folder:/backup \
+      -e 'MONGO_URI=mongodb://mongo:27017' \
+      -e 'CRON_SCHEDULE=0 1 * * *' \
+      -e 'TARGET_S3_FOLDER=s3://my_bucket/backup/' \
+      -e 'AWS_ACCESS_KEY_ID=my_aws_key' \
+      -e 'AWS_SECRET_ACCESS_KEY=my_aws_secret' \
+      istepanov/mongodump:4.2
+
+Same, but runs once, no cron job:
+
+    docker run -ti \
+      -v /path/to/target/folder:/backup \
+      -e 'MONGO_URI=mongodb://mongo:27017' \
+      -e 'TARGET_S3_FOLDER=s3://my_bucket/backup/' \
+      -e 'AWS_ACCESS_KEY_ID=my_aws_key' \
+      -e 'AWS_SECRET_ACCESS_KEY=my_aws_secret' \
+      istepanov/mongodump:4.2
+
+Run container with cron job (once a day at 1am), upload backups to AWS S3 folder, do not create local backups:
+
+    docker run -d \
+      -e 'MONGO_URI=mongodb://mongo:27017' \
+      -e 'CRON_SCHEDULE=0 1 * * *' \
+      -e 'TARGET_FOLDER=' \
+      -e 'TARGET_S3_FOLDER=s3://my_bucket/backup/' \
+      -e 'AWS_ACCESS_KEY_ID=my_aws_key' \
+      -e 'AWS_SECRET_ACCESS_KEY=my_aws_secret' \
+      istepanov/mongodump:4.2
+
+Docker Compose example - run container with cron job (once a day at 1am), save backup to `mongo-backup` volume:
 
     version: '3'
 

backup.sh

@@ -1,21 +1,34 @@
 #!/bin/bash
 
-set -e
+set -eo pipefail
 
 echo "Job started: $(date)"
 
 DATE=$(date +%Y%m%d_%H%M%S)
-FILE="/backup/backup-$DATE.tar.gz"
 
-mkdir -p dump
-mongodump --uri "$MONGO_URI"
-tar -zcvf "$FILE" dump/
-rm -rf dump/
+if [[ -z "$TARGET_FOLDER" ]]; then
+    # dump directly to AWS S3
 
-if [[ "$TARGET_S3_FOLDER" ]]; then
-    aws s3 cp "$FILE" "$TARGET_S3_FOLDER"
-    echo "$FILE uploaded to $TARGET_S3_FOLDER"
-    rm -rf "$FILE"
+    if [[ -z "$TARGET_S3_FOLDER" ]]; then
+        >&2 echo "If TARGET_FOLDER is null/unset, TARGET_S3_FOLDER must be set"
+        exit 1
+    fi
+
+    mongodump --uri "$MONGO_URI" --gzip --archive | aws s3 cp - "$TARGET_S3_FOLDER/backup-$DATE.tar.gz"
+    echo "Mongo dump uploaded to $TARGET_S3_FOLDER"
+else
+    # save dump locally (and optionally to AWS S3)
+
+    FILE="$TARGET_FOLDER/backup-$DATE.tar.gz"
+
+    mkdir -p "$TARGET_FOLDER"
+    mongodump --uri "$MONGO_URI" --gzip --archive="$FILE"
+    echo "Mongo dump saved to $FILE"
+
+    if [[ -n "$TARGET_S3_FOLDER" ]]; then
+        aws s3 cp "$FILE" "$TARGET_S3_FOLDER"
+        echo "$FILE uploaded to $TARGET_S3_FOLDER"
+    fi
 fi
 
 echo "Job finished: $(date)"

entrypoint.sh

@@ -3,6 +3,7 @@
 set -e
 
 export MONGO_URI=${MONGO_URI:-mongodb://mongo:27017}
+export TARGET_FOLDER=${TARGET_FOLDER-/backup}   # can be set to null
 
 # Optional env vars:
 # - CRON_SCHEDULE
@@ -16,6 +17,9 @@ if [[ "$CRON_SCHEDULE" ]]; then
         mkfifo "$LOGFIFO"
     fi
     CRON_ENV="MONGO_URI='$MONGO_URI'"
+    if [[ "$TARGET_FOLDER" ]]; then
+        CRON_ENV="$CRON_ENV\nTARGET_FOLDER='$TARGET_FOLDER'"
+    fi
     if [[ "$TARGET_S3_FOLDER" ]]; then
         CRON_ENV="$CRON_ENV\nTARGET_S3_FOLDER='$TARGET_S3_FOLDER'"
     fi