Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes #27 and corrects comments from pull #27 #31

Merged
merged 5 commits into from
Jun 15, 2018
Merged

Fixes #27 and corrects comments from pull #27 #31

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
83413ac
Close #27 Prepost strategies refactored taking in consideration the c…
SrZorro Jun 8, 2018
21ea207
Updated MySQL prepost strategy with changes from #27. Also corrected …
SrZorro Jun 8, 2018
12f7339
Fixes #33 #30 dependency problem.
Jun 11, 2018
5e3caa1
Fixed review in #31
SrZorro Jun 14, 2018
e8cfe65
Prepost strategy for MongoDB
SrZorro Jun 14, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,11 +46,11 @@ RUN apk upgrade --update && \
touch /etc/volumerize/remove-all-inc-of-but-n-full /etc/volumerize/remove-all-but-n-full /etc/volumerize/startContainers /etc/volumerize/stopContainers \
/etc/volumerize/backup /etc/volumerize/backupIncremental /etc/volumerize/backupFull /etc/volumerize/restore \
/etc/volumerize/periodicBackup /etc/volumerize/verify /etc/volumerize/cleanup /etc/volumerize/remove-older-than /etc/volumerize/cleanCacheLocks \
/etc/volumerize/postexecute /etc/volumerize/prexecute && \
/etc/volumerize/prepoststrategy && \
chmod +x /etc/volumerize/remove-all-inc-of-but-n-full /etc/volumerize/remove-all-but-n-full /etc/volumerize/startContainers /etc/volumerize/stopContainers \
/etc/volumerize/backup /etc/volumerize/backupIncremental /etc/volumerize/backupFull /etc/volumerize/restore \
/etc/volumerize/periodicBackup /etc/volumerize/verify /etc/volumerize/cleanup /etc/volumerize/remove-older-than /etc/volumerize/cleanCacheLocks \
/etc/volumerize/postexecute /etc/volumerize/prexecute && \
/etc/volumerize/prepoststrategy && \
# Install Jobber
export JOBBER_HOME=/tmp/jobber && \
export JOBBER_LIB=$JOBBER_HOME/lib && \
Expand Down
18 changes: 12 additions & 6 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -407,13 +407,20 @@ $ docker run -d \

# Post scripts and pre scripts (prepost strategies)

If you mount a folder in `/prexecute` containing `.sh` files, they will be executed in alphabetical order before the backup and after the docker stop process.

If you mount a folder in `/postexecute` containing `.sh` files, they will be executed in alphabetical order after the backup and before the docker start process.
Pre-scripts must be located at `/preexecute/$duplicity_action/$your_scripts_here`.

In this scripts the global variable `BACKUP_TYPE` contains the type of backup or restore, so it will contain one of this options `backup, backupIncremental, backupFull or restore` so you can customize your scripts to do one thing or another depending of the type of backup or if its a restore.
Post-scripts must be located at `/postexecute/$duplicity_action/$your_scripts_here`.

For an example and implementation for mysqldump and restore look at /prepost_strategies/mysql/README.md
`$duplicity_action` folder must be named `backup`, `restore` or `verify`.

> Note: `backup` action is the same for the scripts `backup`, `backupFull`, `backupIncremental` and `periodicBackup`.

All `.sh` files located in the `$duplicity_action` folder will be executed in alphabetical order.

When using prepost strategies, this will be the execution flow: `pre-scripts -> stop containers -> duplicity action -> start containers -> post-scripts`.

Some premade strategies are available at [prepost strategies](prepost_strategies).

# Container Scripts

Expand All @@ -431,8 +438,7 @@ This image creates at container startup some convenience scripts.
| stopContainers | Stops the specified Docker containers |
| remove-older-than | Delete older backups |
| cleanCacheLocks | Cleanup of old Cache locks. |
| prexecute | Execute all .sh files in /prexecute folder. |
| postexecute | Execute all .sh files in /postexecute folder. |
| prepoststrategy `$path` | Execute all `.sh` files in `$path` folder in alphabetical order. |

Example triggering script inside running container:

Expand Down
4 changes: 4 additions & 0 deletions imagescripts/create_jobber.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,13 @@ cat > ${JOBBER_SCRIPT_DIR}/periodicBackup <<_EOF_

set -o errexit

PREPOSTSTRATEGY=/preexecute/backup
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Magic variable. How and where will this be interpreted?

Better: Two variables DUPLICITY_ACTION, e.g. backup, verify, restore. Then EXECUTION_PHASE, e.g. preAction and postAction.

Variables will be passed directly: prepoststrategy $DUPLICITY_ACTION $EXECUTION_PHASE.

The logic using those variables will be implemented inside prepoststrategy.

This is more maintainable for me.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The magic variable was used inside the prepoststrategy script and was the path to the folder to execute.

But you are right, better with two variables, but instead of declaring the variables in the backup, backupRestore, periodicBackup, etc scripts im going to change it so instead of declaring the variable and then using it in the prepoststrategy script, the prepoststrategy script is going to accept two arguments, execution phase (preAction and postAction) and duplicity action (backup, restore, verify).

This way we remove duplicated code and the user can execute prepost strategies in a more readeable way, example: prepoststrategy preAction verify instead of the old prepoststrategy /preexecute/verify

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Last commit should have this fixed.

sourc ${VOLUMERIZE_SCRIPT_DIR}/prepoststrategy
source ${VOLUMERIZE_SCRIPT_DIR}/stopContainers
${DUPLICITY_COMMAND} ${JOBBER_PARAMETER_PROXY} ${DUPLICITY_MODE} ${DUPLICITY_OPTIONS} ${VOLUMERIZE_INCUDES} ${VOLUMERIZE_SOURCE} ${VOLUMERIZE_TARGET}
source ${VOLUMERIZE_SCRIPT_DIR}/startContainers
PREPOSTSTRATEGY=/postexecute/backup
source ${VOLUMERIZE_SCRIPT_DIR}/prepoststrategy
_EOF_

JOBBER_CRON_SCHEDULE='0 0 4 * * *'
Expand Down
57 changes: 24 additions & 33 deletions imagescripts/create_scripts.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,13 @@ source $CUR_DIR/base.sh

readonly PARAMETER_PROXY='$@'

cat > ${VOLUMERIZE_SCRIPT_DIR}/prexecute << '_EOF_'
cat > ${VOLUMERIZE_SCRIPT_DIR}/prepoststrategy << '_EOF_'
#!/bin/bash

set -o errexit

if [ -d "/prexecute" ]; then
for f in /prexecute/*; do
if [ -d "$PREPOSTSTRATEGY" ]; then
for f in $PREPOSTSTRATEGY/*; do
case "$f" in
*.sh) echo "running $f"; . "$f" ;;
*) echo "ignoring $f" ;;
Expand All @@ -24,81 +24,72 @@ if [ -d "/prexecute" ]; then
fi
_EOF_

cat > ${VOLUMERIZE_SCRIPT_DIR}/postexecute << '_EOF_'
#!/bin/bash

set -o errexit

if [ -d "/postexecute" ]; then
for f in /postexecute/*; do
case "$f" in
*.sh) echo "running $f"; . "$f" ;;
*) echo "ignoring $f" ;;
esac
echo
done
fi
_EOF_


cat > ${VOLUMERIZE_SCRIPT_DIR}/backup <<_EOF_
#!/bin/bash

set -o errexit

PREPOSTSTRATEGY=/preexecute/backup
source ${VOLUMERIZE_SCRIPT_DIR}/prepoststrategy
source ${VOLUMERIZE_SCRIPT_DIR}/stopContainers
export BACKUP_TYPE=\$(basename -- "\$0")
source ${VOLUMERIZE_SCRIPT_DIR}/prexecute
${DUPLICITY_COMMAND} ${PARAMETER_PROXY} ${DUPLICITY_OPTIONS} ${VOLUMERIZE_INCUDES} ${VOLUMERIZE_SOURCE} ${VOLUMERIZE_TARGET}
source ${VOLUMERIZE_SCRIPT_DIR}/postexecute
source ${VOLUMERIZE_SCRIPT_DIR}/startContainers
PREPOSTSTRATEGY=/postexecute/backup
source ${VOLUMERIZE_SCRIPT_DIR}/prepoststrategy
_EOF_

cat > ${VOLUMERIZE_SCRIPT_DIR}/backupIncremental <<_EOF_
#!/bin/bash

set -o errexit

PREPOSTSTRATEGY=/preexecute/backup
source ${VOLUMERIZE_SCRIPT_DIR}/prepoststrategy
source ${VOLUMERIZE_SCRIPT_DIR}/stopContainers
export BACKUP_TYPE=\$(basename -- "\$0")
source ${VOLUMERIZE_SCRIPT_DIR}/prexecute
${DUPLICITY_COMMAND} incremental ${PARAMETER_PROXY} ${DUPLICITY_OPTIONS} ${VOLUMERIZE_INCUDES} ${VOLUMERIZE_SOURCE} ${VOLUMERIZE_TARGET}
source ${VOLUMERIZE_SCRIPT_DIR}/postexecute
source ${VOLUMERIZE_SCRIPT_DIR}/startContainers
PREPOSTSTRATEGY=/postexecute/backup
source ${VOLUMERIZE_SCRIPT_DIR}/prepoststrategy
_EOF_

cat > ${VOLUMERIZE_SCRIPT_DIR}/backupFull <<_EOF_
#!/bin/bash

set -o errexit

PREPOSTSTRATEGY=/preexecute/backup
source ${VOLUMERIZE_SCRIPT_DIR}/prepoststrategy
source ${VOLUMERIZE_SCRIPT_DIR}/stopContainers
export BACKUP_TYPE=\$(basename -- "\$0")
source ${VOLUMERIZE_SCRIPT_DIR}/prexecute
${DUPLICITY_COMMAND} full ${PARAMETER_PROXY} ${DUPLICITY_OPTIONS} ${VOLUMERIZE_INCUDES} ${VOLUMERIZE_SOURCE} ${VOLUMERIZE_TARGET}
source ${VOLUMERIZE_SCRIPT_DIR}/postexecute
source ${VOLUMERIZE_SCRIPT_DIR}/startContainers
PREPOSTSTRATEGY=/postexecute/backup
source ${VOLUMERIZE_SCRIPT_DIR}/prepoststrategy
_EOF_

cat > ${VOLUMERIZE_SCRIPT_DIR}/restore <<_EOF_
#!/bin/bash

set -o errexit

PREPOSTSTRATEGY=/preexecute/restore
source ${VOLUMERIZE_SCRIPT_DIR}/prepoststrategy
source ${VOLUMERIZE_SCRIPT_DIR}/stopContainers
export BACKUP_TYPE=\$(basename -- "\$0")
source ${VOLUMERIZE_SCRIPT_DIR}/prexecute
${DUPLICITY_COMMAND} restore --force ${PARAMETER_PROXY} ${DUPLICITY_OPTIONS} ${VOLUMERIZE_INCUDES} ${VOLUMERIZE_TARGET} ${VOLUMERIZE_SOURCE}
source ${VOLUMERIZE_SCRIPT_DIR}/postexecute
source ${VOLUMERIZE_SCRIPT_DIR}/startContainers
PREPOSTSTRATEGY=/postexecute/restore
source ${VOLUMERIZE_SCRIPT_DIR}/prepoststrategy
_EOF_

cat > ${VOLUMERIZE_SCRIPT_DIR}/verify <<_EOF_
#!/bin/bash

set -o errexit

exec ${DUPLICITY_COMMAND} verify --compare-data ${PARAMETER_PROXY} ${DUPLICITY_OPTIONS} ${VOLUMERIZE_INCUDES} ${VOLUMERIZE_TARGET} ${VOLUMERIZE_SOURCE}
PREPOSTSTRATEGY=/preexecute/verify
source ${VOLUMERIZE_SCRIPT_DIR}/prepoststrategy
${DUPLICITY_COMMAND} verify --compare-data ${PARAMETER_PROXY} ${DUPLICITY_OPTIONS} ${VOLUMERIZE_INCUDES} ${VOLUMERIZE_TARGET} ${VOLUMERIZE_SOURCE}
PREPOSTSTRATEGY=/postexecute/verify
source ${VOLUMERIZE_SCRIPT_DIR}/prepoststrategy
_EOF_

cat > ${VOLUMERIZE_SCRIPT_DIR}/cleanup <<_EOF_
Expand Down
2 changes: 1 addition & 1 deletion prepost_strategies/mysql/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ services:
- MYSQL_DATABASE=somedatabase
volumes:
- volumerize-cache:/volumerize-cache
- .backup:/backup
- backup:/backup
depends_on:
- mariadb

Expand Down
9 changes: 0 additions & 9 deletions prepost_strategies/mysql/postexecute/mysqlimport.sh
View file
Open in desktop

This file was deleted.

7 changes: 7 additions & 0 deletions prepost_strategies/mysql/postexecute/restore/mysqlimport.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
source /preexecute/utils/check-env.sh

check_env "mysqlimport" "MYSQL_PASSWORD" "MYSQL_USERNAME" "MYSQL_HOST" "MYSQL_DATABASE"

echo "mysql import starts"
pv ${VOLUMERIZE_SOURCE}/dump-${MYSQL_DATABASE}.sql | mysql -u ${MYSQL_USERNAME} -p${MYSQL_PASSWORD} $MYSQL_DATABASE
echo "Import done"
12 changes: 12 additions & 0 deletions prepost_strategies/mysql/preexecute/backup/mysqldump.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
source /preexecute/utils/check-env.sh

check_env "Mysqldump" "MYSQL_PASSWORD" "MYSQL_USERNAME" "MYSQL_HOST" "MYSQL_DATABASE"

echo "Creating $VOLUMERIZE_SOURCE folder if not exists"
mkdir -p $VOLUMERIZE_SOURCE

# Based on this answer https://stackoverflow.com/a/32361604
SIZE_BYTES=$(mysql --skip-column-names -u ${MYSQL_USERNAME} -p${MYSQL_PASSWORD} $MYSQL_DATABASE -e "SELECT ROUND(SUM(data_length * 0.8), 0) FROM information_schema.TABLES WHERE table_schema='${MYSQL_DATABASE}';")

echo "mysqldump starts (Progress is aproximated)"
mysqldump --databases "${MYSQL_DATABASE}" --single-transaction --add-drop-database --user="${MYSQL_USERNAME}" --password="${MYSQL_PASSWORD}" --host="${MYSQL_HOST}" | pv --progress --size "$SIZE_BYTES" > ${VOLUMERIZE_SOURCE}/dump-${MYSQL_DATABASE}.sql
10 changes: 5 additions & 5 deletions ...tegies/mysql/prexecute/utils/check-env.sh → ...egies/mysql/preexecute/utils/check-env.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@
# @param1 - Name of the failed environment variable
##
function _check_env_failed {
echo "[!] ENV VAR $1 IS NOT SET"
printf "\n====ENVIRONMENT VARIABLES FAILED====\n\n"
echo "Environment variable $1 is not set."
echo "Environment variables failed, exit 1"
exit 1
}

Expand All @@ -17,7 +17,7 @@ function _check_env_failed {
# @param1 - Name of the environment variable
##
function _check_env_ok {
echo "Env var $1 OK"
echo "Env var $1 ok."
}

##
Expand All @@ -26,7 +26,7 @@ function _check_env_ok {
# @param2 to ∞ - Environment variables to check
##
function check_env {
printf "\n====CHECKING ENVIRONMENT VARIABLES FOR $1====\n\n"
echo "Checking environment variables for $1."

for e_var in "$@"; do
if [ $e_var = $1 ]; then continue; fi # Jump first arg
Expand All @@ -39,5 +39,5 @@ function check_env {
fi

done
printf "\n====ENVIRONMENT VARIABLES OK====\n\n"
echo "Environment variables ok."
}
11 changes: 0 additions & 11 deletions prepost_strategies/mysql/prexecute/mysqldump.sh
View file
Open in desktop

This file was deleted.