From d2abd360fcf7760dadb2dc7cfce968a069b89c85 Mon Sep 17 00:00:00 2001 From: Renan Rangel Date: Thu, 7 Jul 2022 18:27:24 +0100 Subject: [PATCH 1/8] add documentation about backup related flag options Signed-off-by: Renan Rangel --- .../backup-and-restore/backup-and-restore.md | 56 +++++++++++++++++++ 1 file changed, 56 insertions(+) diff --git a/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md b/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md index b06814358..dce864303 100644 --- a/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md +++ b/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md @@ -87,6 +87,58 @@ The following options can be used to configure VTTablet and Vtctld for backups: twice. + compression_level + Select what it the compression level (from `1..9`) to be used with the builtin compressors. + It doesn't have any effect if you are using an external compressor. Defaults to + 1 (fastest compression). + + + + builtin_compressor + + This sets the compression engine used to compress backups. Supported compressors are: + pgzip, pargzip, lz4 and zstd. + Defaults to pgzip.
+ Not used if --external_compressor is present. + + + + builtin_decompressor + + Set which decompressor engine to use for backups. Defaults to auto, which causes + it to be infered from the MANIFEST (for the builtinbackupengine) or from the + file extension (for the xtrabackupengine). Not used if --external_decompressor is present. + + + + external_compressor + + Instead of compressing inside the vttablet process, use the external command to + compress the input; this overrides the --builtin_compressor. The compressed stream + needs to be written to STDOUT.

+ An example command to compress with an external compressor using the fastest mode and lowest CPU priority:
+ --external_compressor "nice -n 19 pigz -1 -c" + + + + external_compressor_extension + + If using the --external_compressor_extension, this will use the correct extension when + writing the file, which can also be used by a builtin decompressor if supported (e.g. you can use an + external compressor like pigz but choose to decompress with pargzip + inside the vttablet).

+ Example: --external_compressor_extension ".gz" + + + + external_decompressor + + Use an external decompressor to process the backups. This will override --builtin_decompressor. + The decompressed stream needs to be written to STDOUT.

+ An example of how to use an external decompressor:
+ --external_decompressor "pigz -d -c" + + file_backup_storage_root For the file plugin, this identifies the root directory @@ -193,3 +245,7 @@ The backup and restore processes simultaneously copy and either compress or deco * vttablet uses the `--restore_concurrency` flag. If the network link is fast enough, the concurrency matches the CPU usage of the process during the backup or restore process. + +### Backup Compression + +WIP From eeb629bd504817dffb8b21a9bc4b7f1690a7b2b8 Mon Sep 17 00:00:00 2001 From: Renan Rangel Date: Thu, 14 Jul 2022 16:59:45 +0100 Subject: [PATCH 2/8] wip Signed-off-by: Renan Rangel --- .../backup-and-restore/backup-and-restore.md | 66 ++++++++++++------- 1 file changed, 44 insertions(+), 22 deletions(-) diff --git a/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md b/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md index dce864303..4a9ac214f 100644 --- a/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md +++ b/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md @@ -87,56 +87,49 @@ The following options can be used to configure VTTablet and Vtctld for backups: twice. - compression_level + compression-level Select what it the compression level (from `1..9`) to be used with the builtin compressors. It doesn't have any effect if you are using an external compressor. Defaults to 1 (fastest compression). - builtin_compressor + builtin-compressor This sets the compression engine used to compress backups. Supported compressors are: pgzip, pargzip, lz4 and zstd. Defaults to pgzip.
- Not used if --external_compressor is present. + Not used if --external-compressor is present. - builtin_decompressor - - Set which decompressor engine to use for backups. Defaults to auto, which causes - it to be infered from the MANIFEST (for the builtinbackupengine) or from the - file extension (for the xtrabackupengine). Not used if --external_decompressor is present. - - - - external_compressor + external-compressor Instead of compressing inside the vttablet process, use the external command to - compress the input; this overrides the --builtin_compressor. The compressed stream + compress the input; this overrides the --builtin-compressor. The compressed stream needs to be written to STDOUT.

An example command to compress with an external compressor using the fastest mode and lowest CPU priority:
- --external_compressor "nice -n 19 pigz -1 -c" + --external-compressor "nice -n 19 pigz -1 -c" - external_compressor_extension + external-compressor-extension - If using the --external_compressor_extension, this will use the correct extension when - writing the file, which can also be used by a builtin decompressor if supported (e.g. you can use an + Using the --external-compressor-extension flag will set the correct extension when + writing the file, which can also be used by a builtin decompressor if supported (e.g. you use an external compressor like pigz but choose to decompress with pargzip inside the vttablet).

- Example: --external_compressor_extension ".gz" + Example: --external-compressor-extension ".gz" - external_decompressor + external-decompressor - Use an external decompressor to process the backups. This will override --builtin_decompressor. + Use an external decompressor to process the backups. This overrides the builtin + decompressor which would be automatically select the best engine based on the MANIFEST information. The decompressed stream needs to be written to STDOUT.

An example of how to use an external decompressor:
- --external_decompressor "pigz -d -c" + --external-decompressor "pigz -d -c" @@ -248,4 +241,33 @@ If the network link is fast enough, the concurrency matches the CPU usage of the ### Backup Compression -WIP +By default, `vttablet` backups are compressed using `pargzip` that generates `gzip` compatible files. +You can select other builtin engines that are supported, or choose to use an external process to do the +compression/decompression for you. There are some advantages of doing this, like being able to set the +scheduling priority or even to choose dedicated CPU cores to do the compression, things that are not possible when runningg inside the `vttablet` process. + +The built-in supported engines are: + +__Compression:__ +- `pargzip` +- `pgzip` +- `lz4` +- `zstd` + +__Decompression:__ +- `pgzip` +- `lz4` +- `zstd` + +To change which compression engine to use, you can use the `--builtin-compressor` flag. The compression +engine is also saved to the backup manifest, which is read during the decompression process to select +the right engine to decompress. + +If you want to use an external compressor/decompressor, you can you can do this by setting: +- `--external-compressor` with the command that will actually compress the stream; +- `--external-compressor-extension` with the extension to be used. this will be saved in the backup MANIFEST; +- `--external-decompressor` with the command used to decompress the files; + +The `vttablet` process will launch the external process and pass the input stream via STDIN and expects +the process will write the compressed/decompressed stream to STDOUT. + From 467d42826bba8b941340100fb8bfd938e494a9a3 Mon Sep 17 00:00:00 2001 From: Renan Rangel Date: Mon, 22 Aug 2022 17:50:53 +0100 Subject: [PATCH 3/8] wip Signed-off-by: Renan Rangel --- .../backup-and-restore/backup-and-restore.md | 29 ++++++++++--------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md b/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md index 4a9ac214f..a45c992c9 100644 --- a/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md +++ b/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md @@ -94,12 +94,11 @@ The following options can be used to configure VTTablet and Vtctld for backups: - builtin-compressor + compression-engine-name - This sets the compression engine used to compress backups. Supported compressors are: - pgzip, pargzip, lz4 and zstd. - Defaults to pgzip.
- Not used if --external-compressor is present. + This indicates which compression engine to use. The default value is pargzip. + If using an external compressor (see below), this should be a compatible compression engine as the + value will be saved to the MANIFEST when creating the backup and can be used to decompress it. @@ -109,16 +108,16 @@ The following options can be used to configure VTTablet and Vtctld for backups: compress the input; this overrides the --builtin-compressor. The compressed stream needs to be written to STDOUT.

An example command to compress with an external compressor using the fastest mode and lowest CPU priority:
- --external-compressor "nice -n 19 pigz -1 -c" + --external-compressor "nice -n 19 pigz -1 -c"

+ If the backup is supported by one of the builtin engines, make sure to pass --compression-engine-name + so it can be restored without requiring to pass external-compressor-extension Using the --external-compressor-extension flag will set the correct extension when - writing the file, which can also be used by a builtin decompressor if supported (e.g. you use an - external compressor like pigz but choose to decompress with pargzip - inside the vttablet).

+ writing the file. Only used for the xtrabackupengine.

Example: --external-compressor-extension ".gz" @@ -249,7 +248,7 @@ scheduling priority or even to choose dedicated CPU cores to do the compression, The built-in supported engines are: __Compression:__ -- `pargzip` +- `pargzip` (default) - `pgzip` - `lz4` - `zstd` @@ -259,13 +258,15 @@ __Decompression:__ - `lz4` - `zstd` -To change which compression engine to use, you can use the `--builtin-compressor` flag. The compression -engine is also saved to the backup manifest, which is read during the decompression process to select -the right engine to decompress. +To change which compression engine to use, you can use the `--compression-engine-name` flag. The compression +engine is also be saved to the backup manifest, which is read during the decompression process to select +the right engine to decompress (so even if it gets changed, the `vttablet` will still be able to restore +previous backups). If you want to use an external compressor/decompressor, you can you can do this by setting: - `--external-compressor` with the command that will actually compress the stream; -- `--external-compressor-extension` with the extension to be used. this will be saved in the backup MANIFEST; +- `--external-compressor-extension` (only if using xtrabackupengine): this will let you use the extension of the file saved +- `--compression-engine-name` with the compatible engine that can decompress it. This will be saved in the backup MANIFEST; If you don't add this, backups won't be able to restore unless you pass the parameter below: - `--external-decompressor` with the command used to decompress the files; The `vttablet` process will launch the external process and pass the input stream via STDIN and expects From b70805617dec977003d9309679fd0b56b4e0bbdf Mon Sep 17 00:00:00 2001 From: Renan Rangel Date: Thu, 25 Aug 2022 16:24:40 +0100 Subject: [PATCH 4/8] PR feedback Signed-off-by: Renan Rangel --- .../backup-and-restore/backup-and-restore.md | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md b/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md index a45c992c9..1d04f3dcd 100644 --- a/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md +++ b/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md @@ -88,7 +88,7 @@ The following options can be used to configure VTTablet and Vtctld for backups: compression-level - Select what it the compression level (from `1..9`) to be used with the builtin compressors. + Select what is the compression level (from `1..9`) to be used with the builtin compressors. It doesn't have any effect if you are using an external compressor. Defaults to 1 (fastest compression). @@ -105,11 +105,10 @@ The following options can be used to configure VTTablet and Vtctld for backups: external-compressor Instead of compressing inside the vttablet process, use the external command to - compress the input; this overrides the --builtin-compressor. The compressed stream - needs to be written to STDOUT.

+ compress the input. The compressed stream needs to be written to STDOUT.

An example command to compress with an external compressor using the fastest mode and lowest CPU priority:
--external-compressor "nice -n 19 pigz -1 -c"

- If the backup is supported by one of the builtin engines, make sure to pass --compression-engine-name + If the backup is supported by one of the builtin engines, make sure to use --compression-engine-name so it can be restored without requiring to pass @@ -259,16 +258,24 @@ __Decompression:__ - `zstd` To change which compression engine to use, you can use the `--compression-engine-name` flag. The compression -engine is also be saved to the backup manifest, which is read during the decompression process to select +engine will also be saved to the backup manifest, which is read during the decompression process to select the right engine to decompress (so even if it gets changed, the `vttablet` will still be able to restore previous backups). If you want to use an external compressor/decompressor, you can you can do this by setting: - `--external-compressor` with the command that will actually compress the stream; - `--external-compressor-extension` (only if using xtrabackupengine): this will let you use the extension of the file saved -- `--compression-engine-name` with the compatible engine that can decompress it. This will be saved in the backup MANIFEST; If you don't add this, backups won't be able to restore unless you pass the parameter below: +- `--compression-engine-name` with the compatible engine that can decompress it. Use `external` if you are using an external engine not included in the above supported list. This value will be saved to the backup +MANIFEST; If it is not added (or engine is `external`), backups won't be able to restore unless you pass the parameter below: - `--external-decompressor` with the command used to decompress the files; The `vttablet` process will launch the external process and pass the input stream via STDIN and expects the process will write the compressed/decompressed stream to STDOUT. +If you are using an external compressor and want to move to a builtin engine: +- If the engine is supported according to the list above, you just need to make sure your `--compression-engine-name` is correct and you can remove +the `--external-compressor` parameter +- If you want to move away from an unsupported engine to a builtin one, then you have to: + - First change the `--compression-engine-name` to a supported one and remove the `--external-compressor` + - Once the first backup is completed, you can then remove `--external-decompressor` + - After this all new backups will be done using the new engine. Older backup will still required the `--external-decompressor` From ba20a239caf383f1c7339a1d0c3e97170c5f9df4 Mon Sep 17 00:00:00 2001 From: Renan Rangel Date: Sun, 28 Aug 2022 11:04:00 +0100 Subject: [PATCH 5/8] fix typo Signed-off-by: Renan Rangel --- .../operating-vitess/backup-and-restore/backup-and-restore.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md b/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md index 1d04f3dcd..867d70ebc 100644 --- a/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md +++ b/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md @@ -262,7 +262,7 @@ engine will also be saved to the backup manifest, which is read during the decom the right engine to decompress (so even if it gets changed, the `vttablet` will still be able to restore previous backups). -If you want to use an external compressor/decompressor, you can you can do this by setting: +If you want to use an external compressor/decompressor, you can do this by setting: - `--external-compressor` with the command that will actually compress the stream; - `--external-compressor-extension` (only if using xtrabackupengine): this will let you use the extension of the file saved - `--compression-engine-name` with the compatible engine that can decompress it. Use `external` if you are using an external engine not included in the above supported list. This value will be saved to the backup From 59ee01d7150c678c87f2be3ab264239b78b5ef7b Mon Sep 17 00:00:00 2001 From: Renan Rangel Date: Wed, 31 Aug 2022 18:41:23 +0100 Subject: [PATCH 6/8] wip Signed-off-by: Renan Rangel --- .../operating-vitess/backup-and-restore/backup-and-restore.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md b/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md index 867d70ebc..febd97465 100644 --- a/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md +++ b/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md @@ -109,7 +109,7 @@ The following options can be used to configure VTTablet and Vtctld for backups: An example command to compress with an external compressor using the fastest mode and lowest CPU priority:
--external-compressor "nice -n 19 pigz -1 -c"

If the backup is supported by one of the builtin engines, make sure to use --compression-engine-name - so it can be restored without requiring to pass + so it can be restored without requiring --external-decompressor to be defined. From 79bc56d48bcb6c920c61dd0000cc5b4e3ca03f33 Mon Sep 17 00:00:00 2001 From: Renan Rangel Date: Thu, 8 Sep 2022 18:22:11 +0100 Subject: [PATCH 7/8] Update content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md Co-authored-by: Deepthi Sigireddi Signed-off-by: Renan Rangel --- .../operating-vitess/backup-and-restore/backup-and-restore.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md b/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md index febd97465..f42ffe8e6 100644 --- a/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md +++ b/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md @@ -278,4 +278,4 @@ the `--external-compressor` parameter - If you want to move away from an unsupported engine to a builtin one, then you have to: - First change the `--compression-engine-name` to a supported one and remove the `--external-compressor` - Once the first backup is completed, you can then remove `--external-decompressor` - - After this all new backups will be done using the new engine. Older backup will still required the `--external-decompressor` + - After this all new backups will be done using the new engine. Restoring an older backup will still require the `--external-decompressor` flag to be provided From 8f71e6caa5a7d4dc97db54dd0426721151e60ff0 Mon Sep 17 00:00:00 2001 From: Renan Rangel Date: Thu, 8 Sep 2022 18:22:21 +0100 Subject: [PATCH 8/8] Update content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md Co-authored-by: Deepthi Sigireddi Signed-off-by: Renan Rangel --- .../operating-vitess/backup-and-restore/backup-and-restore.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md b/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md index f42ffe8e6..6ccdb488f 100644 --- a/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md +++ b/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md @@ -242,7 +242,7 @@ If the network link is fast enough, the concurrency matches the CPU usage of the By default, `vttablet` backups are compressed using `pargzip` that generates `gzip` compatible files. You can select other builtin engines that are supported, or choose to use an external process to do the compression/decompression for you. There are some advantages of doing this, like being able to set the -scheduling priority or even to choose dedicated CPU cores to do the compression, things that are not possible when runningg inside the `vttablet` process. +scheduling priority or even to choose dedicated CPU cores to do the compression, things that are not possible when running inside the `vttablet` process. The built-in supported engines are: