diff --git a/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md b/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md index b06814358..6ccdb488f 100644 --- a/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md +++ b/content/en/docs/15.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md @@ -87,6 +87,49 @@ The following options can be used to configure VTTablet and Vtctld for backups: twice. + compression-level + Select what is the compression level (from `1..9`) to be used with the builtin compressors. + It doesn't have any effect if you are using an external compressor. Defaults to + 1 (fastest compression). + + + + compression-engine-name + + This indicates which compression engine to use. The default value is pargzip. + If using an external compressor (see below), this should be a compatible compression engine as the + value will be saved to the MANIFEST when creating the backup and can be used to decompress it. + + + + external-compressor + + Instead of compressing inside the vttablet process, use the external command to + compress the input. The compressed stream needs to be written to STDOUT.

+ An example command to compress with an external compressor using the fastest mode and lowest CPU priority:
+ --external-compressor "nice -n 19 pigz -1 -c"

+ If the backup is supported by one of the builtin engines, make sure to use --compression-engine-name + so it can be restored without requiring --external-decompressor to be defined. + + + + external-compressor-extension + + Using the --external-compressor-extension flag will set the correct extension when + writing the file. Only used for the xtrabackupengine.

+ Example: --external-compressor-extension ".gz" + + + + external-decompressor + + Use an external decompressor to process the backups. This overrides the builtin + decompressor which would be automatically select the best engine based on the MANIFEST information. + The decompressed stream needs to be written to STDOUT.

+ An example of how to use an external decompressor:
+ --external-decompressor "pigz -d -c" + + file_backup_storage_root For the file plugin, this identifies the root directory @@ -193,3 +236,46 @@ The backup and restore processes simultaneously copy and either compress or deco * vttablet uses the `--restore_concurrency` flag. If the network link is fast enough, the concurrency matches the CPU usage of the process during the backup or restore process. + +### Backup Compression + +By default, `vttablet` backups are compressed using `pargzip` that generates `gzip` compatible files. +You can select other builtin engines that are supported, or choose to use an external process to do the +compression/decompression for you. There are some advantages of doing this, like being able to set the +scheduling priority or even to choose dedicated CPU cores to do the compression, things that are not possible when running inside the `vttablet` process. + +The built-in supported engines are: + +__Compression:__ +- `pargzip` (default) +- `pgzip` +- `lz4` +- `zstd` + +__Decompression:__ +- `pgzip` +- `lz4` +- `zstd` + +To change which compression engine to use, you can use the `--compression-engine-name` flag. The compression +engine will also be saved to the backup manifest, which is read during the decompression process to select +the right engine to decompress (so even if it gets changed, the `vttablet` will still be able to restore +previous backups). + +If you want to use an external compressor/decompressor, you can do this by setting: +- `--external-compressor` with the command that will actually compress the stream; +- `--external-compressor-extension` (only if using xtrabackupengine): this will let you use the extension of the file saved +- `--compression-engine-name` with the compatible engine that can decompress it. Use `external` if you are using an external engine not included in the above supported list. This value will be saved to the backup +MANIFEST; If it is not added (or engine is `external`), backups won't be able to restore unless you pass the parameter below: +- `--external-decompressor` with the command used to decompress the files; + +The `vttablet` process will launch the external process and pass the input stream via STDIN and expects +the process will write the compressed/decompressed stream to STDOUT. + +If you are using an external compressor and want to move to a builtin engine: +- If the engine is supported according to the list above, you just need to make sure your `--compression-engine-name` is correct and you can remove +the `--external-compressor` parameter +- If you want to move away from an unsupported engine to a builtin one, then you have to: + - First change the `--compression-engine-name` to a supported one and remove the `--external-compressor` + - Once the first backup is completed, you can then remove `--external-decompressor` + - After this all new backups will be done using the new engine. Restoring an older backup will still require the `--external-decompressor` flag to be provided