From dca8abb0b60c47b3dc6ed3d31010c8c88f6e4105 Mon Sep 17 00:00:00 2001 From: Max Englander Date: Tue, 17 Jan 2023 19:15:31 -0500 Subject: [PATCH 1/3] content/en/docs/16.0: add reference program docs for vtbackup Signed-off-by: Max Englander --- .../docs/16.0/reference/programs/vtbackup.md | 232 ++++++++++++++++++ 1 file changed, 232 insertions(+) create mode 100644 content/en/docs/16.0/reference/programs/vtbackup.md diff --git a/content/en/docs/16.0/reference/programs/vtbackup.md b/content/en/docs/16.0/reference/programs/vtbackup.md new file mode 100644 index 000000000..51dc7931b --- /dev/null +++ b/content/en/docs/16.0/reference/programs/vtbackup.md @@ -0,0 +1,232 @@ +--- +title: vtbackup +description: The Vitess Batch Command for Backup Maintenance +--- + +`vtbackup` is a batch comand to perform a single pass of backup maintenance for a shard. + +When run periodically for each shard, `vtbackup` can ensure these configurable policies: + - There is always a recent backup for the shard. + - Old backups for the shard are removed. + +Whatever system launches `vtbackup` is responsible for the following: + - Running `vtbackup` with similar flags that would be used for a vttablet and + mysqlctld in the target shard to be backed up. + - Provisioning as much disk space for `vtbackup` as would be given to vttablet. + The data directory MUST be empty at startup. Do NOT reuse a persistent disk. + - Running `vtbackup` periodically for each shard, for each backup storage location. + - Ensuring that at most one instance runs at a time for a given pair of shard + and backup storage location. + - Retrying `vtbackup` if it fails. + - Alerting human operators if the failure is persistent. + +## Example Usage + +The following example launches the `vtbackup`. + +```bash +export TOPOLOGY_FLAGS="--topo_implementation etcd2 --topo_global_server_address localhost:2379 --topo_global_root /vitess/global" +export VTROOT="/tmp" + +mkdir -p $VTROOT/{backups,socket} + +vtbackup \ + $TOPOLOGY_FLAGS \ + --backup_storage_implementation file \ + --file_backup_storage_root $VTROOT/backups/vitess-local/ \ + --logtostderr=true \ + --mysql_socket $VTROOT/socket/mysql.sock \ + --port 15500 \ + --mysql_port 33306 \ + --init_shard=0 \ + --init_keyspace=commerce \ + --db_dba_user=vt_dba +``` + +While it is running, `vtbackup` serves debugging info and metrics on port `15500`, and starts a mysqld daemon serving on port `33306`. + +## Options + +| Name | Type | Definition | +| :------------------------------------ | :--------- | :----------------------------------------------------------------------------------------- | +| --allow_first_backup | boolean | Allow this job to take the first backup of an existing shard. | +| --alsologtostderr | boolean | log to standard error as well as files | +| --azblob_backup_account_key_file | string | Path to a file containing the Azure Storage account key; if this flag is unset, the environment variable VT_AZBLOB_ACCOUNT_KEY will be used as the key itself (NOT a file path). | +| --azblob_backup_account_name | string | Azure Storage Account name for backups; if this flag is unset, the environment variable VT_AZBLOB_ACCOUNT_NAME will be used. | +| --azblob_backup_container_name | string | Azure Blob Container Name. | +| --azblob_backup_parallelism | int | Azure Blob operation parallelism (requires extra memory when increased). (default 1) | +| --azblob_backup_storage_root | string | Root prefix for all backup-related Azure Blobs; this should exclude both initial and trailing '/' (e.g. just 'a/b' not '/a/b/'). | +| --backup_engine_implementation | string | Specifies which implementation to use for creating new backups (builtin or xtrabackup). Restores will always be done with whichever engine created a given backup. (default "builtin") | +| --backup_storage_block_size | int | if backup_storage_compress is true, backup_storage_block_size sets the byte size for each block while compressing (default is 250000). (default 250000) | +| --backup_storage_compress | boolean | if set, the backup files will be compressed. (default true) | +| --backup_storage_implementation | string | Which backup storage implementation to use for creating and restoring backups. | +| --backup_storage_number_blocks | int | if backup_storage_compress is true, backup_storage_number_blocks sets the number of blocks that can be processed, at once, before the writer blocks, during compression (default is 2). It should be equal to the number of CPUs available for compression. (default 2) | +| --ceph_backup_storage_config | string | Path to JSON config file for ceph backup storage. (default "ceph_backup_config.json") | +| --compression-engine-name | string | compressor engine used for compression. (default "pargzip") | +| --compression-level | int | what level to pass to the compressor. (default 1) | +| --concurrency | int | (init restore parameter) how many concurrent files to restore at once (default 4) | +| --consul_auth_static_file | string | JSON File to read the topos/tokens from. | +| --db-credentials-file | string | db credentials file; send SIGHUP to reload this file | +| --db-credentials-server | string | db credentials server type ('file' - file implementation; 'vault' - HashiCorp Vault implementation) (default "file") | +| --db-credentials-vault-addr | string | URL to Vault server | +| --db-credentials-vault-path | string | Vault path to credentials JSON blob, e.g.: secret/data/prod/dbcreds | +| --db-credentials-vault-role-mountpoint | string | Vault AppRole mountpoint; can also be passed using VAULT_MOUNTPOINT environment variable (default "approle") | +| --db-credentials-vault-role-secretidfile | string Path to file containing Vault AppRole secret_id; can also be passed using VAULT_SECRETID environment variable | +| --db-credentials-vault-roleid | string | Vault AppRole id; can also be passed using VAULT_ROLEID environment variable | +| --db-credentials-vault-timeout | duration | Timeout for vault API operations (default 10s) | +| --db-credentials-vault-tls-ca | string | Path to CA PEM for validating Vault server certificate | +| --db-credentials-vault-tokenfile | string | Path to file containing Vault auth token; token can also be passed using VAULT_TOKEN environment variable | +| --db-credentials-vault-ttl | duration | How long to cache DB credentials from the Vault server (default 30m0s) | +| --db_allprivs_password | string | db allprivs password | +| --db_allprivs_use_ssl | boolean | Set this flag to false to make the allprivs connection to not use ssl (default true) | +| --db_allprivs_user | string | db allprivs user userKey (default "vt_allprivs") | +| --db_app_password | string | db app password | +| --db_app_use_ssl | boolean | Set this flag to false to make the app connection to not use ssl (default true) | +| --db_app_user | string | db app user userKey (default "vt_app") | +| --db_appdebug_password | string | db appdebug password | +| --db_appdebug_use_ssl | boolean | Set this flag to false to make the appdebug connection to not use ssl (default true) | +| --db_appdebug_user | string | db appdebug user userKey (default "vt_appdebug") | +| --db_charset | string | Character set used for this tablet. (default "utf8mb4") | +| --db_conn_query_info | boolean | enable parsing and processing of QUERY_OK info fields | +| --db_connect_timeout_ms | int | connection timeout to mysqld in milliseconds (0 for no timeout) | +| --db_dba_password | string | db dba password | +| --db_dba_use_ssl | boolean | Set this flag to false to make the dba connection to not use ssl (default true) | +| --db_dba_user | string | db dba user userKey (default "vt_dba") | +| --db_erepl_password | string | db erepl password | +| --db_erepl_use_ssl | boolean | Set this flag to false to make the erepl connection to not use ssl (default true) | +| --db_erepl_user | string | db erepl user userKey (default "vt_erepl") | +| --db_filtered_password | string | db filtered password | +| --db_filtered_use_ssl | boolean | Set this flag to false to make the filtered connection to not use ssl (default true) | +| --db_filtered_user | string | db filtered user userKey (default "vt_filtered") | +| --db_flags | uint | Flag values as defined by MySQL. | +| --db_flavor | string | Flavor overrid. Valid value is FilePos. | +| --db_host | string | The host name for the tcp connection. | +| --db_port | int | tcp port | +| --db_repl_password | string | db repl password | +| --db_repl_use_ssl | boolean | Set this flag to false to make the repl connection to not use ssl (default true) | +| --db_repl_user | string | db repl user userKey (default "vt_repl") | +| --db_server_name | string | server name of the DB we are connecting to. | +| --db_socket | string | The unix socket to connect on. If this is specified, host and port will not be used. | +| --db_ssl_ca | string | connection ssl ca | +| --db_ssl_ca_path | string | connection ssl ca path | +| --db_ssl_cert | string | connection ssl certificate | +| --db_ssl_key | string | connection ssl key | +| --db_ssl_mode | SslMode | SSL mode to connect with. One of disabled, preferred, required, verify_ca & verify_identity. | +| --db_tls_min_version | string | Configures the minimal TLS version negotiated when SSL is enabled. Defaults to TLSv1.2. Options: TLSv1.0, TLSv1.1, TLSv1.2, TLSv1.3. | +| --detach | boolean | detached mode - run backups detached from the terminal | +| --disable-redo-log | boolean | Disable InnoDB redo log during replication-from-primary phase of backup. | +| --emit_stats | boolean | If set, emit stats to push-based monitoring and stats backends | +| --external-compressor | string | command with arguments to use when compressing a backup. | +| --external-compressor-extension | string | extension to use when using an external compressor. | +| --external-decompressor | string | command with arguments to use when decompressing a backup. | +| --file_backup_storage_root | string | Root directory for the file backup storage. | +| --gcs_backup_storage_bucket | string | Google Cloud Storage bucket to use for backups. | +| --gcs_backup_storage_root | string | Root prefix for all backup-related object names. | +| --grpc_auth_static_client_creds | string | When using grpc_static_auth in the server, this file provides the credentials to use to authenticate with server. | +| --grpc_compression | string | Which protocol to use for compressing gRPC. Default: nothing. Supported: snappy | +| --grpc_enable_tracing | boolean | Enable gRPC tracing. | +| --grpc_initial_conn_window_size | int | gRPC initial connection window size | +| --grpc_initial_window_size | int | gRPC initial window size | +| --grpc_keepalive_time | duration | After a duration of this time, if the client doesn't see any activity, it pings the server to see if the transport is still alive. (default 10s) | +| --grpc_keepalive_timeout | duration | After having pinged for keepalive check, the client waits for a duration of Timeout and if no activity is seen even after that the connection is closed. (default 10s) | +| --grpc_max_message_size | int | Maximum allowed RPC message size. Larger messages will be rejected by gRPC with the error 'exceeding the max size'. (default 16777216) | +| --grpc_prometheus | boolean | Enable gRPC monitoring with Prometheus. | +| --incremental_from_pos | string | Position of previous backup. Default: empty. If given, then this backup becomes an incremental backup from given position. If value is 'auto', backup taken from last successful backup position | +| --init_db_name_override | string | (init parameter) override the name of the db used by vttablet | +| --init_db_sql_file | string | path to .sql file to run after mysql_install_db | +| --init_keyspace | string | (init parameter) keyspace to use for this tablet | +| --init_shard | string | (init parameter) shard to use for this tablet | +| --initial_backup | boolean | Instead of restoring from backup, initialize an empty database with the provided init_db_sql_file and upload a backup of that for the shard, if the shard has no backups yet. This can be used to seed a brand new shard with an initial, empty backup. If any backups already exist for the shard, this will be considered a successful no-op. This can only be done before the shard exists in topology (i.e. before any tablets are deployed). | +| --keep-alive-timeout | duration | Wait until timeout elapses after a successful backup before shutting down. | +| --keep_logs | boolean | keep logs for this long (using ctime) (zero to keep forever) | +| --keep_logs_by_mtime | duration | keep logs for this long (using mtime) (zero to keep forever) | +| --lock-timeout | duration | Maximum time for which a shard/keyspace lock can be acquired for (default 45s) | +| --log_backtrace_at | traceLocation | when logging hits line file:N, emit a stack trace (default :0) | +| --log_dir | string | If non-empty, write log files in this directory | +| --log_err_stacks | boolean | log stack traces for errors | +| --log_rotate_max_size | uint | size in bytes at which logs are rotated (glog.MaxSize) (default 1887436800) | +| --logtostderr | boolean | log to standard error instead of files | +| --min_backup_interval | duration | Only take a new backup if it's been at least this long since the most recent backup. | +| --min_retention_count | int | Always keep at least this many of the most recent backups in this backup storage location, even if some are older than the min_retention_time. This must be at least 1 since a backup must always exist to allow new backups to be made (default 1) | +| --min_retention_time | duration | Keep each old backup for at least this long before removing it. Set to 0 to disable pruning of old backups. | +| --mycnf-file | string | path to my.cnf, if reading all config params from there | +| --mycnf_bin_log_path | string | mysql binlog path | +| --mycnf_data_dir | string | data directory for mysql | +| --mycnf_error_log_path | string | mysql error log path | +| --mycnf_general_log_path | string | mysql general log path | +| --mycnf_innodb_data_home_dir | string | Innodb data home directory | +| --mycnf_innodb_log_group_home_dir | string | Innodb log group home directory | +| --mycnf_master_info_file | string | mysql master.info file | +| --mycnf_mysql_port | int | port mysql is listening on | +| --mycnf_pid_file | string | mysql pid file | +| --mycnf_relay_log_index_path | string | mysql relay log index path | +| --mycnf_relay_log_info_path | string | mysql relay log info path | +| --mycnf_relay_log_path | string | mysql relay log path | +| --mycnf_secure_file_priv | string | mysql path for loading secure files | +| --mycnf_server_id | int | mysql server id of the server (if specified, mycnf-file will be ignored) | +| --mycnf_slow_log_path | string | mysql slow query log path | +| --mycnf_socket_file | string | mysql socket file | +| --mycnf_tmp_dir | string | mysql tmp directory | +| --mysql_port | int | mysql port (default 3306) | +| --mysql_server_version | string | MySQL server version to advertise. | +| --mysql_socket | string | path to the mysql socket | +| --mysql_timeout | duration | how long to wait for mysqld startup (default 5m0s) | +| --port | int | port for the server | +| --pprof | strings | enable profiling | +| --purge_logs_interval | boolean | how often try to remove old logs (default 1h0m0s) | +| --remote_operation_timeout | duration | time to wait for a remote operation (default 15s) | +| --restart_before_backup | boolean | Perform a mysqld clean/full restart after applying binlogs, but before taking the backup. Only makes sense to work around xtrabackup bugs. | +| --s3_backup_aws_endpoint | string | endpoint of the S3 backend (region must be provided). | +| --s3_backup_aws_region | string | AWS region to use. (default "us-east-1") | +| --s3_backup_aws_retries | int | AWS request retries. (default -1) | +| --s3_backup_force_path_style | | force the s3 path style. | +| --s3_backup_log_level | string | determine the S3 loglevel to use from LogOff, LogDebug, LogDebugWithSigning, LogDebugWithHTTPBody, LogDebugWithRequestRetries, LogDebugWithRequestErrors. (default "LogOff") | +| --s3_backup_server_side_encryption | string | server-side encryption algorithm (e.g., AES256, aws:kms, sse_c:/path/to/key/file). | +| --s3_backup_storage_bucket | string | S3 bucket to use for backups. | +| --s3_backup_storage_root | string | root prefix for all backup-related object names. | +| --s3_backup_tls_skip_verify_cert | | skip the 'certificate is valid' check for SSL connections. | +| --security_policy | string | the name of a registered security policy to use for controlling access to URLs - empty means allow all for anyone (built-in policies: deny-all, read-only) | +| --sql-max-length-errors | int | truncate queries in error logs to the given length (default unlimited) | +| --sql-max-length-ui | int | truncate queries in debug UIs to the given length (default 512) (default 512) | +| --stats_backend | string | The name of the registered push-based monitoring/stats backend to use | +| --stats_combine_dimensions | string | List of dimensions to be combined into a single "all" value in exported stats vars | +| --stats_common_tags | strings | Comma-separated list of common tags for the stats backend. It provides both label and values. Example: label1:value1,label2:value2 | +| --stats_drop_variables | string | Variables to be dropped from the list of exported variables. | +| --stats_emit_period | duration | Interval between emitting stats to all registered backends (default 1m0s) | +| --stderrthreshold | severity | logs at or above this threshold go to stderr (default 1) | +| --tablet_manager_grpc_ca | string | the server ca to use to validate servers when connecting | +| --tablet_manager_grpc_cert | string | the cert to use to connect | +| --tablet_manager_grpc_concurrency | int | concurrency to use to talk to a vttablet server for performance-sensitive RPCs (like ExecuteFetchAs{Dba,AllPrivs,App}) (default 8) | +| --tablet_manager_grpc_connpool_size | int | number of tablets to keep tmclient connections open to (default 100) | +| --tablet_manager_grpc_crl | string | the server crl to use to validate server certificates when connecting | +| --tablet_manager_grpc_key | string | the key to use to connect | +| --tablet_manager_grpc_server_name | string | the server name to use to validate server certificate | +| --tablet_manager_protocol | string | Protocol to use to make tabletmanager RPCs to vttablets. (default "grpc") | +| --topo_consul_lock_delay | duration | LockDelay for consul session. (default 15s) | +| --topo_consul_lock_session_checks | string | List of checks for consul session. (default "serfHealth") | +| --topo_consul_lock_session_ttl | string | TTL for consul session. | +| --topo_consul_watch_poll_duration | duration | time of the long poll for watch queries. (default 30s) | +| --topo_etcd_lease_ttl | int | Lease TTL for locks and leader election. The client will use KeepAlive to keep the lease going. (default 30) | +| --topo_etcd_tls_ca | string | path to the ca to use to validate the server cert when connecting to the etcd topo server | +| --topo_etcd_tls_cert | string | path to the client cert to use to connect to the etcd topo server, requires topo_etcd_tls_key, enables TLS | +| --topo_etcd_tls_key | string | path to the client key to use to connect to the etcd topo server, enables TLS | +| --topo_global_root | string | the path of the global topology data in the global topology server | +| --topo_global_server_address | string | the address of the global topology server | +| --topo_implementation | string | the topology implementation to use | +| --topo_zk_auth_file | string | auth to use when connecting to the zk topo server, file contents should be :, e.g., digest:user:pass | +| --topo_zk_base_timeout | duration | zk base timeout (see zk.Connect) (default 30s) | +| --topo_zk_max_concurrency | int | maximum number of pending requests to send to a Zookeeper server. (default 64) | +| --topo_zk_tls_ca | string | the server ca to use to validate servers when connecting to the zk topo server | +| --topo_zk_tls_cert | string | the cert to use to connect to the zk topo server, requires topo_zk_tls_key, enables TLS | +| --topo_zk_tls_key | string | the key to use to connect to the zk topo server, enables TLS | +| --v | Level | log level for V logs | +| -v, --version | | print binary version | +| --vmodule | moduleSpec | comma-separated list of pattern=N settings for file-filtered logging | +| --xbstream_restore_flags | string | Flags to pass to xbstream command during restore. These should be space separated and will be added to the end of the command. These need to match the ones used for backup e.g. --compress / --decompress, --encrypt / --decrypt | +| --xtrabackup_backup_flags | string | Flags to pass to backup command. These should be space separated and will be added to the end of the command | +| --xtrabackup_prepare_flags | string | Flags to pass to prepare command. These should be space separated and will be added to the end of the command | +| --xtrabackup_root_path | string | Directory location of the xtrabackup and xbstream executables, e.g., /usr/bin | +| --xtrabackup_stream_mode | string | Which mode to use if streaming, valid values are tar and xbstream. Please note that tar is not supported in XtraBackup 8.0 (default "tar") | +| --xtrabackup_stripe_block_size | uint | Size in bytes of each block that gets sent to a given stripe before rotating to the next stripe (default 102400) | +| --xtrabackup_stripes | uint | If greater than 0, use data striping across this many destination files to parallelize data transfer and decompression | +| --xtrabackup_user | string | User that xtrabackup will use to connect to the database server. This user must have all necessary privileges. For details, please refer to xtrabackup documentation. | From fc3bdf44faefb17c40ef21d3cabae87f705849db Mon Sep 17 00:00:00 2001 From: Max Englander Date: Wed, 18 Jan 2023 10:52:09 -0500 Subject: [PATCH 2/3] content/en/docs/16.0: incorporate vtbackup into backup-and-restore guide Signed-off-by: Max Englander --- .../{backup-and-restore.md => overview.md} | 64 +++++++++++++------ 1 file changed, 46 insertions(+), 18 deletions(-) rename content/en/docs/16.0/user-guides/operating-vitess/backup-and-restore/{backup-and-restore.md => overview.md} (91%) diff --git a/content/en/docs/16.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md b/content/en/docs/16.0/user-guides/operating-vitess/backup-and-restore/overview.md similarity index 91% rename from content/en/docs/16.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md rename to content/en/docs/16.0/user-guides/operating-vitess/backup-and-restore/overview.md index 651cca1aa..eb0aee684 100644 --- a/content/en/docs/16.0/user-guides/operating-vitess/backup-and-restore/backup-and-restore.md +++ b/content/en/docs/16.0/user-guides/operating-vitess/backup-and-restore/overview.md @@ -1,5 +1,5 @@ --- -title: Backup and Restore +title: Overview weight: 1 aliases: ['/docs/user-guides/backup-and-restore/'] --- @@ -10,8 +10,6 @@ Backup and Restore are integrated features provided by tablets managed by Vitess Vitess supports pluggable interfaces for both [Backup Storage Services](https://github.com/vitessio/vitess/blob/main/go/vt/mysqlctl/backupstorage/interface.go) and [Backup Engines](https://github.com/vitessio/vitess/blob/main/go/vt/mysqlctl/backupengine.go). -Before backing up or restoring a tablet, you need to ensure that the tablet is aware of the Backup Storage system and Backup engine that you are using. To do so, use the following command-line flags when starting a vttablet or vtctld that has access to the location where you are storing backups. - ### Backup Storage Services Currently, Vitess has plugins for: @@ -28,14 +26,29 @@ The engine is the techology used for generating the backup. Currently Vitess has * Builtin: Shutdown an instance and copy all the database files (default) * XtraBackup: An online backup using Percona's [XtraBackup](https://www.percona.com/software/mysql-database/percona-xtrabackup) -## VTTablet and Vtctld configuration +## Vtbackup, VTTablet and Vtctld + +Vtbackup, VTTablet, and Vtctld may all participate in backups and restores. + + * Vtbackup is a standalone program that restores the last backup into an empty mysqld installation, replicates new changes into that installation, and takes a new backup from that installation. + * VTTablet can be configured to restore from a backup, or to take a new backup. + * Vtctld can be instructed to take backups with commands like `Backup` and `BackupShard`. + +### Configuration + +Before backing up or restoring a tablet, you need to ensure that the tablet is aware of the Backup Storage system and Backup Engine that you are using. -The following options can be used to configure VTTablet and Vtctld for backups: +To do so, use command-line flags to configure vtbackup, vttablet, or vtctld programs that have access to the location where you are storing backups. + +__Common flags:__ + +All three programs can be made aware of Backup Engine and Backup Storage using these common flags. - + + @@ -166,18 +179,6 @@ The following options can be used to configure VTTablet and Vtctld for backups: executing any `FLUSH` statements. This enables users to work around xtrabackup DDL issues. - - - - - - - - @@ -213,6 +214,33 @@ The following options can be used to configure VTTablet and Vtctld for backups:
FlagsNameDefinition
restore_from_backupIndicates that, when started with an empty MySQL instance, the - tablet should restore the most recent backup from the specified - storage plugin.
restore_from_backup_tsIf set, restore the latest backup taken at or before this timestamp - rather than using the most recent one. Example: ‘2021-04-29.133050’. - (Vitess 12.0+)
xbstream_restore_flags The flags to pass to the xbstream command during restore. These should be space separated and will be added to the end of the command. These need to match the ones used for backup e.g. --compress / --decompress, --encrypt / --decrypt
+__Restore flags:__ + +Only VTTablet can be configured to restore from a previous backup. The flags below only apply to VTTablet. + + + + + + + + + + + + + + + + + + +
NameDefinition
restore_from_backupIndicates that, when started with an empty MySQL instance, the + tablet should restore the most recent backup from the specified + storage plugin. This flag only applies to VTTablet.
restore_from_backup_tsIf set, restore the latest backup taken at or before this timestamp + rather than using the most recent one. Example: ‘2021-04-29.133050’. + (Vitess 12.0+)
+ ### Authentication Note that for the Google Cloud Storage plugin, we currently only support From 3570f02b4b8a77052f8295ec5be1224b3e7fbe74 Mon Sep 17 00:00:00 2001 From: Max Englander Date: Wed, 18 Jan 2023 11:24:27 -0500 Subject: [PATCH 3/3] content/en/docs/16.0: re-organize backup/restore docs Signed-off-by: Max Englander --- .../bootstrap-and-restore.md | 34 +++++ .../backup-and-restore/creating-a-backup.md | 128 +++++------------- .../backup-and-restore/managing-backups.md | 44 ++++++ 3 files changed, 111 insertions(+), 95 deletions(-) create mode 100644 content/en/docs/16.0/user-guides/operating-vitess/backup-and-restore/bootstrap-and-restore.md create mode 100644 content/en/docs/16.0/user-guides/operating-vitess/backup-and-restore/managing-backups.md diff --git a/content/en/docs/16.0/user-guides/operating-vitess/backup-and-restore/bootstrap-and-restore.md b/content/en/docs/16.0/user-guides/operating-vitess/backup-and-restore/bootstrap-and-restore.md new file mode 100644 index 000000000..5ab721ed7 --- /dev/null +++ b/content/en/docs/16.0/user-guides/operating-vitess/backup-and-restore/bootstrap-and-restore.md @@ -0,0 +1,34 @@ +--- +title: Bootstrap and Restore +weight: 3 +aliases: ['/docs/user-guides/backup-and-restore/'] +--- + +## Restoring a backup + +When a tablet starts, Vitess checks the value of the `--restore_from_backup` command-line flag to determine whether to restore a backup to that tablet. Restores will always be done with whichever engine was used to create the backup. + +* If the flag is present, Vitess tries to restore the most recent backup from the [BackupStorage](../overview/#backup-storage-services) system when starting the tablet or if the `--restore_from_backup_ts` flag (Vitess 12.0+) is also set then using the latest backup taken at or before this timestamp instead. Example: '2021-04-29.133050' +* If the flag is absent, Vitess does not try to restore a backup to the tablet. This is the equivalent of starting a new tablet in a new shard. + +This flag is generally enabled all of the time for all of the tablets in a shard. By default, if Vitess cannot find a backup in the Backup Storage system, the tablet will start up empty. This behavior allows you to bootstrap a new shard before any backups exist. + +If the `--wait_for_backup_interval` flag is set to a value greater than zero, the tablet will instead keep checking for a backup to appear at that interval. This can be used to ensure tablets launched concurrently while an initial backup is being seeded for the shard (e.g. uploaded from cold storage or created by another tablet) will wait until the proper time and then pull the new backup when it's ready. + +``` sh +vttablet ... --backup_storage_implementation=file \ + --file_backup_storage_root=/nfs/XXX \ + --restore_from_backup +``` + +## Bootstrapping a new tablet + +Bootstrapping a new tablet is almost identical to restoring an existing tablet. The only thing you need to be cautious about is that the tablet specifies its keyspace, shard and tablet type when it registers itself in the topology. Specifically, make sure that the following additional vttablet parameters are set: + +``` sh + --init_keyspace + --init_shard + --init_tablet_type replica|rdonly +``` + +The bootstrapped tablet will restore the data from the backup and then apply changes, which occurred after the backup, by restarting replication. diff --git a/content/en/docs/16.0/user-guides/operating-vitess/backup-and-restore/creating-a-backup.md b/content/en/docs/16.0/user-guides/operating-vitess/backup-and-restore/creating-a-backup.md index ae6e5fc6c..e336e47df 100644 --- a/content/en/docs/16.0/user-guides/operating-vitess/backup-and-restore/creating-a-backup.md +++ b/content/en/docs/16.0/user-guides/operating-vitess/backup-and-restore/creating-a-backup.md @@ -4,22 +4,24 @@ weight: 2 aliases: ['/docs/user-guides/backup-and-restore/'] --- -## Creating a backup +## Using xtrabackup The default backup implementation is `builtin`, however we strongly recommend using the `xtrabackup` engine as it is more robust and allows for non-blocking backups. Restores will always be done with whichever engine was used to create the backup. -### Prerequisite +### Prerequisites A compatible version of [xtrabackup](https://www.percona.com/doc/percona-xtrabackup/LATEST/index.html) and [xbstream](https://www.percona.com/doc/percona-xtrabackup/LATEST/xtrabackup_bin/backup.streaming.html), if needed, must be present in your `$PATH` prior to running the `Backup[Shard]` command. -### Supported Versions of Xtrabackup +### Supported Versions * [For MySQL 5.7](https://www.percona.com/doc/percona-xtrabackup/2.4/index.html#installation) * [MySQL 8.0](https://www.percona.com/doc/percona-xtrabackup/8.0/index.html#installation) -### Basic VTTablet and Vtctld Configuration +### Configuration -Required vttablet and vtctld flags: +To use `xtrabackup` with Vtbackup, VTTablet or Vtctld, the following flags must be set. + +__Required flags:__ * `--backup_engine_implementation=xtrabackup` * `--xtrabackup_user string` @@ -30,95 +32,13 @@ Additionally required for MySQL 8.0: * `--xtrabackup_stream_mode=xbstream` -### Run the following vtctl command to create a backup: - -``` sh -vtctldclient --server=: Backup -``` - -If the engine is `builtin`, replication will be stopped prior to shutting down mysqld for the backup. - -If the engine is `xtrabackup`, the tablet can continue to serve traffic while the backup is running. - -### Run the following vtctl command to backup a specific shard: - -``` sh -vtctldclient --server=: BackupShard [--allow_primary=false] -``` - -## Restoring a backup - -When a tablet starts, Vitess checks the value of the `--restore_from_backup` command-line flag to determine whether to restore a backup to that tablet. - -* If the flag is present, Vitess tries to restore the most recent backup from the [BackupStorage](../backup-and-restore/#backup-storage-services) system when starting the tablet or if the `--restore_from_backup_ts` flag (Vitess 12.0+) is also set then using the latest backup taken at or before this timestamp instead. Example: '2021-04-29.133050' -* If the flag is absent, Vitess does not try to restore a backup to the tablet. This is the equivalent of starting a new tablet in a new shard. - -As noted in the [Configuration](#basic-vttablet-and-vtctld-configuration) section, the flag is generally enabled all of the time for all of the tablets in a shard. By default, if Vitess cannot find a backup in the Backup Storage system, the tablet will start up empty. This behavior allows you to bootstrap a new shard before any backups exist. - -If the `--wait_for_backup_interval` flag is set to a value greater than zero, the tablet will instead keep checking for a backup to appear at that interval. This can be used to ensure tablets launched concurrently while an initial backup is being seeded for the shard (e.g. uploaded from cold storage or created by another tablet) will wait until the proper time and then pull the new backup when it's ready. - -``` sh -vttablet ... --backup_storage_implementation=file \ - --file_backup_storage_root=/nfs/XXX \ - --restore_from_backup -``` - -## Managing backups - -**vtctldclient** provides two commands for managing backups: - -* [GetBackups](https://vitess.io/docs/reference/programs/vtctldclient/vtctldclient_getbackups/) displays the existing backups for a keyspace/shard in chronological order. - - ``` sh - vtctldclient --server=: GetBackups - ``` - -* [RemoveBackup](https://vitess.io/docs/reference/programs/vtctldclient/vtctldclient_removebackup/) deletes a specified backup for a keyspace/shard. + - ``` sh - vtctldclient --server=: RemoveBackup - ``` +### Common Errors and Resolutions -You can also confirm your backup finished by viewing the files in your configured `--_backup_storage_root` location. You will still need to test and verify these backups for completeness. Note that backups are stored by keyspace and shard under `--_backup_storage_root`. For example, when using `--file_backup_storage_root=/vt/vtdataroot/backups`: - -```sh -/vt/vtdataroot/backups/commerce/0/2021-03-10.205419.zone1-0000000102: -backup.xbstream.gz MANIFEST -``` - -Each backup contains a manifest file with general information about the backup: - -```sh -MySQL 8.0 xbstream Manifest -{ - "BackupMethod": "xtrabackup", - "Position": "MySQL56/c022ad67-81fc-11eb-aa0e-1c1bb572885f:1-50", - "BackupTime": "2021-03-11T00:01:37Z", - "FinishedTime": "2021-03-11T00:01:42Z", - "FileName": "backup.xbstream.gz", - "ExtraCommandLineParams": "--no-server-version-check", - "StreamMode": "xbstream", - "NumStripes": 0, - "StripeBlockSize": 102400, - "SkipCompress": false -} -``` - -## Bootstrapping a new tablet - -Bootstrapping a new tablet is almost identical to restoring an existing tablet. The only thing you need to be cautious about is that the tablet specifies its keyspace, shard and tablet type when it registers itself in the topology. Specifically, make sure that the following additional vttablet parameters are set: - -``` sh - --init_keyspace - --init_shard - --init_tablet_type replica|rdonly -``` - -The bootstrapped tablet will restore the data from the backup and then apply changes, which occurred after the backup, by restarting replication. - -## Common Errors and Resolutions - -### No xtrabackup User passed to vttablet +__No xtrabackup User passed to vttablet:__ ``` E0310 08:15:45.336083 197442 main.go:72] remote error: rpc error: code = Unknown desc = TabletManager.Backup on zone1-0000000102 error: xtrabackupUser must be specified.: xtrabackupUser must be specified @@ -126,7 +46,7 @@ E0310 08:15:45.336083 197442 main.go:72] remote error: rpc error: code = Unknow Fix: Set the vtctld and vttablet flag `--xtrabackup_user` -### xtrabackup binary not found in $PATH +__xtrabackup binary not found in $PATH:__ ``` E0310 08:22:22.260044 200147 main.go:72] remote error: rpc error: code = Unknown desc = TabletManager.Backup on zone1-0000000102 error: unable to start backup: exec: "xtrabackup": executable file not found in $PATH: unable to start backup: exec: "xtrabackup": executable file not found in $PATH @@ -137,7 +57,7 @@ Fixes: * Ensure the xtrabackup binary is in the $PATH for the $USER running vttablet * Alternatively, set --xtrabackup_root_path on vttablet provide path to xtrabackup/xbstream binaries via vtctld and vttablet flags -### Tar format no longer supported in 8.0 +__Tar format no longer supported in 8.0:__ ``` I0310 12:34:47.900363 211809 backup.go:163] I0310 20:34:47.900004 xtrabackupengine.go:310] xtrabackup stderr: Invalid --stream argument: tar @@ -146,7 +66,7 @@ Streaming in tar format is no longer supported in 8.0; use xbstream instead Fix: Set the `--xtrabackup_stream_mode` flag to to xbstream on vttablets and vtctlds -### Unsupported mysql server version +__Unsupported mysql server version:__ ``` I0310 12:49:32.279729 215835 backup.go:163] I0310 20:49:32.279435 xtrabackupengine.go:310] xtrabackup stderr: Error: Unsupported server version 8.0.23-0ubuntu0.20.04.1. @@ -155,6 +75,24 @@ I0310 12:49:32.279773 215835 backup.go:163] I0310 20:49:32.279485 xtrabackupeng To continue with risk: Set `--xtrabackup_backup_flags=--no-server-version-check`. Note this occurs when your MySQL server version is technically unsupported by `xtrabackup`. +## Create backups with vtctl + +__Run the following vtctl command to create a backup:__ + +``` sh +vtctldclient --server=: Backup +``` + +If the engine is `builtin`, replication will be stopped prior to shutting down mysqld for the backup. + +If the engine is `xtrabackup`, the tablet can continue to serve traffic while the backup is running. + +__Run the following vtctl command to backup a specific shard:__ + +``` sh +vtctldclient --server=: BackupShard [--allow_primary=false] +``` + ## Backing up Topology Server The Topology Server stores metadata (and not tablet data). It is recommended to create a backup using the method described by the underlying plugin: diff --git a/content/en/docs/16.0/user-guides/operating-vitess/backup-and-restore/managing-backups.md b/content/en/docs/16.0/user-guides/operating-vitess/backup-and-restore/managing-backups.md new file mode 100644 index 000000000..0b4f8182f --- /dev/null +++ b/content/en/docs/16.0/user-guides/operating-vitess/backup-and-restore/managing-backups.md @@ -0,0 +1,44 @@ +--- +title: Managing Backups +weight: 4 +aliases: ['/docs/user-guides/backup-and-restore/'] +--- + +**vtctldclient** provides two commands for managing backups: + +* [GetBackups](https://vitess.io/docs/reference/programs/vtctldclient/vtctldclient_getbackups/) displays the existing backups for a keyspace/shard in chronological order. + + ``` sh + vtctldclient --server=: GetBackups + ``` + +* [RemoveBackup](https://vitess.io/docs/reference/programs/vtctldclient/vtctldclient_removebackup/) deletes a specified backup for a keyspace/shard. + + ``` sh + vtctldclient --server=: RemoveBackup + ``` + +You can also confirm your backup finished by viewing the files in your configured `--_backup_storage_root` location. You will still need to test and verify these backups for completeness. Note that backups are stored by keyspace and shard under `--_backup_storage_root`. For example, when using `--file_backup_storage_root=/vt/vtdataroot/backups`: + +```sh +/vt/vtdataroot/backups/commerce/0/2021-03-10.205419.zone1-0000000102: +backup.xbstream.gz MANIFEST +``` + +Each backup contains a manifest file with general information about the backup: + +```sh +MySQL 8.0 xbstream Manifest +{ + "BackupMethod": "xtrabackup", + "Position": "MySQL56/c022ad67-81fc-11eb-aa0e-1c1bb572885f:1-50", + "BackupTime": "2021-03-11T00:01:37Z", + "FinishedTime": "2021-03-11T00:01:42Z", + "FileName": "backup.xbstream.gz", + "ExtraCommandLineParams": "--no-server-version-check", + "StreamMode": "xbstream", + "NumStripes": 0, + "StripeBlockSize": 102400, + "SkipCompress": false +} +```