Point-in-time-recovery by deepthi · Pull Request #4857 · vitessio/vitess

deepthi · 2019-05-07T19:36:01Z

Partial fix for #4886

CreateKeyspace now takes three additional (optional) parameters
- keyspace_type: NORMAL/SNAPSHOT
- base_keyspace: keyspace that you are trying to recover
- snapshot_time: for SNAPSHOT keyspaces, the time for which it is being recovered
when creating a keyspace, copy vschema from base_keyspace
Vttablet recovery mode
- init_keyspace: this should be a SNAPSHOT keyspace
- snapshot_time will be used as a starting point to find a backup to restore (most recent backup from snapshot_time)
- replication will not be setup when restoring a snapshot keyspace
Vtgate changes
- tables and vindexes of SNAPSHOT keyspaces are not added to the uniqueTables and uniqueVindexes maps when building vschema. This is sufficient to hide them from the global routing.
It is not necessary to bring up a master tablet for PITR. All tablets can be replicas.
If a master is desired for any reason, it should be initialized using TabletExternallyReparented, not InitShardMaster or PlannedReparentShard.
How to access recovered data
- use @replica followed by select ... from ks.table
- use ks@replica or use ks:shard@replica followed by select .. from table
added unit tests for CreateKeyspace
integration test to test recovery of vttablet + vtgate routing

Signed-off-by: deepthi deepthi@planetscale.com

- keyspace_type: NORMAL/SNAPSHOT - snapshot_time: for SNAPSHOT keyspaces, the time for which it is being recovered - also when creating a keyspace, create an empty vschema * Vttablet recovery mode - recovery_keyspace: which keyspace to recover - init_keyspace: this should be a SNAPSHOT keyspace - snapshot_time will be used as a starting point to find a backup to restore (most recent backup from snapshot_time) * Vtgate changes - tables and vindexes of SNAPSHOT keyspaces are not added to the uniqueTables and uniqueVindexes maps when building vschema * integration test to test recovery of vttablet Signed-off-by: deepthi <deepthi@planetscale.com>

Signed-off-by: deepthi <deepthi@planetscale.com>

…ime is not set Signed-off-by: deepthi <deepthi@planetscale.com>

…ion for CreateKeyspace call RebuildVSchemaGraph after copying vschema for snapshot keyspace Signed-off-by: deepthi <deepthi@planetscale.com>

Signed-off-by: deepthi <deepthi@planetscale.com>

…nd which gtid we restored to Signed-off-by: deepthi <deepthi@planetscale.com>

sougou

Still need to do a deeper dive. Have one high level comment so far.

sougou · 2019-06-02T00:51:48Z

go/vt/vttablet/tabletmanager/rpc_backup.go

 	dir := fmt.Sprintf("%v/%v", tablet.Keyspace, tablet.Shard)
-	name := fmt.Sprintf("%v.%v", time.Now().UTC().Format("2006-01-02.150405"), topoproto.TabletAliasString(tablet.Alias))
-	returnErr := mysqlctl.Backup(ctx, agent.Cnf, agent.MysqlDaemon, l, dir, name, concurrency, agent.hookExtraEnv())
+	backupTime := time.Now().UTC()


I think the backup time should be pulled from the server's timestamp. Ideally, as of the snapshot time. But I can see someone make an argument for the current time as seen by vttablet. @rafael: thoughts?

that is a good point. until now this time was only used to make the backup directory, so it didn't matter, but for purposes of knowing where to replay binlogs from, it should be server time.

Signed-off-by: deepthi <deepthi@planetscale.com>

distribute backup tests among jobs Signed-off-by: deepthi <deepthi@planetscale.com>

Signed-off-by: deepthi <deepthi@planetscale.com>

go/vt/mysqlctl/backup.go

enisoc · 2019-07-04T00:35:55Z

go/cmd/vtbackup/vtbackup.go


 	log.Infof("Restoring latest backup from directory %v", backupDir)
-	restorePos, err := mysqlctl.Restore(ctx, mycnf, mysqld, backupDir, *concurrency, extraEnv, map[string]string{}, logutil.NewConsoleLogger(), true, dbName)
+	restorePos, _, err := mysqlctl.Restore(ctx, mycnf, mysqld, backupDir, *concurrency, extraEnv, map[string]string{}, logutil.NewConsoleLogger(), true, dbName, time.Unix(0, 0).UTC())


Traditionally we use the zero value time.Time{} to represent "unspecified". Is there any reason we can't make Restore() accept that?

no reason, I just didn't know there was a convention. I will change it.

Still need to change this to time.Time{} (and a few more places below). Also, I noticed that you were calling .UTC() in a few places. That should not be necessary because time.Unix() will return the correct value whether it's UTC or not.

sougou · 2019-07-28T04:34:35Z

proto/time.proto

+// to convert times.
+message Time {
+  int64 seconds = 1;
+  int32 nanoseconds = 2;


I'd say go with int64. There's no great benefit to using int32.

this is an existing message that I moved into a new file. Is it safe to change the data type?

we'll keep this as int32 because changing the type will not be backwards-compatible

sougou · 2019-07-28T04:57:00Z

go/cmd/vtbackup/vtbackup.go


 	log.Infof("Restoring latest backup from directory %v", backupDir)
-	restorePos, err := mysqlctl.Restore(ctx, mycnf, mysqld, backupDir, *concurrency, extraEnv, map[string]string{}, logutil.NewConsoleLogger(), true, dbName)
+	restorePos, _, err := mysqlctl.Restore(ctx, mycnf, mysqld, backupDir, *concurrency, extraEnv, map[string]string{}, logutil.NewConsoleLogger(), true, dbName, time.Unix(0, 0).UTC())


Still need to change this to time.Time{} (and a few more places below). Also, I noticed that you were calling .UTC() in a few places. That should not be necessary because time.Unix() will return the correct value whether it's UTC or not.

Signed-off-by: deepthi <deepthi@planetscale.com>

deepthi · 2019-09-04T21:39:16Z

Closing this in favor of #5160.
@enisoc if you have a review in progress on this, please go ahead and submit it and I will look at it.

deepthi requested a review from sougou as a code owner May 7, 2019 19:36

deepthi added 3 commits May 14, 2019 14:23

unit test for CreateKeyspace, fix CI test failures

359b2f2

Signed-off-by: deepthi <deepthi@planetscale.com>

fix xtrabackup backup file checking to work correctly when snapshot_t…

4bbd80c

…ime is not set Signed-off-by: deepthi <deepthi@planetscale.com>

deepthi force-pushed the ds-pitr-tablet branch from eedff48 to 4bbd80c Compare May 14, 2019 21:24

deepthi added 2 commits May 29, 2019 16:04

remove -recovery_keyspace option for vttablet, add -base_keyspace opt…

5c33729

…ion for CreateKeyspace call RebuildVSchemaGraph after copying vschema for snapshot keyspace Signed-off-by: deepthi <deepthi@planetscale.com>

test PITR data access through vtgate

ae06bc1

Signed-off-by: deepthi <deepthi@planetscale.com>

deepthi changed the title ~~WIP: Features needed to facilitate Point-in-time-recovery~~ Features needed to facilitate Point-in-time-recovery May 30, 2019

fix test failures

3bb0c73

Signed-off-by: deepthi <deepthi@planetscale.com>

deepthi changed the title ~~Features needed to facilitate Point-in-time-recovery~~ Point-in-time-recovery May 30, 2019

deepthi added 3 commits May 30, 2019 17:31

fix time formats and help text

b81a10a

Signed-off-by: deepthi <deepthi@planetscale.com>

revert wrapping errors to fix test failure

2cb4c0d

Signed-off-by: deepthi <deepthi@planetscale.com>

add fields to Tablet record to store which backup we restored from, a…

e56f5d1

…nd which gtid we restored to Signed-off-by: deepthi <deepthi@planetscale.com>

sougou reviewed Jun 2, 2019

View reviewed changes

deepthi and others added 9 commits June 20, 2019 10:46

Merge branch 'master' into ds-pitr-tablet

c2c3b04

pitr: fix compile errors, refactor after merge

6b27741

Signed-off-by: deepthi <deepthi@planetscale.com>

add multi-recovery integration test

873818b

distribute backup tests among jobs Signed-off-by: deepthi <deepthi@planetscale.com>

simplify recovery test setup

603be9e

Signed-off-by: deepthi <deepthi@planetscale.com>

Merge branch 'master' into ds-pitr-tablet

10c7d08

fix after merge

5c0bef0

Signed-off-by: deepthi <deepthi@planetscale.com>

Merge branch 'master' into ds-pitr-tablet

3aeffbc

sharded recovery testcase

07fe385

Signed-off-by: deepthi <deepthi@planetscale.com>

use vtctl SplitClone instead of vtworker

485431d

Signed-off-by: deepthi <deepthi@planetscale.com>

deepthi requested a review from enisoc July 3, 2019 16:26

deepthi commented Jul 3, 2019

View reviewed changes

go/vt/mysqlctl/backup.go Show resolved Hide resolved

enisoc reviewed Jul 4, 2019

View reviewed changes

sougou reviewed Jul 28, 2019

View reviewed changes

deepthi added 2 commits August 13, 2019 11:42

Merge branch 'master' into ds-pitr-tablet

382d3c7

Signed-off-by: deepthi <deepthi@planetscale.com>

address review comments

40ee700

Signed-off-by: deepthi <deepthi@planetscale.com>

deepthi requested a review from derekperkins as a code owner August 13, 2019 23:46

deepthi requested a review from dkhenry as a code owner August 13, 2019 23:46

deepthi mentioned this pull request Sep 4, 2019

Point in time recovery (part 1) #5160

Merged

deepthi closed this Sep 4, 2019

deepthi deleted the ds-pitr-tablet branch July 6, 2020 19:01

arindamnayak mentioned this pull request Jul 18, 2020

Implemented PITR-2 #6408

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Point-in-time-recovery#4857

Point-in-time-recovery#4857
deepthi wants to merge 20 commits intovitessio:masterfrom
planetscale:ds-pitr-tablet

deepthi commented May 7, 2019 •

edited

Loading

Uh oh!

sougou left a comment

Uh oh!

sougou Jun 2, 2019

Uh oh!

deepthi Jun 3, 2019

Uh oh!

Uh oh!

enisoc Jul 4, 2019

Uh oh!

deepthi Jul 8, 2019

Uh oh!

sougou Jul 28, 2019

Uh oh!

sougou Jul 28, 2019

Uh oh!

deepthi Aug 13, 2019

Uh oh!

deepthi Sep 4, 2019

Uh oh!

sougou Jul 28, 2019

Uh oh!

deepthi commented Sep 4, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

deepthi commented May 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sougou left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deepthi commented Sep 4, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

deepthi commented May 7, 2019 •

edited

Loading