Implemented PITR-2 by arindamnayak · Pull Request #6408 · vitessio/vitess

arindamnayak · 2020-07-03T18:47:24Z

Implemented PITR-2
Fixes #6267, fixes #4886

Here are the changes I did.

vttablet will accept binlog server parameter
all those parameters will be used while restoring the tablet and it need a snapshot keyspace.
from the snapshot keyspace type, we will get the snapshot_time
we will get the GTID from the snapshot time
then we will replicate from binlog server upto that GTID
once we reach that GTID , the slave connection will be closed and reset.

- Added flags for restore to time - Added flags for binlog server details Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

- added ability to fetch the GTID from the restore time - added ability to apply the binlogs till the above GTID Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

…chan Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

go/vt/vttablet/tabletmanager/restore.go

arindamnayak · 2020-07-14T17:41:01Z

go/vt/vttablet/tabletserver/schema/engine.go

 	notifiers   map[string]notifier

+	// SkipMetaCheck skips the metadata about the database and table information
+	SkipMetaCheck bool


@sougou , is it ok if we add this flag here? This flag will skip the sql query to get table information ( form information_schema). We need this when we need to get the GTID from timestamp , as the binlog server does not support the query and we also don't need that info in that process.

Sounds good for now. I see that you're just using it to covert timestamp to gtid. There may be a simpler way to do that without having to use vstream, but we can use this method until we figure that out.

…le names Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

deepthi

Mostly comments and documentation, but there is an important logic issue as well.

.github/workflows/docker_test_2.yml

go/mysql/flavor_mysql.go

go/test/endtoend/recovery/pitr/pitr_test.go

deepthi · 2020-07-16T01:45:43Z

go/vt/vttablet/tabletmanager/restore.go

+	binlogPort           = flag.Int("binlog_port", 0, "(PITR restore parameter) port of binlog server.")
+	binlogUser           = flag.String("binlog_user", "", "(PITR restore parameter) username of binlog server.")
+	binlogPwd            = flag.String("binlog_password", "", "(PITR restore parameter) password of binlog server.")
+	timeoutForGTIDLookup = flag.Duration("binlog_lookup_timeout", 60*time.Second, "(PITR restore parameter) timeout for fetching gtid from timestamp.")


what do you think of gtid_lookup_timeout for this flag name?

that does not signifies which gtid we are referring to, let me know if we should use that.

recovery_gtid_lookup_timeout?

go/vt/vttablet/tabletmanager/restore.go

go/vt/vttablet/tabletmanager/vreplication/replica_connector.go

go/vt/vttablet/tabletserver/schema/engine.go

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

arindamnayak · 2020-07-16T14:43:00Z

Mostly comments and documentation, but there is an important logic issue as well.

I have implemented the suggestion you mentioned.

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

aquarapid · 2020-07-17T16:54:12Z

go/mysql/flavor_mysql.go

 // masterGTIDSet is part of the Flavor interface.
 func (mysqlFlavor) masterGTIDSet(c *Conn) (GTIDSet, error) {
-	qr, err := c.ExecuteFetch("SELECT @@GLOBAL.gtid_executed", 1, false)
+	// making @@global as lowercase, as the PITR depends on binlog server, which honours only lowercase `global` value


Let's just make this:

// keep @@global as lowercase, as some servers like the Ripple binlog server only honors a lowercase `global` value

Otherwise people might get confused (since this is not PITR-specific code)

Agreed, I have updated the comment.

deepthi · 2020-07-17T18:20:41Z

Can you balance tests between docker_test_1 and docker_test_2?

docker test 2 / Docker Test 2 (pull_request) Successful in 19m
docker_test_1 / Docker Test 1 (pull_request) Successful in 9m

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

arindamnayak · 2020-07-18T12:21:46Z

Can you balance tests between docker_test_1 and docker_test_2?
docker test 2 / Docker Test 2 (pull_request) Successful in 19m
docker_test_1 / Docker Test 1 (pull_request) Successful in 9m

I have balanced the tests

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

deepthi

The new logic looks good.
A few more nits. We can merge once those are addressed.

go/test/endtoend/cluster/cluster_util.go

go/test/endtoend/recovery/pitr/main_test.go

deepthi · 2020-07-18T16:29:18Z

go/test/endtoend/recovery/pitr/main_test.go

+		if proc, err := tablet.MysqlctlProcess.StartProcess(); err != nil {
+			return err
+		} else {
+			// ignore golint warning, we need the else block to use proc


don't you need a //nolint here? otherwise CI will fail.

not sure why golangci-lint did not capture this earlier, it is passing without //nolint, to be on safe side I have added it.

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

arindamnayak · 2020-07-18T18:41:16Z

The new logic looks good.
A few more nits. We can merge once those are addressed.

Implemented the suggested changes and also updated the flag name.

deepthi · 2020-07-18T20:29:12Z

Very nice work! This has been a long-standing request.

arindamnayak added 11 commits June 14, 2020 00:23

Initial commit for PITR

b1113c7

- Added flags for restore to time - Added flags for binlog server details Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

Initial commit for PITR -2

21b7a04

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

PITR changes

28df598

- added ability to fetch the GTID from the restore time - added ability to apply the binlogs till the above GTID Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

Added some validation and fixes while testing

ddf1ee9

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

updated vstream to work with go routine and return the gitd back via …

94e6529

…chan Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

added fix for replication

be17562

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

Added timeout of gtid to time lookup

0b7a255

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

added last method to gtid

6805b62

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

added wait for gtid catch up

f4b8375

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

added go version of wait till postion

de7c4c0

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

Fix the pos.atleast with required gtid

51864ca

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

arindamnayak requested a review from sougou as a code owner July 3, 2020 18:47

arindamnayak linked an issue Jul 3, 2020 that may be closed by this pull request

RFC: Point in time recovery (PITR) Part 2 #6267

Closed

arindamnayak marked this pull request as draft July 3, 2020 18:48

arindamnayak requested a review from deepthi July 3, 2020 18:48

arindamnayak added 3 commits July 4, 2020 16:17

sync with vitess/master

239ebb0

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

Added mariadb last gtid impl

ca96cbc

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

fix build issue

d5bc793

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

arindamnayak force-pushed the pitr-2 branch from e2fefaf to d5bc793 Compare July 5, 2020 11:02

deepthi requested a review from aquarapid July 6, 2020 15:53

added rippled dependency in docker

1a800d9

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

arindamnayak force-pushed the pitr-2 branch 2 times, most recently from 1bffb9e to dc8af79 Compare July 9, 2020 12:12

added pitr testcase

47bce65

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

arindamnayak force-pushed the pitr-2 branch 2 times, most recently from fb065be to a989cf2 Compare July 10, 2020 18:29

checked in the binary for testing

a9bbf8e

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

arindamnayak force-pushed the pitr-2 branch from a989cf2 to a9bbf8e Compare July 10, 2020 18:50

Added rippled as extra bin path

4acba2e

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

arindamnayak force-pushed the pitr-2 branch from 2aa48ef to 4acba2e Compare July 11, 2020 10:37

arindamnayak added 2 commits July 14, 2020 11:30

merged with latest master

53dded5

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

rearranged the testcases

943903c

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

arindamnayak commented Jul 14, 2020

View reviewed changes

go/vt/vttablet/tabletmanager/restore.go Outdated Show resolved Hide resolved

arindamnayak commented Jul 14, 2020

View reviewed changes

go/vt/vttablet/tabletmanager/restore.go Outdated Show resolved Hide resolved

arindamnayak commented Jul 14, 2020

View reviewed changes

arindamnayak added 5 commits July 15, 2020 23:55

include changes for improving the error messages and comments, variab…

d24e79e

…le names Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

merge docker tests to matrix

653ad37

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

updated workflow

f694c31

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

Rervert back to individual action

ff5a0d2

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

remove the skipmetacheck from unneeded place

7bb7863

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

deepthi reviewed Jul 16, 2020

View reviewed changes

Implemented suggestion in PR comments

c500654

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

handle the case where upto snapshot time there is no binlog event

eaf2454

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

aquarapid reviewed Jul 17, 2020

View reviewed changes

arindamnayak added 2 commits July 18, 2020 00:03

fix the vstream end of events

6abb509

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

improved the comment and rearrange the tests

ecf278b

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

Replace string check with gtid.pos.atleast

5b64ddc

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

deepthi approved these changes Jul 18, 2020

View reviewed changes

updated flag name and improvement on code comments

878c871

Signed-off-by: Arindam Nayak <arindam.nayak@outlook.com>

deepthi merged commit e86a967 into vitessio:master Jul 18, 2020

arindamnayak deleted the pitr-2 branch July 21, 2020 18:21

deepthi added this to the v7.0 milestone Jul 27, 2020

ameetkotian mentioned this pull request Aug 20, 2020

Slack vitess 2020.08.19.r0 tinyspeck/vitess#180

Merged

shlomi-noach mentioned this pull request Sep 15, 2022

RFC: incremental backup and point-in-time recovery #11227

Closed

shlomi-noach mentioned this pull request Aug 28, 2024

Backup/restore: remove legacy binlog-server based PITR code #16673

Closed

Conversation

arindamnayak commented Jul 3, 2020 • edited by deepthi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

arindamnayak Jul 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deepthi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

arindamnayak commented Jul 16, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deepthi commented Jul 17, 2020

Uh oh!

arindamnayak commented Jul 18, 2020

Uh oh!

deepthi left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arindamnayak commented Jul 18, 2020

Uh oh!

deepthi commented Jul 18, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

arindamnayak commented Jul 3, 2020 •

edited by deepthi

Loading

arindamnayak Jul 14, 2020 •

edited

Loading

deepthi left a comment •

edited

Loading