VDiff2: Auto retry to continue on error

[mattlord]

Description

We now record and display errors for a VDiff on a per shard basis:

$ vtctlclient VDiff -- --v2 customer.commerce2customer show last

VDiff Summary for customer.p1c2 (c6e8698e-01b3-11ed-b76d-920702940ee0)
State:        error
              Error: (shard -80) vttablet: rpc error: code = Unknown desc = (errno 1213) (sqlstate 40001): Deadlock found when trying to get lock; try restarting transaction
              Error: (shard 80-) vttablet: rpc error: code = Unknown desc = (errno 1213) (sqlstate 40001): Deadlock found when trying to get lock; try restarting transaction
RowsCompared: 105
HasMismatch:  false
StartedAt:    2022-07-12 07:25:17

Use "--format=json" for more detailed output.


$ vtctlclient VDiff -- --v2 --format=json customer.commerce2customer show last
{
	"Workflow": "p1c2",
	"Keyspace": "customer",
	"State": "error",
	"UUID": "c6e8698e-01b3-11ed-b76d-920702940ee0",
	"RowsCompared": 106,
	"HasMismatch": false,
	"Shards": "-80,80-",
	"StartedAt": "2022-07-12 07:25:17",
	"Errors": {
		"-80": "vttablet: rpc error: code = Unknown desc = (errno 1213) (sqlstate 40001): Deadlock found when trying to get lock; try restarting transaction",
		"80-": "vttablet: rpc error: code = Unknown desc = (errno 1213) (sqlstate 40001): Deadlock found when trying to get lock; try restarting transaction"
	}
}

And we automatically retry those VDiffs (every 30 seconds) that ended with an error, when the auto-retry option has been set for it (now defaults to true) and we can see that it's an ephemeral error which warrants an immediate retry. For example (from a manual local test):

$ vtctlclient VDiff -- --v2 --format=json --auto-retry customer.commerce2customer show last
	"Workflow": "commerce2customer",
	"Keyspace": "customer",
	"State": "error",
	"UUID": "da4c5e14-fece-11ec-a217-920702940ee0",
	"RowsCompared": 2,
	"HasMismatch": false,
	"Shards": "0",
	"StartedAt": "2022-07-08 15:01:32",
	"Errors": {
		"0": "vttablet: rpc error: code = Unknown desc = (errno 1213) (sqlstate 40001): Deadlock found when trying to get lock; try restarting transaction"
	}
}

$ vtctlclient VDiff -- --v2 --format=json --auto-retry customer.commerce2customer show last
{
	"Workflow": "commerce2customer",
	"Keyspace": "customer",
	"State": "completed",
	"UUID": "da4c5e14-fece-11ec-a217-920702940ee0",
	"RowsCompared": 4,
	"HasMismatch": false,
	"Shards": "0",
	"StartedAt": "2022-07-08 15:02:31",
	"CompletedAt": "2022-07-08 15:02:32"
}

ℹ️ NOTE: The resume command lets you manually pick up a successfully completed VDiff where it left off (based on the last PK) and no longer resets the VDiff state (rows compared, diff reports, etc) when it starts. The old behavior of resetting that state was not documented and I can see now that it would be unintuitive in a number of ways, along with preventing reasonable progress updates. The new behavior makes resume align nicely with the automatic retry on error behavior added here, which picks the VDiff up where it left off (based on the last PK) — for a run that ended with an error — and retains the state from the run up to the point that it encountered an error.

We also add progress reporting to the VDiff show summary output when the State is started. For example:

$ vtctlclient VDiff -- --v2 --format=json customer.p1c2 show last
{
	"Workflow": "p1c2",
	"Keyspace": "customer",
	"State": "started",
	"UUID": "daf1f03a-03ed-11ed-9ab8-920702940ee0",
	"RowsCompared": 51,
	"HasMismatch": false,
	"Shards": "-80,80-",
	"StartedAt": "2022-07-15 03:26:03",
	"Progress": {
		"Percentage": 48.57,
		"ETA": "2022-07-15 03:26:10"
	}
}

$ vtctlclient VDiff -- --v2 --format=json customer.p1c2 show last
{
	"Workflow": "p1c2",
	"Keyspace": "customer",
	"State": "started",
	"UUID": "daf1f03a-03ed-11ed-9ab8-920702940ee0",
	"RowsCompared": 104,
	"HasMismatch": false,
	"Shards": "-80,80-",
	"StartedAt": "2022-07-15 03:26:03",
	"Progress": {
	"Percentage": 99.05,
		"ETA": "2022-07-15 03:26:11"
	}
}

$ vtctlclient VDiff -- --v2 --format=json customer.p1c2 show last
{
	"Workflow": "p1c2",
	"Keyspace": "customer",
	"State": "started",
	"UUID": "daf1f03a-03ed-11ed-9ab8-920702940ee0",
	"RowsCompared": 105,
	"HasMismatch": false,
	"Shards": "-80,80-",
	"StartedAt": "2022-07-15 03:26:03",
	"Progress": {
		"Percentage": 100,
		"ETA": "2022-07-15 03:26:12"
	}
}

$ vtctlclient VDiff -- --v2 --format=json customer.p1c2 show last
{
	"Workflow": "p1c2",
	"Keyspace": "customer",
	"State": "completed",
	"UUID": "daf1f03a-03ed-11ed-9ab8-920702940ee0",
	"RowsCompared": 105,
	"HasMismatch": false,
	"Shards": "-80,80-",
	"StartedAt": "2022-07-15 03:26:03",
	"CompletedAt": "2022-07-15 03:26:13"
}

Related Issue(s)

• Fixes: VDiff2: Automatic retry on error #10618
• Fixes: Add Progress field to Show output when the Status == started to display a % of work done and ETA for completion #10666

Checklist

• "Backport me!" label has been added if this change should be backported
• Tests were added or are not required
• Documentation: Document https://github.com/vitessio/vitess/pull/10639 website#1108

[vitess-bot]

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

• Ensure that the Pull Request has a descriptive title.
• If this is a change that users need to know about, please apply the release notes (needs details) label so that merging is blocked unless the summary release notes document is included.
• If a new flag is being introduced, review whether it is really needed. The flag names should be clear and intuitive (as far as possible), and the flag's help should be descriptive.
• If a workflow is added or modified, each items in Jobs should be named in order to mark it as required. If the workflow should be required, the GitHub Admin should be notified.

Bug fixes

• There should be at least one unit or end-to-end test.
• The Pull Request description should either include a link to an issue that describes the bug OR an actual description of the bug and how to reproduce, along with a description of the fix.

Non-trivial changes

• There should be some code comments as to why things are implemented the way they are.

New/Existing features

• Should be documented, either by modifying the existing documentation or creating new documentation.
• New features should have a link to a feature request issue or an RFC that documents the use cases, corner cases and test cases.

Backward compatibility

• Protobuf changes should be wire-compatible.
• Changes to _vt tables and RPCs need to be backward compatible.
• vtctl command output order should be stable and awk-able.

[rohit-nayak-ps]

lgtm
Nice work!