Skip to content

Vdiff formatting improvements#8079

Merged
rafael merged 6 commits intovitessio:masterfrom
tinyspeck:vdiff-formatting-improvements
May 10, 2021
Merged

Vdiff formatting improvements#8079
rafael merged 6 commits intovitessio:masterfrom
tinyspeck:vdiff-formatting-improvements

Conversation

@rafael
Copy link
Copy Markdown
Member

@rafael rafael commented May 7, 2021

Description

The following change improves how diffs are reported to users. In paticular it adds the following:

  • Adds sample rows to the diff report. This allows us to show the diff in a way that is more structured and with better formatting.
  • Adds a debug option that will add a Query to the diff report. This query can then later be used to check the actual vitess shards and perform further debugging.
  • Adds a onlyPks option to only report this information in the diffs. This is useful for cases where the data could have PII information.
  • Last but not least, all this info is used to provide a more comprehensive report at the end of the run.

These are the kind of reports that now can be generated with these changes:

vtctl VDiff  -debug test.copy_rafael_test
Summary for table test_rafael_materialized:
	ProcessedRows: 39055
	MatchingRows: 39053
	MismatchedRows: 0
	ExtraRowsSource: 0
	ExtraRowsTarget: 2
	Sample extra row in target 0:
		Row:
			 `column`: null (NULL_TYPE)
			 `data`: "3232" (BLOB)
			 id: 23232 (INT32)
			 id1: 1 (INT32)
			 id2: "1" (VARCHAR)
			 id3: 0.5 (FLOAT64)
			 weight_string(`column`): null (NULL_TYPE)
			 weight_string(id2): "\x001" (VARBINARY)
		DebugQuery: select id, `data`, id1, id2, id3, `column`, weight_string(id2), weight_string(`column`) from test_rafael_materialized where id=23232 AND id1=1 AND id2='1' AND id3=0.5;
	Sample extra row in target 1:
		Row:
			 `column`: null (NULL_TYPE)
			 `data`: "33123123123131312323...[TRUNCATED]" (BLOB)
			 id: 232232 (INT32)
			 id1: 1 (INT32)
			 id2: "1" (VARCHAR)
			 id3: 0.5 (FLOAT64)
			 weight_string(`column`): null (NULL_TYPE)
			 weight_string(id2): "\x001" (VARBINARY)
		DebugQuery: select id, `data`, id1, id2, id3, `column`, weight_string(id2), weight_string(`column`) from test_rafael_materialized where id=232232 AND id1=1 AND id2='1' AND id3=0.5;

or

vtctl VDiff  test.copy_rafael_test -only_pks

Summary for table test_rafael_materialized:
	ProcessedRows: 39055
	MatchingRows: 39053
	MismatchedRows: 0
	ExtraRowsSource: 0
	ExtraRowsTarget: 2
	Sample extra row in target 0:
		Row:
			 id: 23232 (INT32)
			 id1: 1 (INT32)
			 id2: "1" (VARCHAR)
			 id3: 0.5 (FLOAT64)
	Sample extra row in target 1:
		Row:
			 id: 232232 (INT32)
			 id1: 1 (INT32)
			 id2: "1" (VARCHAR)
			 id3: 0.5 (FLOAT64)

Checklist

  • Tests were added or are not required
  • Documentation was added or is not required

Rafael Chacon added 3 commits May 7, 2021 14:53
…ticular it adds the following:

* Adds sample rows to the diff report. This allows us to show the diff in a way that is more structured and with better formatting.
* Adds a debug option that will add a Query to the diff report. This query can then later be used to check the actual vitess shards and perform further debugging.
* Adds a onlyPks option to only report this information in the diffs. This is useful for cases where the data could have PII information.
* Last but not least, all this info is used to provide a more comprehensive report at the end of the run.

Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
@rafael rafael changed the title [draft] Vdiff formatting improvements Vdiff formatting improvements May 7, 2021
Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
Copy link
Copy Markdown
Member

@rohit-nayak-ps rohit-nayak-ps left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice! lgtm

@rohit-nayak-ps
Copy link
Copy Markdown
Member

@rafael , can you also create a PR to update the docs for vdiff at https://vitess.io/docs/reference/vreplication/vdiff/ (https://github.com/vitessio/website) with the new options and the sample? Thx

@systay systay added Component: VReplication Type: Enhancement Logical improvement (somewhere between a bug and feature) labels May 9, 2021
…f-formatting-improvements

Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
@rafael rafael merged commit 493eafb into vitessio:master May 10, 2021
@rafael
Copy link
Copy Markdown
Member Author

rafael commented May 10, 2021

@rafael , can you also create a PR to update the docs for vdiff at https://vitess.io/docs/reference/vreplication/vdiff/ (https://github.com/vitessio/website) with the new options and the sample? Thx

Yes, will do!

@rafael rafael deleted the vdiff-formatting-improvements branch May 10, 2021 16:52
rafael pushed a commit to tinyspeck/vitess that referenced this pull request Jun 8, 2021
Signed-off-by: Rafael Chacon <rafael@slack-corp.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Component: VReplication Type: Enhancement Logical improvement (somewhere between a bug and feature)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants