[release-15.0] [Backport]: VReplication: Prevent Orphaned VDiff2 Jobs (#11768)

[mattlord]

Description

Note This is a backport of #11768. We're doing this because 1) the changes are isolated to vdiff2 2) vdiff2 is currently marked as experimental in v15 and we want more testing of it in v15 (this was the origination of the bug report) in the hopes of marking it production ready in v16.

This addresses two ways a VDiff could become "orphaned" and require a manual Stop and Resume step:

1. A tablet and/or mysqld are not shutdown normally or otherwise cannot save the final state
   • Fix: when opening the engine, restart any vdiffs that are in the started state as this indicates it did not complete and was unable to save the final state and must be restarted.
2. The vdiff run fails with an error and the DB connection is closed/lost
   • Fix: when a vdiff run fails, retry saving the error state with an exponential backoff until the engine shuts down. This way the normal retry mechanism will eventually kick in OR the behavior in item 1 will kick in when the engine is next opened on the primary tablet.

We also begin to build out the unit testing framework for VDiff2 here and use that to test the related behavior.

Manual Tests

Note In both of these manual tests, before this PR the VDiff was stuck/orphaned.

Failure before the vdiff fully initializes

Test:

./101_initial_cluster.sh; ./201_customer_tablets.sh;
mysql commerce -e "insert into customer values (1, 'mlord@planetscale.com'), (2, 'mlord2@planetscale.com'), (3, 'mlord3@planetscale.com')"

sleep 5
vtctlclient -- InitShardPrimary --force customer/0 zone1-200
./202_move_tables.sh

mysql -u root --socket=${VTDATAROOT}/vt_0000000200/mysql.sock  -e "set global max_execution_time=500"
vtctlclient -- VDiff --v2 customer.commerce2customer
sleep 2
vtctlclient -- VDiff --v2 customer.commerce2customer show last

mysql -u root --socket=${VTDATAROOT}/vt_0000000200/mysql.sock  -e "lock tables _vt.vdiff_table write; do sleep (5);" &

vtctlclient -- VDiff --v2 customer.commerce2customer
sleep 5
vtctlclient -- VDiff --v2 customer.commerce2customer show last

mysql -u root --socket=${VTDATAROOT}/vt_0000000200/mysql.sock  -e "set global max_execution_time=50000"

for i in {1..20}; do vtctlclient -- VDiff --v2 customer.commerce2customer show last; sleep 2; done

Results:

VDiff Summary for customer.commerce2customer (ffe52378-6791-11ed-8c12-920702940ee1)
State:        error
              Error: (shard 0) buildPlan: Query execution was interrupted, maximum statement execution time exceeded (errno 3024) (sqlstate HY000) during query: select vdt.lastpk as lastpk, vdt.mismatch as mismatch, vdt.report as report
                                                from _vt.vdiff as vd inner join _vt.vdiff_table as vdt on (vd.id = vdt.vdiff_id)
                                                where vdt.vdiff_id = 2 and vdt.table_name = 'corder'
RowsCompared: 0
HasMismatch:  false
StartedAt:    2022-11-18 22:40:27

Use "--format=json" for more detailed output.


VDiff Summary for customer.commerce2customer (ffe52378-6791-11ed-8c12-920702940ee1)
State:        completed
RowsCompared: 3
HasMismatch:  false
StartedAt:    2022-11-18 22:40:48
CompletedAt:  2022-11-18 22:40:48

Use "--format=json" for more detailed output.

Failure after vdiff initialization

Test:

./101_initial_cluster.sh; ./201_customer_tablets.sh;
mysql commerce -e "insert into customer values (1, 'mlord@planetscale.com'), (2, 'mlord2@planetscale.com'), (3, 'mlord3@planetscale.com')"

sleep 5
vtctlclient -- InitShardPrimary --force customer/0 zone1-200
./202_move_tables.sh

vtctlclient -- VDiff --v2 customer.commerce2customer
sleep 2
vtctlclient -- VDiff --v2 customer.commerce2customer show last

vtctlclient -- VDiff --v2 customer.commerce2customer
mysql -u root --socket=${VTDATAROOT}/vt_0000000200/mysql.sock  -e "lock tables _vt.vdiff_table write; do sleep (1);" &
kill -9 $(ps auxww | grep "${VTDATAROOT}/vt_0000000200/mysql.sock" | grep mysqld | awk '{print $2}')
vtctlclient -- VDiff --v2 customer.commerce2customer show last

CELL=zone1 TABLET_UID=200 ./scripts/mysqlctl-up.sh

for i in {1..20}; do vtctlclient -- VDiff --v2 customer.commerce2customer show last; sleep 1; done

Results:

VDiff Summary for customer.commerce2customer (3dab68ae-6790-11ed-89b4-920702940ee1)
State:        started
              Error: (shard 0) conn 112: Write(packet) failed: write unix ->/opt/vtdataroot/vt_0000000200/mysql.sock: write: broken pipe (errno 2006) (sqlstate HY000) during query: update _vt.vdiff_table set state = 'error' where vdiff_id = 2 and table_name = 'corder'
RowsCompared: 0
HasMismatch:  false
StartedAt:    2022-11-18 22:27:52
Progress:     0.00%

Use "--format=json" for more detailed output.


VDiff Summary for customer.commerce2customer (3dab68ae-6790-11ed-89b4-920702940ee1)
State:        pending
RowsCompared: 0
HasMismatch:  false
StartedAt:    2022-11-18 22:28:12

Use "--format=json" for more detailed output.


VDiff Summary for customer.commerce2customer (3dab68ae-6790-11ed-89b4-920702940ee1)
State:        completed
RowsCompared: 3
HasMismatch:  false
StartedAt:    2022-11-18 22:28:12
CompletedAt:  2022-11-18 22:28:13

Use "--format=json" for more detailed output.

Related Issue(s)

• Fixes: Bug Report: VDiff2 becoming stuck #11742
• Backports: VReplication: Prevent Orphaned VDiff2 Jobs #11768

Checklist

• "Backport to:" labels have been added if this change should be back-ported
• Tests were added
• Documentation is not required

[arthurschreiber]

To expand on the reasoning for the backport, we've been running VDiff2 on fairly large dataset (10+TB), and have seen tremendous reductions in diff execution time compared to VDiff1. Unfortunately, reliability was way below our expectations, partially due to the bugs fixed by these changes. Backporting this will make it easier to actually test VDiff2 for other users with similar data sizes as us.

[vitess-bot]

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

• Ensure that the Pull Request has a descriptive title.
• If this is a change that users need to know about, please apply the release notes (needs details) label so that merging is blocked unless the summary release notes document is included.

If a new flag is being introduced:

• Is it really necessary to add this flag?
• Flag names should be clear and intuitive (as far as possible)
• Help text should be descriptive.
• Flag names should use dashes (-) as word separators rather than underscores (_).

If a workflow is added or modified:

• Each item in Jobs should be named in order to mark it as required.
• If the workflow should be required, the maintainer team should be notified.

Bug fixes

• There should be at least one unit or end-to-end test.
• The Pull Request description should include a link to an issue that describes the bug.

Non-trivial changes

• There should be some code comments as to why things are implemented the way they are.

New/Existing features

• Should be documented, either by modifying the existing documentation or creating new documentation.
• New features should have a link to a feature request issue or an RFC that documents the use cases, corner cases and test cases.

Backward compatibility

• Protobuf changes should be wire-compatible.
• Changes to _vt tables and RPCs need to be backward compatible.
• vtctl command output order should be stable and awk-able.
• RPC changes should be compatible with vitess-operator
• If a flag is removed, then it should also be removed from VTop, if used there.