Online DDL, CancelMigration: distinguish user-issued vs. internally-issued cancellation

[shlomi-noach]

Description

#10900 introduced cancelled_timestamp column, which is populated when a migration is cancelled. Then, based on the value of this column, and assuming a problem in the migration, a migration transitions to either failed (if the column is NULL) or to cancelled (if the column is NOT NULL).

As quick recap, gh-ost, pt-osc and even vreplication itself, to some extent, act as 3rd party tools for Online DDL. To cancel a migration is to fail the migration; #10900 was made so that we can make a distinction between a "legitimately failed" migration to a "user cancelled" migration.

However, we left out a couple use cases, where the Online DDL executor itself may cancel a migration:

• if the migration is stale
• if there's an unrecoverable error in the migration flow

In these scenarios, the executor calls CancelMigration, but we expect the terminal state to be failed, not cancelled.

In this PR we make the distinction between a user-generated CANCEL (e.g. user issued a ALTER VITESS_MIGRATION ... CANCEL command), and an internal-generated cancellation. the latter now leads to failed state.

Related Issue(s)

• Online DDL: tracking issue #6926

Checklist

• "Backport me!" label has been added if this change should be backported
• Tests were added or are not required
• Documentation was added or is not required

Deployment Notes

[vitess-bot]

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

• Ensure that the Pull Request has a descriptive title.
• If this is a change that users need to know about, please apply the release notes (needs details) label so that merging is blocked unless the summary release notes document is included.
• If a new flag is being introduced, review whether it is really needed. The flag names should be clear and intuitive (as far as possible), and the flag's help should be descriptive.
• If a workflow is added or modified, each items in Jobs should be named in order to mark it as required. If the workflow should be required, the GitHub Admin should be notified.

Bug fixes

• There should be at least one unit or end-to-end test.
• The Pull Request description should either include a link to an issue that describes the bug OR an actual description of the bug and how to reproduce, along with a description of the fix.

Non-trivial changes

• There should be some code comments as to why things are implemented the way they are.

New/Existing features

• Should be documented, either by modifying the existing documentation or creating new documentation.
• New features should have a link to a feature request issue or an RFC that documents the use cases, corner cases and test cases.

Backward compatibility

• Protobuf changes should be wire-compatible.
• Changes to _vt tables and RPCs need to be backward compatible.
• vtctl command output order should be stable and awk-able.

[deepthi]

LGTM
What about tests?

[shlomi-noach]

Mentioned in a private channel, tests are complex. I'm not sure right now. Reason is that the scenarios where a migration gets cancelled by the scheduler are a bit extreme and will be difficult to reproduce synthetically.

[deepthi]

Mentioned in a private channel, tests are complex. I'm not sure right now. Reason is that the scenarios where a migration gets cancelled by the scheduler are a bit extreme and will be difficult to reproduce synthetically.

You may need to build a unit test framework to simulate vreplication stream errors. We can let this through for now, because that will surely take time to do.