Testing upgrade path from / downgrade path to v8.0.0

[shlomi-noach]

Description

Followup to team discussion, we wish to add CI (endtoend) tests that validate an upgrade from a previous tagged version.

This PR introduces endtoend version upgrade test, specifically testing an upgrade path from v8.0.0. (side note, v9.0.0 will be released later this month, and then we will add a test for a v9.0.0 upgrade path).

The test verifies that a vitess cluster which was built, started and populated by a v8.0.0 release, is readable by a HEAD (the PR branch) build.

The way this works:

• .github/workflows/cluster_endtoend_upgrade.yml is the workflow file
• it checks out v8.0.0, and builds the binaries
• stores binaries in a safe place
• checks out HEAD
• gets the binaries we built for v8.0.0
• runs go/test/endtoend/versionupgrade/upgrade_test.go
  • the test creates sharded and unsharded keyspaces and tablets
  • creates tables in keyspaces and shards
  • populated the tables
  • verifies data is populated
  • shuts down without removing data (-keep-data)
• Checks out HEAD again, cleaning up the workflow running directory
• Builds HEAD
• Starts vitess on top of existing data, which means:
  • etcd2 loads from existing data
  • mysqld starts with existing data
  • vttablets start with existing data (and are already in SERVING mode)
• Very data is still populated.

So this test basically starts a "latest" version of Vitess on top of preexisting v8.0.0 data, and SELECTs data from tables to prove the upgrade is good.
We may add additional tests to verify the upgrade path is good.

What kind of changes were made:

• Existing endtoend tests intentionally randomized paths, ports, etc. But we have to be able to run two vitess clusters on exact same paths, ports etc. To that effect we introduce these flags:
  • -force-vtdataroot (path)
  • -force-port-start (port number)
  • -force-base-tablet-uid (number)
    With these three supplied, all locations and ports are idempotent.
• test.go has a -skip-build flag; this is required because we want to be able to run the latest upgrade test, on a past release, which may not have this test (specifically, v8.0.0 does not have this test). We use this to run a latest test on top of pre-built v8.0.0 binaries.
• LocalProcessCluster has a field called ReusingVTDATAROOT which indicates whether this cluster started from empty, and initialized itself (false), or whether the cluster reused an existing dataset (true)
• cluster_process.go uses that field to decide whether it should initialize mysql, or create keyspaces, or what state it expects tablet to start with, etc.
  -go/test/endtoend/cluster/vtctlclient_process.go uses that field to determine if it needs to populate database tables and data or not.

EDIT

Symmetrically, the workflow now also runs a downgrade path test.

Related Issue(s)

Checklist

• Should this PR be backported?
• Tests were added or are not required
• Documentation was added or is not required

Deployment Notes

Impacted Areas in Vitess

Components that this PR will affect:

• Query Serving
• VReplication
• Cluster Management
• Build
• VTAdmin

[shlomi-noach]

A "fun" loophole I got into is this:

I've added a an endtoend upgrade test. In CI, I check out v8.0.0 to run the test, and then I check out pr/branch HEAD to run the test again.
Problem is... the test does not exist in v8.0.0. Next problem is that I can't precompile the binaries because test.go itself does that. So it seems like the endtoend test is coupled with the vitess binary test. I need to override that.

[shlomi-noach]

Also copying this to main comment

---

This PR introduced endtoend version upgrade test, specifically testing an upgrade path from v8.0.0. (side note, v9.0.0 will be released later this month, and then we will add a test for a v9.0.0 upgrade path).

The test verifies that a vitess cluster which was built, started and populated by a v8.0.0 release, is readable by a HEAD (the PR branch) build.

The way this works:

• .github/workflows/cluster_endtoend_upgrade.yml is the workflow file
• it checks out v8.0.0, and builds the binaries
• stores binaries in a safe place
• checks out HEAD
• gets the binaries we built for v8.0.0
• runs go/test/endtoend/versionupgrade/upgrade_test.go
  • the test creates sharded and unsharded keyspaces and tablets
  • creates tables in keyspaces and shards
  • populated the tables
  • verifies data is populated
  • shuts down without removing data (-keep-data)
• Checks out HEAD again, cleaning up the workflow running directory
• Builds HEAD
• Starts vitess on top of existing data, which means:
  • etcd2 loads from existing data
  • mysqld starts with existing data
  • vttablets start with existing data (and are already in SERVING mode)
• Verify data is still populated.

So this test basically starts a "latest" version of Vitess on top of preexisting v8.0.0 data, and SELECTs data from tables to prove the upgrade is good.
We may add additional tests to verify the upgrade path is good.

What kind of changes were made:

• Existing endtoend tests intentionally randomized paths, ports, etc. But we have to be able to run two vitess clusters on exact same paths, ports etc. To that effect we introduce these flags:
  • -force-vtdataroot (path)
  • -force-port-start (port number)
  • -force-base-tablet-uid (number)
    With these three supplied, all locations and ports are idempotent.
• test.go has a -skip-build flag; this is required because we want to be able to run the latest upgrade test, on a past release, which may not have this test (specifically, v8.0.0 does not have this test). We use this to run a latest test on top of pre-built v8.0.0 binaries.
• LocalProcessCluster has a field called ReusingVTDATAROOT which indicates whether this cluster started from empty, and initialized itself (false), or whether the cluster reused an existing dataset (true)
• cluster_process.go uses that field to decide whether it should initialize mysql, or create keyspaces, or what state it expects tablet to start with, etc.
  -go/test/endtoend/cluster/vtctlclient_process.go uses that field to determine if it needs to populate database tables and data or not.

[shlomi-noach]

Ready for review.

[shlomi-noach]

I'll create a separate PR to test a downgrade; basically it's a duplication of existing test. EDIT: eventually this was done in this PR

[shlomi-noach]

This PR should be backported to v9.x

[shlomi-noach]

This PR now also includes a downgrade path test

[deepthi]

This is excellent! Just one typo, can merge after fixing that.

[deepthi]

Suggested change

	# We pass -skip-build so that we use he v8.0.0 binaries, not HEAD binaries
	# We pass -skip-build so that we use the v8.0.0 binaries, not HEAD binaries

[deepthi]

This is a great start towards completely automating the upgrade/downgrade testing.
Other scenarios that need to be tested:

• mixed mode: only one or two of the 3 main components (vttablet/vtgate/vtctld) is/are upgraded / downgraded

[shlomi-noach]

#7323 is a resubmission as a backport to 9.0