Backport: Testing version upgrade/downgrade path from/to 8.0

[shlomi-noach]

Description

Backport of #7294

Copy of comment from #7294 :

Followup to team discussion, we wish to add CI (endtoend) tests that validate an upgrade from a previous tagged version.

This PR introduces endtoend version upgrade test, specifically testing an upgrade path from v8.0.0. (side note, v9.0.0 will be released later this month, and then we will add a test for a v9.0.0 upgrade path).

The test verifies that a vitess cluster which was built, started and populated by a v8.0.0 release, is readable by a HEAD (the PR branch) build.

The way this works:

• .github/workflows/cluster_endtoend_upgrade.yml is the workflow file
• it checks out v8.0.0, and builds the binaries
• stores binaries in a safe place
• checks out HEAD
• gets the binaries we built for v8.0.0
• runs go/test/endtoend/versionupgrade/upgrade_test.go
  • the test creates sharded and unsharded keyspaces and tablets
  • creates tables in keyspaces and shards
  • populated the tables
  • verifies data is populated
  • shuts down without removing data (-keep-data)
• Checks out HEAD again, cleaning up the workflow running directory
• Builds HEAD
• Starts vitess on top of existing data, which means:
  • etcd2 loads from existing data
  • mysqld starts with existing data
  • vttablets start with existing data (and are already in SERVING mode)
• Very data is still populated.

So this test basically starts a "latest" version of Vitess on top of preexisting v8.0.0 data, and SELECTs data from tables to prove the upgrade is good.
We may add additional tests to verify the upgrade path is good.

What kind of changes were made:

• Existing endtoend tests intentionally randomized paths, ports, etc. But we have to be able to run two vitess clusters on exact same paths, ports etc. To that effect we introduce these flags:
  • -force-vtdataroot (path)
  • -force-port-start (port number)
  • -force-base-tablet-uid (number)
    With these three supplied, all locations and ports are idempotent.
• test.go has a -skip-build flag; this is required because we want to be able to run the latest upgrade test, on a past release, which may not have this test (specifically, v8.0.0 does not have this test). We use this to run a latest test on top of pre-built v8.0.0 binaries.
• LocalProcessCluster has a field called ReusingVTDATAROOT which indicates whether this cluster started from empty, and initialized itself (false), or whether the cluster reused an existing dataset (true)
• cluster_process.go uses that field to decide whether it should initialize mysql, or create keyspaces, or what state it expects tablet to start with, etc.
  -go/test/endtoend/cluster/vtctlclient_process.go uses that field to determine if it needs to populate database tables and data or not.

Symmetrically, the workflow also tests a downgrade path.

Checklist

• Should this PR be backported? This is the backport
• Tests were added or are not required
• Documentation was added or is not required

Impacted Areas in Vitess

Components that this PR will affect:

• Query Serving
• VReplication
• Cluster Management
• Build
• VTAdmin