Skip to content

Enable VTOrc in backup tests #11410

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
GuptaManan100 merged 4 commits into from
Oct 1, 2022
Merged

Enable VTOrc in backup tests #11410

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
665e834
feat: enable vtorc in backup tests
GuptaManan100 Sep 30, 2022
8051fb6
feat: reorder verifying initial replication to happen first
GuptaManan100 Sep 30, 2022
aa7ba34
Merge remote-tracking branch 'upstream/main' into vtorc-in-backup-tests
GuptaManan100 Sep 30, 2022
316fcdf
test: increase timeout for VTOrc recovery to be successful and improv…
GuptaManan100 Oct 1, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 18 additions & 2 deletions go/test/endtoend/backup/vtctlbackup/backup_utils.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -763,10 +763,21 @@ func terminatedRestore(t *testing.T) {
// Args:
// tablet_type: 'replica' or 'rdonly'.
func vtctlBackup(t *testing.T, tabletType string) {
restoreWaitForBackup(t, tabletType, nil, true)
// Start vtorc before running backups
vtorcProcess := localCluster.NewVTOrcProcess(cluster.VTOrcConfiguration{})
err := vtorcProcess.Setup()
require.NoError(t, err)
localCluster.VTOrcProcesses = append(localCluster.VTOrcProcesses, vtorcProcess)

// StopReplication on replica1. We verify that the replication works fine later in
// verifyInitialReplication. So this will also check that VTOrc is running.
err = localCluster.VtctlclientProcess.ExecuteCommand("StopReplication", replica1.Alias)
require.Nil(t, err)

verifyInitialReplication(t)
restoreWaitForBackup(t, tabletType, nil, true)

err := localCluster.VtctlclientProcess.ExecuteCommand("Backup", replica1.Alias)
err = localCluster.VtctlclientProcess.ExecuteCommand("Backup", replica1.Alias)
require.Nil(t, err)

backups := localCluster.VerifyBackupCount(t, shardKsName, 1)
Expand All @@ -781,6 +792,11 @@ func vtctlBackup(t *testing.T, tabletType string) {
cluster.VerifyLocalMetadata(t, replica2, keyspaceName, shardName, cell)
verifyAfterRemovingBackupNoBackupShouldBePresent(t, backups)

// Stop VTOrc
err = localCluster.VTOrcProcesses[0].TearDown()
localCluster.VTOrcProcesses = nil
require.NoError(t, err)

err = replica2.VttabletProcess.TearDown()
require.Nil(t, err)

Expand Down
2 changes: 1 addition & 1 deletion go/test/endtoend/cluster/cluster_util.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ func GetPrimaryPosition(t *testing.T, vttablet Vttablet, hostname string) (strin
// VerifyRowsInTabletForTable verifies the total number of rows in a table.
// This is used to check that replication has caught up with the changes on primary.
func VerifyRowsInTabletForTable(t *testing.T, vttablet *Vttablet, ksName string, expectedRows int, tableName string) {
timeout := time.Now().Add(10 * time.Second)
timeout := time.Now().Add(1 * time.Minute)
Copy link
Copy Markdown
Contributor Author

@GuptaManan100 GuptaManan100 Oct 1, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This timeout has been increased for VTOrc. With the defaults, it takes VTOrc 15 seconds to discover the tablets and then it can start repairing them. This wait was for 10 seconds which turned out to be too small. I have improved VTOrc performance ☝️, but it doesn't hurt to increase the timeout.

for time.Now().Before(timeout) {
// ignoring the error check, if the newly created table is not replicated, then there might be error and we should ignore it
// but eventually it will catch up and if not caught up in required time, testcase will fail
Expand Down
2 changes: 2 additions & 0 deletions go/test/endtoend/cluster/vtorc_process.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,8 @@ func (orc *VTOrcProcess) Setup() (err error) {
// This parameter is overriden from the config file, added here to just verify that we indeed use the config file paramter over the flag
"--recovery-period-block-duration", "10h",
"--instance-poll-time", "1s",
// Faster topo information refresh speeds up the tests. This doesn't add any significant load either
"--topo-information-refresh-duration", "3s",
"--orc_web_dir", path.Join(os.Getenv("VTROOT"), "web", "vtorc"),
)
if *isCoverage {
Expand Down