Skip to content

Fix for vttablet going unhealthy after failed backup #4805

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sougou merged 1 commit into from
Apr 16, 2019
Merged

Fix for vttablet going unhealthy after failed backup #4805

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 7 additions & 6 deletions go/vt/mysqlctl/builtinbackupengine.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -301,8 +301,8 @@ func (be *BuiltinBackupEngine) ExecuteBackup(ctx context.Context, cnf *Mycnf, my
backupErr := be.backupFiles(ctx, cnf, mysqld, logger, bh, replicationPosition, backupConcurrency, hookExtraEnv)
usable := backupErr == nil

// Try to restart mysqld
err = mysqld.Start(ctx, cnf)
// Try to restart mysqld, use background context in case we timed out the original context
err = mysqld.Start(context.Background(), cnf)
if err != nil {
return usable, vterrors.Wrap(err, "can't restart mysqld")
}
Expand Down Expand Up @@ -420,21 +420,22 @@ func (be *BuiltinBackupEngine) backupFile(ctx context.Context, cnf *Mycnf, mysql
return err
}

logger.Infof("Backing up file: %v", fe.Name)
// Open the destination file for writing, and a buffer.
wc, err := bh.AddFile(ctx, name, fi.Size())
if err != nil {
return vterrors.Wrapf(err, "cannot add file: %v", name)
return vterrors.Wrapf(err, "cannot add file: %v,%v", name, fe.Name)
}
defer func() {
defer func(name, fileName string) {
if rerr := wc.Close(); rerr != nil {
if err != nil {
// We already have an error, just log this one.
logger.Errorf2(rerr, "failed to close file %v", name)
logger.Errorf2(rerr, "failed to close file %v,%v", name, fe.Name)
} else {
err = rerr
}
}
}()
}(name, fe.Name)
dst := bufio.NewWriterSize(wc, writerBufferSize)

// Create the hasher and the tee on top.
Expand Down
10 changes: 8 additions & 2 deletions go/vt/vttablet/tabletmanager/rpc_backup.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,8 +84,14 @@ func (agent *ActionAgent) Backup(ctx context.Context, concurrency int, logger lo
returnErr := mysqlctl.Backup(ctx, agent.Cnf, agent.MysqlDaemon, l, dir, name, concurrency, agent.hookExtraEnv())

if builtin != nil {

bgCtx := context.Background()
// Starting from here we won't be able to recover if we get stopped by a cancelled
// context. It is also possible that the context already timed out during the
// above call to Backup. Thus we use the background context to get through to the finish.

Copy link
Copy Markdown
Contributor

@sougou sougou Apr 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you create a separate ctx := context.Background() here and use it in both places.
Ideally, they say you have to use context.TODO() (because you're not supposed to create background contexts outside of main.go). But I'm not very hung up on that rule.

Copy link
Copy Markdown
Collaborator Author

@deepthi deepthi Apr 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. I squashed it with the previous commit.

// change our type back to the original value
_, err = topotools.ChangeType(ctx, agent.TopoServer, tablet.Alias, originalType)
_, err = topotools.ChangeType(bgCtx, agent.TopoServer, tablet.Alias, originalType)
if err != nil {
// failure in changing the topology type is probably worse,
// so returning that (we logged the snapshot error anyway)
Expand All @@ -96,7 +102,7 @@ func (agent *ActionAgent) Backup(ctx context.Context, concurrency int, logger lo
}

// let's update our internal state (start query service and other things)
if err := agent.refreshTablet(ctx, "after backup"); err != nil {
if err := agent.refreshTablet(bgCtx, "after backup"); err != nil {
return err
}
}
Expand Down