Skip to content

Add timeout for mysqld_shutdown#6849

Merged
deepthi merged 10 commits intovitessio:masterfrom
tinyspeck:am_mysqld_shutdown_timeout
Oct 22, 2020
Merged

Add timeout for mysqld_shutdown#6849
deepthi merged 10 commits intovitessio:masterfrom
tinyspeck:am_mysqld_shutdown_timeout

Conversation

@ajm188
Copy link
Contributor

@ajm188 ajm188 commented Oct 10, 2020

Resolves #6848

Notable changes:

  • hooks can now be passed contexts via the ExecuteContext method. All hooks are now implemented with exec.CommandContext under the hood instead of exec.Command. Current callers are unaffected, and can opt in to switching over at their discretion.
  • adds a new HookResult.ExitStatus type for hooks that exceed their context deadline/timeout.
  • FakeMysqlDaemon.Start() and FakeMysqlDaemon.Shutdown() are configurable via the FakeMysqlDaemon.StartupTime and FakeMysqlDaemon.ShutdownTime respectively, to artificially cause delay in tests.

Andrew Mason added 6 commits October 10, 2020 00:53
…down before aborting

Signed-off-by: Andrew Mason <amason@slack-corp.com>
Signed-off-by: Andrew Mason <amason@slack-corp.com>
Signed-off-by: Andrew Mason <amason@slack-corp.com>
Signed-off-by: Andrew Mason <amason@slack-corp.com>
…sses

Signed-off-by: Andrew Mason <amason@slack-corp.com>
Signed-off-by: Andrew Mason <amason@slack-corp.com>
// shutdown mysqld
err = params.Mysqld.Shutdown(ctx, params.Cnf, true)
shutdownCtx, cancel := context.WithTimeout(ctx, *builtinBackupMysqldDeadline)
err = params.Mysqld.Shutdown(shutdownCtx, params.Cnf, true)
Copy link
Contributor Author

@ajm188 ajm188 Oct 10, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see an easy way to test this. I can't use fakemysqldaemon to create a mysqld that hangs on shutdown, because that package imports this package, which causes circular import errors. 🤔

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solved this with blackbox testing ✅

@ajm188 ajm188 marked this pull request as ready for review October 10, 2020 15:51
@ajm188 ajm188 requested review from deepthi and sougou as code owners October 10, 2020 15:51
Andrew Mason added 3 commits October 11, 2020 16:00
Signed-off-by: Andrew Mason <amason@slack-corp.com>
Signed-off-by: Andrew Mason <amason@slack-corp.com>
…behavior

Signed-off-by: Andrew Mason <amason@slack-corp.com>
Copy link
Collaborator

@deepthi deepthi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! Just the one comment.

)

var (
// BuiltinBackupMysqldDeadline is how long ExecuteBackup should wait for response from mysqld.Shutdown.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typically deadline is used for timestamps and timeout for durations. This looks like a timeout.

Signed-off-by: Andrew Mason <amason@slack-corp.com>
@ajm188 ajm188 requested a review from morgo as a code owner October 19, 2020 21:24
@deepthi deepthi merged commit d3f66bd into vitessio:master Oct 22, 2020
@deepthi
Copy link
Collaborator

deepthi commented Oct 22, 2020

@ajm188 would you please create another PR to merge this into release-8.0?

ajm188 pushed a commit to tinyspeck/vitess that referenced this pull request Oct 22, 2020
…meout

Add timeout for mysqld_shutdown

Signed-off-by: Andrew Mason <amason@slack-corp.com>
deepthi added a commit that referenced this pull request Oct 22, 2020
Backport pull request #6849 (add timeout for mysqld_shutdown) to 8.0
@ajm188 ajm188 deleted the am_mysqld_shutdown_timeout branch January 14, 2021 15:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Wedged mysqld shutdown delays error handling in builtinbackup.ExecuteBackup

2 participants