Skip to content

Add Gauge For CheckMySQL Running#11524

Merged
GuptaManan100 merged 2 commits intovitessio:mainfrom
planetscale:check-mysql-metrics
Oct 20, 2022
Merged

Add Gauge For CheckMySQL Running#11524
GuptaManan100 merged 2 commits intovitessio:mainfrom
planetscale:check-mysql-metrics

Conversation

@GuptaManan100
Copy link
Copy Markdown
Contributor

Description

We have been seeing a failure of DemotePrimary wherein it just gets stuck indefinitely. The failure was traced to a call to acquiring a lock which was held by the CheckMySQL function which too is stuck indefinitely. This PR aims to add a gauge which tells us if CheckMySQL is running or not. If the gauge is constantly 1 for more than 10 minutes, then it is an indication that something is wrong and we can take a look.

Related Issue(s)

Checklist

  • "Backport me!" label has been added if this change should be backported
  • Tests were added or are not required
  • Documentation was added or is not required

Deployment Notes

Signed-off-by: Manan Gupta <manan@planetscale.com>
@vitess-bot
Copy link
Copy Markdown
Contributor

vitess-bot bot commented Oct 18, 2022

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

  • Ensure that the Pull Request has a descriptive title.
  • If this is a change that users need to know about, please apply the release notes (needs details) label so that merging is blocked unless the summary release notes document is included.

If a new flag is being introduced:

  • Is it really necessary to add this flag?
  • Flag names should be clear and intuitive (as far as possible)
  • Help text should be descriptive.
  • Flag names should use dashes (-) as word separators rather than underscores (_).

If a workflow is added or modified:

  • Each item in Jobs should be named in order to mark it as required.
  • If the workflow should be required, the maintainer team should be notified.

Bug fixes

  • There should be at least one unit or end-to-end test.
  • The Pull Request description should include a link to an issue that describes the bug.

Non-trivial changes

  • There should be some code comments as to why things are implemented the way they are.

New/Existing features

  • Should be documented, either by modifying the existing documentation or creating new documentation.
  • New features should have a link to a feature request issue or an RFC that documents the use cases, corner cases and test cases.

Backward compatibility

  • Protobuf changes should be wire-compatible.
  • Changes to _vt tables and RPCs need to be backward compatible.
  • vtctl command output order should be stable and awk-able.
  • RPC changes should be compatible with vitess-operator
  • If a flag is removed, then it should also be removed from VTop, if used there.

Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100 GuptaManan100 merged commit e25c152 into vitessio:main Oct 20, 2022
@GuptaManan100 GuptaManan100 deleted the check-mysql-metrics branch October 20, 2022 09:49
notfelineit pushed a commit to planetscale/vitess that referenced this pull request Nov 16, 2022
* feat: add gauge for checkMySQL running

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: fix data race by converting bool to atomic bool

Signed-off-by: Manan Gupta <manan@planetscale.com>

Signed-off-by: Manan Gupta <manan@planetscale.com>

Signed-off-by: Manan Gupta <manan@planetscale.com>
timvaillancourt pushed a commit to slackhq/vitess that referenced this pull request Feb 19, 2023
* feat: add gauge for checkMySQL running

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: fix data race by converting bool to atomic bool

Signed-off-by: Manan Gupta <manan@planetscale.com>

Signed-off-by: Manan Gupta <manan@planetscale.com>
timvaillancourt pushed a commit to slackhq/vitess that referenced this pull request Jun 29, 2023
* feat: add gauge for checkMySQL running

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: fix data race by converting bool to atomic bool

Signed-off-by: Manan Gupta <manan@planetscale.com>

Signed-off-by: Manan Gupta <manan@planetscale.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Component: Cluster management Component: Query Serving Type: Enhancement Logical improvement (somewhere between a bug and feature)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants