Toggle primary read-only when disk capacity hits threshold #59
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
davissp14
commented
Feb 2, 2023
internal/flycheck/pg.go
Outdated
| return "", fmt.Errorf("failed to turn primary readonly: %s", err) | ||
| } | ||
|
|
||
| return "", fmt.Errorf("%0.1f%% - extend your volume to re-enable writes", usedPercentage) |
There was a problem hiding this comment.
Not exactly sure why, but i'm getting MISSING here.
[✗] disk-capacity: 93.2%!e(MISSING)xtend your volume to re-enable writes (2.92ms)
There was a problem hiding this comment.
This has been fixed with:
DAlperin
reviewed
Feb 2, 2023
Comment on lines
+13
to
+15
| // Primary will be made read-only when disk capacity reaches this percentage. | ||
| const diskCapacityPercentageThreshold = 90.0 | ||
|
|
There was a problem hiding this comment.
I've been meaning to make health-check stuff configurable but that can be out of scope of this PR
DAlperin
reviewed
Feb 2, 2023
| return connectionCount(ctx, localConn) | ||
| }) | ||
|
|
||
| if member.Role == flypg.PrimaryRoleName && member.Active { |
There was a problem hiding this comment.
Do we want to expose this healthcheck on replicas without setting them to readonly?
There was a problem hiding this comment.
I feel like it might be useful info
There was a problem hiding this comment.
There's a VM check that communicates general capacity that should cover that. It makes me think though that maybe we need a new name for the check.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR ensures the primary is made read-only when disk capacity exceeds 90%.
I've introduced a new PG health check called
disk-capacity. This health check has quite a bit of overlap with the existing VM check communicating disk usage, but given the implications and message I feel it should be separated.How it works
On an interval defined by the health check, we will calculate disk usage and evaluate whether capacity has exceeded our pre-defined threshold of 90% ( should be made configurable in the future ).
When capacity exceeds the pre-defined threshold, we will set a
readonly.lockfile on the filesystem and establish a connection to each user-defined table ( anything that's not Postgres or Repmgr) and make it readonly. When disk usage falls below the defined threshold, either through file cleanup or volume extension, we will work to turn all tables back to read/write and then clear thereadonly.lockfile on success.The
readonly.lockfile is used for two things:If the
readonly.lockfile is present, we know the database(s) have already been made readonly so there's no need to needlessly establish any more connections.It allows us to coordinate intentions across the codebase.
Limitations
Read-only mode is currently enabled at runtime and will be cleared on restart. That being said, there will be a small window on boot where the primary will accept writes before it is reconfigured back to read-only. If this becomes a problem, we can look into addressing this.
#56