fingerprint: convert consul and vault fingerprinters to be reloadable #24526
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
shoenig
force-pushed
the
cv-reloadable-fingerprints
branch
from
488237b to
f926c8e
Compare
shoenig
force-pushed
the
cv-reloadable-fingerprints
branch
from
f926c8e to
39a0327
Compare
shoenig
force-pushed
the
cv-reloadable-fingerprints
branch
from
39a0327 to
21ed04c
Compare
This PR changes the Consul and Vault fingerprint implementations to be reloadable rather than periodic. Reasons described in the issue. Closes: #24049
shoenig
force-pushed
the
cv-reloadable-fingerprints
branch
from
21ed04c to
0689759
Compare
tgross
added a commit
that referenced
this pull request
Feb 12, 2025
In #24526 we updated the Consul and Vault fingerprints so that they are no longer periodic. This fixed a problem that cluster admins reported where rolling updates of Vault or Consul would cause a thundering herd of fingerprint updates across the whole cluster. But if Consul/Vault is not available during the initial fingerprint, it will never get fingerprinted again. This is challenging for cluster updates and black starts because the implicit service startup ordering may require reloads. Instead, have the fingerprinter run periodically but mark that it has made its first successful fingerprint of all Consul/Vault clusters. At that point, we can skip further periodic updates. The `Reload` method will reset the mark and allow the subsequent fingerprint to run normally. Fixes: #25097 Ref: #24526 Ref: #24049
6 tasks
tgross
added a commit
that referenced
this pull request
Feb 12, 2025
In #24526 we updated the Consul and Vault fingerprints so that they are no longer periodic. This fixed a problem that cluster admins reported where rolling updates of Vault or Consul would cause a thundering herd of fingerprint updates across the whole cluster. But if Consul/Vault is not available during the initial fingerprint, it will never get fingerprinted again. This is challenging for cluster updates and black starts because the implicit service startup ordering may require reloads. Instead, have the fingerprinter run periodically but mark that it has made its first successful fingerprint of all Consul/Vault clusters. At that point, we can skip further periodic updates. The `Reload` method will reset the mark and allow the subsequent fingerprint to run normally. Fixes: #25097 Ref: #24526 Ref: #24049
tgross
added a commit
that referenced
this pull request
Feb 12, 2025
In #24526 we updated the Consul and Vault fingerprints so that they are no longer periodic. This fixed a problem that cluster admins reported where rolling updates of Vault or Consul would cause a thundering herd of fingerprint updates across the whole cluster. But if Consul/Vault is not available during the initial fingerprint, it will never get fingerprinted again. This is challenging for cluster updates and black starts because the implicit service startup ordering may require reloads. Instead, have the fingerprinter run periodically but mark that it has made its first successful fingerprint of all Consul/Vault clusters. At that point, we can skip further periodic updates. The `Reload` method will reset the mark and allow the subsequent fingerprint to run normally. Fixes: #25097 Ref: #24526 Ref: #24049
tgross
added a commit
that referenced
this pull request
Feb 13, 2025
…25102) In #24526 we updated the Consul and Vault fingerprints so that they are no longer periodic. This fixed a problem that cluster admins reported where rolling updates of Vault or Consul would cause a thundering herd of fingerprint updates across the whole cluster. But if Consul/Vault is not available during the initial fingerprint, it will never get fingerprinted again. This is challenging for cluster updates and black starts because the implicit service startup ordering may require reloads. Instead, have the fingerprinter run periodically but mark that it has made its first successful fingerprint of all Consul/Vault clusters. At that point, we can skip further periodic updates. The `Reload` method will reset the mark and allow the subsequent fingerprint to run normally. Fixes: #25097 Ref: #24526 Ref: #24049
6 tasks
tgross
added a commit
that referenced
this pull request
Feb 21, 2025
In #24526 we updated Consul and Vault fingerprinting so that we no longer periodically fingerprint. In #25102 we made it so that we fingerprint periodically on start until the first fingerprint, in order to tolerate Consul or Vault not being available on start. For clusters not running Consul, this leads to a warn-level log every 15s. This same log exists for Vault, but Vault support is opt-in via `vault.enable = true` whereas you have to manually disable the fingerprinter for Consul. Make it so that we only log a failed Consul fingerprint once per Consul cluster. Reset the gate on this once we have a successful fingerprint, so that we get the logs after a reload if Consul is unavailable. Ref: #24526 Ref: #25102 Fixes: #25181
tgross
added a commit
that referenced
this pull request
Feb 21, 2025
In #24526 we updated Consul and Vault fingerprinting so that we no longer periodically fingerprint. In #25102 we made it so that we fingerprint periodically on start until the first fingerprint, in order to tolerate Consul or Vault not being available on start. For clusters not running Consul, this leads to a warn-level log every 15s. This same log exists for Vault, but Vault support is opt-in via `vault.enable = true` whereas you have to manually disable the fingerprinter for Consul. Make it so that we only log a failed Consul fingerprint once per Consul cluster. Reset the gate on this once we have a successful fingerprint, so that we get the logs after a reload if Consul is unavailable. Ref: #24526 Ref: #25102 Fixes: #25181
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR changes the Consul and Vault fingerprint implementations to be
reloadable rather than periodic. Reasons described in the issue.
Closes: #24049