Master babysit retry #5036
Merged
Master babysit retry #5036
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Even if they're dying, we want to keep trying - for any temporary condition, this is better than waiting forever.
This helps with log monitoring
elliefm
reviewed
Sep 16, 2024
Comment on lines
535
to
537
| xlog $self, "check that the error was syslogged"; | ||
| my @loglines = $self->{instance}->getsyslog(); | ||
| $self->assert(grep { m/too many failures for service/ } @loglines); |
There was a problem hiding this comment.
This will give false failures if the syslog replacement injection had failed. Have a look at $self->assert_syslog_matches()... I can't remember if it's exactly suitable here, but if it's not, at least the implementation will show the right way to handle this sort of thing.
There was a problem hiding this comment.
yes, can probably just use that :) Thanks
There was a problem hiding this comment.
I added this!
We can test new behaviour by hand by killing something every second and watching the logs
elliefm
approved these changes
Sep 16, 2024
ksmurchison
approved these changes
Sep 16, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I recently moved sync_client to be managed by Cyrus rather than a separate "monitorsync" script.
Likewise, we have other things started under the DAEMON block which are supposed to run forever at Fastmail:
pusher, calalarmd, squatter, idled, bayesrpcd and one day again: promstatd.
This change means that if we hit the restart limit, we retry every 10 seconds forever for anything with the 'babysit' flag - which includes all daemons. We only try once per 10 seconds unless we get a clean 10 seconds, in which case it resets back to allowing 5 failures with immediate restart. This should stop it causing a totally crazy amount of restarts.
Finally: I added the string "ERROR" to the messages about hitting the limits, which will help with Fastmail's log tracking.