fix: plugin grpc server retry method #1096
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: googs1025 <[email protected]>
|
friendly ping @elezar /PTAL thanks |
elezar
approved these changes
Jan 9, 2025
|
@cdesiniotis can you help this ? |
elezar
requested changes
Feb 12, 2025
| @@ -183,11 +183,13 @@ func (plugin *nvidiaDevicePlugin) Serve() error { | |||
| go func() { | |||
| lastCrashTime := time.Now() | |||
| restartCount := 0 | |||
| maxRestarts := 5 | |||
| crashTimeoutSeconds := float64(3600) // 1 hour in seconds | |||
There was a problem hiding this comment.
Suggested change
| crashTimeoutSeconds := float64(3600) // 1 hour in seconds | |
| crashTimeout := 1 * time.Hour |
| @@ -198,17 +200,29 @@ func (plugin *nvidiaDevicePlugin) Serve() error { | |||
| break | |||
| } | |||
|
|
|||
| klog.Infof("GRPC server for '%s' crashed with error: %v", plugin.rm.Resource(), err) | |||
|
|
|||
| timeSinceLastCrash := time.Since(lastCrashTime).Seconds() | |||
There was a problem hiding this comment.
Suggested change
| timeSinceLastCrash := time.Since(lastCrashTime).Seconds() | |
| timeSinceLastCrash := time.Since(lastCrashTime) |
| timeSinceLastCrash, | ||
| ) | ||
|
|
||
| if timeSinceLastCrash > crashTimeoutSeconds { |
There was a problem hiding this comment.
Suggested change
| if timeSinceLastCrash > crashTimeoutSeconds { | |
| if timeSinceLastCrash > crashTimeout { |
| plugin.rm.Resource(), | ||
| err, | ||
| restartCount+1, | ||
| timeSinceLastCrash, |
There was a problem hiding this comment.
You probably need:
Suggested change
| timeSinceLastCrash, | |
| timeSinceLastCrash.Seconds(), |
or need to adjust the format string.
| // add a small delay before restarting to prevent tight loops | ||
| retryDelay := 5 * time.Second | ||
| klog.Infof("Waiting for %v before attempting to restart GRPC server for '%s'", retryDelay, plugin.rm.Resource()) | ||
| time.Sleep(retryDelay) // Wait for 5 seconds before attempting to restart |
There was a problem hiding this comment.
Suggested change
| time.Sleep(retryDelay) // Wait for 5 seconds before attempting to restart | |
| time.Sleep(retryDelay) |
| // it has been one hour since the last crash.. reset the count | ||
| // to reflect on the frequency | ||
| restartCount = 0 | ||
| } else { | ||
| restartCount++ | ||
| } | ||
|
|
||
| // add a small delay before restarting to prevent tight loops |
There was a problem hiding this comment.
Is there a reason that you chose 5 seconds specifically? This seems quite long.
Comment on lines
215
to
216
| // it has been one hour since the last crash.. reset the count | ||
| // to reflect on the frequency |
There was a problem hiding this comment.
Let's rather just log the crashTimeout instead of a comment that is now further from the place where the one hour is specified.
Comment on lines
190
to
191
| // quite if it has been restarted too often | ||
| // i.e. if server has crashed more than 5 times and it didn't last more than one hour each time |
There was a problem hiding this comment.
This comment should be moved to the docstring or at the point where we declare the variables.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
To improve observability and facilitate debugging, this PR enhances logging after the GRPC server crashes