Bridge GRPC logs to logrus #6517
Conversation
twz123
added
the
backport/release-1.34
internal/pkg/log/grpc.go
Outdated
| func newGRPCLogger(log logrus.FieldLogger) grpclog.LoggerV2 { | ||
| // https://github.com/grpc/grpc-go/blob/v1.76.0/grpclog/loggerv2.go#L65-L79 | ||
|
|
||
| levelValue := os.Getenv("GRPC_GO_LOG_SEVERITY_LEVEL") |
There was a problem hiding this comment.
We have --logging flag, shouldn't it be there as well?
There was a problem hiding this comment.
Hmm, right now, the logging flag is only for embedded binaries, not for the logging of stuff happening in the k0s executable itself.
There was a problem hiding this comment.
Then why we need a separate log level for it? Can we just use the k0s log level here?
There was a problem hiding this comment.
We could, but the warning would still be displayed because k0s logs warnings and errors by default. This cannot be turned off. You can only request info logs (--verbose) or debug logs (--debug). Logrus itself is not flexible enough to support different log levels for different components either.
Kimmo was trying to troubleshoot this and found the environment variable that usually controls it. That's why I thought it wouldn't be a bad idea for k0s to honor those settings. However, it currently doesn't because the API server overwrites grpc-go's defaults (which is error) in one of its package's init functions.
So yes, we can remove the separate log level, but that wouldn't silence the warning. I'm trying to figure out if this can be fixed upstream because either etcd is doing something unusual or GRPC is logging at the wrong log level. However, I'm not convinced that this will lead anywhere.
There was a problem hiding this comment.
Another option would be to shift the GRPC log levels by one: Error -> Warning, Warning -> Info, Info -> Debug. The GRPC logger doesn't have a Debug level, it has a V threshold instead.
There was a problem hiding this comment.
Hold on! Maybe there's really a chance to fix it upstream. In that case, it'd probably be best to just use k0s log levels.
There was a problem hiding this comment.
@makhov I've removed all the custom handling and made this a simple adapter, just as we do it for cfssl. I'm trying my best to get rid of the warning upstream. However, if that won't work out, then we can still opt to add some means of changing the GRPC log level separately from the general k0s log level.
twz123
changed the title
The Kubernetes API server installs a klog backend for GRPC logs. In k0s, this means that GRPC warnings will be printed to stderr. Install our own logrus adapter for GRPC instead. See: https://github.com/kubernetes/kubernetes/blob/v1.34.1/staging/src/k8s.io/apiserver/pkg/storage/etcd3/logger.go#L27 Signed-off-by: Tom Wieczorek <[email protected]>
2 participants
Description
By default, grpc-go only logs errors. However, the Kubernetes API server installs a klog backend for GRPC logs. In k0s, this means that GRPC warnings will be printed to stderr. Install our own logrus backend instead. Set the default log level to "error," as vanilla grpc-go does. Support the same environment variables to change that behavior.
Note that to see GRPC info logs, one must set the GRPC log level to info and run k0s with either the --verbose or --debug flag.
See:
Type of change
How Has This Been Tested?
Checklist