LOGGING-3416 Enable verbose on log forwarder #164
Conversation
cmd/newrelic-infra/newrelic-infra.go
Outdated
| @@ -298,6 +298,7 @@ func initializeAgentAndRun(c *config.Config, logFwCfg config.LogForward) error { | |||
| FluentBitExePath: c.FluentBitExePath, | |||
| FluentBitNRLibPath: c.FluentBitNRLibPath, | |||
| FluentBitParsersPath: c.FluentBitParsersPath, | |||
| FluentBitVerbose: c.Verbose != 0, | |||
There was a problem hiding this comment.
I wasn't sure whether to enable it when Verbose != 0 or just when Verbose = 1. Also, I am afraid that when Verbose = 3, when shipping the infra-agent logs (and those logs containing the logs being shipped, due that is in verbose mode), this might cause an infinite loop. Maybe it would be wiser to just enable it when Verbose = 1? Any thoughts @carlosroman @varas @cristianciutea ?
There was a problem hiding this comment.
verbose: 2 is a special "smart" verbosity mode, see https://docs.newrelic.com/docs/infrastructure/infrastructure-troubleshooting/troubleshoot-logs/infrastructure-agent-logging-behavior#smart-verbose-mode
There was a problem hiding this comment.
@jsubirat did you experience a retro feedback loop enabling verbosity on both agent and FluentBit?
if that's the case, we should add a circuit breaker
There was a problem hiding this comment.
I tested it now and actually doesn't cause an infinite loop, thanks to the fact that we're not outputting the forwarded records. However, a hacky mind could define an external fluent-bit configuration file to be included (via the @INCLUDE tag) with the following contents:
[OUTPUT]
Name stdout
Match *
And this does cause lots of stuff being written to the infra-agent log, as the records themselves are sent to the infra-agent log, which in turn are captured by fluent-bit, which in turn sends that to the log. It has an exponential growth (about 250MB/min in my VM).
I'd recommend, then, to only enable the verbose mode of Fluent Bit when running with Verbose = 2, what do you think?
There was a problem hiding this comment.
I added a change for FB verbose to be enabled as a feature trace
Pull Request Test Coverage Report for Build 306280047
💛 - Coveralls |
There was a problem hiding this comment.
I'm wondering whether we should forward FB verbosity as:
- log.Debug entries
- or as a feature trace
Waiting for team output on this one.
In the later users would need to enable verbose + the feature trace to get FB verbosity logged.
Config for the later would be like trace: [ log-forwarder ].
This is the same approach we use for other features to avoid flooding the agent output.
Helpers per feature are available at https://github.com/newrelic/infrastructure-agent/blob/master/pkg/trace/features.go#L13-L21
varas
force-pushed
the
LOGGING-3416_enable_verbose_on_log_forwarder
branch
2 times, most recently
from
2afbcd5 to
9a6a70e
Compare
varas
force-pushed
the
LOGGING-3416_enable_verbose_on_log_forwarder
branch
from
9a6a70e to
1180fe3
Compare
| @@ -18,6 +18,7 @@ const ( | |||
| DM_SUBMISSION Feature = "dm.submission" // dimensional metrics submission | |||
| METRIC_MATCHER Feature = "metric.match" // match metric by rule | |||
| INVENTORY Feature = "inventory" | |||
| LOG_FWD Feature = "log.fw" | |||
There was a problem hiding this comment.
this is what needs to be written by the user, ie:
trace:
- log.fw
maybe we should be more verbose on the name? like log.forwarder?
|
✅ |
The purpose of this PR is to be able to enable the verbose mode of Fluent Bit when running the agent in verbose mode. This will greatly help when diagnosing problems in the Log Forwarder.
Also, I took the opportunity to remove the logs/logs folder, which I think was accidentally copied twice and has no use.