Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log non-zero expvars periodically #1955

Merged
merged 1 commit into from
Jul 5, 2016

Conversation

tsg
Copy link
Contributor

@tsg tsg commented Jul 4, 2016

Prints the delta since the last period, if it's non-zero. The idea is
to help with troubleshooting while still creating relatively little output
in the logs. 30s seems like a good compromise.

Part of/related to #1931

@tsg tsg added discuss Issue needs further discussion. review and removed review labels Jul 4, 2016
@tsg
Copy link
Contributor Author

tsg commented Jul 4, 2016

Not ready for merging, but discussion/reviews welcome.

@tsg
Copy link
Contributor Author

tsg commented Jul 4, 2016

Example log line:

2016/07/04 12:16:10.821047 logp.go:194: INFO Non-zero metrics in the last 30s:  libbeatEsPublishedAndAckedEvents=18 libbeatEsPublishWriteBytes=10045 libbeatEsPublishReadBytes=1032 libbeatMessagesInWorkerQueues=3 libbeatPublishedEvents=18 libbeatEsPublishEventsCallCount=3

@tsg tsg added in progress Pull request is currently in progress. review labels Jul 4, 2016
@@ -21,8 +24,18 @@ type Logging struct {
ToSyslog *bool `config:"to_syslog"`
ToFiles *bool `config:"to_files"`
Level string
Metrics LoggingMetricsConfig `config:"metrics"`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. Should we enable this by default?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is enabled by default with a 30s interval.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now I see it on line 170. Would probably make sense to also use a ucfg defaultConfig.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tsg For the above: I didn't mean as part of this PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried initially but didn't work out due to circular imports (cfgfile imports logp).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:-( @urso interesting issue.

@ruflin
Copy link
Member

ruflin commented Jul 4, 2016

LGTM

@tsg tsg force-pushed the log_expvars_periodically branch from 6d38f6a to 5746349 Compare July 5, 2016 09:27
@tsg
Copy link
Contributor Author

tsg commented Jul 5, 2016

Added docs and a basic system test. @ruflin, can you have a look before I squash?

@ruflin
Copy link
Member

ruflin commented Jul 5, 2016

LGTM

Prints the delta since the last period, if it's non-zero. The idea is
to help with troubleshooting while still creating relatively little output
in the logs. 30s seems like a good compromise.

Part of/related to elastic#1931
@tsg tsg force-pushed the log_expvars_periodically branch from 2b881d2 to b378ed8 Compare July 5, 2016 10:07
@tsg tsg removed discuss Issue needs further discussion. in progress Pull request is currently in progress. labels Jul 5, 2016
@ruflin ruflin merged commit 2f3885f into elastic:master Jul 5, 2016
func snapshotExpvars(varsMap map[string]int64) {
expvar.Do(func(kv expvar.KeyValue) {
switch kv.Value.(type) {
case *expvar.Int:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this PR covers the most important metrics for the time being so no action is required at the moment IMO. We can improve this when we add "metrics" within beats. We have some expvar metrics that are nested within a map that won't be printed by this. For example, in Metricbeat we have a fetches.mysql-info.failures metric that is nested two levels deep in maps.

@tsg tsg deleted the log_expvars_periodically branch August 25, 2016 10:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants