Start adding log correlation for Stackdriver Logging. #1212

sebright · 2018-05-25T08:53:53Z

This is a rough PR for early feedback. It adds a library for inserting tracing data into Stackdriver log entries, through the google-cloud-logging library. The tests aren't complete, and it has some TODOs.

I also included a demo application. It contains SLF4J log statements and OpenCensus tracing instrumentation. It configures logging with a Logback XML configuration and exports logs using the Stackdriver Logging Logback appender. It also exports traces using opencensus-exporter-trace-stackdriver, so that Stackdriver can show the log entries associated with each trace.

The demo requires a Google Cloud project with Stackdriver Logging enabled. It can be run with

$ ./gradlew :opencensus-contrib-log-correlation-stackdriver-demo:installDist
$ ./contrib/log_correlation/stackdriver_demo/build/install/opencensus-contrib-log-correlation-stackdriver-demo/bin/OpenCensusTraceLoggingEnhancerDemo

Here is a screenshot of a trace with log statements in Stackdriver:

The demo is currently a subproject of opencensus-java, but I'm planning to move it.

codecov-io · 2018-05-25T09:09:46Z

Codecov Report

Merging #1212 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@             Coverage Diff              @@
##             master    #1212      +/-   ##
============================================
+ Coverage     82.14%   82.15%   +<.01%     
- Complexity     1252     1256       +4     
============================================
  Files           193      194       +1     
  Lines          6100     6109       +9     
  Branches        562      562              
============================================
+ Hits           5011     5019       +8     
  Misses          936      936              
- Partials        153      154       +1

Impacted Files	Coverage Δ	Complexity Δ
...on/stackdriver/OpenCensusTraceLoggingEnhancer.java	`100% <100%> (ø)`	`4 <4> (?)`
...census/implcore/trace/export/SpanExporterImpl.java	`91.78% <0%> (-1.37%)`	`8% <0%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5926a85...6ce8f89. Read the comment docs.

sebright · 2018-05-31T03:41:42Z

I also mentioned that the log correlation feature is experimental in the Javadoc.

I'm planning to move the demo to https://github.com/census-ecosystem/opencensus-experiments before I merge this PR.

g-easy

LGTM. Please also get review from a Java perspective.

bogdandrutu · 2018-05-31T14:11:33Z

...in/java/io/opencensus/contrib/logcorrelation/stackdriver/OpenCensusTraceLoggingEnhancer.java

+  }
+
+  private static String formatTraceId(TraceId traceId) {
+    // TODO(sebright): Should we cache the project ID, for efficiency?


Yes please. You should probably cache "projects/" + projectId + "/traces/"

bogdandrutu · 2018-05-31T14:14:34Z

...in/java/io/opencensus/contrib/logcorrelation/stackdriver/OpenCensusTraceLoggingEnhancer.java

+
+  private static String formatTraceId(TraceId traceId) {
+    // TODO(sebright): Should we cache the project ID, for efficiency?
+    String projectId = ServiceOptions.getDefaultProjectId();


I would accept the projectId as a configuration argument for this LogginEnhancer. Reason is because users may configure a different projectId for trace so they should use the same value here. And if not set use default.

Sebruck · 2018-06-01T06:25:47Z

This is great and I didn't knew that it is even possible with Stackdriver! I will definitely use it 👍

Sebruck · 2018-06-01T06:33:58Z

...in/java/io/opencensus/contrib/logcorrelation/stackdriver/OpenCensusTraceLoggingEnhancer.java

+
+  @Override
+  public void enhanceLogEntry(LogEntry.Builder builder) {
+    SpanContext span = tracer.getCurrentSpan().getContext();


Is there any possibility to access the args of a log message within an enhancer and take the span from there (if present)? This would allow people who do not use scoped spans to use this as well.

In Scala we can't use scoped spans in most of the cases since the Future based code is executed on arbitrary threads.

@Sebruck Thanks for the feedback!

Is there any possibility to access the args of a log message within an enhancer and take the span from there (if present)? This would allow people who do not use scoped spans to use this as well.

I don't see any getters on the builder, so I think that would require building the LogEntry within enhanceLogEntry to call the LogEntry's getters. How would the span be set on the log message before it is passed to enhanceLogEntry?

In Scala we can't use scoped spans in most of the cases since the Future based code is executed on arbitrary threads.

What library runs the Futures? The opencensus-java agent (https://github.com/census-instrumentation/opencensus-java/tree/master/contrib/agent) propagates the context to threads or executors, though I don't know whether it could be extended to work with other libraries.

sebright · 2018-06-12T02:00:35Z

I updated the PR to set the LogEntry spanId field after googleapis/google-cloud-java#3332 was released. Now the log messages are shown under the correct spans:

…tifacts list.

The demo project will be moved to https://github.com/census-ecosystem/opencensus-experiments.

sebright · 2018-06-12T22:47:24Z

@g-easy suggested that I merge this PR as is, since it is already large. I added TODOs for the two code review comments, added a few new TODOs, removed the new artifact from the list of released artifacts, and removed the demo application, so it should be ready to merge now.

sebright · 2018-06-12T22:50:27Z

/cc @songy23 @rghetia

g-easy

Looks good!

sebright · 2018-06-12T23:11:34Z

Thanks!

…r (Java). This commit adds a demo for log correlation using OpenCensus, Logback, and Stackdriver. It is very similar to the demo that was originally contained in census-instrumentation/opencensus-java#1212. The demo can be run with the following command in the `java/log_correlation/stackdriver/logback/` directory: `./gradlew clean installDist && ./build/install/logback/bin/OpenCensusTraceLoggingEnhancerDemo`

…r (Java). This commit adds a demo for log correlation using OpenCensus, Logback, and Stackdriver. It is very similar to the demo that was originally contained in census-instrumentation/opencensus-java#1212. The demo can be run with `./gradlew run` in the `java/log_correlation/stackdriver/logback/` directory.

…ackdriver (Java). This commit adds a demo for log correlation using OpenCensus, Logback, and Stackdriver. It is very similar to the demo that was originally contained in census-instrumentation/opencensus-java#1212. The demo can be run with `./gradlew run` in the `java/log_correlation/stackdriver/logback/` directory.

…ackdriver (Java). (#51) This commit adds a demo for log correlation using OpenCensus, Logback, and Stackdriver. It is very similar to the demo that was originally contained in census-instrumentation/opencensus-java#1212. The demo can be run with `./gradlew run` in the `java/log_correlation/stackdriver/logback/` directory. `gradle-wrapper.jar`, `gradle-wrapper.properties`, `gradlew`, and `gradlew.bat` are generated by gradle.

sebright added the tracing label May 25, 2018

googlebot added the cla: yes label May 25, 2018

sebright added the DO NOT MERGE label May 26, 2018

sebright requested review from bogdandrutu and g-easy May 26, 2018 01:53

sebright assigned bogdandrutu and g-easy May 26, 2018

g-easy approved these changes May 31, 2018

View reviewed changes

sebright mentioned this pull request May 31, 2018

google-cloud-logging LogEntry is missing a "spanId" field. googleapis/google-cloud-java#3325

Closed

bogdandrutu reviewed May 31, 2018

View reviewed changes

sebright assigned sebright and unassigned bogdandrutu and g-easy May 31, 2018

sebright added the action required The pull request is blocked by something other than a need for code review. label May 31, 2018

Sebruck reviewed Jun 1, 2018

View reviewed changes

sebright force-pushed the stackdriver-log-correlation branch from 1f898aa to d1bb669 Compare June 12, 2018 01:41

sebright added 10 commits June 12, 2018 15:38

Start adding log correlation for Stackdriver Logging.

d34e4b2

Mention that the log correlation feature is experimental in the Javadoc.

9a5deb0

Remove TODO about ignoring blank spans.

36ad77a

Upgrade google-cloud-logging and set new LogEntry spanId field.

9d4b3e5

Update TODO about caching the project ID.

4f801e3

Add a TODO about overriding the project ID.

9fbd97e

Temporarily remove opencensus-exporter-trace-logging from released ar…

941361b

…tifacts list.

Remove opencensus-contrib-log-correlation-stackdriver-demo.

73c0787

The demo project will be moved to https://github.com/census-ecosystem/opencensus-experiments.

Add a TODO for configuring the LoggingEnhancer with Java properties.

23faa1e

Add a link to the Stackdriver LoggingEnhancer.

6ce8f89

sebright force-pushed the stackdriver-log-correlation branch from aba05bd to 6ce8f89 Compare June 12, 2018 22:39

sebright changed the title ~~[WIP] Start adding log correlation for Stackdriver Logging.~~ Start adding log correlation for Stackdriver Logging. Jun 12, 2018

sebright removed DO NOT MERGE action required The pull request is blocked by something other than a need for code review. labels Jun 12, 2018

sebright assigned g-easy and unassigned sebright Jun 12, 2018

g-easy approved these changes Jun 12, 2018

View reviewed changes

sebright merged commit b00051b into census-instrumentation:master Jun 12, 2018

sebright deleted the stackdriver-log-correlation branch June 12, 2018 23:11

sebright mentioned this pull request Jun 15, 2018

Add support for correlating traces and logs. census-instrumentation/opencensus-specs#123

Closed

sebright mentioned this pull request Jul 23, 2018

Start adding a Logback demo for opencensus-contrib-log-correlation-stackdriver (Java). census-ecosystem/opencensus-experiments#51

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Start adding log correlation for Stackdriver Logging. #1212

Start adding log correlation for Stackdriver Logging. #1212

sebright commented May 25, 2018

codecov-io commented May 25, 2018 •

edited

Loading

sebright commented May 31, 2018

g-easy left a comment

bogdandrutu May 31, 2018

bogdandrutu May 31, 2018

Sebruck commented Jun 1, 2018

Sebruck Jun 1, 2018

sebright Jun 12, 2018

sebright commented Jun 12, 2018

sebright commented Jun 12, 2018

sebright commented Jun 12, 2018

g-easy left a comment

sebright commented Jun 12, 2018

Start adding log correlation for Stackdriver Logging. #1212

Start adding log correlation for Stackdriver Logging. #1212

Conversation

sebright commented May 25, 2018

codecov-io commented May 25, 2018 • edited Loading

Codecov Report

sebright commented May 31, 2018

g-easy left a comment

Choose a reason for hiding this comment

bogdandrutu May 31, 2018

Choose a reason for hiding this comment

bogdandrutu May 31, 2018

Choose a reason for hiding this comment

Sebruck commented Jun 1, 2018

Sebruck Jun 1, 2018

Choose a reason for hiding this comment

sebright Jun 12, 2018

Choose a reason for hiding this comment

sebright commented Jun 12, 2018

sebright commented Jun 12, 2018

sebright commented Jun 12, 2018

g-easy left a comment

Choose a reason for hiding this comment

sebright commented Jun 12, 2018

codecov-io commented May 25, 2018 •

edited

Loading