Migrate from OpenCensus to OpenTelemetry

[yegle]

TL;DR

OpenCensus and OpenTracing has been merged into OpenTelemetry. The new project only promises 2 years of security fixes and we should not expect feature improvement in the OpenCensus libraries.

Design

Proposal
Looking into OpenTelemetry, investigate the backward compatible layer from OpenTelemetry, and migrate to OpenTelemetry.

[yegle]

Related bug: open-telemetry/opentelemetry-go#93

[yegle]

I briefly looked into this. My current understanding is that we either 1) wait for the opencensus-to-opentelemetry bridge (open-telemetry/opentelemetry-go#93) so that we can reuse the OpenCensus instrumentation code, or 2) simply rewrite our observability code to use OpenTelemetry library.

For option 2: rewrite in OpenTelemetry is fine for our custom metrics, but we also have many metrics to gain visibility into third party libraries, e.g. sql and grpc. There's no built-in support in OpenTelemetry for these packages.

Option 1 is more approachable but that'll depend on the implementation of the bridge/shim. The code is not completed yet so we'll have to wait.

[sethvargo]

I'm personally fine dropping the o11y in sql.

[yegle]

I've found the grpc metrics we got for free useful in the past, that would be something nice to have if supported.

https://github.com/open-telemetry/opentelemetry-go-contrib/tree/master/instrumentation is the list of officially supported instrumentation libraries.

I'll start working on implementing our customer metrics in OpenTelemetry. We can continue exporting grpc/sql metrics using OpenCensus. It should work just fine exporting using both metrics.

[yegle]

Tagging upstream issues:

• database/sql metrics support: Opentelemetry for database/sql open-telemetry/opentelemetry-go-contrib#5
• grpc metrics support: Update gRPC instrumentation open-telemetry/opentelemetry-go-contrib#194

[yegle]

Currently blocked by GoogleCloudPlatform/opentelemetry-operations-go#117

[yegle]

Looks like GoogleCloudPlatform/opentelemetry-operations-go#114 is trying to solve this fundamental problem.

The TLDR of the issue: OpenTelemetry's model of "monitored resources" has a mismatch to Cloud Monitoring's monitored resources. The current conversion between these two "resources" is to use heuristics. Adding more heuristic to support generic_task doesn't seem to be long term supportable.

I'll stop working on this for now and await for the upstream to sort this out.

[yegle]

I've made some progress getting Cloud Run resource detector into opentelemetry-go-contrib, and I'm working on improving opentelemetry-operations-go to support exporting from Cloud Run.

Reopen to track the progress.

[sethvargo]

@yegle you mentioned something about this on standup. Do we still want to pursue this, or should we stick with OC?

[yegle]

Let's close this for now. OpenTelemetry has yet to have feature parity with OpenCensus (their focus is currently on implementing OpenTracing features).