stats/internal: OpenTelemetry tracing GRPCTraceBinPropagator

[purnesh42H]

This PR adds the GRPCTraceBinPropagator that implements TextMapPropagator as proposed in gRFC A72 which outlines a custom propagator to handle both binary and text formats for trace context propagation. This will allow gRPC to keep using grpc-trace-bin header for context propagation and also support any other text propagators. When using grpc-trace-bin the OpenCensus SpanContext and OpenTelemetry SpanContext are identical, therefore a gRPC OpenCensus client can speak with a gRPC OpenTelemetry server and vice versa. It is encouraged to use GRPCTraceBinPropagator for the migration. Using the same header greatly simplifies rollout.

Injecting trace context

• GRPCTraceBinPropagator first attempts to retrieve any existing binary trace data from the provided context using stats.Trace(). If found, it means that a trace data was injected by a system using gRPC OpenCensus plugin. Hence, we inject this trace data into the carrier, allowing trace from the systems which are using gRPC OpenCensus, to continue to propagate downstream. However, we set the value in string format against grpc-trace-bin key so that downstream systems which are using gRPC OpenTelemetry plugin are able to extract it using GRPCTraceBinPropagator.

• GRPCTraceBinPropagator then attempts to retrieve an OpenTelemetry span context from the provided context. If not found, that means either there is no trace context or previous system had injected it using gRPC OpenCensus plugin. Therefore, it returns early without doing anymore modification to carrier. If found, that means previous system had injected using OpenTelemetry plugin so it converts the span context to binary and set in carrier in string format against grpc-trace-bin key.

Extracting trace context

• GRPCTraceBinPropagator first attempts to read grpc-trace-bin header value from carrier. If found, that means the trace data was injected using gRPC OpenTelemetry plugin using GRPCTraceBinPropagator. It then set the trace data into context using stats.SetTrace for downstream systems still using gRPC OpenCensus plugin to be able to use this context.

• GRPCTraceBinPropagator then also extracts the OpenTelemetry span context from binary header value. If span context is not valid, it just returns the context as parent. If span context is valid, it creates a new context containing the extracted OpenTelemetry span context marked as remote so that downstream systems using OpenTelemetry are able use this context.

RELEASE NOTES: None

[codecov]

Codecov Report

Attention: Patch coverage is 87.93103% with 7 lines in your changes missing coverage. Please review.

Project coverage is 81.85%. Comparing base (e2b98f9) to head (ee2869c).

Files with missing lines	Patch %	Lines
stats/opentelemetry/grpc_trace_bin_propagator.go	87.50%	4 Missing and 1 partial ⚠️
...s/opentelemetry/internal/tracing/custom_carrier.go	88.88%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7677      +/-   ##
==========================================
- Coverage   81.91%   81.85%   -0.07%     
==========================================
  Files         374      376       +2     
  Lines       37930    37988      +58     
==========================================
+ Hits        31071    31095      +24     
- Misses       5566     5588      +22     
- Partials     1293     1305      +12     

Files with missing lines	Coverage Δ
...s/opentelemetry/internal/tracing/custom_carrier.go	88.88% <88.88%> (ø)
stats/opentelemetry/grpc_trace_bin_propagator.go	87.50% <87.50%> (ø)

... and 18 files with indirect coverage changes

[dfawley]

I'll let @zasweq do the primary review here and just take a 2nd pass.

[zasweq]

Some comments on implementation; didn't review tests yet. My next pass will be more on correctness, this pass was more general Go style/API and docstring semantics.

[zasweq]

Here and elsewhere: I don't know what these comments semantically refer to. What does "cross-cutting concerns" mean? I don't think this is a strict requirement "within a carrier that travels in-band across process boundaries."

[purnesh42H]

both "cross-cutting concerns" and "within a carrier that travels in-band across process boundaries." are taken directly from OpenTelemetry propagation.TextMapCarrier interface https://pkg.go.dev/go.opentelemetry.io/otel/propagation#TextMapPropagator.

"cross-cutting concerns" simply refers to functionalities that are not central to the core logic of your application but are needed across different parts of it (eg. tracing, logging etc.) which are transported "in-band" (within the main data flow) across process boundaries (separate applications).

However, you are right to point out that we should keep the docstrings simplified for anyone to understand the purpose of the method. I have simplified the comments everwhere now. Let me know what you think.

[zasweq]

Is this the correct error message? I think both in the case it's not present in md and if it's set to an empty byte string it'll be not found. Is an empty byte string distinct from nil in the return? Is the empty byte string a valid thing to transmit for this header? Do we want this to return an error at all?

[purnesh42H]

Removed returning error from here to be consistent with Get(..)

[zasweq]

I'm fine with the error. What is the intended usage of this function? Do you intended to treat not set and set to an empty string equivalent? Is empty string a valid key for this? In that case you should make the error message fail with that. But that relates to the intended usage of this, and the failure modes (i.e. what happens in the error case).

[purnesh42H]

Here we are only looking trace context which needs to conform to following struct. So the propagator will not allow to set any such value (like empty) as it won't be a valid SpanContext. Hence, the only reason for returning nil is client didn't set the grpc-trace-bin header.

// Fields:
        //
        // TraceId: (field_id = 0, len = 16, default = "0000000000000000") -    
        // 16-byte array representing the trace_id.
        //
        // SpanId: (field_id = 1, len = 8, default = "00000000") - 8-byte array   
        // representing the span_id.
        //
        // TraceOptions: (field_id = 2, len = 1, default = "0") - 1-byte array 
           representing the trace_options.

[purnesh42H]

Is empty string a valid key for this?

Binary methods will not deal with string keys because we are using existing stats package

grpc-go/stats/stats.go

Line 316 in d9d8f34

func SetTrace(ctx context.Context, b []byte) context.Context {

to set bytes which sets a custom struct key

[zasweq]

I don't get the point of this API. It seems to do things by deferring to operations on a context either to the stats package or the metadata package? What is the function of this API?

[purnesh42H]

yeah this requires a bit more background on migration path from opencensus to opentelemetry (see A72).

The design proposes that while gRPC OpenCensus directly interacts with metadata API, gRPC Open Telemetry will use standardized https://pkg.go.dev/go.opentelemetry.io/otel/propagation package for context propagation by encoding them in metadata for the following benefits:

• Full integration with OpenTelemetry APIs that is easier for users to reason about.
• Make it possible to plugin other propagators that the community supports.
• Flexible API that allows clean and simple migration paths to a different propagator.

As of today, OpenTelemetry propagator API only supports https://pkg.go.dev/go.opentelemetry.io/otel/propagation#TextMapPropagator, that is to send string key/value pairs between the client and server, which is different from the binary header that gRPC currently uses. The future roadmap to support binary propagators at OpenTelemetry is unclear. So, gRPC will use propagator API in TextMap format with optimization path to work around the binary propagator API. Once the Open Telemetry binary propagator API is available in the future, we can continuously integrate with those API with little effort.

Therefore, we need a custom implementation for the Carrier that supports both binary and text values. For binary header grpc-trace-bin we are reusing the existing stats package to conform with existing grpc design.

[zasweq]

It also returns empty string if it's set in the context but has no values. But I don't know if this is the right behavior, should we return an error? What is the intended usage of this?

[purnesh42H]

Actually, I replied above the reason why empty value is not possible. We check if the SpanContext is valid in propagator's Inject() method before setting in carrier

span := oteltrace.SpanFromContext(ctx)
	if !span.SpanContext().IsValid() {
		return
	}
	```

[zasweq]

Please add to documentation the case if it's set in context but has no values. We don't assume this operation comes after that in the callsite, this is a documentation for this layer/operation.

[purnesh42H]

Added

[zasweq]

Please make t-test.

See https://github.com/grpc/grpc-go/blob/master/stats/opentelemetry/csm/observability_test.go#L200 for an example.

[purnesh42H]

yeah i had tabular test before but both tests have different veriffication logic so I wrote in this way

[purnesh42H]

yeah i had them as tabular test before but verification logic is significantly different because of binary and string format so i wrote it this way

[purnesh42H]

I saw there are some existing tests written in similar way

[zasweq]

You can declare the want in the t-test as a variable.

[purnesh42H]

If you see logic to retrieve got is different as well. For fast path which will use SetBinary we get it through stats package because we are using CustomCarrier where as for slow path which will use Set, we just retrieve directly using carrier's Get because we are using a different carrier (not implemented by us). Would you prefer to have separate tests altogether like TestInject_FastPath, TestInject_SlowPath?

[zasweq]

Yeah separate tests are preferred over t.Run(). Thanks.

[purnesh42H]

Separated the fast and slow path tests

[zasweq]

Same comments on this test as above.

[purnesh42H]

Done

[purnesh42H]

@dfawley as we discussed offline, I have modified the Carrier and GRPCTraceBinPropagator to directly set the binary header as string in carrier's metadata. However, I still use stats package stats package is to check if trace data was inserted using opencensus during injection and to insert binary header to outgoing context during extraction

[purnesh42H]

@dfawley as discussed offline. I have removed the stats package usages because both opencensus and opentelemtry plugin set's the binary header in grpc-trace-bin in metadata. Hence, Otel plugin server can read the trace context from OC plugin client and vice-versa.