exporter-collector-grpc traceId and spanId values in protobuf are incorrect

[blumamir]

What version of OpenTelemetry are you using?

v0.12.0

What version of Node are you using?

v12.18.3

Please provide the code you used to setup the OpenTelemetry SDK

import { CollectorTraceExporter } from '@opentelemetry/exporter-collector-grpc';
const collectorTraceExporter = new CollectorTraceExporter({
    url:'localhost:55680',
    serviceName,
});

What did you do?

Used the exporter-collector-grpc to export traces to the latest version of opentelemetry-collector.
Then I used the loggingExporter to log the traces as they received in the collector, and kinesisexporter to export traces from the collector.

What did you expect to see?

Expected the trace id and span id to match the value I see in JS when they are created.

What did you see instead?

In the logging exporter I see different values then the values in JS.
For example, instead of the following trace id: 0c3a85a5506373fd7448be50881ebf58 (valid length of 32 hex -> 16 bytes), I get the following trace id in the collector: d1cddaf396b9e74eb7ef77ddef8e3c6dee74f3cd5e6dfe7c (48 hex -> 24 bytes).

When I try to export the data to kinesis in Jaeger format, I get the following error:
2020-10-25T14:59:21.808+0200 ERROR [email protected]/exporter.go:54 error translating span batch {"metadata": {"appName":"opentelemetry-collector","appVersion":"latest","env":"","hostname":"Amirs-MacBook-Pro.local"}, "component_kind": "exporter", "component_type": "kinesis", "component_name": "kinesis", "error": "TraceID does not have 16 bytes"}

Additional context

When printing the incoming protobuf message in the otlpreceiver, the value is already incorrect, so the issue is probably in the JS exporter. Also, the rest of the span as logged in the the collector do look ok, it is just the trace and span ids which are incorrect.

I suspect that the issue is that when transforming the traces in js to proto format, the traceId and spanId are used as strings where in the proto file they are defined as bytes
I can see that in the metrics definitions they are not defined as string but as UInt8Array which I believe is the right type.
I am not familiar enough with protobut and grpc, but suspect that there is a missing transformation from the string representation of the trace where each hex digit is saved as ascii code, to the bytes array representation of the actual values.

[vmarchaud]

@blumamir Which collector version are using ? If you are using the latest (0.13.0), could you try with 0.12.0 ? I have a setup working with this version.

[blumamir]

I tried with collector version v.0.12.0 and still see wrong traceId.
This is how I run the docker:
docker run --rm -p 55680:55680 -v "${PWD}":/otel-local-config --name otelcol otel/opentelemetry-collector:0.12.0 --config otel-local-config/otel-config.yaml

and this is the collector config:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:55680
      http:
        endpoint: 0.0.0.0:55681

processors:
  batch:

exporters:
  logging:
    logLevel: debug

service:

  pipelines:

    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging]

[blumamir]

tried to run it now agains the basic-trace-node modified to export with grpc like this:

const {
  CollectorTraceExporter,
} = require("@opentelemetry/exporter-collector-grpc");

const provider = new BasicTracerProvider({ logger: console });

// Configure span processor to send spans to the exporter
const exporter = new CollectorTraceExporter({ serviceName: "basic-service" });
provider.addSpanProcessor(new SimpleSpanProcessor(exporter));
provider.addSpanProcessor(new SimpleSpanProcessor(new ConsoleSpanExporter()));

What I get in JS is:

{
  traceId: 'b2685323bfd7fcb29f663d259cc14578',
  parentId: '7700ed8e54dea186',
  name: 'doWork',
  id: 'd1583ba517c280b4',
  kind: 0,
  timestamp: 1603639516438789,
  duration: 862,
  attributes: { key: 'value' },
  status: { code: 0 },
  events: [ { name: 'invoking doWork', attributes: undefined, time: [Array] } ]
}

traceId is valid 32 hex digits.

In collector I get this:

ResourceSpans #1
Resource labels:
     -> service.name: STRING(basic-service)
     -> telemetry.sdk.language: STRING(nodejs)
     -> telemetry.sdk.name: STRING(opentelemetry)
     -> telemetry.sdk.version: STRING(0.12.0)
InstrumentationLibrarySpans #0
InstrumentationLibrary example-basic-tracer-node *
Span #0
    Trace ID       : 6f6ebce77db76df77b7dc6f6f5febaddddb9f5c735e39efc
    Parent ID      : efbd3479df1ee7875e6b5f3a
    ID             : e78ef8efd75c79ae5ed3bedc
    Name           : doWork
    Kind           : SPAN_KIND_INTERNAL
    Start time     : 2020-10-25 15:25:16.453477888 +0000 UTC
    End time       : 2020-10-25 15:25:16.453595392 +0000 UTC
    Status code    : STATUS_CODE_OK
    Status message :
Attributes:
     -> key: STRING(value)
Events:
SpanEvent #0
     -> Name: invoking doWork
     -> Timestamp: 1603639516453588480
     -> DroppedAttributesCount: 0

traceId: 6f6ebce77db76df77b7dc6f6f5febaddddb9f5c735e39efc is 48 hex digits.

[dyladan]

I think this is most likely related to #1513 (implemented in #1588, released in https://github.com/open-telemetry/opentelemetry-js/releases/tag/v0.12.0).

The collector recently changed how it interprets trace and span ids. Either we incorrectly implemented the fix or you are using an incompatible collector version. Can you clarify exactly which collector version you're using?

[blumamir]

I tested it both with opentelemetry-collector v0.12.0 and with collector v0.13.0 (latest)
The docker run command I used to run the collector is attached above

[dyladan]

@obecny Seems like there may be a bug in the id conversion using grpc.

[obecny]

so this is funny, when I added back conversion from hex to base64 grpc and proto works fine. So what is wrong with collector ? Was the spec https://github.com/open-telemetry/opentelemetry-specification/pull/911 not implemented yet to collector when using proto ?

[obecny]

@blumamir can you please check if this fix -> #1627, works for you ?

[blumamir]

@obecny Tested your fix and it works well now, both with gRPC and HTTP/JSON. Thank you for fixing it.

The one issue that I still think is problematic is the types in this file.
traceId and spanId are sometimes typed as Uint8Array, (traceId: Uint8Array;) and sometimes as string (traceId: string;)
Is it by design? I think it should be consistent, as they are the same type (bytes) in the proto file:
here here and here

[alolita]

@dyladan @obecny - we'd like to get a release with this fix in it. When is the next release scheduled for? Can we get it this week please?

[obecny]

Hi @alolita from our last Sig meeting we should prepare next version this week.