New opentelemetry source and sink

[binarylogic]

OpenTelemetry is a specification for collecting observability data.

Their collector and libraries are of questionable quality. We'd like Vector to support OT through their various protobufs and become the best OT collector.

We should break this down into smaller tasks, likely around their various data type (logs, metrics, and traces). I'd like to start with tracing, if possible, to introduce that type into our data model.

• Support ingesting OpenTelemetry logs #13551
• Support ingesting OpenTelemetry metrics #17309
• Support ingesting OpenTelemetry traces #17307
• Support sending OpenTelemetry logs #13622
• Support sending OpenTelemetry metrics #17310
• Support sending OpenTelemetry traces #17308

[loony-bean]

Looks relevant #576

[szibis]

As OpenTracing merged with OpenCensus in one OpenTelemetry project it is considered to support that feature on in and / out ??
It is important to make a Vector replacement for Datadog on the Tracing layer. https://www.datadoghq.com/blog/opentelemetry-instrumentation/ and may also help to build one layer for logs, metrics, and traces.
This also may help to build an architecture that is not vendor locked and allow to switch to other providers easily.

[binarylogic]

Thanks @szibis. Agree, that's the idea with thee OpenTelemetry components. We also want them to enforce our tracing data model when we start to implement it.

This also may help to build an architecture that is not vendor locked and allow to switch to other providers easily.

Agree! That's the primary idea behind Vector. Although, Vector wants to acknowledge current state and help users migrate towards open standards.

[LukeMathWalker]

Hi!
We have been using Vector for a while as a log exporter (stdout -> AWS Kinesis -> ElasticSearch) while we have a separate pipeline for OpenTelemetry traces (application pushes to a Jaeger collector).
We were considering the option of moving to Honeycomb for our observability needs and I noticed Vector provides a honeycomb sink, but it does not provide any OpenTelemetry source. Would we therefore be losing information by using stdout as source and honeycomb as sink compared to pushing our OpenTelemetry data directly into the OpenTelemetry collector and using that to push the data into Honeycomb?

I'd prefer to have a single agent for all our telemetry needs, but it'd be interesting to understand better 👀

[kaarolch]

Any update? I see that tasks related with opentelemetry were removed from different milestones?

[spencergilbert]

Hi @KFearsoff 👋

I do have adding metrics support to the existing opentelemetry source as a work item later this quarter, but otherwise we don't have any existing PRs to add functionality. Was there a particular area of functionality you were looking for first?

[KFearsoff]

I do have adding metrics support to the existing opentelemetry source as a work item later this quarter

This thread consists of "next quarter". I don't mean any offense to you or the Vector team, but I'd like to take the initiative given the opportunity 😄

Was there a particular area of functionality you were looking for first?

I'm particularly interested in tracing source and sink. I don't really care about metrics or logs, but I'm guessing there will be some overlap (like trace -> log transformation). I'd like to get the sources and sinks done before worrying about transforms, though.

otherwise we don't have any existing PRs to add functionality

I don't mind opening ones 😉
Will be referring to the merged OTel parts, Datadog Agent's code and the development docs from this repo. Hopefully I'll be able to come up with something that gets the job done.

[spencergilbert]

This thread consists of "next quarter". I don't mean any offense to you or the Vector team, but I'd like to take the initiative given the opportunity 😄

No offense taken! I've been disappointed personally with not being able to work on it, but that's just how prioritization goes some times 🙂

I'm particularly interested in tracing source and sink. I don't really care about metrics or logs, but I'm guessing there will be some overlap (like trace -> log transformation). I'd like to get the sources and sinks done before worrying about transforms, though.

It looks like we have a few issues open for transforms:

• log_to_trace New transform: log to trace #12029
• log_to_span New transform: log to span #13260
• trace_to_metric New transform: trace to metric #12710

I'd agree that we need more integrations for receiving and sending trace events before the demands for transforms hits a breaking point. I'm not even sure all of the existing transforms have been updated to support receiving traces.

I don't mind opening ones 😉 Will be referring to the merged OTel parts, Datadog Agent's code and the development docs from this repo. Hopefully I'll be able to come up with something that gets the job done.

👍 sounds good! I think adding trace support to the existing opentelemetry source would make the most sense, and be an easier undertaking than adding a completely new sink. If you'd like to collaborate/discuss I'll open a specific issue for enhancing the source or we can chat in our Discord server.

I suspect much of the code for trace support should be straight forward, currently the trace data model is identical to our log data model (type wise) so I don't expect there will be too many surprises from the existing log implementation.

[ericsampson]

Godspeed @KFearsoff 🫡

[KFearsoff]

👍 sounds good! I think adding trace support to the existing opentelemetry source would make the most sense, and be an easier undertaking than adding a completely new sink. If you'd like to collaborate/discuss I'll open a specific issue for enhancing the source or we can chat in our Discord server.

I suspect much of the code for trace support should be straight forward, currently the trace data model is identical to our log data model (type wise) so I don't expect there will be too many surprises from the existing log implementation.

I'm mostly unsure if it would be best to create a dummy source that would take in traces in the current trace data model, or if it's best to start with changing the trace data model (as described in RFC about OTLP traces). I've considered starting with the trace data model, because I think traces and logs are quite different, but I've looked over it and I think starting with that might be a little too ambitious for me 😅

Not sure if treating traces like logs is fine, though (especially because it would make it quite hard to create a sink). What do you think?

I'll join Discord a little bit later, eager to start!

[spencergilbert]

Not sure if treating traces like logs is fine, though (especially because it would make it quite hard to create a sink). What do you think?

I think given that's how we're handling Datadog traces today it should be fine, I'd agree that updating the data model is a much bigger task and would require a lot more consensus and coordination. I'll check what the rest of the team thinks and get back to you.

[spencergilbert]

I'm going to switch over to #17307 for further discussion specific to adding trace support here 👍

[fzyzcjy]

Hi, is there any updates? Thanks!

[benjaminhuo]

@spencergilbert I'm curious that now vector's OpenTelemetry Source can only ingest logs for now, metrics and traces are not yet supported?

[spencergilbert]

@spencergilbert I'm curious that now vector's OpenTelemetry Source can only ingest logs for now, metrics and traces are not yet supported?

Hi @benjaminhuo - metrics and traces are not yet supported, but we plan to support metrics in the near future.

[srstrickland]

Any updates or timeline here? Looking to build out an observability pipeline, and I was excited to use Vector for everything until I noticed the minimal support for OTel. An option we'll be evaluating now is building the entire observability pipeline with OTel collectors & exporters, since in theory they can handle logs, metrics, and traces (though the support for logs via OTel is still very young). My experimentation with vector for logs has been great so far, and it was a huge disappointment to find that we'll likely need another solution for traces (and maybe metrics). Here's the landscape as I see it:

For traces, there's exactly one source (datadog) and one sink (datadog).

For metrics, I am primarily interested in capturing (code-instrumented) application metrics. Ignoring the datadog libs/agent for a moment, it seems like my options for getting these into vector are:

1. StatsD (ugh)
2. Prometheus / scraping (not great for batch jobs)
3. Prometheus / remote write
4. Logs to metrics (feels like this should be used for edge cases, not the primary mechanism)

Publishing vector-shipped metrics to some endpoint are also a bit limited, but I could probably get by with what's available. The transformations for metrics seem very useful. In particular, the tag cardinality transform seems critical for maintaining downstream health (especially certain DBs). I have yet to find a similar OTel processor, so that's a big selling point for vector metrics, but I'm not sold on the mechanisms for getting metrics into vector. I would love to use OTel.

Am I missing something? Has anyone faced a similar situation? My options seem to be: vector for logs (and maybe metrics), OTel for tracing (and maybe metrics)... Or OTel for everything. (I am not sure about the OTel mechanism & data model for logs, though).

Reading through the history on this thread, it's hard for me to discount the likelihood that this work will never receive internal prioritization because it's counter to datadog's business model. I get it. Datadog needs to sell datadog. Maybe the answer here is for community contribution, as some folks here have already discussed. It would be great to get an idea on where things stand, so we're not just waiting for Godot.

[cboudereau]

Hi @srstrickland,

I had barely the same feelings for vector vs otel. Here is my comparison to help the vector team to enhance OTEL support.

You will find a comparison and working demo (end to end from signal to grafana dashboard for services and agents https://github.com/cboudereau/consul-demo/blob/main/service-discovery/up.sh). I made this demo during DataDog evaluation where I showed the solution to the DataDog Engineer solution from parsing with agent observability to monitor agents for vector and otelcol-contrib:
https://github.com/cboudereau/consul-demo/tree/main/service-discovery/agents

In our case we choose both but we would like to reduce our complexity.

[gaby]

I also ran into the same issue. All this time I thought Vector supported OTEL, to find out it only supports part of it.

[fpytloun]

Any plan to increase priority for implementing full OTEL support?

[hobbyhorse]

This is the most depressing thing I have seen on github in a while.

[pront]

Basic support for ingesting logs/metrics/traces was added here. Future work involves gRPC support and native event conversion.

Leaving this ticket open until we add support for metrics and traces to the OpenTelemetry source. Then we can close this and the community can start creating more targeted issues for bugs/enhancements.

[fpytloun]

Basic support for ingesting logs/metrics/traces was added here. Future work involves gRPC support and native event conversion.

Leaving this ticket open until we add support for metrics and traces to the OpenTelemetry source. Then we can close this and the community can start creating more targeted issues for bugs/enhancements.

Nice. That's a good start. I was more thinking about having source than sink to have Vector in role of aggregator.

[pront]

I was more thinking about having source than sink to have Vector in role of aggregator.

This should work for logs and traces. Did you try it? Not sure how well tested it is. Feel free open a discussion or a new issue.

Again, it would be easier if we stopped using this issue as an OpenTelemetry/Vector catch-all.