Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support sending OpenTelemetry logs #13622

Open
3 tasks
Tracked by #1444
spencergilbert opened this issue Jul 19, 2022 · 5 comments
Open
3 tasks
Tracked by #1444

Support sending OpenTelemetry logs #13622

spencergilbert opened this issue Jul 19, 2022 · 5 comments
Assignees
Labels
domain: logs Anything related to Vector's log events sink: opentelemetry Anything `opentelemetry` sink related type: feature A value-adding code addition that introduce new functionality.

Comments

@spencergilbert
Copy link
Contributor

spencergilbert commented Jul 19, 2022

Subtasks:

  • Support gRPC to the OpenTelemetry collector
  • Support HTTP (binary-encoded Protobuf format) to the OpenTelemetry collector
  • Integration tests

gRPC can be modeled after the vector (v2) sink.
HTTP can refer to the datadog sinks.

Ref: https://opentelemetry.io/docs/reference/specification/protocol/exporter/
Ref: https://opentelemetry.io/docs/reference/specification/protocol/otlp/

@spencergilbert
Copy link
Contributor Author

One outstanding question that needs to be answered is how we want to model the configuration for the sink, allowing for either HTTP or gRPC based sending. Perhaps something like:

otel:
  type: opentelemetry
  inputs: ["source0"]
  address: http://otelcol2:4318
  protocol: grpc # grpc OR http
  http:
    path: "/custom/logs" # changing from the default "/v1/logs" path
  tls:
    # ...

Generally want to ensure it's clear what options are valid for http vs grpc, or for both, as well as staying similar/familiar with other sinks.

@caibirdme
Copy link
Contributor

And there's another question, how to fulfill the opentelemetry proto object by vector Event model?
For example:

                          sources                                           sinks
otel-sdk --(opentelemetry proto)--> vector --(opentelemetry proto)> otel-collector(or any other services that support otel)

We'd better make sure vector outputs what it received. And If vector ingest data from file, kafka or any other non-otel sources, how to convert those data(EventLog) to otel proto object

@spencergilbert
Copy link
Contributor Author

And there's another question, how to fulfill the opentelemetry proto object by vector Event model?

We have some inflight work that can help ease the translation between data models, but I think we'd start with something similar to our gelf encoder.

@spencergilbert spencergilbert self-assigned this Jul 21, 2022
@spencergilbert spencergilbert added the sink: opentelemetry Anything `opentelemetry` sink related label Jul 21, 2022
@spencergilbert spencergilbert removed their assignment Aug 18, 2022
@spencergilbert spencergilbert added domain: logs Anything related to Vector's log events source: new A request for a new source type: feature A value-adding code addition that introduce new functionality. and removed source: new A request for a new source labels May 4, 2023
@pront pront self-assigned this Nov 6, 2024
@pront
Copy link
Member

pront commented Nov 11, 2024

Implementation plan

Config

# ...
sinks: 
  sink0:
    type: opentelemetry
    protocol: grpc # or "http"
    grpc:
      # ... like the OTEL source
    # other common sink config params 

@pront
Copy link
Member

pront commented Dec 2, 2024

Note to self: #21866 is a good first step. Keeping this open until we support gRPC. Format mapping will be left as an enhancement for later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain: logs Anything related to Vector's log events sink: opentelemetry Anything `opentelemetry` sink related type: feature A value-adding code addition that introduce new functionality.
Projects
None yet
Development

No branches or pull requests

3 participants