Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New transform: log to span #13260

Open
kamalmarhubi opened this issue Jun 21, 2022 · 0 comments
Open

New transform: log to span #13260

kamalmarhubi opened this issue Jun 21, 2022 · 0 comments
Labels
domain: logs Anything related to Vector's log events domain: traces Anything related to Vectors' trace events domain: transforms Anything related to Vector's transform components type: feature A value-adding code addition that introduce new functionality.

Comments

@kamalmarhubi
Copy link

A more basic version of #12029 that I'd love to have: simply generating spans from logs.

If my logs have trace_id and span_id on them, then I can just spit out spans and have some other part of the pipeline (possibly within vector, possibly not) group them by trace for tail based sampling, or stuff like the sql_query_count example in #12710.

Specific use cases for this simpler version:

  • generating spans from GCP load balancer logs; the LogEntry object includes trace info in a GCP-specific format*—a bit of VRL could pluck those out
  • if you instrument sql clients to insert trace context in a SQL comment, you can generate server spans from database server logs with a bit of remap to parse out the trace context from the comment

This might not even by the same transform, but it's a huge missing piece of our nascent pipeline. For load balancer logs, we currently have a cloud function doing the conversion and exporting to our opentelemetry collector. It's a lot more operational overhead than I'd like, and a simple way to turn logs into spans would tidy it all away.


‌* .spanId as hex, .trace as projects/<project_id>/traces/<trace_id_hex>

Originally posted by @kamalmarhubi in #12029 (comment)

@jszwedko jszwedko added domain: transforms Anything related to Vector's transform components domain: logs Anything related to Vector's log events domain: traces Anything related to Vectors' trace events type: feature A value-adding code addition that introduce new functionality. labels Jun 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain: logs Anything related to Vector's log events domain: traces Anything related to Vectors' trace events domain: transforms Anything related to Vector's transform components type: feature A value-adding code addition that introduce new functionality.
Projects
None yet
Development

No branches or pull requests

2 participants