Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add design doc for collector processor experience #41

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

anuraaga
Copy link

@anuraaga anuraaga commented Nov 9, 2021

No description provided.


The processors implementing this use case are `metricsgenerationprocessor`, `spanmetricsprocessor`.

### Grouping
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should cumulative to delta and delta to rate be included here?

telemetry signal. For metrics, the structure presented for processing is actual data points, e.g. `NumberDataPoint`,
`HistogramDataPoint`, with the information from higher levels like `Metric` or the data type available as virtual fields.

Navigation can then be used with a simple expression language for identifying telemetry to operate on.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this expression language designed from scratch or is it a specialization of some existing language to our domain? Is there a formal grammar?


```
create_histogram("duration", end_time_nanos - start_time_nanos) where type = span
keep(attributes, "http.method") where type = metric and descriptor.metric_name = "duration
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
keep(attributes, "http.method") where type = metric and descriptor.metric_name = "duration
keep(attributes, "http.method") where type = metric and descriptor.metric_name = "duration"

keep(attributes, "http.method") where type = metric and descriptor.metric_name = "duration
```

Group spans by trace ID
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this do? I.e., are the grouped traces aggregated in some way or dumped out as Pair<TraceId, List<Span>>? What does the 2m do?


## Declarative configuration

The telemetry query language presents an SQL-like experience for defining telemetry transformations - it is made up of
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A syntax even closer to SQL would be more familiar/approachable to many potential users. It may be worth documenting why we're considering a custom syntax instead.

@jmacd
Copy link

jmacd commented Nov 17, 2021

Tangentially, I've researched this topic and have a prototype I'm trying to extract from Google, see my plea: https://gist.github.com/jmacd/17a6f2885d92d7864ba76680a8ee367a

@erichsueh3 erichsueh3 added the documentation Improvements or additions to documentation label Mar 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants