Tempo: Include OpenTelemetry Resource Processor as an extra pipeline for trace configuration

[hedss]

Grafana Agent currently supports the OpenTelemetry Collector Attribute processor (https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/attributesprocessor/README.md) as a configurable part of it's span processing pipeline.

There is another processor that pertains to the Resource Attributes that would be useful in some circumstances (https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/resourceprocessor/README.md). The Prometheus SD processor already has the ability to make extra resource attributes based upon resources already set, and users should be able to configure specific resource properties as well.

[rfratto]

👋 Are we talking about something for Flow mode-only, or both Flow and static mode?

[hedss]

Heya! Hmm. Probably both atm, given that we support the Attributes processor in static mode and I assume we want to support both in Flow too? Happy to further discuss if that might be problematic though!

[ptodev]

A Flow component for this has been requested by @cyrille-leclerc. IMO we should consider this issue (#454) to only have the Flow component in scope, and we should only consider working on a Static mode feature for this if there is a strong need for it, via another GitHub issue.

[kago-dk]

Access to the Resource Attributes (in the flow component) would also ease the "Configuration via attribute hints" for the Loki Exporter.
https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/lokiexporter#labels

I encountered that challenge when migrating from OTEL Collector configuration to Grafana Agent (Flow mode), as essential OTEL collector components are still missing in Grafana Agent.

[ptodev]

Although the Agent hasn't ported the Collector's resource processor, it did port over the transform processor in the form of otelcol.processor.transform.

As this comment says, the transform processor is set to replace the resource processor over time. I believe the main advantage of the resource processor at the moment is that its stability is "beta" as opposed to transform processor's "alpha" stability. I believe this is mostly due to OTTL syntax changing a lot.

otelcol.processor.transform in the Agent is labeled as "experimental" because the Collector's transform processor is labeled as "alpha".

There is an open issue in the OTel-Contrib repo to relabel the transform processor as "beta". When that happens, I think we will be very unlikely to port over the resource processor to the Agent and we can safely close this issue.

[cyrille-leclerc]

Thanks @ptodev .
@mar4uk shall we align the OTel Collector Loki exporter to document usage of the Otel Collector Transform Processor (alpha status) as a replacement for Otel Collector Resource Processor (beta status) and Attributes processor (beta status)?

[mar4uk]

I would wait till the transform processor moves to beta

[cyrille-leclerc]

As discussed with @mar4uk

1. We are happy to use the Transform processor in our recommended Grafana Agent and OTel Collector configs
2. We will keep the existing documentation of the Loki exporter using the Resource and Attributes processors as long as the Transform processor is "alpha" status.

[ptodev]

I'm closing this issue because otelcol.processor.transform is considered "stable" in Alloy.

[etiennep]

How would you handle the case where you have an Alloy collector sitting behind an ALB or a proxy and you want to be able to add the k8s.pod.ip resource attribute in order for the k8sattributes processor to add all the kubernetes metadata. Is there a way to do that with the transform processor?

I've figured a pretty hacky way of doing this with the attribute processor, followed by the transform processor.

Step 1. add the XFF header as a span attribute using attribute processor.
Step 2. copy the XFF attribute from the span to the resource, and delete it from the span using the transform processor.
Step 3. let k8sattributes processor do its thing.

There must be a better way to do that, and I believe the resource processor would make my life easier. Unless there's something available in the transform processor already that lets me access the HTTP transport metadata?

[ptodev]

Hi, @etiennep! Your point about the resource processor having the from_context is a valid one. I'll check if there are plans to support it in the transform processor. If there are no such plans, then I agree that it might be good to consider adding the resource processor.

Regarding your specific k8sattributes issue, I believe the "clean" way to do this is to add the k8s.pod.ip resource attribute prior to the ALB/proxy layer. I could think of two ways:

• Using the OTEL_RESOURCE_ATTRIBUTES env var.
• Using a otelcol.processor.k8sattributes configured with passthrough = true. If you're not currently running collectors before the ALB/proxy layer, this could be inconvenient.

[ptodev]

@etiennep I opened a new issue in the upstream Collector repo for the transform processor to have an equivalent to the resource processor's from_context config argument. The transform processor's maintainers mentioned on the CNCF slack that they would consider it.

I do agree that using the XFF header + a from_context to update the resource attributes straight away (without involving span attributes) could also be considered "clean", as long as the XFF header is a reliable source of truth.