otelcol.connector.spanmetrics spanmetrics count is growing infinitely over the period

[ashish0x02]

What's wrong?

i can see spanmetrics count is growing over the period, once grafana-agent restart then only i can see some drop on traces_spanmetrics_calls_total.
i've configured metrics_flush_interval = "5s" and i'm using docker.io/grafana/agent:v0.39.2

issue is similar to grafana/agent#5271

Steps to reproduce

config for otelcol.connector.spanmetrics

// The Spanmetrics Connector will generate RED metrics based on the incoming trace span data.
    otelcol.connector.spanmetrics "tracemetrics" {
        // The namespace explicit adds a prefix to all the generated span metrics names.
        // In this case, we'll ensure they match as closely as possible those generated by Tempo.
        namespace = "traces.spanmetrics"

        // Each extra dimension (metrics label) to be added to the generated metrics from matching span attributes. These
        // need to be defined with a name and optionally a default value (in the following cases, we do not want a default
        // value if the span attribute is not present).
        dimension {
            name = "http.method"
        }
        dimension {
            name = "http.status_code"
        }
        metrics_flush_interval = "5s"

        // A histogram block must be present, either explicitly defining bucket values or via an exponential block.
        // We do the latter here.
        histogram {
            explicit {
              buckets=  ["10ms", "50ms", "100ms", "200ms", "400ms", "1s"]
            }
        }

        // The exemplar block is added to ensure we generate exemplars for traces on relevant metric values.
        exemplars {
            enabled = true
        }

        // Generated metrics data is in OTLP format. We send this data to the OpenTelemetry Prometheus exporter to ensure
        // it gets transformed into Prometheus format data.
        output {
            metrics = [otelcol.processor.filter.default.input]
        }
    }

System information

No response

Software version

agent v0.39.2

Configuration

No response

Logs

No response

[hainenber]

I think this upstream PR would resolve the issue, ofc when it's merged and GA has upgraded to the next OTEL version :D

[ashish0x02]

is there any expected timeline to merge above upstream PR

[hainenber]

It depends on the upstream OpenTelemetry repository authors, fyi

[hainenber]

The PR is merged into upstream but it's not bundled into a release yet so there would be quite sometime to have this landed in the Agent.

[rfratto]

Hi there 👋

On April 9, 2024, Grafana Labs announced Grafana Alloy, the spirital successor to Grafana Agent and the final form of Grafana Agent flow mode. As a result, Grafana Agent has been deprecated and will only be receiving bug and security fixes until its end-of-life around November 1, 2025.

To make things easier for maintainers, we're in the process of migrating all issues tagged variant/flow to the Grafana Alloy repository to have a single home for tracking issues. This issue is likely something we'll want to address in both Grafana Alloy and Grafana Agent, so just because it's being moved doesn't mean we won't address the issue in Grafana Agent :)

[SatyKrish]

Could this be a related issue? We observed spanmetrics keep getting produced when there is no new spans being received.

open-telemetry/opentelemetry-collector-contrib#30559

[rfratto]

The upstream fix (open-telemetry/opentelemetry-collector-contrib#31106) was brought into Alloy with #711 and released with Alloy v1.1; closing as completed.