[tempo-distributed] Querier.queryRangeRecent: empty ring error with rate function

[vaibhhavv]

Hi Team,
I'm using Tempo Version: 2.6.0, Tempo-Distributed Helm Chart Version: 1.21.1

When I query traces on grafana, I get the output.

But when I run some queries with the rate and count_over_time function, it shows a weird error. The errors tell it's the ring issue, but it's not because when you run the query without these functions, it works perfectly.

Is it the intended behaviour, or am I doing it wrong?

From here, I got to know we have to enable the metrics-generator and enable a few configurations. But we are generating metrics from traces from open telemetry processors and do not want to use metrics-generator but want to use functions like rate() etc to find how many spans are success and how many are failed.

Do we have any suggestions for this use case?

[meSATYA]

From the documentation, it appears like to run the rate or count_over_time or any TraceQL metric functions, metrics generator should be enabled.

https://grafana.com/docs/tempo/latest/traceql/metrics-queries/solve-problems-metrics-queries/#how-the-query-looks-in-promql
https://grafana.com/docs/tempo/latest/traceql/metrics-queries/solve-problems-metrics-queries/#how-the-query-looks-in-traceql

From the above links, it can be inferred that the TraceQL metric function uses "traces_spanmetrics_calls_total" metric in the backend when we use this functions in TraceQL.

The developers can share more insights on this.