Labels and series lookups only query ingesters

[lostmimic]

https://github.com/cortexproject/cortex/blob/master/pkg/ingester/ingester.go#L833-L834

// TODO Right now we ignore start and end.

This breaks away from the Prometheus API standard for /api/v1/series (https://prometheus.io/docs/prometheus/latest/querying/api/#finding-series-by-label-matchers) which causes problems with things like Grafana generating variable values from labels (https://grafana.com/docs/grafana/latest/features/datasources/prometheus/#templating) for something not occurring in the current timeframe

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[pracucci]

This is still valid. Both applies to chunks and blocks storages.

[containerpope]

In our case, grafana is not able to fetch label values for its variables in case we have no metrics for a while. When metrics are coming in this works fine. In our scenario, we are currently collecting metrics from an iot device that is not running constantly.

If I understand this issue correctly, tools like Grafana cannot send the date range of the dashboard to cortex and it is using a default instead. Am I right with my assumption?

[lostmimic]

If I understand this issue correctly, tools like Grafana cannot send the date range of the dashboard to cortex and it is using a default instead. Am I right with my assumption?

No, Grafana should still be sending the date range, but Cortex is not using that data in it's query (see the link to Cortex Project source in my original post).

You can also confirm that by running developer window in your browser and looking at the requests it sends out.

[containerpope]

@lostmimic yeah, got you. Basically I wanted to say, there is no chance of sending the time range into Cortex. That is why we don't get variables in case there are no metrics for a while, even if Grafana sends the right time range.

[gouthamve]

Hey, this is a hard problem to fix because the way the current index is structured, it is hard to compute the labels/series responses without loading a lot of chunks from the backend store. So we're only limiting the lookup to the data in the ingesters, hence the older data is not looked up.

This problem also exists in the blocks storage engine as loading too much data causes OOM issues. But this can be solved in the blocks storage engine much more easily than in the chunks store. So we'll likely fix this in the blocks engine first. But ideas welcome on how to efficiently solve this for the chunks engine.

[containerpope]

@gouthamve sounds interesting. Adhoc I also don't know a proper solution to this issue, but I understand now, why a full scan is possibly not the right strategy here.