Split cortex_api recording rule group into three groups.

CHANGELOG.md

@@ -31,6 +31,7 @@
   * `_config` > `queryFrontend` > `shard_factor`
   * `_config` > `queryFrontend` > `sharded_queries_enabled`
   * `_config` > `queryFrontend` > `query_split_factor`
+* [CHANGE] Split `cortex_api` recording rule group into three groups. This is a workaround for large clusters where this group can become slow to evaluate. #401
 * [ENHANCEMENT] Add overrides config to compactor. This allows setting retention configs per user. #386
 * [ENHANCEMENT] cortex-mixin: Make `cluster_namespace_deployment:kube_pod_container_resource_requests_{cpu_cores,memory_bytes}:sum` backwards compatible with `kube-state-metrics` v2.0.0. #317
 * [ENHANCEMENT] Cortex-mixin: Include `cortex-gw-internal` naming variation in default `gateway` job names. #328

cortex-mixin/recording_rules.libsonnet

@@ -11,10 +11,18 @@ local utils = import 'mixin-utils/utils.libsonnet';
   prometheusRules+:: {
     groups+: [
       {
-        name: 'cortex_api',
+        name: 'cortex_api_1',
+        rules:
+          utils.histogramRules('cortex_request_duration_seconds', ['cluster', 'job']),
+      },
+      {
+        name: 'cortex_api_2',
+        rules:
+          utils.histogramRules('cortex_request_duration_seconds', ['cluster', 'job', 'route']),
+      },
+      {
+        name: 'cortex_api_3',
         rules:
-          utils.histogramRules('cortex_request_duration_seconds', ['cluster', 'job']) +
-          utils.histogramRules('cortex_request_duration_seconds', ['cluster', 'job', 'route']) +
           utils.histogramRules('cortex_request_duration_seconds', ['cluster', 'namespace', 'job', 'route']),
       },
       {