Increase CortexIngesterReachingSeriesLimit warning for duration

As it turns out, during normal shuffle-sharding operation, the 70%
mark is often exceeded, but not by much. Rather than increasing the
threshold to 75%, this commit increases the `for` duration to 3h,
following the thought that we want this alert to fire if ingesters are
constantly above the threshold even after stale series are flushed
(which occurs every 2h, when the TSDB head is compacted). We flush
series with a timestamp between [-3h, -1h] after the last compaction,
so the worst case scenario is that it takes 3h to flush a stale
series.

Signed-off-by: beorn7 <[email protected]>

Author: beorn7

CHANGELOG.md

@@ -29,6 +29,7 @@
 * [ENHANCEMENT] cortex-mixin: Added `alert_excluded_routes` config to exclude specific routes from alerts. #338
 * [ENHANCEMENT] Added `CortexMemcachedRequestErrors` alert. #346
 * [ENHANCEMENT] Ruler dashboard: added "Per route p99 latency" panel in the "Configuration API" row. #353
+* [ENHANCEMENT] Increased the `for` duration of the `CortexIngesterReachingSeriesLimit` warning alert to 3h. #362
 * [BUGFIX] Fixed `CortexIngesterHasNotShippedBlocks` alert false positive in case an ingester instance had ingested samples in the past, then no traffic was received for a long period and then it started receiving samples again. #308
 * [BUGFIX] Alertmanager: fixed `--alertmanager.cluster.peers` CLI flag passed to alertmanager when HA is enabled. #329
 * [BUGFIX] Fixed `CortexInconsistentRuntimeConfig` metric. #335

cortex-mixin/alerts/alerts.libsonnet

@@ -257,7 +257,7 @@
                 (cortex_ingester_instance_limits{limit="max_series"} > 0)
             ) > 0.7
           |||,
-          'for': '5m',
+          'for': '3h',
           labels: {
             severity: 'warning',
           },