Skip to content

Commit

Permalink
Improve comments around resending resolved alerts (prometheus#13990)
Browse files Browse the repository at this point in the history
Signed-off-by: George Robinson <[email protected]>
  • Loading branch information
grobinson-grafana authored Apr 25, 2024
1 parent 4ac7806 commit dde2e5e
Showing 1 changed file with 11 additions and 2 deletions.
13 changes: 11 additions & 2 deletions rules/alerting.go
Original file line number Diff line number Diff line change
Expand Up @@ -457,8 +457,17 @@ func (r *AlertingRule) Eval(ctx context.Context, ts time.Time, query QueryFunc,
}
}

// If the alert was previously firing, keep it around for a given
// retention time so it is reported as resolved to the AlertManager.
// If the alert is resolved (was firing but is now inactive) keep it for
// at least the retention period. This is important for a number of reasons:
//
// 1. It allows for Prometheus to be more resilient to network issues that
// would otherwise prevent a resolved alert from being reported as resolved
// to Alertmanager.
//
// 2. It helps reduce the chance of resolved notifications being lost if
// Alertmanager crashes or restarts between receiving the resolved alert
// from Prometheus and sending the resolved notification. This tends to
// occur for routes with large Group intervals.
if a.State == StatePending || (!a.ResolvedAt.IsZero() && ts.Sub(a.ResolvedAt) > resolvedRetention) {
delete(r.active, fp)
}
Expand Down

0 comments on commit dde2e5e

Please sign in to comment.