Skip to content
This repository was archived by the owner on Nov 15, 2023. It is now read-only.

Add a Prometheus alert on no incoming connection #7517

Merged
tomaka merged 4 commits into from
Feb 23, 2021
Merged

Add a Prometheus alert on no incoming connection #7517

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
9f71cac
Add a Prometheus alert on no incoming connection
tomaka Nov 11, 2020
afa0a58
Merge remote-tracking branch 'upstream/master' into alert-no-inc
tomaka Nov 11, 2020
3980a03
Merge remote-tracking branch 'upstream/master' into alert-no-inc
tomaka Jan 15, 2021
5e6e283
Merge remote-tracking branch 'upstream/master' into alert-no-inc
tomaka Feb 23, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .maintain/monitoring/alerting-rules/alerting-rules.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,13 @@ groups:
annotations:
message: 'The node {{ $labels.instance }} has less than 3 peers for more
than 15 minutes'
- alert: NoIncomingConnection
expr: increase(polkadot_sub_libp2p_incoming_connections_total[20m]) == 0
Copy link
Contributor

@mxinden mxinden Nov 12, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without a for would this not fire each time a new node starts as the first value would be 0 and there are no other values in the past 20m in the timeseries?

Copy link
Contributor Author

@tomaka tomaka Nov 12, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested this with my own server, where my node was offline for a few hours:

image

Instead of 20m I put 6h. In the period before I started the node again, the values were simply missing, and I assume missing values wouldn't trigger the alert.

Interestingly, however, between 1am (when my node crashed) and 7am, values were still present and decreasing linearly.

Copy link
Contributor

@mxinden mxinden Nov 17, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interestingly, however, between 1am (when my node crashed) and 7am, values were still present and decreasing linearly.

Just to make sure there is no confusion, the increase rate decreased.

labels:
severity: warning
annotations:
message: 'The node {{ $labels.instance }} has not received any new incoming
TCP connection in the past 20 minutes. Is it connected to the Internet?'

##############################################################################
# System
Expand Down