feature: require N check-ins

[adamdecaf]

Often we may be running N instances of a service and expect them all to checkin. If not all services checkin then we'd want to alert somehow.

Use-case: If there are N kafka consumers staging work (e.g. achgateway) and none of them have work to commit (e.g. upload) then we could notify folks to expect zero commited work. (e.g. if there are zero pending files then expect no file uploaded)

[alovak]

How would you distinguish instances between each other?

[adamdecaf]

How would you distinguish instances between each other?

Good question. I was thinking we could store some state in PagerDuty and check-in requests could de-dup based on the request headers/contents.

[adamdecaf]

Similarly we may want to accept a checking with a score of N and fire alerts if N is below some threshold.

[adamdecaf]

Still not sure how to do this in a provider agnostic way. We've got PD working (finally) and will just split out -0 and -1 into different checks.