add ClusterResourceQuota prometheus alert for overprovisioning#1311
Conversation
anmolsachan
force-pushed
the
overprovision_alert
branch
3 times, most recently
from
fc5aa97 to
93c4f30
Compare
|
@umangachapagain @synarete Please review. |
anmolsachan
changed the title
anmolsachan
force-pushed
the
overprovision_alert
branch
from
93c4f30 to
7953a0d
Compare
openshift-ci
bot
added
the
needs-rebase
anmolsachan
force-pushed
the
overprovision_alert
branch
from
7953a0d to
a9a3b1c
Compare
openshift-ci
bot
removed
the
needs-rebase
anmolsachan
force-pushed
the
overprovision_alert
branch
from
a9a3b1c to
cf91cfa
Compare
jarrpa
left a comment
jarrpa
left a comment
There was a problem hiding this comment.
Looks sane at first glance, but I'm not an expert in any of this. Will let others LGTM.
metrics/mixin/config.libsonnet
Outdated
|
|
||
| // Duration to raise various Alerts | ||
| clusterObjectStoreStateAlertTime: '15s', | ||
| clusterResourceQuotaAlertTime: '0s', |
There was a problem hiding this comment.
0s feels too aggressive. Any reason why it can't be 5s or 10s?
There was a problem hiding this comment.
Because the PVC requests are absolute like 50GB or 100GB, and won't grow gradually like usage.
Also taking an example :
If the set quota is 1TB, and already provisioned is 750GB, then if at this moment, another PVC of 100 GB is requested, the total will go to 850G. Then users should immediately be informed about it because all PVC requests above the remaining 150GB will be failed immediately by the ClusterResourceQuota.
umangachapagain
left a comment
umangachapagain
left a comment
There was a problem hiding this comment.
good to merge but holding off until dependent PR is merged.
openshift-ci
bot
added
the
approved
|
/hold |
openshift-ci
bot
added
the
do-not-merge/hold
|
Dependent PR merged. /hold cancel |
openshift-ci
bot
added
needs-rebase
anmolsachan
force-pushed
the
overprovision_alert
branch
from
cf91cfa to
4eb569f
Compare
openshift-ci
bot
removed
the
needs-rebase
This commit adds prometheus alert to notify the users when the PVC request for a particular storageclass goes beyond 80% of the limit set by the user through the ClusterResourceQuota resource. Signed-off-by: Anmol Sachan <anmol13694@gmail.com>
Signed-off-by: Anmol Sachan <anmol13694@gmail.com>
anmolsachan
force-pushed
the
overprovision_alert
branch
from
4eb569f to
845484e
Compare
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: agarwal-mudit, umangachapagain The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
6 participants
Depends on #1282 . This PR is required to alert users about the thresholds being reached set via the functionality in #1282 .
Signed-off-by: Anmol Sachan anmol13694@gmail.com