Define and apply a priority class to critical components

[p1-bot-repo-sync]

Summary

While updating a Bigbang deployment we noticed the monitoring release was failing to reconcile on repeated attempts with a timeout error.

The underlying cause of the issue was that prometheus-node-exporter pods were unable to deploy to all nodes due to insufficient resources. This was indicated in the failed helm release:

      DaemonSet is not ready: monitoring/monitoring-monitoring-prometheus-node-exporter. 0 out of 10 expected pods have been scheduled
      DaemonSet is not ready: monitoring/monitoring-monitoring-prometheus-node-exporter. 1 out of 10 expected pods have been scheduled
      DaemonSet is not ready: monitoring/monitoring-monitoring-prometheus-node-exporter. 2 out of 10 expected pods have been scheduled
      warning: Upgrade "monitoring-monitoring" failed: timed out waiting for the condition
    reason: UpgradeFailed

As well as the pod and events:

Warning  FailedScheduling   39s (x6 over 2m18s)  default-scheduler   0/10 nodes are available: 1 Insufficient cpu, 9 node(s) didn't match Pod's node affinity/selector

Temporary resolution

The cluster we were deploying into happened to have a pre-defined PolicyClass. We manually added this to the daemonset's pod spec so that the pods were deployed and the release was able to reconcile as expected.

Notional feature request

There are several daemonsets deployed by Bigbang which I would imagine could run into similar scenarios. Promtail, twistlock defenders, velero/restic all come to mind. Bigbang could define a PolicyClass (perhaps as part of /base) and add it to daemonsets and other critical components as appropriate to ensure that reconciliation of BB managed helm releases is able to complete without hanging on resource scheduling constraints.