[FEATURE] Have default priorityClass to prevent unexpected longhorn pods eviction

[innobead]

Is your feature request related to a problem? Please describe (👍 if you like this request)

Have a default priority class with the highest value to ensure longhorn pods have not been evicted when low system resources by the Kube scheduler. Right now, the setting is optional.

https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/

Describe the solution you'd like

Describe alternatives you've considered

Additional context

cc @longhorn/dev

[harridu]

Will existing deployments be migrated to the new default priority class on the fly? I am not sure if this is possible.

[innobead]

Will existing deployments be migrated to the new default priority class on the fly? I am not sure if this is possible.

In the current implementation, it's not possible. However, we should improve this part to have the following support to prevent any interruption for running volumes.

• When changing any volume-related Pod configuration (ex: IM pod, share manager pod, upcoming object storage/s3gw pod) like priority class, the change will only take effect when doing volume upgrade or volume reattachement.

Extra info from @PhanLe1010

cc @shuo-wu @derekbit @c3y1huang

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at:

1. Install Longhorn from scratch.
2. There is a priority class named longhorn-critical (kubectl get pc)
3. Longhorn setting priority-class has a default value longhorn-critical
4. The priorityClass of the spec should be longhorn-critical for all longhorn pods.

• Is there a workaround for the issue? If so, where is it documented?
  The workaround is at:

• Does the PR include the explanation for the fix or the feature?

• Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
  The PR for the YAML change is at:
  The PR for the chart change is at:
  feat: default priority class #7262

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at
  feat: default priority class longhorn-manager#2332
  feat(setting): lazy update for volume-related settings longhorn-manager#2411

• Which areas/issues this PR might have potential impacts on?
  Area
  Issues

• If labeled: require/LEP Has the Longhorn Enhancement Proposal PR submitted?
  The LEP PR is at
  feat(lep): default priority class #7252

• If labeled: area/ui Has the UI issue filed or ready to be merged (including backport-needed/*)?
  The UI issue/PR is at

• If labeled: require/doc Has the necessary document PR submitted or merged (including backport-needed/*)?
  The documentation issue/PR is at
  doc: default priority class website#808

• If labeled: require/automation-e2e Has the end-to-end test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue (including backport-needed/*)
  The automation skeleton PR is at
  The automation test case PR is at
  The issue of automation test case implementation is at (please create by the template)

• If labeled: require/automation-engine Has the engine integration test been merged (including backport-needed/*)?
  The engine automation PR is at

• If labeled: require/manual-test-plan Has the manual test plan been documented?
  The updated manual test plan is at

• If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?
  The compatibility issue is filed at

[chriscchien]

Verified pass on longhorn v1.6.0-rc1 with test stpes

After install v1.6.0-rc1, can see priority class longhorn-critical in all longhorn pods

> k get priorityclass
NAME                      VALUE        GLOBAL-DEFAULT   AGE
system-node-critical      2000001000   false            8m54s
system-cluster-critical   2000000000   false            8m54s
longhorn-critical         1000000000   false            8m4s
> k get setting -n longhorn-system | grep priority
priority-class                                                    longhorn-critical                                 8m4s