[FEATURE] Selective V2 Data Engine Activation

[derekbit]

Is your improvement request related to a feature? Please describe (👍 if you like this request)

In a large cluster, both powerful nodes and low-spec nodes exist, and there are lots of kinds of applications running inside it. Currently, the v2-data-engine enables the instance-manager pod for v2 volumes on all Longhorn nodes regardless of machines' specs. To address the issue, the v2 data engines can be activated through the global setting v2-dat-engine and per-node labels, annotations, or spec fields.

The ticket can be extended to instance-manager pod for v1 volumes in the future.

Describe the solution you'd like

Describe alternatives you've considered

Additional context

cc @shuo-wu @innobead

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at:

1. Create a 3-node LH volume and enable v1-data-engine and v2-data-engine
2. Create v1 and v2 volumes with 2 replicas
3. Add label node.longhorn.io/disable-v2-data-engine: "true" to two of kubernetes nodes
4. IMs and their pods for v1 volumes should not be impacted.
5. For IMs and their pods for v2 volumes, it there is no replicas and engine in the IM, the IM with the label should be deleted.
6. Detach all v2 volumes. After the v2 volumes are detached, the IMs and pods with the label should be deleted.
7. Remove the label, the deleted IMs and pods should recreated.

• Does the PR include the explanation for the fix or the feature?

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at

longhorn/longhorn-manager#2292

• Which areas/issues this PR might have potential impacts on?
  Area: v2 volume, instance manager
  Issues

[chriscchien]

Verified pass on longhorn master(longhorn-manager 970ba4)

Following test steps, did not encounter problem, close this ticket, thank you.

[khushboo-rancher]

@chriscchien Some priority 1/2 scenarios that can be further tested:

Prerequisite: Have a v1 volume with three replicas and data checksum computed.

1. Disable all the node for v2 data engine, create a v2 volume. The volume should show unscheduling, Delete the v2 disable label. Verify the volume become schedulable.
2. Create a v2 volume, add v2 disable label to one of the replica node. Crash the IM on replica node. Check the replica behavior.
3. Create a v2 volume, add v2 disable label to one of the volume attached node. Crash the IM on attached node. Check the replica behavior.
4. Create a v2 volume with 2 replicas. Trigger the replica rebuilding. When the replica is progress, add the v2 disable label to the rebuilding replica node`. Verify that the replica finishes successfully.

v1 volume should not be impacted in any of the above scenarios.

[chriscchien]

@chriscchien Some priority 1/2 scenarios that can be further tested:

Prerequisite: Have a v1 volume with three replicas and data checksum computed.

1. Disable all the node for v2 data engine, create a v2 volume. The volume should show unscheduling, Delete the v2 disable label. Verify the volume become schedulable.
2. Create a v2 volume, add v2 disable label to one of the replica node. Crash the IM on replica node. Check the replica behavior.
3. Create a v2 volume, add v2 disable label to one of the volume attached node. Crash the IM on attached node. Check the replica behavior.
4. Create a v2 volume with 2 replicas. Trigger the replica rebuilding. When the replica is progress, add the v2 disable label to the rebuilding replica node`. Verify that the replica finishes successfully.

v1 volume should not be impacted in any of the above scenarios.

Hi @khushboo-rancher,

1. v2 volume become schedulable after remove the label or set label to false
2. The replica on crashed IM gone, volume become degraded, after remove label or set label to false then do offline rebuilding, all replicas ready and data intact.
3. The replica on crashed IM gone, volume try to reattach to the same node but stuck at attaching, after remove label or set label to false, volume attached, data intact.
4. Following steps, the replica rebuild success.

And after above tests complete, v1 volume healthy, data intact

[innobead]

@chriscchien let's automate these cases it they haven't been implemented. Create a ticket for it.