[FEATURE] Add a brand new/empty bdev with WriteOnly mode to the RAID1 bdev

[DamiaSan]

This feature is a step to implement the replica rebuilding without stopping the volume for all the time of the rebuild process

[innobead]

cc @longhorn/dev-data-plane

[DamiaSan]

Some implementations details:

• function raid_bdev_is_mode_supported, only raid1 supports add
• function raid_bdev_update to manage incremented num_base_bdevs
  • must update raid1 channel base_bdev_read_bw
• function bdev_raid_add_base_bdev
  • call raid_bdev_add_base_device
  • for each channel, call a function to update raid and module channel with increased num_base_bdevs

Must implement a revert operation too, because memory allocation on per-channel add function can fail?
...
Probably not, even in blobstore (blob_resize) and nvme_ctrlr multiple realloc failure are not reverted.
If the operation fails with -ENOMEM error, we have 2 option:

• try again after a bit, and in that case the already reallocated array will have the right size ( in the GNU C Library, if the new size is the same as the old, realloc and reallocarray are guaranteed to change nothing and return the same address that you gave )
• restart spdk tgt or the entire node

[DamiaSan]

Some notes on tests:

• add to non used raid in new slot
• add to non used raid in existing slot
• add to non used raid and after use it
• add to used raid
• add to raid with superblock in new slot
• add to raid with superblock in existing slot
• load an updated raid superblock

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Does the PR include the explanation for the fix or the feature?
  A new JSON-RPC has been created, here the documentation to use it
  https://github.com/DamiaSan/spdk/blob/raid1_rw_mode/doc/jsonrpc.md#bdev_raid_add_base_bdev-rpc_bdev_raid_add_base_bdev

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at Raid1 rw mode spdk#12

• Which areas/issues this PR might have potential impacts on?
  Area SPDK

[DamiaSan]

With the introduction of the external snapshot, write only mode is no more necessary

[shuo-wu]

The related discussion: #5866 (comment)