HDDS-8914. Datanode may fail to start due to duplicate VolumeInfoMetrics

[adoroszlai]

What changes were proposed in this pull request?

If failures happen during volume initialization, the volume object is abandoned, and a failed volume created instead. The original object should be cleaned up.

1. StorageVolume#initialize may throw IOException. By this time it may have started background thread for disk usage check. HddsVolume also created and registered metrics objects.
2. MutableVolumeSet#initializeVolumeSet also may throw IOException if storage directory does not exist and cannot be created.

To reproduce, simply add /root as an additional data or DB directory (assuming non-root user runs datanode):

OZONE-SITE.XML_hdds.datanode.dir=/root,/data/hdds
OZONE-SITE.XML_hdds.datanode.container.db.dir=/root,/data/metadata/db

Failed data volume results in datanode stopping during startup:

datanode_1  | [main] INFO volume.HddsVolume: Creating HddsVolume: /root/hdds of storage type : DISK capacity : 499596230656
datanode_1  | [main] ERROR ozone.HddsDatanodeService: Exception in HddsDatanodeService.
datanode_1  | org.apache.hadoop.metrics2.MetricsException: Metrics source VolumeInfoMetrics-/root already exists!
datanode_1  | 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
datanode_1  | 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
datanode_1  | 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
datanode_1  | 	at org.apache.hadoop.ozone.container.common.volume.VolumeInfoMetrics.init(VolumeInfoMetrics.java:50)
datanode_1  | 	at org.apache.hadoop.ozone.container.common.volume.VolumeInfoMetrics.<init>(VolumeInfoMetrics.java:45)
datanode_1  | 	at org.apache.hadoop.ozone.container.common.volume.HddsVolume.<init>(HddsVolume.java:140)
datanode_1  | 	at org.apache.hadoop.ozone.container.common.volume.HddsVolume.<init>(HddsVolume.java:73)
datanode_1  | 	at org.apache.hadoop.ozone.container.common.volume.HddsVolume$Builder.build(HddsVolume.java:115)
datanode_1  | 	at org.apache.hadoop.ozone.container.common.volume.HddsVolumeFactory.createFailedVolume(HddsVolumeFactory.java:60)
datanode_1  | 	at org.apache.hadoop.ozone.container.common.volume.MutableVolumeSet.initializeVolumeSet(MutableVolumeSet.java:187)
datanode_1  | 	at org.apache.hadoop.ozone.container.common.volume.MutableVolumeSet.<init>(MutableVolumeSet.java:135)
datanode_1  | 	at org.apache.hadoop.ozone.container.common.volume.MutableVolumeSet.<init>(MutableVolumeSet.java:99)
datanode_1  | 	at org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.<init>(OzoneContainer.java:146)
datanode_1  | 	at org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.<init>(DatanodeStateMachine.java:173)
datanode_1  | 	at org.apache.hadoop.ozone.HddsDatanodeService.start(HddsDatanodeService.java:306)

Failed DB volume only results in leftover thread for the abandoned volume (/root here):

$ jstack $(jps | grep Datanode | awk '{ print $1 }') | grep DiskUsage
"DiskUsage-/data/hdds-
"DiskUsage-/data/metadata/ratis-
"DiskUsage-/root-
"DiskUsage-/data/metadata/db-

https://issues.apache.org/jira/browse/HDDS-8914

How was this patch tested?

Verified that datanode starts successfully with additional read-only volume, and that DiskUsage thread for the abandoned volume is not running.

$ jstack $(jps | grep Datanode | awk '{ print $1 }') | grep DiskUsage
"DiskUsage-/data/hdds-
"DiskUsage-/data/metadata/ratis-
"DiskUsage-/data/metadata/db-

[adoroszlai]

@ArafatKhan2198 please take a look

[ArafatKhan2198]

Thanks for working on this @adoroszlai it's a good find!
LGTM +1

[duongkame]

Thanks for the patch @adoroszlai .

[adoroszlai]

Thanks @ArafatKhan2198, @duongkame for the review.