HDDS-10117. ChunkKeyHandler does not close XceiverClient in case of exception

[adoroszlai]

What changes were proposed in this pull request?

ChunkKeyHandler may not release xceiverClient (and thus close ManagedChannel) if exception is encountered.

This can be reproduced by running TestOzoneDebugShell with Java 17 with the fix for HDDS-10123. ChunkKeyHandler throws exception due to GSON not being able to set some fields via reflection (HDDS-10122). xceiverClient is not released, and test log shows:

SEVERE: *~*~*~ Previous channel ManagedChannelImpl{logId=209, target=127.0.0.1:15011} was not shutdown properly!!! ~*~*~*
    Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.
java.lang.RuntimeException: ManagedChannel allocation site
	at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:102)
	at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:60)
	at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:51)
	at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:631)
	at org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:297)
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connectToDatanode(XceiverClientGrpc.java:178)
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.reconnect(XceiverClientGrpc.java:574)
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.checkOpen(XceiverClientGrpc.java:565)
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandAsync(XceiverClientGrpc.java:495)
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandOnAllNodes(XceiverClientGrpc.java:279)
	at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlockFromAllNodes(ContainerProtocolCalls.java:690)
	at org.apache.hadoop.ozone.debug.ChunkKeyHandler.execute(ChunkKeyHandler.java:141)

https://issues.apache.org/jira/browse/HDDS-10117

How was this patch tested?

Tested the fix on top of HDDS-10123. GSON still doesn't work, but xceiverClient is no longer leaked.

CI:
https://github.com/adoroszlai/ozone/actions/runs/7510787779

[sadanand48]

Thanks @adoroszlai for the fix, LGTM

[adoroszlai]

Thanks @sadanand48 for the review.