HDDS-10256. Retry block allocation when SCM is in safe mode.

[ashishkumar50]

What changes were proposed in this pull request?

When SCM is in safe mode during restart(or rolling restart), block allocation fails. Instead of failing operation immediately, OM can retry few times(5 times every 3 seconds) to get block info from SCM before failing the operation.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-10256

How was this patch tested?

Integration test

[sadanand48]

Thanks @ashishkumar50 for the patch, Left few minor comments. otherwise looks good.

[jojochuang]

This code is executed in the OM's request handler when OM receives the request from the client. Client --> OM --> SCM.

IMO It's probably not a good idea to sleep inside a request handler, because if there are many clients, their request will all get blocked because the thread pool for the request handlers is exhausted, which may cause other requests to fail.

Would it make more sense to retry at client side?

[ashishkumar50]

This code is executed in the OM's request handler when OM receives the request from the client. Client --> OM --> SCM.

IMO It's probably not a good idea to sleep inside a request handler, because if there are many clients, their request will all get blocked because the thread pool for the request handlers is exhausted, which may cause other requests to fail.

Would it make more sense to retry at client side?

Yes makes sense. Moved retry logic at client side.

[smengcl]

Thanks @ashishkumar50 . Overall lgtm

[adoroszlai]

Is this change specific to the new feature, i.e. does it need to go into the feature branch instead of master?

[ashishkumar50]

Is this change specific to the new feature, i.e. does it need to go into the feature branch instead of master?

It's not related to new feature but issue is seen during hbase testing and impacting hbase.
@jojochuang, Can you please help to confirm whether it can be merged into master?

[jojochuang]

IMO it is specific to HBase but I don't think we've experienced this problem with other applications. I'm fine to forward port it to master branch (assuming no code conflicts)

[jojochuang]

Looks good. This should be enough to deal with transient issues. Applications like HBase should have its own retry mechanism if they want to deal with longer down time.

For future reference: HDFS client would retry 10 times in a roll, non-stop, before abort. So I think Ozone is more resilient than HDFS in this regard.