HDDS-5252. PipelinePlacementPolicy filter out datanodes with not enough space.

[guihecheng]

What changes were proposed in this pull request?

Pipeline placement policy should filter out those datanodes that do not even have enough space for a single container, otherwise even if the pipeline creation succeeds, a datanode may quickly report that it fails to create a container and there will be successive failures because we will choose that node again and again.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-5252

How was this patch tested?

a new ut.
manual test.

[guihecheng]

@bshashikant @ChenSammi please help review this one, thanks~

[GlenGeng-awx]

Thanks @guihecheng for the fix!
I agree with the necessity of the solution. My concern here is the sizeRequired which is 5GB by default, which seems to be not quite viable for a production cluster.

How about adjust it to base on usage percent of the Datanode? For example, a Datanode with usage great then 80% is not a good candidate for new pipeline.

[guihecheng]

@GlenGeng Thanks for your comments, I understand your concern, but I think it is a bit different here.
A datanode with usage > 80% is not a good candidate indeed, and should have a lower priority to be chosen, but it should still have a chance to be chosen. There is a possibility that all datanodes have > 80% space in the cluster, then if we filter out them completely then we don't have a writable cluster while actually there are ~20% space left.

Here we try to solve the problem that if there not not even space for a single container then that datanode should not be chosen anyway.
Maybe we could implement a new placement policy for pipelines that have a priority defined according to available space of datanodes to solve the problem you mentioned ?

[GlenGeng-awx]

Thanks @guihecheng for the clarificaiton.

One concern about the solution based on sizeRequired is, assume there are 3 DN dockers for test, each has 1GB storage space, will scm be able to create a pipeline from them ?

[guihecheng]

@GlenGeng oh, I think not, after this one we should have to set a proper container size for test deploy.

[bshashikant]

I think the solution is reasonable. Any container allocation will fail at datanode if it does not have space equal to container size anyways. Its better to to not choose such nodes while allocating a new container.
For docker tests, i think the container size is already set to a lower threshold but we can double check on the same.

[guihecheng]

@bshashikant Ah, just now I've a talk with @GlenGeng , he suggested that we could do better here.

We could make the sizeRequired to a new config item such as 'pipeline.reserved.space', may be default to 0.
So, with this new item, we could do the following things:

1. use default, then we keep the same behavior as before, so users/developers who just want to do small tests or try out ozone will not be annoyed by some warning says that they have no space for pipeline creation.
2. config it to some value for space reservation (should better be > container size), so then we could solve the problem here, too.
   Moreover, when we have this 'reserved.space' more than container size, say 20% of the volume capacity, then we can filter out the nodes with all its volumes with usage > 80%, which may not be a good choice for pipeline allocation in production as Glen mentioned.
   It seems like we could got overall controll over the space usage of the cluster with this new config item.
   And this should not be conflict with the datanode config 'du.reserved', since its use case is that we have multiple consumers for a single disk.

[bshashikant]

@bshashikant Ah, just now I've a talk with @GlenGeng , he suggested that we could do better here.

We could make the sizeRequired to a new config item such as 'pipeline.reserved.space', may be default to 0.
So, with this new item, we could do the following things:

1. use default, then we keep the same behavior as before, so users/developers who just want to do small tests or try out ozone will not be annoyed by some warning says that they have no space for pipeline creation.
2. config it to some value for space reservation (should better be > container size), so then we could solve the problem here, too.
   Moreover, when we have this 'reserved.space' more than container size, say 20% of the volume capacity, then we can filter out the nodes with all its volumes with usage > 80%, which may not be a good choice for pipeline allocation in production as Glen mentioned.
   It seems like we could got overall controll over the space usage of the cluster with this new config item.
   And this should not be conflict with the datanode config 'du.reserved', since its use case is that we have multiple consumers for a single disk.

I guess, we need to be more precise here. A container creation won't succeed on a datanode unless it has remaining space >= container size as there is hard check there(check RoundRobinVolumeChoosingPolicy#chooseVolume()). Its better to to not choose such datanodes upfront during pipeline allocation in any case.

The pipeline reserved space notion should consider ratis volumes size specifically as well as so as to ensure not allocate any new pipelines on nodes which are very low on ratis log disk space, and should have enough space on all constituent datanodes of the pipeline fo the existing pipelines to get closed as closing a pipeline takes a snapshot and does IO on the raft log disk as well.

This patch itself looks good to me and can be committed and pipeline reserve semantic can be then built on top of this.

[guihecheng]

@bshashikant Ah, just now I've a talk with @GlenGeng , he suggested that we could do better here.
We could make the sizeRequired to a new config item such as 'pipeline.reserved.space', may be default to 0.
So, with this new item, we could do the following things:

1. use default, then we keep the same behavior as before, so users/developers who just want to do small tests or try out ozone will not be annoyed by some warning says that they have no space for pipeline creation.
2. config it to some value for space reservation (should better be > container size), so then we could solve the problem here, too.
   Moreover, when we have this 'reserved.space' more than container size, say 20% of the volume capacity, then we can filter out the nodes with all its volumes with usage > 80%, which may not be a good choice for pipeline allocation in production as Glen mentioned.
   It seems like we could got overall controll over the space usage of the cluster with this new config item.
   And this should not be conflict with the datanode config 'du.reserved', since its use case is that we have multiple consumers for a single disk.

I guess, we need to be more precise here. A container creation won't succeed on a datanode unless it has remaining space >= container size as there is hard check there(check RoundRobinVolumeChoosingPolicy#chooseVolume()). Its better to to not choose such datanodes upfront during pipeline allocation in any case.

Sure, even if we have the proposed config item, we should ensure that if configed value < container size, we should drop it and use container size instead(with a warning log).

The pipeline reserved space notion should consider ratis volumes size specifically as well as so as to ensure not allocate any new pipelines on nodes which are very low on ratis log disk space, and should have enough space on all constituent datanodes of the pipeline fo the existing pipelines to get closed as closing a pipeline takes a snapshot and does IO on the raft log disk as well.

Yes, ratis log disk should be taken into consideration.

This patch itself looks good to me and can be committed and pipeline reserve semantic can be then built on top of this.

Ok, then I'm not going to update this but build another pr for the proposed solution.
Thanks very much @bshashikant