HDDS-7738. SCM terminates when adding container to a closed pipeline

[duongkame]

What changes were proposed in this pull request?

SCM should allow adding a container to a CLOSED pipeline as a pipeline state can be changed while the container creating transaction is waiting to be processed by SCM.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-7738

How was this patch tested?

Unit tests.
Standard CI: https://github.com/duongkame/ozone/actions/runs/3857497185/jobs/6574962174

[duongkame]

@szetszwo @errose28 @aswinshakil please have a look. Some CI integration steps are failing but they're fine when running locally from my laptop, or by the branch CI, guess they're just flaky. Would be nice if one of you can help retry them individually.

[sodonnel]

Does the SCM terminate on the active SCM, or is this on the follower SCMs?

If we allow an open container on a closed pipeline, what will close the open container? The normal close flow is triggered when either the container fills up and the DN triggers a close, or the pipeline is closed and it triggers a close to all containers on the pipeline.

I am also wondering, what happens to a container which is allocated on SCM, but never gets anything written to it. It will never get replicas on a DN, and hence will never have any replicas reported. Will it get cleaned up or will it hang around forever?

[duongkame]

Thanks for having a look @sodonnel .

Does the SCM terminate on the active SCM, or is this on the follower SCMs?

The same transactions get replayed in all SCMs and result the same errors preventing SCM to start up.

If we allow an open container on a closed pipeline, what will close the open container? The normal close flow is triggered when either the container fills up and the DN triggers a close, or the pipeline is closed and it triggers a close to all containers on the pipeline.

I think such containers will be closed by the pipeline scrubber, which periodically scans and closes containers associated with closed pipelines.

I am also wondering, what happens to a container which is allocated on SCM, but never gets anything written to it. It will never get replicas on a DN, and hence will never have any replicas reported. Will it get cleaned up or will it hang around forever?

I'm not sure about this. Basically, I can't find any process that cleans up empty containers and it looks like a container can only be removed via admin CLI.

Alternatively, SCM can also just reject the transaction (throwing a non-terminus) and move on. Yet, I'm not confident about the consequences.

[szetszwo]

@duongkame , thanks a lot for working on this!

• We should print a WARN message when it happens; see below.
• Also, let's keep the addContainerToPipelineSCMStart method for now so that it is easier to back port this change. We may do the code refactoring later.

+ b/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineStateMap.java
@@ -106,9 +106,10 @@ void addContainerToPipeline(PipelineID pipelineID, ContainerID containerID)
 
     Pipeline pipeline = getPipeline(pipelineID);
     if (pipeline.isClosed()) {
-      throw new IOException(String
-          .format("Cannot add container to pipeline=%s in closed state",
-              pipelineID));
+      LOG.warn("Adding container {} to pipeline={} in CLOSED state."
+          + "  This happens only for some exceptional cases."
+          + "  Check for the previous exceptions.",
+          containerID, pipelineID);
     }
     pipeline2container.get(pipelineID).add(containerID);
   }

[duongkame]

• We should print a WARN message when it happens; see below.
• Also, let's keep the addContainerToPipelineSCMStart method for now so that it is easier to back port this change. We may do the code refactoring later.

Thanks for the suggestions, @szetszwo. I've made the updates.
Once this is merged I'll submit another PR to clean up the code.

[szetszwo]

+1 the change looks good.