HDDS-6137. [Ozone-Streaming] Refactor KeyDataStreamOutput. #3195

guohao-rosicky · 2022-03-15T07:59:53Z

What changes were proposed in this pull request?

Optimized the putBlock method of BlockDataStreamOutput

PutBlock does not use RaftRpc, but through the stream close() method

// putBlock metadata

ByteBuffer buf =
    ContainerCommandRequestMessage.toMessage(putBlockRequestProto, null)
        .getContent().asReadOnlyByteBuffer();

out.writeAsync(buf, StandardWriteOption.CLOSE);

2.Mark the stream data boundary by sending an empty stream packet

// Mark the stream data boundary

out.writeAsync(ByteBuffer.allocateDirect(0).asReadOnlyBuffer());

Flush is also in the form of async RPC, because putBlock needs to fetch bcsId (Raft log index)

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-6137

How was this patch tested?

Use the existing UT

captainzmc

Thanks @guohao-rosicky Improve this. LGTM over all. TestBlockDataStreamOutput has some error, can you confirm?

szetszwo · 2022-03-16T07:33:20Z

hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java

@guohao-rosicky, Writing an empty buffer may not work as expected. In the streaming model, data is continuous but not block-by-block. Both sender and receiver must consider that the data is a stream of bytes.

For example, the sender may write 3 times with lengths 3, 4, 5. Since there could be buffers/encoder/decoder in between, the receiver may receive a single buffer with length 12.

Let me think about how to make it work.

Thanks for your comment @szetszwo . I wrote a piece of pseudo-code, Ideas about the modification of org.apache.ratis.netty.server.DataStreamManagement#writeTo.

What do you think of this idea? There is no need to send this empty buffer rpc.

@guohao-rosicky , this is a standard problem in streaming. The standard solution is to write the length with the data. Let's don't change the API for it.

Hi, @szetszwo
I have a question, about

For example, the sender may write 3 times with lengths 3, 4, 5. Since there could be buffers/encoder/decoder in between, the receiver may receive a single buffer with length 12.

I noticed that in HDFS, marking the end of stream is also represented by an empty package.
In Ratis Stream, sticky packets have already been dealt with.

org.apache.ratis.netty.NettyDataStreamUtils#encodeDataStreamRequestHeader
https://github.com/apache/ratis/blob/master/ratis-netty/src/main/java/org/apache/ratis/netty/NettyDataStreamUtils.java#L90

@captainzmc tested the code from some time ago and did not find this problem

szetszwo · 2022-03-16T10:17:34Z

@guohao-rosicky , the idea is to write the proto length at the end as shown below; see also https://issues.apache.org/jira/secure/attachment/13041230/3195_review.patch

    //   readerIndex   protoIndex   lengthIndex    readerIndex+readableBytes
    //         V            V             V                              V
    // format: |--- data ---|--- proto ---|--- proto length (4 bytes) ---|

guohao-rosicky · 2022-03-16T12:59:47Z

@guohao-rosicky , the idea is to write the proto length at the end as shown below; see also https://issues.apache.org/jira/secure/attachment/13041230/3195_review.patch
    //   readerIndex   protoIndex   lengthIndex    readerIndex+readableBytes
    //         V            V             V                              V
    // format: |--- data ---|--- proto ---|--- proto length (4 bytes) ---|

Thanks @szetszwo, Patch has been applied

szetszwo

@guohao-rosicky , please fix the checkstyle warnings and test the change manually. I have not tested it. Thanks.

captainzmc · 2022-03-17T04:30:33Z

Thanks @guohao-rosicky @szetszwo update this. I used this PR test and found that the file failed to write. In DataNode IndexOutOfBoundsException will appear.

szetszwo · 2022-03-17T04:56:50Z

@captainzmc , thanks. Let me test the serialization/deserialization code.

szetszwo · 2022-03-17T05:16:09Z

Found the bug: ByteBu.nioBuffers(index, length) somehow is not working. Let me fix it.

szetszwo · 2022-03-17T05:33:52Z

@captainzmc , @guohao-rosicky, Fixed the bug; see https://issues.apache.org/jira/secure/attachment/13041255/3195_TestStreamDataChannel.patch

captainzmc · 2022-03-17T06:46:13Z

@captainzmc , @guohao-rosicky, Fixed the bug; see https://issues.apache.org/jira/secure/attachment/13041255/3195_TestStreamDataChannel.patch

Thanks @szetszwo for the update. Using new patch I found IndexOutOfBoundsException still appear.
I found that the ERROR only appeared when writing with high concurrency. I was successful with 30 concurrent writes, but I get the exception with 600 concurrent writes.

szetszwo · 2022-03-17T13:01:46Z

@captainzmc , thanks a lot for testing it!

I added one more test for the serialization and buffers. It worked fine. See https://issues.apache.org/jira/secure/attachment/13041275/3195_testBuffers.patch

I found that the ERROR only appeared when writing with high concurrency. I was successful with 30 concurrent writes, but I get the exception with 600 concurrent writes.

It seems there are some synchronization problems. Let me check.

szetszwo · 2022-03-17T13:19:40Z

@captainzmc , could you post the test code?

captainzmc · 2022-03-17T13:29:43Z

@szetszwo yes here is my test code. Is written according to the ratis Filestore test.

Actually there a problem with my previous description. I noticed that the when I use 30 threads writing, we’ll get same error.

captainzmc · 2022-03-17T13:33:51Z

@szetszwo Here is my datanode log. https://issues.apache.org/jira/secure/attachment/13041276/ozone-root-datanode-9-29-173-57.log
In the test, I wrote 30 files concurrently. Each file is 128MB and the chunk size is 1MB

szetszwo · 2022-03-17T13:52:52Z

@captainzmc , @guohao-rosicky , just found a bug in BlockDataStreamOutput. Could you test https://issues.apache.org/jira/secure/attachment/13041279/3195_bugfix.patch to see if it can fix the bug?

szetszwo · 2022-03-17T13:57:35Z

@szetszwo yes here is my test code. Is written according to the ratis Filestore test.

Thanks for sharing the test code. It looks good.

captainzmc · 2022-03-17T14:32:58Z

@captainzmc , @guohao-rosicky , just found a bug in BlockDataStreamOutput. Could you test https://issues.apache.org/jira/secure/attachment/13041279/3195_bugfix.patch to see if it can fix the bug?

Thanks @szetszwo for update this. I had just test this patch, we still get same error. Just as the same as this.

szetszwo

Let's add an abstract class instead static methods. Then, KeyDataStreamOutput can extends it. If we need to add SmallFileDataStreamOutput, it can extends it too.

package org.apache.hadoop.ozone.client.io;

import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
import org.apache.hadoop.hdds.scm.client.HddsClientUtils;
import org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException;
import org.apache.hadoop.hdds.scm.storage.ByteBufferStreamOutput;
import org.apache.hadoop.io.retry.RetryPolicies;
import org.apache.hadoop.io.retry.RetryPolicy;
import org.apache.ratis.protocol.exceptions.AlreadyClosedException;
import org.apache.ratis.protocol.exceptions.RaftRetryFailureException;

import java.io.IOException;
import java.io.InterruptedIOException;
import java.util.Map;
import java.util.function.Function;
import java.util.stream.Collectors;

public abstract class AbstractDataStreamOutput implements ByteBufferStreamOutput {
  private final Map<Class<? extends Throwable>, RetryPolicy> retryPolicyMap;
  private int retryCount;
  private boolean isException;

  protected AbstractDataStreamOutput() {
    this.retryPolicyMap = HddsClientUtils.getExceptionList()
        .stream()
        .collect(Collectors.toMap(Function.identity(),
            e -> RetryPolicies.TRY_ONCE_THEN_FAIL));
  }

  @VisibleForTesting
  public int getRetryCount() {
    return retryCount;
  }

  void resetRetryCount() {
    retryCount = 0;
  }

  boolean isException() {
    return isException;
  }

  /**
   * Checks if the provided exception signifies retry failure in ratis client.
   * In case of retry failure, ratis client throws RaftRetryFailureException
   * and all succeeding operations are failed with AlreadyClosedException.
   */
  boolean checkForRetryFailure(Throwable t) {
    return t instanceof RaftRetryFailureException
        || t instanceof AlreadyClosedException;
  }

  // Every container specific exception from datatnode will be seen as
  // StorageContainerException
  boolean checkIfContainerToExclude(Throwable t) {
    return t instanceof StorageContainerException;
  }

  private void setExceptionAndThrow(IOException ioe) throws IOException {
    isException = true;
    throw ioe;
  }

  void handleRetry(IOException exception) throws IOException {
    RetryPolicy retryPolicy = retryPolicyMap
        .get(HddsClientUtils.checkForException(exception).getClass());
    if (retryPolicy == null) {
      retryPolicy = retryPolicyMap.get(Exception.class);
    }
    handleRetry(exception, retryPolicy);
  }

  private void handleRetry(IOException exception, RetryPolicy retryPolicy)
      throws IOException {
    RetryPolicy.RetryAction action = null;
    try {
      action = retryPolicy.shouldRetry(exception, retryCount, 0, true);
    } catch (Exception e) {
      setExceptionAndThrow(new IOException(e));
    }
    if (action.action == RetryPolicy.RetryAction.RetryDecision.FAIL) {
      String msg = "";
      if (action.reason != null) {
        msg = "Retry request failed. " + action.reason;
        //LOG.error(msg, exception);
      }
      setExceptionAndThrow(new IOException(msg, exception));
    }

    // Throw the exception if the thread is interrupted
    if (Thread.currentThread().isInterrupted()) {
      //LOG.warn("Interrupted while trying for retry");
      setExceptionAndThrow(exception);
    }
    Preconditions.checkArgument(
        action.action == RetryPolicy.RetryAction.RetryDecision.RETRY);
    if (action.delayMillis > 0) {
      try {
        Thread.sleep(action.delayMillis);
      } catch (InterruptedException e) {
        Thread.currentThread().interrupt();
        IOException ioe = (IOException) new InterruptedIOException(
            "Interrupted: action=" + action + ", retry policy=" + retryPolicy)
            .initCause(e);
        setExceptionAndThrow(ioe);
      }
    }
    retryCount++;
  }
}

guohao-rosicky · 2022-03-25T06:41:44Z

Ok, I'll change it

guohao-rosicky · 2022-03-25T11:34:51Z

Has been modified. @szetszwo Please take a look.Thanks.

szetszwo · 2022-03-25T15:28:55Z

@guohao-rosicky , please review it yourself -- go through the change and remove the unused code (e.g. ConsumerWithIOException). Thanks.

szetszwo

@guohao-rosicky , thanks for the update. Some comments inlined.

szetszwo · 2022-03-26T14:37:22Z

...a/org/apache/hadoop/ozone/container/common/transport/server/ratis/ContainerStateMachine.java

+    });
+  }
+
+  private CompletableFuture<ContainerCommandResponseProto> runCommandAsync(


runCommandAsync is not used. Let's add it later.

szetszwo · 2022-03-26T14:39:39Z

...tainer-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java

+      LOG.error(
+          "Get stream data channel error Malformed request " +
+              "containerID: {} msg: {}",
+          container.getContainerData().getContainerID(), msg);


Let's remove LOG.error(..) since it already throws an exception.

szetszwo · 2022-03-26T14:41:20Z

...tainer-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java

      ContainerCommandRequestProto request, KeyValueContainer kvContainer,
      DispatcherContext dispatcherContext) {
-    if (!request.hasWriteChunk()) {
+    BlockID blockID;


szetszwo · 2022-03-26T14:53:02Z

hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/KeyDataStreamOutput.java

   * A constructor for testing purpose only.
   */
  @VisibleForTesting
  public KeyDataStreamOutput() {


This method is not used anywhere. Let's remove it.

szetszwo · 2022-03-26T14:54:06Z

hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/KeyDataStreamOutput.java

-    this.retryPolicyMap = HddsClientUtils.getRetryPolicyByException(
-        config.getMaxRetryCount(), config.getRetryInterval());


We should pass the retryPolicyMap to the super constructor.

szetszwo · 2022-03-27T08:39:14Z

hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/KeyDataStreamOutput.java

+    super(HddsClientUtils.getExceptionList()
+        .stream()
+        .collect(Collectors.toMap(Function.identity(),
+            e -> RetryPolicies.TRY_ONCE_THEN_FAIL)));


The map is passed incorrectly. Below is the original map.

- this.retryPolicyMap = HddsClientUtils.getRetryPolicyByException( - config.getMaxRetryCount(), config.getRetryInterval());

captainzmc

+1 Thanks @guohao-rosicky. The change looks good.

szetszwo

+1 the change looks good.

(cherry picked from commit 88c4d59) (cherry picked from commit 0a432aa0eb57bd25d63e656d0ec6e96a05689ac7)

(cherry picked from commit 88c4d59) (cherry picked from commit 0a432aa0eb57bd25d63e656d0ec6e96a05689ac7) (cherry picked from commit 99b414d)

(cherry picked from commit 06bc968)

guohao-rosicky changed the title ~~Guohao hdds 6137 dev~~ HDDS-6137. [Ozone-Streaming] Optimized the putBlock method of BlockDataStreamOutput Mar 15, 2022

guohao-rosicky force-pushed the guohao-HDDS-6137-dev branch from 942e87f to b1df777 Compare March 15, 2022 08:18

captainzmc requested review from captainzmc and szetszwo March 15, 2022 14:01

captainzmc reviewed Mar 15, 2022

View reviewed changes

szetszwo reviewed Mar 16, 2022

View reviewed changes

guohao-rosicky force-pushed the guohao-HDDS-6137-dev branch from 38b64ff to 35b89e5 Compare March 16, 2022 12:58

guohao-rosicky requested review from captainzmc and szetszwo March 16, 2022 12:59

szetszwo reviewed Mar 16, 2022

View reviewed changes

szetszwo force-pushed the HDDS-4454 branch from b6f1921 to f826d12 Compare March 16, 2022 17:30

guohao-rosicky force-pushed the guohao-HDDS-6137-dev branch from ba84300 to c2e7646 Compare March 17, 2022 03:01

guohao-rosicky force-pushed the guohao-HDDS-6137-dev branch from c2e7646 to cc72e81 Compare March 17, 2022 07:14

captainzmc closed this Mar 17, 2022

captainzmc reopened this Mar 17, 2022

guohao-rosicky force-pushed the guohao-HDDS-6137-dev branch from cc72e81 to 539eea6 Compare March 17, 2022 15:40

guohao-rosicky requested a review from captainzmc March 25, 2022 03:53

szetszwo requested changes Mar 25, 2022

View reviewed changes

guohao-rosicky force-pushed the guohao-HDDS-6137-dev branch from 7c6bf0d to 65ce44e Compare March 25, 2022 11:33

guohao-rosicky requested a review from szetszwo March 25, 2022 11:33

common code

d0cab2b

guohao-rosicky force-pushed the guohao-HDDS-6137-dev branch from c630ed9 to d0cab2b Compare March 26, 2022 06:12

szetszwo reviewed Mar 26, 2022

View reviewed changes

common code

8dd3c3f

szetszwo reviewed Mar 27, 2022

View reviewed changes

common code

833674b

guohao-rosicky requested a review from szetszwo March 28, 2022 02:04

captainzmc approved these changes Mar 28, 2022

View reviewed changes

szetszwo approved these changes Mar 28, 2022

View reviewed changes

szetszwo merged commit 9546d4c into apache:HDDS-4454 Mar 28, 2022

szetszwo changed the title ~~HDDS-6137. [Ozone-Streaming] Optimized the putBlock method of BlockDataStreamOutput~~ HDDS-6137. [Ozone-Streaming] Refactor KeyDataStreamOutput. Mar 28, 2022

szetszwo pushed a commit to szetszwo/ozone that referenced this pull request May 6, 2022

HDDS-6137. [Ozone-Streaming] Refactor KeyDataStreamOutput. (apache#3195)

b23fbcf

szetszwo pushed a commit that referenced this pull request May 13, 2022

HDDS-6137. [Ozone-Streaming] Refactor KeyDataStreamOutput. (#3195)

8995a69

szetszwo pushed a commit that referenced this pull request May 24, 2022

HDDS-6137. [Ozone-Streaming] Refactor KeyDataStreamOutput. (#3195)

bfb9216

szetszwo pushed a commit that referenced this pull request Jun 9, 2022

HDDS-6137. [Ozone-Streaming] Refactor KeyDataStreamOutput. (#3195)

6c0f95f

captainzmc pushed a commit to captainzmc/hadoop-ozone that referenced this pull request Jul 4, 2022

HDDS-6137. [Ozone-Streaming] Refactor KeyDataStreamOutput. (apache#3195)

88c4d59

szetszwo pushed a commit that referenced this pull request Oct 25, 2022

HDDS-6137. [Ozone-Streaming] Refactor KeyDataStreamOutput. (#3195)

99b414d

(cherry picked from commit 88c4d59) (cherry picked from commit 0a432aa0eb57bd25d63e656d0ec6e96a05689ac7)

szetszwo pushed a commit that referenced this pull request Nov 7, 2022

HDDS-6137. [Ozone-Streaming] Refactor KeyDataStreamOutput. (#3195)

06bc968

(cherry picked from commit 88c4d59) (cherry picked from commit 0a432aa0eb57bd25d63e656d0ec6e96a05689ac7) (cherry picked from commit 99b414d)

szetszwo pushed a commit that referenced this pull request Dec 1, 2022

HDDS-6137. [Ozone-Streaming] Refactor KeyDataStreamOutput. (#3195)

46c5184

(cherry picked from commit 06bc968)

szetszwo pushed a commit that referenced this pull request Dec 16, 2022

HDDS-6137. [Ozone-Streaming] Refactor KeyDataStreamOutput. (#3195)

95dbea6

(cherry picked from commit 06bc968)

nishitpatira pushed a commit to nishitpatira/ozone that referenced this pull request Dec 16, 2022

HDDS-6137. [Ozone-Streaming] Refactor KeyDataStreamOutput. (apache#3195)

8cc1354

(cherry picked from commit 06bc968)

		this.retryPolicyMap = HddsClientUtils.getRetryPolicyByException(
		config.getMaxRetryCount(), config.getRetryInterval());

HDDS-6137. [Ozone-Streaming] Refactor KeyDataStreamOutput. #3195

HDDS-6137. [Ozone-Streaming] Refactor KeyDataStreamOutput. #3195

Uh oh!

Conversation

guohao-rosicky commented Mar 15, 2022

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

captainzmc left a comment

Choose a reason for hiding this comment

Uh oh!

szetszwo Mar 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guohao-rosicky Mar 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

szetszwo Mar 16, 2022

Choose a reason for hiding this comment

Uh oh!

guohao-rosicky Mar 24, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

szetszwo commented Mar 16, 2022

Uh oh!

guohao-rosicky commented Mar 16, 2022

Uh oh!

szetszwo left a comment

Choose a reason for hiding this comment

Uh oh!

captainzmc commented Mar 17, 2022

Uh oh!

szetszwo commented Mar 17, 2022

Uh oh!

szetszwo commented Mar 17, 2022

Uh oh!

szetszwo commented Mar 17, 2022

Uh oh!

captainzmc commented Mar 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

szetszwo commented Mar 17, 2022

Uh oh!

szetszwo commented Mar 17, 2022

Uh oh!

captainzmc commented Mar 17, 2022

Uh oh!

captainzmc commented Mar 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

szetszwo commented Mar 17, 2022

Uh oh!

szetszwo commented Mar 17, 2022

Uh oh!

captainzmc commented Mar 17, 2022

Uh oh!

szetszwo left a comment

Choose a reason for hiding this comment

Uh oh!

guohao-rosicky commented Mar 25, 2022

Uh oh!

guohao-rosicky commented Mar 25, 2022

Uh oh!

szetszwo commented Mar 25, 2022

Uh oh!

szetszwo left a comment

Choose a reason for hiding this comment

Uh oh!

szetszwo Mar 26, 2022

Choose a reason for hiding this comment

Uh oh!

szetszwo Mar 26, 2022

Choose a reason for hiding this comment

Uh oh!

szetszwo Mar 26, 2022

Choose a reason for hiding this comment

szetszwo Mar 16, 2022 •

edited

Loading

guohao-rosicky Mar 16, 2022 •

edited

Loading

guohao-rosicky Mar 24, 2022 •

edited

Loading

captainzmc commented Mar 17, 2022 •

edited

Loading

captainzmc commented Mar 17, 2022 •

edited

Loading