HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer to avoid buffer copying #2557

captainzmc · 2021-08-20T10:44:06Z

What changes were proposed in this pull request?

In BlockDataStreamOutput, we should write directly to the underlying stream. We should not use BufferPool and ChunkBuffer in BlockDataStreamOutput anymore since we want to avoid buffer copying.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-5599

How was this patch tested?

Existing UT

captainzmc · 2021-08-20T10:47:58Z

Hi @szetszwo @bshashikant @mukul1987
I've removed the BufferPool and ChunkBuffer from BlockDataStreamOutput in this PR.
Also, since BufferPool is removed, Retry and Flush policy will be different, and I think we can discuss their implementations in this PR.

szetszwo

The change looks good. Some comments inlined. Thanks.

szetszwo · 2021-08-22T23:59:23Z

hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java

+  // TODO We are no longer using the bufferPool, so we do not need to flush the
+  //  data in it. We can discuss whether add other behaviors here.


flush() should wait for all the futures to complete. Move the code below

//executePutBlock(..) try { CompletableFuture.allOf(futures.toArray(EMPTY_FUTURE_ARRAY)).get(); } catch (Exception e) { LOG.warn("Failed to write all chunks through stream: " + e); throw new IOException(e); }

from executePutBlock(..) to flush().

szetszwo · 2021-08-23T00:02:34Z

hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java

-    } else {
-      byteBufferList = null;
-    }
-


Move the try-catch below to flush() and then call flush() here.

szetszwo · 2021-08-23T00:09:19Z

hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/CommitWatcher.java

    futureMap = new ConcurrentHashMap<>();
  }

+  public CommitWatcher(XceiverClientSpi xceiverClient) {


Let's copy CommitWatcher to a new class, say StreamCommitWatcher? In StreamCommitWatcher, we don't need bufferPool. The commitIndex2flushedDataMap should be different -- it should not use ChunkBuffer anymore.

szetszwo

+1, the change looks good.

There are some test failures. Could you take a look?

captainzmc · 2021-08-25T11:15:55Z

+1, the change looks good.

There are some test failures. Could you take a look?

CI runs correctly on my personal branch. It seems the error is not caused by this PR. I will try to re-trigger the CI to make sure this.

captainzmc · 2021-08-25T15:19:21Z

hi @szetszwo, CI has been running correct. Could you help take another look this PR?

szetszwo

+1

kaijchen · 2021-08-26T05:52:31Z

We no longer copy the data into buffer pool. But we still need to retain the ByteBuf until it gets flushed. This retain is for the retry.

szetszwo · 2021-08-26T07:23:01Z

@bshashikant , @kaijchen , @captainzmc , I suggest that we first make the streaming write working (without retry, checksum, error handling, etc) so that we have something to test. Then, we can add those features. What do you think?

captainzmc · 2021-08-26T08:48:09Z

Agree with @szetszwo.
At present the writing flow of Ozone streaming has working. And our client also had avoided buffer copy.(Although we are currently using star Topology).
I think we can do a simple performance test now. Currently our implementation is very simple, which makes it easier to locate performance issues. We need to address performance issues first.
@bshashikant
In addition, I will test the performance of streaming with checksum enabled and without checksum enabled. To verify checksum's impact on performance. (The default value of io.bytes.per.checksum in HDFS is 512 bytes)Perhaps checksum's performance overhead isn't that high. Let's make sure by testing it first.

…buffer copying (#2557)

…buffer copying (apache#2557)

…buffer copying (#2557)

…buffer copying (apache#2557)

…buffer copying (#2557) (cherry picked from commit b1f1051) (cherry picked from commit 6b51811d1f0605e95d52441579190865064b9d05)

…buffer copying (#2557) (cherry picked from commit b1f1051) (cherry picked from commit 6b51811d1f0605e95d52441579190865064b9d05) (cherry picked from commit ae462a6)

…buffer copying (#2557) (cherry picked from commit db89077)

…buffer copying (apache#2557) (cherry picked from commit db89077)

kaijchen and others added 12 commits August 18, 2021 15:30

BlockDataStreamOutput: stop using bufferPool to avoid buffer copy

8aa4abb

delete useless method

cf4d291

delete refreshCurrentBuffer

5dfe1e6

delete ChunkBuffer currentBuffer

ecd00fc

new CommitWatcher constructor without bufferPool

9a26812

delete useless bufferPool

f7edbcb

Retry and flush needs to be discuss

7cd73bd

Attempt to remove the ChunkBuffer

10dd97f

Remove bufferList from BlockDataStreamOutput

91d755a

Remove bufferPool from StreamOutput

4a11af0

fix checkstyle

2cbb964

fix pom

73be32d

captainzmc requested review from bshashikant, mukul1987 and szetszwo August 20, 2021 10:44

Remove unnecessary ChunkBuffer

f4e4210

captainzmc force-pushed the drop-buffer-pool branch from d46f6fc to f4e4210 Compare August 21, 2021 08:22

szetszwo reviewed Aug 23, 2021

View reviewed changes

captainzmc added 2 commits August 25, 2021 12:49

fix review issues

b552965

fix CommitWatcher review issues

4a71962

captainzmc force-pushed the drop-buffer-pool branch from 5b377ae to 4a71962 Compare August 25, 2021 07:02

captainzmc added 2 commits August 25, 2021 15:18

remove useless totalAckDataLength

6fb16c2

trigger new CI

67a4298

szetszwo reviewed Aug 25, 2021

View reviewed changes

trigger new CI

42a5e8f

trigger new CI

10e0d46

szetszwo approved these changes Aug 25, 2021

View reviewed changes

szetszwo pushed a commit that referenced this pull request Aug 26, 2021

HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer to avoid …

6e60a16

…buffer copying (#2557)

szetszwo pushed a commit that referenced this pull request Sep 8, 2021

HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer to avoid …

9cf9c4d

…buffer copying (#2557)

szetszwo pushed a commit that referenced this pull request Oct 1, 2021

HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer to avoid …

69464ce

…buffer copying (#2557)

szetszwo pushed a commit that referenced this pull request Oct 19, 2021

HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer to avoid …

d9cbd7d

…buffer copying (#2557)

szetszwo pushed a commit that referenced this pull request Oct 28, 2021

HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer to avoid …

f6f8715

…buffer copying (#2557)

szetszwo pushed a commit that referenced this pull request Nov 15, 2021

HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer to avoid …

51f88cb

…buffer copying (#2557)

szetszwo pushed a commit that referenced this pull request Nov 30, 2021

HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer to avoid …

8fd34de

…buffer copying (#2557)

captainzmc added a commit that referenced this pull request Dec 20, 2021

HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer to avoid …

f54627e

…buffer copying (#2557)

szetszwo pushed a commit that referenced this pull request Dec 30, 2021

HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer to avoid …

e5fdb95

…buffer copying (#2557)

szetszwo pushed a commit that referenced this pull request Jan 20, 2022

HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer to avoid …

61aa979

…buffer copying (#2557)

captainzmc added a commit that referenced this pull request Feb 9, 2022

HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer to avoid …

ed97ac4

…buffer copying (#2557)

szetszwo pushed a commit that referenced this pull request Feb 16, 2022

HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer to avoid …

4830d36

…buffer copying (#2557)

szetszwo pushed a commit that referenced this pull request Mar 15, 2022

HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer to avoid …

c49edfe

…buffer copying (#2557)

szetszwo pushed a commit that referenced this pull request Mar 24, 2022

HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer to avoid …

c1e4e8e

…buffer copying (#2557)

szetszwo pushed a commit to szetszwo/ozone that referenced this pull request May 6, 2022

HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer to avoid …

0790843

…buffer copying (apache#2557)

szetszwo pushed a commit that referenced this pull request May 13, 2022

HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer to avoid …

ebae768

…buffer copying (#2557)

szetszwo pushed a commit that referenced this pull request May 24, 2022

HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer to avoid …

b19f749

…buffer copying (#2557)

szetszwo pushed a commit that referenced this pull request Jun 9, 2022

HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer to avoid …

948e345

…buffer copying (#2557)

captainzmc added a commit to captainzmc/hadoop-ozone that referenced this pull request Jul 4, 2022

HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer to avoid …

b1f1051

…buffer copying (apache#2557)

szetszwo pushed a commit that referenced this pull request Oct 25, 2022

HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer to avoid …

ae462a6

…buffer copying (#2557) (cherry picked from commit b1f1051) (cherry picked from commit 6b51811d1f0605e95d52441579190865064b9d05)

szetszwo pushed a commit that referenced this pull request Dec 1, 2022

HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer to avoid …

cd7cb21

…buffer copying (#2557) (cherry picked from commit db89077)

szetszwo pushed a commit that referenced this pull request Dec 16, 2022

HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer to avoid …

5a294b7

…buffer copying (#2557) (cherry picked from commit db89077)

nishitpatira pushed a commit to nishitpatira/ozone that referenced this pull request Dec 16, 2022

HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer to avoid …

ea33cfa

…buffer copying (apache#2557) (cherry picked from commit db89077)

		// TODO We are no longer using the bufferPool, so we do not need to flush the
		// data in it. We can discuss whether add other behaviors here.

HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer to avoid buffer copying #2557

HDDS-5599. [Ozone-Streaming]drop BufferPool and ChunkBuffer to avoid buffer copying #2557

Uh oh!

Conversation

captainzmc commented Aug 20, 2021

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

captainzmc commented Aug 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

szetszwo left a comment

Choose a reason for hiding this comment

Uh oh!

szetszwo Aug 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

szetszwo Aug 23, 2021

Choose a reason for hiding this comment

Uh oh!

szetszwo Aug 23, 2021

Choose a reason for hiding this comment

Uh oh!

szetszwo left a comment

Choose a reason for hiding this comment

Uh oh!

captainzmc commented Aug 25, 2021

Uh oh!

captainzmc commented Aug 25, 2021

Uh oh!

szetszwo left a comment

Choose a reason for hiding this comment

Uh oh!

kaijchen commented Aug 26, 2021

Uh oh!

szetszwo commented Aug 26, 2021

Uh oh!

captainzmc commented Aug 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

captainzmc commented Aug 20, 2021 •

edited

Loading

szetszwo Aug 22, 2021 •

edited

Loading

captainzmc commented Aug 26, 2021 •

edited

Loading