HDDS-6039.Define a minimum packet size during streaming writes. #2883

sadanand48 · 2021-12-03T06:08:21Z

What changes were proposed in this pull request?

Define a minimum packet size during streaming writes as 64kb default.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-6039

How was this patch tested?

Unit test

bshashikant · 2021-12-06T08:45:06Z

hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java

It's better to allocate buffers of size equal to packet size and keep on using the same buffer until the packet is full or stream is closed/flushed. This will help reducing buffer allocations call.

captainzmc · 2021-12-07T11:24:07Z

Thanks @sadanand48 for the contribution. Can you describe the purpose of adding this PR? Is this PR related to retry optimization?

bshashikant · 2021-12-07T12:41:52Z

Thanks @sadanand48 for the contribution. Can you describe the purpose of adding this PR? Is this PR related to retry optimization?

The idea here is to ensure we are not sending too many packets when the data gets written in very small fragements(in bytes). There has been cases (for example-- word count acceptance test) where the write pattern is in terms of one byte at a time. For such cases, it becomes necessary to avoid sending extremely small packets over the wire. This idea is derived from hdfs where min packet size is defined as 64 kb.

captainzmc · 2021-12-07T12:53:34Z

Thanks @bshashikant for the explanation. I get your point. Yeah, we need this.

bshashikant · 2021-12-09T03:31:57Z

hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/OzoneClientConfig.java

Let's set this size to 1 MB == bytesPerCheckSum by default.

In performance test, I found that the performance of 512K is better than that of IMB. Can we change the default value to 512K?

bshashikant · 2021-12-09T03:41:42Z

hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/StreamBuffer.java

Its better to add position() and other calls in here, instead of getting an instance of underlying buffer directly.

bshashikant · 2021-12-09T03:42:26Z

hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/StreamBuffer.java

Its safe to always return a read only copy of the buffer if required. I would prefer to remove this.

Removed this.

bshashikant · 2021-12-10T11:11:36Z

hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java

Do we really need to track currentBufferRemaining? It can always be deduced from streamBuffer.

bshashikant · 2021-12-10T11:15:20Z

hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java

will this position be always be set to 0?? The starting offset should be deduced rom last ack'd length in case of retry.

We are writing a new chunk here everytime picking each entry from the bufferList, hence the position will be 0. Same logic as this

sadanand48 · 2021-12-20T08:21:38Z

Changed the default of ozone.fs.datastream.enable and created a new jira (HDDS-6126) to track the MR test failure .

captainzmc

+1 the change looks good.
Let's merge this PR first. If there is any problem later, we will modify it.

…che#2883)

(cherry picked from commit 1f56a45) (cherry picked from commit 750c5f475a4a03cdb078699d5f5a2254f1f491b0)

(cherry picked from commit 1f56a45) (cherry picked from commit 750c5f475a4a03cdb078699d5f5a2254f1f491b0) (cherry picked from commit 7e61b1f)

(cherry picked from commit 28fdecd)

…che#2883) (cherry picked from commit 28fdecd)

sadanand48 force-pushed the HDDS-6039 branch from 88d24c5 to 2911207 Compare December 3, 2021 20:22

sadanand48 marked this pull request as ready for review December 3, 2021 20:41

bshashikant reviewed Dec 6, 2021

View reviewed changes

captainzmc requested review from captainzmc and szetszwo December 7, 2021 11:10

sadanand48 marked this pull request as draft December 8, 2021 09:09

sadanand48 marked this pull request as ready for review December 9, 2021 06:29

bshashikant reviewed Dec 10, 2021

View reviewed changes

szetszwo force-pushed the HDDS-4454 branch from 99ddf5a to d1e4eea Compare December 15, 2021 13:17

captainzmc force-pushed the HDDS-4454 branch from d1e4eea to 89a12ad Compare December 20, 2021 06:26

Sadanand Shenoy added 15 commits December 20, 2021 13:27

HDDS-6039.Define a minimum packet size during streaming writes.

946897d

cleanup

fb5c4d0

trigger new CI check

9c34e32

write fixed sized chunks

d74d07d

fix few tests

d4bedad

fix TestBlockDataOutputStream

c0b76a9

cleanup

2d85b61

trigger new CI check

494d9cb

fix test

2383a28

trigger new CI check

f791369

trigger new CI check

30388bc

trigger new CI check

0aab669

fix test

2feac65

address comments

3623c99

fix commitwatcher logic

5f7951a

sadanand48 force-pushed the HDDS-6039 branch from d1e060e to 5f7951a Compare December 20, 2021 08:00

change default of ozone.fs.streaming.enable

93afa56

captainzmc approved these changes Dec 21, 2021

View reviewed changes

captainzmc merged commit cd143c5 into apache:HDDS-4454 Dec 21, 2021

szetszwo pushed a commit that referenced this pull request Dec 30, 2021

HDDS-6039. Define a minimum packet size during streaming writes. (#2883)

839bc63

szetszwo pushed a commit that referenced this pull request Jan 20, 2022

HDDS-6039. Define a minimum packet size during streaming writes. (#2883)

811f879

captainzmc pushed a commit that referenced this pull request Feb 9, 2022

HDDS-6039. Define a minimum packet size during streaming writes. (#2883)

e4c0c0f

szetszwo pushed a commit that referenced this pull request Feb 16, 2022

HDDS-6039. Define a minimum packet size during streaming writes. (#2883)

ce1d880

szetszwo pushed a commit that referenced this pull request Mar 15, 2022

HDDS-6039. Define a minimum packet size during streaming writes. (#2883)

4e21bdf

szetszwo pushed a commit that referenced this pull request Mar 24, 2022

HDDS-6039. Define a minimum packet size during streaming writes. (#2883)

5469877

szetszwo pushed a commit to szetszwo/ozone that referenced this pull request May 6, 2022

HDDS-6039. Define a minimum packet size during streaming writes. (apa…

ac9815d

…che#2883)

szetszwo pushed a commit that referenced this pull request May 13, 2022

HDDS-6039. Define a minimum packet size during streaming writes. (#2883)

40cf7e9

szetszwo pushed a commit that referenced this pull request May 24, 2022

HDDS-6039. Define a minimum packet size during streaming writes. (#2883)

ad505bc

szetszwo pushed a commit that referenced this pull request Jun 9, 2022

HDDS-6039. Define a minimum packet size during streaming writes. (#2883)

d2a51a0

captainzmc pushed a commit to captainzmc/hadoop-ozone that referenced this pull request Jul 4, 2022

HDDS-6039. Define a minimum packet size during streaming writes. (apa…

1f56a45

…che#2883)

szetszwo pushed a commit that referenced this pull request Oct 25, 2022

HDDS-6039. Define a minimum packet size during streaming writes. (#2883)

7e61b1f

(cherry picked from commit 1f56a45) (cherry picked from commit 750c5f475a4a03cdb078699d5f5a2254f1f491b0)

szetszwo pushed a commit that referenced this pull request Dec 1, 2022

HDDS-6039. Define a minimum packet size during streaming writes. (#2883)

67d79b5

(cherry picked from commit 28fdecd)

szetszwo pushed a commit that referenced this pull request Dec 16, 2022

HDDS-6039. Define a minimum packet size during streaming writes. (#2883)

ac9921d

(cherry picked from commit 28fdecd)

nishitpatira pushed a commit to nishitpatira/ozone that referenced this pull request Dec 16, 2022

HDDS-6039. Define a minimum packet size during streaming writes. (apa…

eb4bb4e

…che#2883) (cherry picked from commit 28fdecd)

HDDS-6039.Define a minimum packet size during streaming writes. #2883

HDDS-6039.Define a minimum packet size during streaming writes. #2883

Uh oh!

Conversation

sadanand48 commented Dec 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

captainzmc commented Dec 7, 2021

Uh oh!

bshashikant commented Dec 7, 2021

Uh oh!

captainzmc commented Dec 7, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

captainzmc Dec 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sadanand48 commented Dec 20, 2021

Uh oh!

captainzmc left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sadanand48 commented Dec 3, 2021 •

edited

Loading

captainzmc Dec 15, 2021 •

edited

Loading

captainzmc left a comment •

edited

Loading