HDDS-5961. [Ozone-Streaming] update the usage space of Containers in … #2833

guohao-rosicky · 2021-11-12T04:31:58Z

HDDS-5961. [Ozone-Streaming] update the usage space of Containers in the stream write

jira: https://issues.apache.org/jira/browse/HDDS-5961

guohao-rosicky · 2021-11-12T04:32:52Z

hi @captainzmc , can you help review this patch?

guohao-rosicky · 2021-11-12T09:17:34Z

The old StreamDataChannel implementation did not update the usedBytes in the Container metadata at write time.
There will be multiple Container types in the future, I put the implementation of DataChannel into the Container package.

guohao-rosicky · 2021-11-12T15:02:53Z

hi @szetszwo , can you help review this patch?

szetszwo · 2021-11-15T04:48:00Z

@guohao-rosicky , I just have sync'ed the HDDS-4454 branch with master. Could you update this pull request?

guohao-rosicky · 2021-11-15T09:10:57Z

@szetszwo @captainzmc , I have completed the update.

captainzmc

LGTM overall, Failed UT may not be caused by this pr, it can be triggered again.

captainzmc · 2021-11-15T13:34:06Z

...src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/KeyValueStreamDataChannel.java

Now we had updae metrics.incContainerBytesStats and containerData.updateWriteStats. Can you also add some test case? Just make sure that future changes do not affect this logic.

@captainzmc ,Has been modified.

szetszwo

@guohao-rosicky , thanks a lot for working on this. Some comments inlined; see also https://issues.apache.org/jira/secure/attachment/13036136/2833_review.patch .

BTW, we use 4 spaces for continuation indent in Ozone. Please change the continuation indent to 4 spaces (instead of 8). Thanks.

szetszwo · 2021-11-16T08:29:05Z

...src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/KeyValueStreamDataChannel.java

Let's use a new Type, say StreamWrite?

@szetszwo is container type.

szetszwo · 2021-11-16T08:30:13Z

...src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/KeyValueStreamDataChannel.java

Remove public.

szetszwo · 2021-11-16T08:30:30Z

...src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/KeyValueStreamDataChannel.java

Remove LOG since it is not used.

szetszwo · 2021-11-16T08:30:40Z

...src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/KeyValueStreamDataChannel.java

Remove public.

szetszwo · 2021-11-16T08:37:38Z

...src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/KeyValueStreamDataChannel.java

Use ContainerData.

szetszwo · 2021-11-16T08:38:15Z

...src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/KeyValueStreamDataChannel.java

Use ContainerData so that we do not need to cast it when calling the constructor.

szetszwo · 2021-11-16T08:56:05Z

...iner-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/HddsDispatcher.java

Add @Override.

szetszwo · 2021-11-16T08:59:17Z

...e/src/main/java/org/apache/hadoop/ozone/container/common/interfaces/ContainerDispatcher.java

Add default to throw UnsupportedOperationException so that the tests won't have to override it.

szetszwo · 2021-11-16T09:03:30Z

...a/org/apache/hadoop/ozone/container/common/transport/server/ratis/ContainerStateMachine.java

Do not try-catch and throws StorageContainerException in the getStreamDataChannel(..) declaration.

szetszwo · 2021-11-16T09:04:49Z

...tainer-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java

Add @Override.

…the stream write

guohao-rosicky · 2021-11-17T05:51:48Z

Has been modified, @szetszwo @captainzmc .Please Take A Look.

szetszwo

+1 the latest change looks good.

…the stream write (#2833)

…the stream write (apache#2833)

…the stream write (#2833)

…the stream write (apache#2833)

…the stream write (#2833) (cherry picked from commit ddc8a2c) (cherry picked from commit 65ec5817e22a666a4fa4fe76ac5c0c1c3263865a)

…the stream write (#2833) (cherry picked from commit ddc8a2c) (cherry picked from commit 65ec5817e22a666a4fa4fe76ac5c0c1c3263865a) (cherry picked from commit 8cbadbc)

…the stream write (#2833) (cherry picked from commit 7290ff9)

…the stream write (apache#2833) (cherry picked from commit 7290ff9)

guohao-rosicky force-pushed the HDDS-4454-stream-channel branch from da03fff to 316ef1f Compare November 12, 2021 05:09

captainzmc requested review from captainzmc and szetszwo November 12, 2021 09:19

szetszwo force-pushed the HDDS-4454 branch from 871f2cd to 1d9f870 Compare November 15, 2021 04:46

guohao-rosicky force-pushed the HDDS-4454-stream-channel branch from 316ef1f to 3b3fb1a Compare November 15, 2021 09:08

guohao-rosicky force-pushed the HDDS-4454-stream-channel branch from 3b3fb1a to 7df8c76 Compare November 15, 2021 11:56

captainzmc reviewed Nov 15, 2021

View reviewed changes

szetszwo requested changes Nov 16, 2021

View reviewed changes

HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …

b2f4cce

…the stream write

guohao-rosicky force-pushed the HDDS-4454-stream-channel branch from 7df8c76 to b2f4cce Compare November 16, 2021 14:04

szetszwo approved these changes Nov 17, 2021

View reviewed changes

szetszwo merged commit 869f943 into apache:HDDS-4454 Nov 17, 2021

szetszwo pushed a commit that referenced this pull request Nov 30, 2021

HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …

153ef3e

…the stream write (#2833)

captainzmc pushed a commit that referenced this pull request Dec 20, 2021

HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …

b1753e6

…the stream write (#2833)

szetszwo pushed a commit that referenced this pull request Dec 30, 2021

HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …

c659513

…the stream write (#2833)

szetszwo pushed a commit that referenced this pull request Jan 20, 2022

HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …

8d13099

…the stream write (#2833)

captainzmc pushed a commit that referenced this pull request Feb 9, 2022

HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …

a25e50c

…the stream write (#2833)

szetszwo pushed a commit that referenced this pull request Feb 16, 2022

HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …

6e01261

…the stream write (#2833)

szetszwo pushed a commit that referenced this pull request Mar 15, 2022

HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …

06dbec5

…the stream write (#2833)

szetszwo pushed a commit that referenced this pull request Mar 24, 2022

HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …

6bf06d1

…the stream write (#2833)

szetszwo pushed a commit to szetszwo/ozone that referenced this pull request May 6, 2022

HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …

bb41750

…the stream write (apache#2833)

szetszwo pushed a commit that referenced this pull request May 13, 2022

HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …

628a7a4

…the stream write (#2833)

szetszwo pushed a commit that referenced this pull request May 24, 2022

HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …

4871da7

…the stream write (#2833)

szetszwo pushed a commit that referenced this pull request Jun 9, 2022

HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …

e350e01

…the stream write (#2833)

captainzmc pushed a commit to captainzmc/hadoop-ozone that referenced this pull request Jul 4, 2022

HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …

ddc8a2c

…the stream write (apache#2833)

szetszwo pushed a commit that referenced this pull request Oct 25, 2022

HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …

8cbadbc

…the stream write (#2833) (cherry picked from commit ddc8a2c) (cherry picked from commit 65ec5817e22a666a4fa4fe76ac5c0c1c3263865a)

szetszwo pushed a commit that referenced this pull request Dec 1, 2022

HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …

ad2fcba

…the stream write (#2833) (cherry picked from commit 7290ff9)

szetszwo pushed a commit that referenced this pull request Dec 16, 2022

HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …

9a7edb0

…the stream write (#2833) (cherry picked from commit 7290ff9)

nishitpatira pushed a commit to nishitpatira/ozone that referenced this pull request Dec 16, 2022

HDDS-5961. [Ozone-Streaming] update the usage space of Containers in …

2cc3a78

…the stream write (apache#2833) (cherry picked from commit 7290ff9)

HDDS-5961. [Ozone-Streaming] update the usage space of Containers in … #2833

HDDS-5961. [Ozone-Streaming] update the usage space of Containers in … #2833

Uh oh!

Conversation

guohao-rosicky commented Nov 12, 2021

Uh oh!

guohao-rosicky commented Nov 12, 2021

Uh oh!

guohao-rosicky commented Nov 12, 2021

Uh oh!

guohao-rosicky commented Nov 12, 2021

Uh oh!

szetszwo commented Nov 15, 2021

Uh oh!

guohao-rosicky commented Nov 15, 2021

Uh oh!

captainzmc left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

szetszwo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guohao-rosicky commented Nov 17, 2021

Uh oh!

szetszwo left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants