HDDS-5851. [Ozone-Streaming] Define a PutBlock/maxBuffer fixed boundary for streaming writes. #2866
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bshashikant
reviewed
Nov 29, 2021
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/OzoneClientConfig.java
Outdated
Show resolved
Hide resolved
captainzmc
reviewed
Nov 30, 2021
Member
There was a problem hiding this comment.
Did we need add updateFlushLength(), waitOnFlushFutures() and watchForCommit(false) here? Just as what we did in handleFlush. Currently we only releaseBuffers after watchForCommit.
Contributor
Author
There was a problem hiding this comment.
Thanks @captainzmc for the comment. The plan here is to execute a putBlock at 32MB and not block the client right away by waiting for the flush futures and calling watchForCommit. We will introduce another boundary say at 64MB to check whether atleast 32MB is flushed and committed to all datanodes i.e watchForCommit() and waitForFlushFutures() will occur at 64mb. (I will add this is in a different patch after this PR)
Member
There was a problem hiding this comment.
Thanks for explaining, make sense.
added 4 commits
November 30, 2021 14:59
…ites. (cherry picked from commit 4406ecf99c38b14c6a0260a933df2fde954e5b99)
(cherry picked from commit 1fcac2916ce5c1b1060edcef3714c8ec43bc6605)
(cherry picked from commit f0858e58cb214c1f5792ff54f1d26b72066359d1)
sadanand48
force-pushed
the
HDDS-5851
branch
from
f0858e5 to
b7164cd
Compare
sadanand48
changed the title
bshashikant
reviewed
Nov 30, 2021
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/OzoneClientConfig.java
Show resolved
Hide resolved
added 2 commits
November 30, 2021 15:21
bshashikant
approved these changes
Dec 1, 2021
Contributor
|
Thanks @sadanand48 for the contribution. |
captainzmc
pushed a commit
that referenced
this pull request
Dec 20, 2021
…ry for streaming writes. (#2866)
szetszwo
pushed a commit
that referenced
this pull request
Dec 30, 2021
…ry for streaming writes. (#2866)
szetszwo
pushed a commit
that referenced
this pull request
Jan 20, 2022
…ry for streaming writes. (#2866)
captainzmc
pushed a commit
that referenced
this pull request
Feb 9, 2022
…ry for streaming writes. (#2866)
szetszwo
pushed a commit
that referenced
this pull request
Feb 16, 2022
…ry for streaming writes. (#2866)
szetszwo
pushed a commit
that referenced
this pull request
Mar 15, 2022
…ry for streaming writes. (#2866)
szetszwo
pushed a commit
that referenced
this pull request
Mar 24, 2022
…ry for streaming writes. (#2866)
szetszwo
pushed a commit
to szetszwo/ozone
that referenced
this pull request
May 6, 2022
…ry for streaming writes. (apache#2866)
szetszwo
pushed a commit
that referenced
this pull request
May 13, 2022
…ry for streaming writes. (#2866)
szetszwo
pushed a commit
that referenced
this pull request
May 24, 2022
…ry for streaming writes. (#2866)
szetszwo
pushed a commit
that referenced
this pull request
Jun 9, 2022
…ry for streaming writes. (#2866)
captainzmc
pushed a commit
to captainzmc/hadoop-ozone
that referenced
this pull request
Jul 4, 2022
…ry for streaming writes. (apache#2866)
szetszwo
pushed a commit
that referenced
this pull request
Nov 7, 2022
nishitpatira
pushed a commit
to nishitpatira/ozone
that referenced
this pull request
Dec 16, 2022
…ry for streaming writes. (apache#2866) (cherry picked from commit 9d5ccb3)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Currently, putBlock call gets executed only during close() when a block is written. The aim here is to have a putBlock boundary similar to async path where the putBlock gets executed which updates the key length and help reduce some memory on client. The default boundary here is set at 32 MB where a putBlock is executed. Also a limit is set on the ByteBuffer size the client can send here (4mb).
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5851
How was this patch tested?
Unit test