HDDS-5488. [Ozone-Streaming] Add a new BlockOutputStream/KeyOutputStream to support streaming api #2495
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…eMachine. (apache#2358). Contributed by mingchao zhao
This reverts commit d3fc826.
This reverts commit 0bbe5ab.
szetszwo
pushed a commit
that referenced
this pull request
Aug 26, 2021
…eam to support streaming api (#2495)
szetszwo
pushed a commit
that referenced
this pull request
Sep 8, 2021
…eam to support streaming api (#2495)
szetszwo
pushed a commit
that referenced
this pull request
Oct 1, 2021
…eam to support streaming api (#2495)
szetszwo
pushed a commit
that referenced
this pull request
Oct 19, 2021
…eam to support streaming api (#2495)
szetszwo
pushed a commit
that referenced
this pull request
Oct 28, 2021
…eam to support streaming api (#2495)
szetszwo
pushed a commit
that referenced
this pull request
Nov 15, 2021
…eam to support streaming api (#2495)
szetszwo
pushed a commit
that referenced
this pull request
Nov 30, 2021
…eam to support streaming api (#2495)
captainzmc
pushed a commit
that referenced
this pull request
Dec 20, 2021
…eam to support streaming api (#2495)
szetszwo
pushed a commit
that referenced
this pull request
Dec 30, 2021
…eam to support streaming api (#2495)
szetszwo
pushed a commit
that referenced
this pull request
Jan 20, 2022
…eam to support streaming api (#2495)
captainzmc
pushed a commit
that referenced
this pull request
Feb 9, 2022
…eam to support streaming api (#2495)
szetszwo
pushed a commit
that referenced
this pull request
Feb 16, 2022
…eam to support streaming api (#2495)
szetszwo
pushed a commit
that referenced
this pull request
Mar 15, 2022
…eam to support streaming api (#2495)
szetszwo
pushed a commit
that referenced
this pull request
Mar 24, 2022
…eam to support streaming api (#2495)
szetszwo
pushed a commit
to szetszwo/ozone
that referenced
this pull request
May 6, 2022
…eam to support streaming api (apache#2495)
szetszwo
pushed a commit
that referenced
this pull request
May 13, 2022
…eam to support streaming api (#2495)
szetszwo
pushed a commit
that referenced
this pull request
May 24, 2022
…eam to support streaming api (#2495)
szetszwo
pushed a commit
that referenced
this pull request
Jun 9, 2022
…eam to support streaming api (#2495)
captainzmc
pushed a commit
to captainzmc/hadoop-ozone
that referenced
this pull request
Jul 4, 2022
…eam to support streaming api (apache#2495)
szetszwo
pushed a commit
that referenced
this pull request
Nov 7, 2022
nishitpatira
pushed a commit
to nishitpatira/ozone
that referenced
this pull request
Dec 16, 2022
…eam to support streaming api (apache#2495) (cherry picked from commit e6be97e)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Request for Comments
This is our streaming client implementation. It’s still in the early stages, but at least it works.
This might be a good start to discuss the design, comments are welcomed.
What changes were proposed in this pull request?
New DataStreamOutput were added, compared with OutputStream, there are some outstanding changes:
write(ByteBuf)methods were added, which can be used for zero buffer copy later.write(byte[])methods were kept, so these DataStreamOutput still extends OutputStream for compatibility.write(ByteBuf)is perferred overwrite(byte[]), the latter is implemented by calling the former.BlockDataStreamOutput#write.Also, we have modified the ozone shell to use the streaming api for putting large keys (length >= chunkSize).
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5488
How was this patch tested?
If this design is ok, unit tests should be added.