HDDS-6955. [Ozone-streaming] Add explicit stream flag in ozone shell #3559
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
Author
|
@captainzmc @szetszwo PTAL, thanks. |
szetszwo
reviewed
Jun 30, 2022
Contributor
szetszwo
left a comment
szetszwo
left a comment
There was a problem hiding this comment.
@kaijchen , thanks for working on this. Some comments inlined. See also https://issues.apache.org/jira/secure/attachment/13046110/3559_review.patch
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/PutKeyHandler.java
Outdated
Show resolved
Hide resolved
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/PutKeyHandler.java
Outdated
Show resolved
Hide resolved
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/shell/keys/PutKeyHandler.java
Show resolved
Hide resolved
Member
Author
|
@szetszwo Thanks for the improvement. |
Member
Member
Author
captainzmc
approved these changes
Jul 12, 2022
Member
Author
|
Thanks @szetszwo and @captainzmc for the review. |
szetszwo
pushed a commit
that referenced
this pull request
Oct 25, 2022
szetszwo
pushed a commit
that referenced
this pull request
Nov 7, 2022
szetszwo
pushed a commit
that referenced
this pull request
Dec 1, 2022
szetszwo
pushed a commit
that referenced
this pull request
Dec 16, 2022
nishitpatira
pushed a commit
to nishitpatira/ozone
that referenced
this pull request
Dec 16, 2022
…pache#3559) (cherry picked from commit 6319ce8)
errose28
added a commit
to errose28/ozone
that referenced
this pull request
Dec 21, 2022
* master: (88 commits) HDDS-7463. SCM Pipeline scrubber never able to cleanup allocated pipeline. (apache#4093) HDDS-7683. EC: ReplicationManager - UnderRep maintenance handler should not request nodes if none needed (apache#4109) HDDS-7635. Update failure metrics when allocate block fails in preExecute. (apache#4086) HDDS-7565. FSO purge directory for old bucket can update quota for new bucket (apache#4021) HDDS-7654. EC: ReplicationManager - merge mis-rep queue into under replicated queue (apache#4099) HDDS-7621. Update SCM term in datanode from heartbeat without any commands (apache#4101) HDDS-7649. S3 multipart upload EC release space quota wrong for old version (apache#4095) HDDS-7399. Enable specifying external root ca (apache#4053) HDDS-7398. Tool to remove old certs from the scm db (apache#3972) HDDS-6650. S3MultipartUpload support update bucket usedNamespace. (apache#4081) HDDS-7605. Improve logging in Container Balancer (apache#4067) HDDS-7616. EC: Refactor Unhealthy Replicated Processor (apache#4063) HDDS-7426. Add a new acceptance test for Streaming Pipeline. (apache#4019) HDDS-7478. [Ozone-Streaming] NPE in when creating a file with o3fs. (apache#3949) HDDS-7425. Add documentation for the new Streaming Pipeline feature. (apache#3913) HDDS-7438. [Ozone-Streaming] Add a createStreamKey method to OzoneBucket. (apache#3914) HDDS-7431. [Ozone-Streaming] Disable data steam by default. (apache#3900) HDDS-6955. [Ozone-streaming] Add explicit stream flag in ozone shell (apache#3559) HDDS-6867. [Ozone-Streaming] PutKeyHandler should not use streaming to put EC key. (apache#3516) HDDS-6842. [Ozone-Streaming] Reduce the number of watch requests in StreamCommitWatcher. (apache#3492) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Cluster-level replication for EC was introduced in HDDS-6294.
When client side and bucket level replication config are both missing,
client will pass
nullas replication config to let OM decide.The first problem is NPE in createStreamKey() if replication config is
null.The second problem is, since streaming is not compatible for writing EC keys,
we cannot determine when to use streaming just by key size and client side replication config.
In this PR, I have added a explicit
--streamflag in ozone shell key handler.In stream mode, replication config will fallback to client side defaults when missing.
And it will be checked to ensure not using streaming to write an EC key.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-6955
How was this patch tested?
Manual test