HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mode #2682

captainzmc · 2021-09-27T11:37:37Z

What changes were proposed in this pull request?

Streaming supports writing in Pipline mode

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-5486

captainzmc · 2021-09-28T07:19:57Z

@szetszwo Please ignore the problem I asked earlier, I have found the root cause.

szetszwo · 2021-09-28T07:32:50Z

... I have found the root cause.

That's great!

szetszwo

A minor comment inlined.

szetszwo · 2021-09-28T07:19:37Z

hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java

RoutingTable only needs RaftPeerId. So we can have List<RaftPeerId> instead of List<RaftPeer>.

bshashikant · 2021-09-28T14:43:34Z

@captainzmc , can you add some description on the jira for this? The patch otherwise looks good. Please remove the commented out code.

captainzmc · 2021-09-29T03:54:27Z

@captainzmc , can you add some description on the jira for this? The patch otherwise looks good. Please remove the commented out code.

Yes, I have updated jIRA and add the description.

captainzmc · 2021-09-29T06:23:52Z

hi @szetszwo, The code has been updated and tested. Can you help take another look?

szetszwo

Thanks for the update. Some comments inlined.

szetszwo · 2021-09-29T07:23:52Z

hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java

+
+  public RoutingTable getRoutingTable(Pipeline pipeline) {
+    RaftPeerId primary = null;
+    List<RaftPeerId> raftPeers = new ArrayList();


It should have an "<>", i.e. new ArrayList<>().

szetszwo · 2021-09-29T07:24:42Z

hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java

+      RaftPeer.Builder raftPeerBuilder = RaftPeer.newBuilder();
+      raftPeerBuilder.setId(dn.getUuidString()).setAddress(dn.getIpAddress()
+          + ":" + dn.getPort(DatanodeDetails.Port.Name.RATIS_SERVER)
+          .getValue());
+      raftPeerBuilder.setDataStreamAddress(dn.getIpAddress()
+          + ":" + dn.getPort(DatanodeDetails.Port.Name.RATIS_DATASTREAM)
+          .getValue());
+      raftPeerBuilder.setAdminAddress(dn.getIpAddress()
+          + ":" + dn.getPort(DatanodeDetails.Port.Name.RATIS_ADMIN).getValue());
+      raftPeerBuilder.setClientAddress(dn.getIpAddress()
+          + ":" + dn.getPort(DatanodeDetails.Port.Name.RATIS).getValue());


Since RaftPeer is not needed, let's just create a RaftPeerId.

final RaftPeerId id = RaftPeerId.valueOf(dn.getUuidString());

szetszwo · 2021-09-29T07:25:02Z

hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java

+
+        }
+      } catch (IOException e) {
+        e.printStackTrace();


It should return null when there is an exception.

szetszwo

+1 the change looks good.

captainzmc · 2021-09-29T14:42:46Z

CI is performing well on my personal branch. This failure should be accidental.

…de (#2682)

…de (apache#2682)

…de (#2682)

…de (apache#2682)

…de (#2682) (cherry picked from commit 2e85eeb) (cherry picked from commit 10b14300d274cf224f56763c66dea9c2d8a6f731)

…de (#2682) (cherry picked from commit 2e85eeb) (cherry picked from commit 10b14300d274cf224f56763c66dea9c2d8a6f731) (cherry picked from commit 2791680)

…de (#2682) (cherry picked from commit 8124a1c)

…de (apache#2682) (cherry picked from commit 8124a1c)

captainzmc requested a review from szetszwo September 27, 2021 12:10

szetszwo changed the title ~~HDDS-5705. [Ozone-Streaming] Streaming supports writing in Pipline mode~~ HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mode Sep 28, 2021

szetszwo reviewed Sep 28, 2021

View reviewed changes

Streaming supports writing in Pipline mode

fe7c4e3

captainzmc force-pushed the support-pipline branch from 96f98f0 to fe7c4e3 Compare September 29, 2021 03:39

fix review issues

31db63e

szetszwo reviewed Sep 29, 2021

View reviewed changes

fix review issues

7741ade

captainzmc force-pushed the support-pipline branch from 574fa34 to 7741ade Compare September 29, 2021 11:11

szetszwo approved these changes Sep 29, 2021

View reviewed changes

captainzmc added 2 commits September 29, 2021 21:34

trigger new CI

e541858

trigger new CI

aa875f8

szetszwo merged this pull request into apache:HDDS-4454 Sep 30, 2021

szetszwo pushed a commit that referenced this pull request Oct 1, 2021

HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mo…

420e472

…de (#2682)

sadanand48 pushed a commit that referenced this pull request Oct 2, 2021

HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mo…

5d00641

…de (#2682)

szetszwo pushed a commit that referenced this pull request Oct 19, 2021

HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mo…

c9ca832

…de (#2682)

szetszwo pushed a commit that referenced this pull request Oct 28, 2021

HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mo…

51664a3

…de (#2682)

szetszwo pushed a commit that referenced this pull request Nov 15, 2021

HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mo…

5156051

…de (#2682)

szetszwo pushed a commit that referenced this pull request Nov 30, 2021

HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mo…

836a67c

…de (#2682)

captainzmc added a commit that referenced this pull request Dec 20, 2021

HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mo…

59475f4

…de (#2682)

szetszwo pushed a commit that referenced this pull request Dec 30, 2021

HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mo…

b3bbc00

…de (#2682)

szetszwo pushed a commit that referenced this pull request Jan 20, 2022

HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mo…

c51065e

…de (#2682)

captainzmc added a commit that referenced this pull request Feb 9, 2022

HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mo…

e221851

…de (#2682)

szetszwo pushed a commit that referenced this pull request Feb 16, 2022

HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mo…

7b45206

…de (#2682)

szetszwo pushed a commit that referenced this pull request Mar 15, 2022

HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mo…

aba2b3b

…de (#2682)

szetszwo pushed a commit that referenced this pull request Mar 24, 2022

HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mo…

af5a9f5

…de (#2682)

szetszwo pushed a commit to szetszwo/ozone that referenced this pull request May 6, 2022

HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mo…

7663ee5

…de (apache#2682)

szetszwo pushed a commit that referenced this pull request May 13, 2022

HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mo…

c210ad4

…de (#2682)

szetszwo pushed a commit that referenced this pull request May 24, 2022

HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mo…

a96e868

…de (#2682)

szetszwo pushed a commit that referenced this pull request Jun 9, 2022

HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mo…

55dab06

…de (#2682)

captainzmc added a commit to captainzmc/hadoop-ozone that referenced this pull request Jul 4, 2022

HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mo…

2e85eeb

…de (apache#2682)

szetszwo pushed a commit that referenced this pull request Oct 25, 2022

HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mo…

2791680

…de (#2682) (cherry picked from commit 2e85eeb) (cherry picked from commit 10b14300d274cf224f56763c66dea9c2d8a6f731)

szetszwo pushed a commit that referenced this pull request Dec 1, 2022

HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mo…

c5ef6f7

…de (#2682) (cherry picked from commit 8124a1c)

szetszwo pushed a commit that referenced this pull request Dec 16, 2022

HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mo…

94f8789

…de (#2682) (cherry picked from commit 8124a1c)

nishitpatira pushed a commit to nishitpatira/ozone that referenced this pull request Dec 16, 2022

HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mo…

4f8383b

…de (apache#2682) (cherry picked from commit 8124a1c)

HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mode #2682

HDDS-5486. [Ozone-Streaming] Streaming supports writing in Pipline mode #2682

Uh oh!

Conversation

captainzmc commented Sep 27, 2021

What changes were proposed in this pull request?

What is the link to the Apache JIRA

Uh oh!

captainzmc commented Sep 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

szetszwo commented Sep 28, 2021

Uh oh!

szetszwo left a comment

Choose a reason for hiding this comment

Uh oh!

szetszwo Sep 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bshashikant commented Sep 28, 2021

Uh oh!

captainzmc commented Sep 29, 2021

Uh oh!

captainzmc commented Sep 29, 2021

Uh oh!

szetszwo left a comment

Choose a reason for hiding this comment

Uh oh!

szetszwo Sep 29, 2021

Choose a reason for hiding this comment

Uh oh!

szetszwo Sep 29, 2021

Choose a reason for hiding this comment

Uh oh!

szetszwo Sep 29, 2021

Choose a reason for hiding this comment

Uh oh!

szetszwo left a comment

Choose a reason for hiding this comment

Uh oh!

captainzmc commented Sep 29, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

captainzmc commented Sep 28, 2021 •

edited

Loading

szetszwo Sep 28, 2021 •

edited

Loading