Skip to content

HDDS-5142. Make generic streaming client/service for container re-replication, data read, scm/om snapshot download #2256

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
bshashikant merged 8 commits into from
May 24, 2021
Merged

HDDS-5142. Make generic streaming client/service for container re-replication, data read, scm/om snapshot download #2256

bshashikant merged 8 commits into from
May 24, 2021

Conversation

@elek
Copy link
Member

@elek elek commented May 17, 2021

What changes were proposed in this pull request?

This was discussed offline a few times. (cc @bshashikant @mukul1987)

Today we have server/client utilities/API for Hadoop RPC and GRPC based services but we don't have any unified and tested API for data streaming.

It is suggested to create a generic API for streaming.

As I need to modify the closed container replication I already created a POC based on Netty (instead of GRPC).

This patch shows the first proposed version.

  1. It uses a file-based API (the files should be specified to stream for a given id) because we can use Netty/native streaming without buffering in Java memory.
  2. Offsets can be supported (but not supported yet).
  3. Freon test is included which shows 8x better performance compared to the old GRPC based container replication. (220Mb / sec vs 1.6-1.7 Gb/sec, using one thread + tmpfs)

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-5142

How was this patch tested?

./ozone freon strmg -n 1000 --files 10

It creates a directory with 10*100MB files (1G) and replicates them 1000 times. Supposed to be finished under 10 minutes

@elek elek requested a review from bshashikant May 17, 2021 16:41
bshashikant
bshashikant reviewed May 19, 2021
View reviewed changes
Copy link
Contributor

@bshashikant bshashikant left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall.

...ontainer-service/src/main/java/org/apache/hadoop/ozone/container/stream/StreamingClient.java Outdated
package org.apache.hadoop.ozone.container.stream;

import io.netty.bootstrap.Bootstrap;
import io.netty.channel.*;
Copy link
Contributor

@bshashikant bshashikant May 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove * imports.

Copy link
Member Author

@elek elek May 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

...ontainer-service/src/main/java/org/apache/hadoop/ozone/container/stream/StreamingClient.java
this.port = port;
this.host = host;

group = new NioEventLoopGroup(100);
Copy link
Contributor

@bshashikant bshashikant May 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious, why the event loop group is created with 100 threads? any rationale behind it?

Copy link
Member Author

@elek elek May 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really. I just used a very high number to make sure it's not a limit during the test ;-)

I think we should make both the server and client more configurable in follow-up patches.

Copy link
Contributor

@bshashikant bshashikant May 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks marton. Let's file jiras to make these things configurable.

@elek
Copy link
Member Author

elek commented May 19, 2021

Thanks a lot for the review @bshashikant. Just pushed the fix commit. Can you PTAL?

@bshashikant bshashikant merged commit 8d29be1 into apache:master May 24, 2021
@bshashikant
Copy link
Contributor

bshashikant commented May 24, 2021

Thanks @elek for the contribution.

errose28 added a commit to errose28/ozone that referenced this pull request Jun 1, 2021
@errose28
Merge remote-tracking branch 'upstream/master' into HDDS-3698-nonroll…
1d18f47 
…ing-upgrade-master-merge

* upstream/master: (76 commits)
  HDDS-5280. Make XceiverClientManager creation when necessary in ContainerOperationClient (apache#2289)
  HDDS-5272. Make ozonefs.robot execution repeatable (apache#2280)
  HDDS-5123. Use the pre-created apache/ozone-testkrb5 image during secure acceptance tests (apache#2165)
  HDDS-4993. Add guardrail for reserved buffer size when DN reads a chunk (apache#2058)
  HDDS-4936. Change ozone groupId from org.apache.hadoop to org.apache.ozone (apache#2018)
  HDDS-4043. allow deletion from Trash directory without -skipTrash option (apache#2110)
  HDDS-4927. Determine over and under utilized datanodes in Container Balancer. (apache#2230)
  HDDS-5273. Handle unsecure cluster convert to secure cluster for SCM. (apache#2281)
  HDDS-5158. Add documentation for SCM HA Security. (apache#2205)
  HDDS-5275. Datanode Report Publisher publishes one extra report after DN shutdown (apache#2283)
  HDDS-5241. SCM UI should have leader/follower and Primordial SCM information (apache#2260)
  HDDS-5219. Limit number of bad volumes by dfs.datanode.failed.volumes.tolerated. (apache#2243)
  HDDS-5252. PipelinePlacementPolicy filter out datanodes with not enough space. (apache#2271)
  HDDS-5191. Increase default pvc storage size (apache#2219)
  HDDS-5073. Use ReplicationConfig on client side  (apache#2136)
  HDDS-5250. Build integration tests with Maven cache (apache#2269)
  HDDS-5236. Require block token for more operations (apache#2254)
  HDDS-5266 Misspelt words in S3MultipartUploadCommitPartRequest.java line 202 (apache#2279)
  HDDS-5249. Race Condition between Full and Incremental Container Reports (apache#2268)
  HDDS-5142. Make generic streaming client/service for container re-replication, data read, scm/om snapshot download (apache#2256)
  ...

Conflicts:
	hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocol.java
	hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/StorageContainerLocationProtocolClientSideTranslatorPB.java
	hadoop-hdds/interface-admin/src/main/proto/ScmAdminProtocol.proto
	hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
	hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/container/MockNodeManager.java
	hadoop-ozone/dist/src/main/compose/testlib.sh
	hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestStorageContainerManager.java
	hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto
	hadoop-ozone/ozone-manager/pom.xml
	hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
	hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ratis/utils/OzoneManagerRatisUtils.java
	hadoop-ozone/s3gateway/pom.xml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bshashikant bshashikant bshashikant approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@elek @bshashikant