Skip to content

HDDS-5060. [SCM HA Security] Make InterSCM grpc channel secure. #2187

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
bshashikant merged 3 commits into from
Apr 28, 2021
Merged

HDDS-5060. [SCM HA Security] Make InterSCM grpc channel secure. #2187

bshashikant merged 3 commits into from
Apr 28, 2021

Conversation

@bharatviswa504
Copy link
Contributor

@bharatviswa504 bharatviswa504 commented Apr 27, 2021

What changes were proposed in this pull request?

Make InterSCM grpc channel secure.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-5060

How was this patch tested?

Tested manually on docker ozone-secure-ha

Started Scm3 later, after some transactions are performed.

Config changes for testing:
OZONE-SITE.XML_ozone.scm.ha.ratis.log.purge.enabled=true
OZONE-SITE.XML_ozone.scm.ha.ratis.log.purge.gap=10
OZONE-SITE.XML_ozone.scm.ha.ratis.snapshot.threshold=5

scm3.org_1   | 2021-04-27 07:27:57,895 [grpc-default-executor-3] INFO server.GrpcServerProtocolService: 01c9e353-3886-4c6a-8c9b-f7278a473f30: Completed INSTALL_SNAPSHOT, lastRequest: 5249309c-73b8-4f8f-9b08-5cd73257f1df->01c9e353-3886-4c6a-8c9b-f7278a473f30#0-t2,notify:(t:2, i:43)
scm3.org_1   | 2021-04-27 07:27:57,929 [grpc-default-executor-0] INFO server.RaftServer$Division: 01c9e353-3886-4c6a-8c9b-f7278a473f30@group-EFC7205245FB: Failed appendEntries as snapshot ((t:2, i:43)) installation is in progress
scm3.org_1   | 2021-04-27 07:27:57,930 [grpc-default-executor-0] INFO server.RaftServer$Division: 01c9e353-3886-4c6a-8c9b-f7278a473f30@group-EFC7205245FB: inconsistency entries. Reply:5249309c-73b8-4f8f-9b08-5cd73257f1df<-01c9e353-3886-4c6a-8c9b-f7278a473f30#47:FAIL-t2,INCONSISTENCY,nextIndex=1,followerCommit=-1
scm3.org_1   | 2021-04-27 07:27:57,931 [grpc-default-executor-3] INFO server.RaftServer$Division: 01c9e353-3886-4c6a-8c9b-f7278a473f30@group-EFC7205245FB: receive installSnapshot: 5249309c-73b8-4f8f-9b08-5cd73257f1df->01c9e353-3886-4c6a-8c9b-f7278a473f30#0-t2,notify:(t:2, i:43)
scm3.org_1   | 2021-04-27 07:27:57,931 [grpc-default-executor-3] INFO server.RaftServer$Division: 01c9e353-3886-4c6a-8c9b-f7278a473f30@group-EFC7205245FB: Snapshot Installation by StateMachine is in progress.
scm3.org_1   | 2021-04-27 07:27:57,931 [pool-13-thread-1] INFO ha.SCMHAManagerImpl: Reloaded SCM state with Term: 2 and Index: 48

bshashikant
bshashikant approved these changes Apr 27, 2021
View reviewed changes
Copy link
Contributor

@bshashikant bshashikant left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. @xiaoyuyao , can you please check this one?

xiaoyuyao
xiaoyuyao approved these changes Apr 27, 2021
View reviewed changes
Copy link
Contributor

@xiaoyuyao xiaoyuyao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, +1.

vivekratnavel
vivekratnavel approved these changes Apr 27, 2021
View reviewed changes
Copy link
Contributor

@vivekratnavel vivekratnavel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 LGTM

@bshashikant bshashikant merged commit 6f9c3f2 into apache:master Apr 28, 2021
@bshashikant
Copy link
Contributor

bshashikant commented Apr 28, 2021

Thanks @bharatviswa504 for the fix and @xiaoyuyao / @vivekratnavel for the review. Committed this.

errose28 added a commit to errose28/ozone that referenced this pull request May 4, 2021
@errose28
Merge remote-tracking branch 'upstream/master' into HDDS-3698-nonroll…
253cc55 
…ing-upgrade-master-merge2

* upstream/master: (56 commits)
  HDDS-2212. Genconf tool should generate config files for secure clust… (apache#1788)
  HDDS-5166. Remove duplicate assignment of OZONE_OPTS for freon and sh (apache#2195)
  Revert "HDDS-5144. Create github check to alert when dependency tree is changed (apache#2177)"
  HDDS-4983. Display key offset for each block in command key info (apache#2051)
  HDDS-5144. Create github check to alert when dependency tree is changed (apache#2177)
  HDDS-4585. Support bucket acl operation in S3g (apache#1701)
  HDDS-5153. Decommissioning a dead node should complete immediately (apache#2190)
  HDDS-5147. Intermittent test failure in TestContainerDeletionChoosingPolicy#testRandomChoosingPolicy (apache#2188)
  HDDS-5152. Fix Suggested leader in Client. (apache#2189)
  HDDS-5148. Bump ratis version to 2.1.0-ff8aa66-SNAPSHOT (apache#2184)
  HDDS-4515. Datanodes should be able to persist and load CRL (apache#2181)
  HDDS-5060. [SCM HA Security] Make InterSCM grpc channel secure. (apache#2187)
  HDDS-5051. Ensure failover to suggested leader if any for NotLeaderException. (apache#2141)
  HDDS-5127. Fix getServiceList when SCM HA is enabled (apache#2173)
  HDDS-4889. Add simple CI check for docs (apache#2156)
  HDDS-5131. Use timeout in github actions (apache#2176)
  HDDS-5103. Fix Install Snapshot Mechanism in SCMStateMachine. (apache#2155)
  HDDS-5124. Use OzoneConsts.OZONE_TIME_ZONE instead of "GMT" (apache#2166)
  HDDS-5047. Refactor Pipeline to use ReplicationConfig instead of factor/type (apache#2096)
  HDDS-5083. Bump version of common-compress (apache#2139)
  ...

Conflicts:
	hadoop-hdds/common/pom.xml
	hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java
	hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
	hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/HddsDatanodeService.java
	hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java
	hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineManager.java
	hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMStorageConfig.java
	hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManager.java
	hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestSCMNodeManager.java
	hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneClusterImpl.java
	hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMStorage.java
	hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
	hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java
errose28 added a commit to errose28/ozone that referenced this pull request May 13, 2021
@errose28
Merge branch 'HDDS-3698-nonrolling-upgrade' into HDDS-5138-admin-chec…
080019b 
…k-in-auth

* HDDS-3698-nonrolling-upgrade: (57 commits)
  Fix compilation errors afte merge Update javassist in recon pom Fix changes introduced in merge that failed TestSCMNodeManager upgrade tests Fix checkstyle Fix intermittent test failure TestSCMNodeManager#testSetNodeOpStateAndCommandFired after merge Skip scm init default layout version in TestOzoneConfigurationFields
  HDDS-2212. Genconf tool should generate config files for secure clust… (apache#1788)
  HDDS-5166. Remove duplicate assignment of OZONE_OPTS for freon and sh (apache#2195)
  Revert "HDDS-5144. Create github check to alert when dependency tree is changed (apache#2177)"
  HDDS-4983. Display key offset for each block in command key info (apache#2051)
  HDDS-5144. Create github check to alert when dependency tree is changed (apache#2177)
  HDDS-4585. Support bucket acl operation in S3g (apache#1701)
  HDDS-5153. Decommissioning a dead node should complete immediately (apache#2190)
  HDDS-5147. Intermittent test failure in TestContainerDeletionChoosingPolicy#testRandomChoosingPolicy (apache#2188)
  HDDS-5152. Fix Suggested leader in Client. (apache#2189)
  HDDS-5148. Bump ratis version to 2.1.0-ff8aa66-SNAPSHOT (apache#2184)
  HDDS-4515. Datanodes should be able to persist and load CRL (apache#2181)
  HDDS-5060. [SCM HA Security] Make InterSCM grpc channel secure. (apache#2187)
  HDDS-5051. Ensure failover to suggested leader if any for NotLeaderException. (apache#2141)
  HDDS-5127. Fix getServiceList when SCM HA is enabled (apache#2173)
  HDDS-4889. Add simple CI check for docs (apache#2156)
  HDDS-5131. Use timeout in github actions (apache#2176)
  HDDS-5103. Fix Install Snapshot Mechanism in SCMStateMachine. (apache#2155)
  HDDS-5124. Use OzoneConsts.OZONE_TIME_ZONE instead of "GMT" (apache#2166)
  HDDS-5047. Refactor Pipeline to use ReplicationConfig instead of factor/type (apache#2096)
  ...
bharatviswa504 added a commit to bharatviswa504/hadoop-ozone that referenced this pull request Jul 25, 2021
@bharatviswa504 @bshashikant
HDDS-5060. [SCM HA Security] Make InterSCM grpc channel secure. (apac…
60328d2 
…he#2187)

(cherry picked from commit 6f9c3f2)
Change-Id: I9c17c129cbe3c7fdb20f5c8121740bc5b67ff6f3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vivekratnavel vivekratnavel vivekratnavel approved these changes

@xiaoyuyao xiaoyuyao xiaoyuyao approved these changes

@bshashikant bshashikant bshashikant approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@bharatviswa504 @bshashikant @vivekratnavel @xiaoyuyao