HDDS-6135. SCM Container DB bootstrap on Recon startup for SCM HA. #2972

aswinshakil · 2022-01-10T18:47:57Z

What changes were proposed in this pull request?

When Recon startup, Based on a threshold value we need to get SCM DB using gRPC and populate the Recon DB in SCM HA enabled cluster.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-6135

How was this patch tested?

This patch was tested manually and using integration tests.

JyotinderSingh · 2022-01-10T19:22:16Z

...one/integration-test/src/test/java/org/apache/hadoop/ozone/recon/TestReconScmHASnapshot.java

+    GenericTestUtils.LogCapturer logCapturer = GenericTestUtils.LogCapturer
+        .captureLogs(LoggerFactory.getLogger(
+            ReconStorageContainerManagerFacade.class));
+
+    List<ContainerInfo> reconContainers = cluster.getReconServer()
+        .getReconStorageContainerManager().getContainerManager()
+        .getContainers();
+    assertEquals(0, reconContainers.size());
+
+    ReconNodeManager nodeManager;
+    nodeManager = (ReconNodeManager) cluster.getReconServer()
+        .getReconStorageContainerManager().getScmNodeManager();
+    long keyCountBefore = nodeManager.getNodeDBKeyCount();
+
+    //Stopping Recon to add Containers in SCM
+    cluster.stopRecon();
+
+    ContainerManager containerManager;
+    containerManager = cluster.getStorageContainerManager()
+        .getContainerManager();
+
+    for (int i = 0; i < 10; i++) {
+      containerManager.allocateContainer(new RatisReplicationConfig(
+          HddsProtos.ReplicationFactor.ONE), "testOwner");
+    }
+
+    cluster.startRecon();


Can we move this to a common initializer function shared between the TestReconScmHASnapshot.java and TestReconScmSnapshot.java, it will help reduce code duplication across the two files.

JyotinderSingh · 2022-01-10T19:24:46Z

...one/integration-test/src/test/java/org/apache/hadoop/ozone/recon/TestReconScmHASnapshot.java

+    //PipelineCount after Recon DB is updated with SCM DB
+    PipelineManager scmPipelineManager = cluster.getStorageContainerManager()
+        .getPipelineManager();
+    PipelineManager reconPipelineManager = cluster.getReconServer()
+        .getReconStorageContainerManager().getPipelineManager();
+    assertEquals(scmPipelineManager.getPipelines().size(),
+        reconPipelineManager.getPipelines().size());
+
+    //NodeCount after Recon DB updated with SCM DB
+    nodeManager = (ReconNodeManager) cluster.getReconServer()
+        .getReconStorageContainerManager().getScmNodeManager();
+    long keyCountAfter = nodeManager.getNodeDBKeyCount();
+    assertEquals(keyCountAfter, keyCountBefore);


Can we extract this into a common assertion function shared between the HA and non HA variants of this test?

JyotinderSingh · 2022-01-10T19:27:30Z

Thank you for the contribution @aswinshakil! I have left a few minor comments.

smengcl

Thanks @aswinshakil for working on this.

...econ/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java

...rc/main/java/org/apache/hadoop/ozone/recon/spi/impl/StorageContainerServiceProviderImpl.java

Change-Id: I1afb61b93fc0c0f641d52e5d42c586009120c031

@REPLicated

* master: (27 commits) HDDS-6206. Application errors must not flood system log (apache#3001) HDDS-6181. Change SCMHAInvocationHandler#invokeRatis() logging to TRACE (apache#2992) HDDS-5529. For any IOexception from @REPLicated method we should throw it (apache#2788) HDDS-6201. Fix NPE for DataScanner with scanned container deleted by others. (apache#3005) HDDS-6239. ozonesecure-mr failing with No URLs in mirrorlist (apache#3029) HDDS-6227. Test helpers should observe naming conditions (apache#3020) HDDS-6205. Add CLI command to display the latest Replication Manager report (apache#3013) HDDS-6192. feature/Observability.md translated to Chinese (apache#2994) HDDS-6219. Switch to RATIS ReplicationType from STAND_ALONE in our tests. (apache#3014) HDDS-6211. [Docs] Image styling on deployed site does not replicate local builds. (apache#3007) HDDS-6177. Extend container info command to include replica details (apache#2995) HDDS-6128. CLI tool that downloads all the block replicas and creates a manifest file (apache#2987) HDDS-6191. Intermittent failure in TestDeleteWithSlowFollower (apache#3015) HDDS-6216. Move OMOpenKeysDeleteRequest to package om.request.key (apache#3011) HDDS-6203. CleanUp incomplete gz files during Container move (apache#3000) HDDS-3408. Rename ChunkLayOutVersion to ContainerLayoutVersion. (apache#2983) HDDS-6109. Preserve the underlying exception raised in client lib. (apache#2989) HDDS-6135. SCM Container DB bootstrap on Recon startup for SCM HA. (apache#2972) HDDS-6202. Avoid using jmh-generator-annprocess since it is GPL2.0. (apache#2998) HDDS-6163. Fix PATH in docker image (apache#2967) ... Conflicts: hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainerCheck.java

@REPLicated

* master: (27 commits) HDDS-6206. Application errors must not flood system log (apache#3001) HDDS-6181. Change SCMHAInvocationHandler#invokeRatis() logging to TRACE (apache#2992) HDDS-5529. For any IOexception from @REPLicated method we should throw it (apache#2788) HDDS-6201. Fix NPE for DataScanner with scanned container deleted by others. (apache#3005) HDDS-6239. ozonesecure-mr failing with No URLs in mirrorlist (apache#3029) HDDS-6227. Test helpers should observe naming conditions (apache#3020) HDDS-6205. Add CLI command to display the latest Replication Manager report (apache#3013) HDDS-6192. feature/Observability.md translated to Chinese (apache#2994) HDDS-6219. Switch to RATIS ReplicationType from STAND_ALONE in our tests. (apache#3014) HDDS-6211. [Docs] Image styling on deployed site does not replicate local builds. (apache#3007) HDDS-6177. Extend container info command to include replica details (apache#2995) HDDS-6128. CLI tool that downloads all the block replicas and creates a manifest file (apache#2987) HDDS-6191. Intermittent failure in TestDeleteWithSlowFollower (apache#3015) HDDS-6216. Move OMOpenKeysDeleteRequest to package om.request.key (apache#3011) HDDS-6203. CleanUp incomplete gz files during Container move (apache#3000) HDDS-3408. Rename ChunkLayOutVersion to ContainerLayoutVersion. (apache#2983) HDDS-6109. Preserve the underlying exception raised in client lib. (apache#2989) HDDS-6135. SCM Container DB bootstrap on Recon startup for SCM HA. (apache#2972) HDDS-6202. Avoid using jmh-generator-annprocess since it is GPL2.0. (apache#2998) HDDS-6163. Fix PATH in docker image (apache#2967) ...

@REPLicated

…ansport support (#3074) * HDDS-6149. Remove unused keytabs (#2960) * HDDS-6094. Some unit tests are skipped due to using JUnit4 runner (#2909) * HDDS-6075. OzoneConfiguration constructor overrides input configuration keys. (#2921) * HDDS-4177. SCM Container DB bootstrap on Recon startup (#2942) * HDDS-6086. Compute MD5MD5CRC file checksum using chunk checksums from DataNodes (#2919) * HDDS-6148. Validate ContainerBalancerConfiguration before start ContainerBalancer (#2957) * HDDS-6161. SCM StateMachine failing to reinitialize doesn't terminate the process. (#2971) * HDDS-6134. Move replication-specific config to ReplicationServer (#2943) * HDDS-4010. S3G startup fails when multiple service ids are configured. (#2976) * HDDS-6170. Add metrics to replication manager to track container health states (#2975) * HDDS-3231. Cleanup KeyManagerImpl (#2961) * HDDS-5927. Improve defaults in ContainerBalancerConfiguration (#2892) * HDDS-6157. More consistent synchronization in InputStreams (#2965) * HDDS-4743. [FSO] Add FSO variant of ITestOzoneContractDistcp. (#2980) * HDDS-6114. Intermittent error due to Failed to init RocksDB (#2947) * HDDS-6175. Use s3Auth during proxy during decrypt in RpcClient. (#2981) * HDDS-6175. Use s3Auth during proxy during decrypt in RpcClient. * HDDS-5740. Enable ratis by default for SCM. (#2637) * HDDS-6183. Intermittent failure in TestKeyDeletingService.checkIfDeleteServiceWithFailingSCM. (#2991) * HDDS-4190. Intermittent failure in TestOMVolumeSetOwnerRequest and TestOMVolumeSetQuotaRequest. (#2982) * HDDS-6120. Compute block checksum using chunk checksums (#2930) * HDDS-6147. Add ability in OM to get limited delta updates (#2956) * HDDS-6195. Remove unused jmh-core dependency. (#2997) * HDDS-6167. Update ozone-runner version to 20211202-1 (#2969) * HDDS-6171. Create an API to change Bucket Owner (#2988) * HDDS-6163. Fix PATH in docker image (#2967) * HDDS-6202. Avoid using jmh-generator-annprocess since it is GPL2.0. (#2998) * HDDS-6135. SCM Container DB bootstrap on Recon startup for SCM HA. (#2972) * HDDS-6109. Preserve the underlying exception raised in client lib. (#2989) * HDDS-3408. Rename ChunkLayOutVersion to ContainerLayoutVersion. (#2983) * HDDS-6203. CleanUp incomplete gz files during Container move (#3000) * HDDS-6216. Move OMOpenKeysDeleteRequest to package om.request.key (#3011) * HDDS-6191. Intermittent failure in TestDeleteWithSlowFollower (#3015) * HDDS-6128. CLI tool that downloads all the block replicas and creates a manifest file (#2987) * HDDS-6177. Extend container info command to include replica details (#2995) * HDDS-6211. [Docs] Image styling on deployed site does not replicate local builds. (#3007) * HDDS-6219. Switch to RATIS ReplicationType from STAND_ALONE in our tests. (#3014) * HDDS-6192. feature/Observability.md translated to Chinese (#2994) * HDDS-6205. Add CLI command to display the latest Replication Manager report (#3013) * HDDS-6227. Test helpers should observe naming conditions (#3020) * HDDS-6239. ozonesecure-mr failing with No URLs in mirrorlist (#3029) * HDDS-6201. Fix NPE for DataScanner with scanned container deleted by others. (#3005) * HDDS-5529. For any IOexception from @REPLicated method we should throw it (#2788) * HDDS-6181. Change SCMHAInvocationHandler#invokeRatis() logging to TRACE (#2992) * HDDS-6206. Application errors must not flood system log (#3001) * HDDS-6245. Add BucketLayout logging to Audit Logs (#3040) * HDDS-6238 Reduce memory requirements for list keys. (#3032) * HDDS-2919. Intermittent failure in TestRDBStore (#3028) * HDDS-6253. Unnecessary duplicate smoketest after defaulting to FSO (#3036) * HDDS-6204. Cleanup handling malformed authorization header (#2999) * HDDS-6169. Selective checks: skip junit tests on ozone-runner image update (#2974) * HDDS-6270. Use a dedicated file instead of /etc/passwd for xcompat acceptance test (#3050) * HDDS-6273. Amend doc SecuringTDE.md (#3047) * HDDS-6140. Selective checks: skip unit check for integration-test changes (#2948) * HDDS-6215. Recon get limited delta updates from OM (#3009) * HDDS-6125. Recon get limited delta updates from OM * HDDS-6215. Fix unit test * trigger new CI check * HDDS-6215. Fix typo * trigger new CI check Co-authored-by: Symious <[email protected]> * HDDS-6226. Run tests for selective CI checks in CI (#3019) * HDDS-6247. Avoid logging stack trace for user input problems (#3039) * HDDS-6208. New checkstyle: WhitespaceAround (#3003) * HDDS-6289. Upgrade acceptance test log flooded with parse error (#3063) Co-authored-by: Siyao Meng <[email protected]> * Trigger Build * Fix integration test for added configuation field for selecting OmTransport for s3 gateway - TestOzoneConfigurationFields (added config key not in xml). Co-authored-by: Doroszlai, Attila <[email protected]> Co-authored-by: Lokesh Jain <[email protected]> Co-authored-by: Aswin Shakil Balasubramanian <[email protected]> Co-authored-by: Wei-Chiu Chuang <[email protected]> Co-authored-by: Symious <[email protected]> Co-authored-by: Bharat Viswanadham <[email protected]> Co-authored-by: Stephen O'Donnell <[email protected]> Co-authored-by: GeorgeJahad <[email protected]> Co-authored-by: Siddhant Sangwan <[email protected]> Co-authored-by: Jyotinder Singh <[email protected]> Co-authored-by: Shashikant Banerjee <[email protected]> Co-authored-by: Tsz-Wo Nicholas Sze <[email protected]> Co-authored-by: Kiyoshi Mizumaru <[email protected]> Co-authored-by: Ritesh H Shukla <[email protected]> Co-authored-by: Nibiru <[email protected]> Co-authored-by: Kaijie Chen <[email protected]> Co-authored-by: Zita Dombi <[email protected]> Co-authored-by: Istvan Fajth <[email protected]> Co-authored-by: steinsgateted <[email protected]> Co-authored-by: Gui Hecheng <[email protected]> Co-authored-by: Jackson Yao <[email protected]> Co-authored-by: Keyi Song <[email protected]> Co-authored-by: UENISHI Kota <[email protected]> Co-authored-by: Siyao Meng <[email protected]> Co-authored-by: Symious <[email protected]>

HDDS-6135. SCM Container DB bootstrap on Recon startup for SCM HA.

b1c7095

aswinshakil mentioned this pull request Jan 10, 2022

HDDS-6135. SCM Container DB bootstrap on Recon startup for SCM HA #2970

Closed

JyotinderSingh reviewed Jan 10, 2022

View reviewed changes

swagle requested a review from smengcl January 11, 2022 19:06

smengcl reviewed Jan 18, 2022

View reviewed changes

...econ/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java Outdated Show resolved Hide resolved

...rc/main/java/org/apache/hadoop/ozone/recon/spi/impl/StorageContainerServiceProviderImpl.java Outdated Show resolved Hide resolved

aswinshakil added 4 commits January 18, 2022 15:58

HDDS-6135. Refactored code and tests.

f1091d3

HDDS-6135. Fixed checkstyle.

d9f6489

Trigger CI

1fb7574

Fixed tests

d188eb0

smengcl reviewed Jan 19, 2022

View reviewed changes

...rc/main/java/org/apache/hadoop/ozone/recon/spi/impl/StorageContainerServiceProviderImpl.java Show resolved Hide resolved

smengcl reviewed Jan 19, 2022

View reviewed changes

...rc/main/java/org/apache/hadoop/ozone/recon/spi/impl/StorageContainerServiceProviderImpl.java Outdated Show resolved Hide resolved

Fixed spacing and logging

534637b

smengcl approved these changes Jan 19, 2022

View reviewed changes

Retrigger CI

9ebc777

Change-Id: I1afb61b93fc0c0f641d52e5d42c586009120c031

smengcl merged commit 05773af into apache:master Jan 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDDS-6135. SCM Container DB bootstrap on Recon startup for SCM HA. #2972

HDDS-6135. SCM Container DB bootstrap on Recon startup for SCM HA. #2972

Uh oh!

aswinshakil commented Jan 10, 2022

Uh oh!

JyotinderSingh Jan 10, 2022

Uh oh!

JyotinderSingh Jan 10, 2022

Uh oh!

JyotinderSingh commented Jan 10, 2022

Uh oh!

smengcl left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HDDS-6135. SCM Container DB bootstrap on Recon startup for SCM HA. #2972

HDDS-6135. SCM Container DB bootstrap on Recon startup for SCM HA. #2972

Uh oh!

Conversation

aswinshakil commented Jan 10, 2022

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

JyotinderSingh Jan 10, 2022

Choose a reason for hiding this comment

Uh oh!

JyotinderSingh Jan 10, 2022

Choose a reason for hiding this comment

Uh oh!

JyotinderSingh commented Jan 10, 2022

Uh oh!

smengcl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants