HDDS-4177. SCM Container DB bootstrap on Recon startup #2942

aswinshakil · 2021-12-22T21:13:22Z

What changes were proposed in this pull request?

On Recon startup, get the latest SCM DB (if needed) and bootstrap the container data.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-4177

How was this patch tested?

This patch was manually tested.

avijayanhwx

Thanks for working on this @aswinshakil.

Can we add an integration test for this flow?

avijayanhwx · 2022-01-04T19:36:43Z

...mmon/src/main/java/org/apache/hadoop/hdds/scm/protocol/StorageContainerLocationProtocol.java

   */
  Token<?> getContainerToken(ContainerID containerID) throws IOException;
+
+  long getSCMContainersCount() throws IOException;


nit. getSCMContainersCount --> getContainerCount

avijayanhwx · 2022-01-04T19:37:38Z

hadoop-hdds/interface-admin/src/main/proto/ScmAdminProtocol.proto

  optional ContainerBalancerStatusRequestProto containerBalancerStatusRequest = 35;
  optional FinalizeScmUpgradeRequestProto finalizeScmUpgradeRequest = 36;
  optional QueryUpgradeFinalizationProgressRequestProto queryUpgradeFinalizationProgressRequest = 37;
+  optional GetSCMContainersCountRequestProto getSCMContainersCountRequest = 38;


nit. GetSCMContainersCountRequestProto --> GetContainerCountRequestProto

avijayanhwx · 2022-01-04T19:41:02Z

...econ/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java

+
+      if(Math.abs(scmContainersCount - reconContainerCount) > threshold) {
+        updateReconSCMDBWithNewSnapshot();
+        LOG.info("Update Recon DB with SCM DB");


nit. Log container counts here?

Sure, I'll add it here

avijayanhwx · 2022-01-04T19:43:21Z

...econ/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java

+        KeyValue<UUID, DatanodeDetails> keyValue = iterator.next();
+        newNodeTable.put(keyValue.getKey(), keyValue.getValue());
+      }
+      deleteOldSCMDB();


The deletion of the old SCM DB should be the last step, after we have successfully re-init the different SCM managers.

deleteOldSCMDB() is added here because in the next step we set the new DB to the dbStore. If we keep it as the last step we won't have the reference to the old DB.

...e/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/StorageContainerServiceProvider.java

avijayanhwx · 2022-01-04T19:47:22Z

...rc/main/java/org/apache/hadoop/ozone/recon/spi/impl/StorageContainerServiceProviderImpl.java

+
+    if (policy.isHttpsEnabled()) {
+      scmDBSnapshotUrl = "https://" + scmHttpsAddress +
+              OZONE_OM_DB_CHECKPOINT_HTTP_ENDPOINT;


nit. Rename OZONE_OM_DB_CHECKPOINT_HTTP_ENDPOINT to OZONE_DB_CHECKPOINT_HTTP_ENDPOINT?

Will change it.

aswinshakil · 2022-01-04T20:36:38Z

Thank you for the review @avijayanhwx . I will update the PR with the suggested changes. For the integration test, I was planning to add it as a part of SCM HA enabled cluster, i.e HDDS-6135.

avijayanhwx

LGTM +1.

avijayanhwx · 2022-01-07T19:11:24Z

...econ/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java

  private void initializeSCMDB() {
    try {
-      long scmContainersCount = scmServiceProvider.getSCMContainersCount();
+      long scmContainersCount = scmServiceProvider.getContainerCount();


We should decide on a defined behavior if SCM is down when Recon starts up.

@REPLicated

…ansport support (#3074) * HDDS-6149. Remove unused keytabs (#2960) * HDDS-6094. Some unit tests are skipped due to using JUnit4 runner (#2909) * HDDS-6075. OzoneConfiguration constructor overrides input configuration keys. (#2921) * HDDS-4177. SCM Container DB bootstrap on Recon startup (#2942) * HDDS-6086. Compute MD5MD5CRC file checksum using chunk checksums from DataNodes (#2919) * HDDS-6148. Validate ContainerBalancerConfiguration before start ContainerBalancer (#2957) * HDDS-6161. SCM StateMachine failing to reinitialize doesn't terminate the process. (#2971) * HDDS-6134. Move replication-specific config to ReplicationServer (#2943) * HDDS-4010. S3G startup fails when multiple service ids are configured. (#2976) * HDDS-6170. Add metrics to replication manager to track container health states (#2975) * HDDS-3231. Cleanup KeyManagerImpl (#2961) * HDDS-5927. Improve defaults in ContainerBalancerConfiguration (#2892) * HDDS-6157. More consistent synchronization in InputStreams (#2965) * HDDS-4743. [FSO] Add FSO variant of ITestOzoneContractDistcp. (#2980) * HDDS-6114. Intermittent error due to Failed to init RocksDB (#2947) * HDDS-6175. Use s3Auth during proxy during decrypt in RpcClient. (#2981) * HDDS-6175. Use s3Auth during proxy during decrypt in RpcClient. * HDDS-5740. Enable ratis by default for SCM. (#2637) * HDDS-6183. Intermittent failure in TestKeyDeletingService.checkIfDeleteServiceWithFailingSCM. (#2991) * HDDS-4190. Intermittent failure in TestOMVolumeSetOwnerRequest and TestOMVolumeSetQuotaRequest. (#2982) * HDDS-6120. Compute block checksum using chunk checksums (#2930) * HDDS-6147. Add ability in OM to get limited delta updates (#2956) * HDDS-6195. Remove unused jmh-core dependency. (#2997) * HDDS-6167. Update ozone-runner version to 20211202-1 (#2969) * HDDS-6171. Create an API to change Bucket Owner (#2988) * HDDS-6163. Fix PATH in docker image (#2967) * HDDS-6202. Avoid using jmh-generator-annprocess since it is GPL2.0. (#2998) * HDDS-6135. SCM Container DB bootstrap on Recon startup for SCM HA. (#2972) * HDDS-6109. Preserve the underlying exception raised in client lib. (#2989) * HDDS-3408. Rename ChunkLayOutVersion to ContainerLayoutVersion. (#2983) * HDDS-6203. CleanUp incomplete gz files during Container move (#3000) * HDDS-6216. Move OMOpenKeysDeleteRequest to package om.request.key (#3011) * HDDS-6191. Intermittent failure in TestDeleteWithSlowFollower (#3015) * HDDS-6128. CLI tool that downloads all the block replicas and creates a manifest file (#2987) * HDDS-6177. Extend container info command to include replica details (#2995) * HDDS-6211. [Docs] Image styling on deployed site does not replicate local builds. (#3007) * HDDS-6219. Switch to RATIS ReplicationType from STAND_ALONE in our tests. (#3014) * HDDS-6192. feature/Observability.md translated to Chinese (#2994) * HDDS-6205. Add CLI command to display the latest Replication Manager report (#3013) * HDDS-6227. Test helpers should observe naming conditions (#3020) * HDDS-6239. ozonesecure-mr failing with No URLs in mirrorlist (#3029) * HDDS-6201. Fix NPE for DataScanner with scanned container deleted by others. (#3005) * HDDS-5529. For any IOexception from @REPLicated method we should throw it (#2788) * HDDS-6181. Change SCMHAInvocationHandler#invokeRatis() logging to TRACE (#2992) * HDDS-6206. Application errors must not flood system log (#3001) * HDDS-6245. Add BucketLayout logging to Audit Logs (#3040) * HDDS-6238 Reduce memory requirements for list keys. (#3032) * HDDS-2919. Intermittent failure in TestRDBStore (#3028) * HDDS-6253. Unnecessary duplicate smoketest after defaulting to FSO (#3036) * HDDS-6204. Cleanup handling malformed authorization header (#2999) * HDDS-6169. Selective checks: skip junit tests on ozone-runner image update (#2974) * HDDS-6270. Use a dedicated file instead of /etc/passwd for xcompat acceptance test (#3050) * HDDS-6273. Amend doc SecuringTDE.md (#3047) * HDDS-6140. Selective checks: skip unit check for integration-test changes (#2948) * HDDS-6215. Recon get limited delta updates from OM (#3009) * HDDS-6125. Recon get limited delta updates from OM * HDDS-6215. Fix unit test * trigger new CI check * HDDS-6215. Fix typo * trigger new CI check Co-authored-by: Symious <[email protected]> * HDDS-6226. Run tests for selective CI checks in CI (#3019) * HDDS-6247. Avoid logging stack trace for user input problems (#3039) * HDDS-6208. New checkstyle: WhitespaceAround (#3003) * HDDS-6289. Upgrade acceptance test log flooded with parse error (#3063) Co-authored-by: Siyao Meng <[email protected]> * Trigger Build * Fix integration test for added configuation field for selecting OmTransport for s3 gateway - TestOzoneConfigurationFields (added config key not in xml). Co-authored-by: Doroszlai, Attila <[email protected]> Co-authored-by: Lokesh Jain <[email protected]> Co-authored-by: Aswin Shakil Balasubramanian <[email protected]> Co-authored-by: Wei-Chiu Chuang <[email protected]> Co-authored-by: Symious <[email protected]> Co-authored-by: Bharat Viswanadham <[email protected]> Co-authored-by: Stephen O'Donnell <[email protected]> Co-authored-by: GeorgeJahad <[email protected]> Co-authored-by: Siddhant Sangwan <[email protected]> Co-authored-by: Jyotinder Singh <[email protected]> Co-authored-by: Shashikant Banerjee <[email protected]> Co-authored-by: Tsz-Wo Nicholas Sze <[email protected]> Co-authored-by: Kiyoshi Mizumaru <[email protected]> Co-authored-by: Ritesh H Shukla <[email protected]> Co-authored-by: Nibiru <[email protected]> Co-authored-by: Kaijie Chen <[email protected]> Co-authored-by: Zita Dombi <[email protected]> Co-authored-by: Istvan Fajth <[email protected]> Co-authored-by: steinsgateted <[email protected]> Co-authored-by: Gui Hecheng <[email protected]> Co-authored-by: Jackson Yao <[email protected]> Co-authored-by: Keyi Song <[email protected]> Co-authored-by: UENISHI Kota <[email protected]> Co-authored-by: Siyao Meng <[email protected]> Co-authored-by: Symious <[email protected]>

aswinshakil added 3 commits December 21, 2021 16:57

HDDS-4177. SCM Container DB bootstrap on Recon startup.

a89fe0e

HDDS-4177. Fixed Test.

f57bd16

Trigger Build

5070465

avijayanhwx reviewed Jan 4, 2022

View reviewed changes

HDDS-4177. Added suggestions.

b227fb8

avijayanhwx approved these changes Jan 7, 2022

View reviewed changes

avijayanhwx merged commit ca25193 into apache:master Jan 7, 2022

HDDS-4177. SCM Container DB bootstrap on Recon startup #2942

HDDS-4177. SCM Container DB bootstrap on Recon startup #2942

Uh oh!

Conversation

aswinshakil commented Dec 22, 2021

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

avijayanhwx left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aswinshakil commented Jan 4, 2022

Uh oh!

avijayanhwx left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants