HDDS-12824. Optimize container checksum read during datanode startup #8604

aswinshakil · 2025-06-11T14:38:04Z

What changes were proposed in this pull request?

In #8204 , during data node startup, we read the <containerID>.tree to get the container checksum, which is used to update the in-memory container data. We want to store this dataChecksum in RocksDB, because in a huge cluster, deserializing <containerID>.tree for every container on startup may have an impact. In this patch

We store the dataChecksum in RocksDB so that we don't have to deserialize the .tree file every time during DN restart.
Fixed packing and unpacking of .tree file during container import/export.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-12824

How was this patch tested?

Updated existing tests

errose28

Thanks for fixing this @aswinshakil. I have done a partial review, have not looked at the import/export changes yet.

...e/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerChecksumTreeManager.java

...tainer-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java

.../src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/KeyValueContainerUtil.java

...c/test/java/org/apache/hadoop/ozone/container/checksum/TestContainerChecksumTreeManager.java

...org/apache/hadoop/ozone/container/keyvalue/TestContainerReconciliationWithMockDatanodes.java

errose28

Thanks for the updates @aswinshakil. I reviewed the import/export code as well this round, so there are some more comments in that area.

...tainer-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java

...ner-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/TarContainerPacker.java

.../src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/KeyValueContainerUtil.java

...e/src/test/java/org/apache/hadoop/ozone/container/checksum/ContainerMerkleTreeTestUtils.java

...t/java/org/apache/hadoop/ozone/dn/scanner/TestBackgroundContainerDataScannerIntegration.java

...est/java/org/apache/hadoop/ozone/dn/scanner/TestOnDemandContainerDataScannerIntegration.java

...service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestTarContainerPacker.java

errose28

Thanks for the fixes @aswinshakil. Looks like we should wait until #8565 is merged before proceeding with this one since they overlap in some areas. We should update TestContainerReader or somewhere similar for tests of loading containers when the tree and checksums may or may not exist.

...e/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerChecksumTreeManager.java

.../src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/KeyValueContainerUtil.java

errose28

Left a few more comments mostly based on non-existent tree handling after #8565.

.../src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/KeyValueContainerUtil.java

...r-service/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestContainerReader.java

...service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestTarContainerPacker.java

errose28 · 2025-07-29T17:44:24Z

Test failure was not related to this change. It is being fixed in #8876

errose28

Thanks for the updates. A few more comments, and this one still needs to be addressed.

...r-service/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestContainerReader.java

.../src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/KeyValueContainerUtil.java

...-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainer.java

aswinshakil · 2025-07-30T20:01:49Z

@errose28 testContainerLoadingWithNoChecksumAnywhere and testContainerLoadingWithoutMerkleTree address the comment mentioned above.

errose28 · 2025-07-31T17:53:00Z

@errose28 testContainerLoadingWithNoChecksumAnywhere and testContainerLoadingWithoutMerkleTree address the #8604 (comment).

These tests are in TestContainerReader for startup testing, but that comment is on TestTarContainerPacker for import/export. We are testing that the tree file made it through import/export, but we are not checking that the DB and in-memory ContainerData got updated with its checksum at the destination.

errose28

LGTM

* master: (55 commits) HDDS-13525. Rename configuration property to ozone.om.compaction.service.enabled (apache#8928) HDDS-13519. Reconciliation should continue if a peer datanode is unreachable (apache#8908) HDDS-13566. Fix incorrect authorizer class in ACL documentation (apache#8931) HDDS-13084. Trigger on-demand container scan when a container moves from open to unhealthy. (apache#8904) HDDS-13432. Accelerating Namespace Usage Calculation in Recon using - Materialised Approach (apache#8797) HDDS-13557. Bump jline to 3.30.5 (apache#8920) HDDS-13556. Bump assertj-core to 3.27.4 (apache#8919) HDDS-13543. [Docs] Design doc for OM bootstrapping process with snapshots. (apache#8900) HDDS-13541. Bump sonar-maven-plugin to 5.1.0.4751 (apache#8911) HDDS-13101. Remove duplicate information in datanode list output (apache#8523) HDDS-13528. Handle null paths when the NSSummary is initializing (apache#8901) HDDS-12990. (addendum) Generate tree from metadata when it does not exist during getContainerChecksumInfo call (apache#8881) HDDS-13086. Block duplicate reconciliation requests for the same container and datanode within the datanode. (apache#8905) HDDS-12990. Generate tree from metadata when it doesn't exist during getContainerChecksumInfo call (apache#8881) HDDS-12824. Optimize container checksum read during datanode startup (apache#8604) HDDS-13522. Rename axisLabel for No. of delete request received (apache#8879) HDDS-12196. Document ozone repair cli (apache#8849) HDDS-13514. Intermittent failure in TestNSSummaryMemoryLeak (apache#8889) HDDS-13423. Log reason for triggering on-demand container scan (apache#8854) HDDS-13466. Disable flaky TestOmSnapshotFsoWithNativeLibWithLinkedBuckets ...

HDDS-12824. Optimize container checksum read during datanode startup

923ca71

aswinshakil self-assigned this Jun 11, 2025

aswinshakil added the container-reconciliation label Jun 11, 2025

aswinshakil requested review from errose28 and kerneltime June 11, 2025 14:38

errose28 reviewed Jun 12, 2025

View reviewed changes

Address review comments.

93d396a

aswinshakil changed the base branch from HDDS-10239-container-reconciliation to master June 26, 2025 07:12

Merge branch 'master' of https://github.com/apache/ozone into HDDS-12824

1893f77

errose28 reviewed Jun 26, 2025

View reviewed changes

aswinshakil added 2 commits July 7, 2025 17:04

Address review comments.

2950120

Fix test.

f0a7ab3

aswinshakil force-pushed the HDDS-12824 branch from 4cb42fa to f0a7ab3 Compare July 8, 2025 17:36

Update TempDir reference.

483cee5

errose28 reviewed Jul 22, 2025

View reviewed changes

...e/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerChecksumTreeManager.java Outdated Show resolved Hide resolved

.../src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/KeyValueContainerUtil.java Show resolved Hide resolved

aswinshakil added 2 commits July 23, 2025 12:25

Merge branch 'master' of https://github.com/apache/ozone into HDDS-12824

6cc42a0

Add new tests and address review comments

5c0b0dc

errose28 reviewed Jul 26, 2025

View reviewed changes

Address review comments.

7811e5f

errose28 reviewed Jul 29, 2025

View reviewed changes

Address review comments.

3f9cea6

aswinshakil added 2 commits July 31, 2025 12:03

Address review comments.

e104acf

Modify Rocks DB open for V1/V2 and Update TestKeyValueContainer tests.

ff568fd

errose28 approved these changes Aug 6, 2025

View reviewed changes

errose28 merged commit 5e03af1 into apache:master Aug 6, 2025
42 checks passed

errose28 mentioned this pull request Aug 7, 2025

HDDS-12990. Generate tree from metadata when it doesn't exist during getContainerChecksumInfo call #8881

Merged

HDDS-12824. Optimize container checksum read during datanode startup #8604

HDDS-12824. Optimize container checksum read during datanode startup #8604

Uh oh!

Conversation

aswinshakil commented Jun 11, 2025

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

errose28 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

errose28 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

errose28 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

errose28 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

errose28 commented Jul 29, 2025

Uh oh!

errose28 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aswinshakil commented Jul 30, 2025

Uh oh!

errose28 commented Jul 31, 2025

Uh oh!

errose28 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants