HDDS-13023. Container checksum is missing after container import #8459

Gargi-jais11 · 2025-05-15T03:08:22Z

What changes were proposed in this pull request?

After the changes introduced in PR#7934, DiskBalancer is failing to move containers due to a checksum mismatch error during container import.

The root cause is a regression introduced in the KeyValueContainer#importContainerData method. The method was modified to call KeyValueContainerUtil.parseKVContainerData(...) before setting the container state and metadata (which includes the expected checksum).

Since the checksum is not yet populated in containerData at the time of parsing, the call to ContainerUtils.verifyChecksum(...) inside parseKVContainerData fails with the following error:

2025-04-10 15:58:41 2025-04-10 10:28:41,115 [DiskBalancerService#0] WARN diskbalancer.DiskBalancerService: Failed to move container KeyValueContainerData #8 (CLOSED, non-empty, ri=0, origin=[dn_b916c228-6ce7-4f89-8de3-b29b1d967af1, pipeline_efeea169-e8f7-43fb-9003-270fa058adb2]): {} 
2025-04-10 15:58:41 org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException: Container checksum error for ContainerID: 8.
2025-04-10 15:58:41 Stored Checksum:
2025-04-10 15:58:41 Expected Checksum: 7a0ec671d9f43c8d6dd303961776f383ce767bfa2e365de7119d1c6c7d2c1359
2025-04-10 15:58:41 at org.apache.hadoop.ozone.container.common.helpers.ContainerUtils.verifyChecksum(ContainerUtils.java:214)
2025-04-10 15:58:41 at org.apache.hadoop.ozone.container.keyvalue.helpers.KeyValueContainerUtil.parseKVContainerData(KeyValueContainerUtil.java:210)
2025-04-10 15:58:41 at org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.importContainerData(KeyValueContainer.java:683)
2025-04-10 15:58:41 at org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.importContainerData(KeyValueContainer.java:708)

Proposed fix :
To do checksum verification before the state changes to RECOVERING during unpackContainerTar and not after full containerImport.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13023

How was this patch tested?

Passed the existing unit test.

As well as updated the unit test TestKeyValueContainer#testContainerImportExport to also check whether checksum is not null and not equal to DUMMY_CHECK after container import.

swamirishi

Thanks for identifying the issue @Gargi-jais11 Left some review comments inline

...ainer-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerData.java

...iner-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainer.java

.../src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/KeyValueContainerUtil.java

ChenSammi · 2025-06-17T08:05:35Z

...service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestTarContainerPacker.java


  private static final String TEST_DESCRIPTOR_FILE_CONTENT = "!<KeyValueContainerData>\n" +
-      "checksum: 2215d39f2ae1de89fec837d18dc6387d8cba22fb5943cf4616f80c4b34e2edfe\n" +
+      "checksum: 5e4bea7286f96d88a5b3a745011ff9e4281a5221bfe564413215cd85871dcfd8\n" +


Why update this checksum?

The checksum change was necessary because the point of checksum verification shifted. The checksum embedded in TEST_DESCRIPTOR_FILE_CONTENT must now accurately reflect the content of that string before any state changes occur during the unpackContainerData process. So the new checksum 5e4bea... correctly corresponds to the TEST_DESCRIPTOR_FILE_CONTENT which includes state: CLOSED.

.../src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/KeyValueContainerUtil.java

… container import

ChenSammi · 2025-06-18T12:00:15Z

@swamirishi , the checksum verification of container import is improved in a way we discussed in the meeting. Do you want to take another look?

ChenSammi

The last patch LGTM. Thanks @Gargi-jais11 for the contribution and @swamirishi for the review.

…239-container-reconciliation Commits: 62 da53b5b HDDS-13299. Fix failures related to delete (apache#8665) 8c1b439 HDDS-13296. Integration check always passes due to missing output (apache#8662) 7329859 HDDS-13023. Container checksum is missing after container import (apache#8459) a0af93e HDDS-13292. Change `<? extends KeyValue>` to `<KeyValue>` in test (apache#8657) f3050cf HDDS-13276. Use KEY_ONLY/VALUE_ONLY iterator in SCM/Datanode. (apache#8638) e9c0a45 HDDS-13262. Simplify key name validation (apache#8619) f713e57 HDDS-12482. Avoid using CommonConfigurationKeys (apache#8647) b574709 HDDS-12924. datanode used space calculation optimization (apache#8365) de683aa HDDS-13263. Refactor DB Checkpoint Utilities. (apache#8620) 97262aa HDDS-13256. Updated OM Snapshot Grafana Dashboard to reflect metric updates from HDDS-13181. (apache#8639) 9d2b415 HDDS-13234. Expired secret key can abort leader OM startup. (apache#8601) d9049a2 HDDS-13220. Change Recon 'Negative usedBytes' message loglevel to DEBUG (apache#8648) 6df3077 HDDS-9223. Use protobuf for SnapshotDiffJobCodec (apache#8503) a7fc290 HDDS-13236. Change Table methods not to throw IOException. (apache#8645) 9958f5b HDDS-13287. Upgrade commons-beanutils to 1.11.0 due to CVE-2025-48734 (apache#8646) 48aefea HDDS-13277. [Docs] Native C/C++ Ozone clients (apache#8630) 052d912 HDDS-13037. Let container create command support STANDALONE , RATIS and EC containers (apache#8559) 90ed60b HDDS-13279. Skip verifying Apache Ranger binaries in CI (apache#8633) 9bc53b2 HDDS-11513. All deletion configurations should be configurable without restart (apache#8003) ac511ac HDDS-13259. Deletion Progress - Grafana Dashboard (apache#8617) 3370f42 HDDS-13246. Change `<? extend KeyValue>` to `<KeyValue>` in hadoop-hdds (apache#8631) 7af8c44 HDDS-11454. Ranger integration for Docker Compose environment (apache#8575) 5a3e4e7 HDDS-13273. Bump awssdk to 2.31.63 (apache#8626) 77138b8 HDDS-13254. Change table iterator to optionally read key or value. (apache#8621) ce288b6 HDDS-13265. Simplify the page Access Ozone using HTTPFS REST API (apache#8629) 36fe888 HDDS-13275. Improve CheckNative implementation (apache#8628) d38484e HDDS-13274. Bump sqlite-jdbc to 3.50.1.0 (apache#8627) 3f3ec43 HDDS-13266. `ozone debug checknative` to show OpenSSL lib (apache#8623) 8983a63 HDDS-13272. Bump junit to 5.13.1 (apache#8625) a927113 HDDS-13271. [Docs] Minor text updates, reference links. (apache#8624) 7e77058 HDDS-13112. [Docs] OM Bootstrap can also happen when follower falls behind too much. (apache#8600) fd13300 HDDS-10775. Support bucket ownership verification (apache#8558) 3ecf345 HDDS-13207. [Docs] Third party systems compatible with Ozone S3. (apache#8584) ad5a507 HDDS-13035. SnapshotDeletingService should hold write locks while purging deleted snapshots (apache#8554) 38a9186 HDDS-12637. Increase max buffer size for tar entry read/write (apache#8618) f31c264 HDDS-13045. Implement Immediate Triggering of Heartbeat when Volume Full (apache#8590) 0701d6a HDDS-13248. Remove `ozone debug replicas verify` option --output-dir (apache#8612) ca1afe8 HDDS-13257. Remove separate split for shell integration tests (apache#8616) 5d6fe94 HDDS-13216. Standardize Container[Replica]NotFoundException messages (apache#8599) 1e47217 HDDS-13168. Fix error response format in CheckUploadContentTypeFilter (apache#8614) 6d4d423 HDDS-13181. Added metrics for internal Snapshot Operations. (apache#8606) 4a461b2 HDDS-10490. Intermittent NPE in TestSnapshotDiffManager#testLoadJobsOnStartUp (apache#8596) bf29f7f HDDS-13235. The equals/hashCode methods in anonymous KeyValue classes may not work. (apache#8607) 6ff3ad6 HDDS-12873. Improve ContainerData statistics synchronization. (apache#8305) 09d3b27 HDDS-13244. TestSnapshotDeletingServiceIntegrationTest should close snapshots after deleting them (apache#8611) 931bc2d HDDS-13243. copy-rename-maven-plugin version is missing (apache#8605) 3b5985c HDDS-13244. Disable TestSnapshotDeletingServiceIntegrationTest 6bf009c HDDS-12927. metrics and log to indicate datanode crossing disk limits (apache#8573) 752da2b HDDS-12760. Intermittent Timeout in testImportedContainerIsClosed (apache#8349) 8c32363 HDDS-13050. Update StartFromDockerHub.md. (apache#8586) ba1887c HDDS-13241. Fix some potential resource leaks (apache#8602) bbaf71e HDDS-13130. Rename all instances of Disk Usage to Namespace usage (apache#8571) 0628386 HDDS-13142. Correct SCMPerformanceMetrics for delete operation. (apache#8592) 516bc96 HDDS-13148. [Docs] Update Transparent Data Encryption doc. (apache#8530) 5787135 HDDS-13229. [Doc] Fix incorrect CLI argument order in OM upgrade docs (apache#8598) ba95074 HDDS-13107. Support limiting output of `ozone admin datanode list` (apache#8595) e7f5544 HDDS-13171. Replace pipelineID if nodes are changed (apache#8562) 3c9d4d8 HDDS-13103. Correct transaction metrics in SCMBlockDeletingService. (apache#8516) f62eb8a HDDS-13160. Remove SnapshotDirectoryCleaningService and refactor AbstractDeletingService (apache#8547) b46e6b2 HDDS-13150. Fixed SnapshotLimitCheck when failures occur. (apache#8532) 203c1d3 HDDS-13206. Update documentation for Apache Ranger (apache#8583) 2072ef0 HDDS-13214. populate-cache fails due to unused dependency (apache#8594) Conflicts: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerData.java hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainer.java hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/KeyValueContainerUtil.java hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/statemachine/background/BlockDeletingTask.java

Gargi-jais11 marked this pull request as ready for review May 15, 2025 03:50

adoroszlai requested review from sumitagrawl and swamirishi May 15, 2025 09:55

swamirishi requested changes May 15, 2025

View reviewed changes

...ainer-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerData.java Show resolved Hide resolved

...iner-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainer.java Outdated Show resolved Hide resolved

swamirishi requested a review from errose28 May 15, 2025 18:44

Gargi-jais11 force-pushed the HDDS-13023 branch from 46ea852 to 951620c Compare June 4, 2025 06:29

Gargi-jais11 requested review from ChenSammi and swamirishi June 4, 2025 10:31

ChenSammi reviewed Jun 17, 2025

View reviewed changes

.../src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/KeyValueContainerUtil.java Outdated Show resolved Hide resolved

ChenSammi reviewed Jun 17, 2025

View reviewed changes

.../src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/KeyValueContainerUtil.java Show resolved Hide resolved

Gargi-jais11 force-pushed the HDDS-13023 branch from fc29ef0 to 3bb4d7e Compare June 18, 2025 05:34

Gargi Jaiswal added 4 commits June 18, 2025 11:06

update importContainerData logic to resolve checksum missing

9eb945d

verify checksum before state changes to recovering and skipping after…

2ac4848

… container import

fixed TestTarContainerPacker failure

a25cde5

keep the current parseKVContainerData for most existing code call

cf5751c

Gargi-jais11 force-pushed the HDDS-13023 branch from 3bb4d7e to cf5751c Compare June 18, 2025 05:37

refactor alignment

9bb5c5f

Gargi-jais11 requested a review from ChenSammi June 18, 2025 06:58

ChenSammi approved these changes Jun 20, 2025

View reviewed changes

ChenSammi merged commit 7329859 into apache:master Jun 20, 2025
79 of 80 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDDS-13023. Container checksum is missing after container import #8459

HDDS-13023. Container checksum is missing after container import #8459

Uh oh!

Gargi-jais11 commented May 15, 2025 •

edited

Loading

Uh oh!

swamirishi left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ChenSammi Jun 17, 2025

Uh oh!

Gargi-jais11 Jun 18, 2025

Uh oh!

Uh oh!

ChenSammi commented Jun 18, 2025

Uh oh!

ChenSammi left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

HDDS-13023. Container checksum is missing after container import #8459

HDDS-13023. Container checksum is missing after container import #8459

Uh oh!

Conversation

Gargi-jais11 commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

swamirishi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ChenSammi Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

Gargi-jais11 Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ChenSammi commented Jun 18, 2025

Uh oh!

ChenSammi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Gargi-jais11 commented May 15, 2025 •

edited

Loading