Skip to content

HDDS-12397. Persist putBlock for closed container. #7943

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
kerneltime merged 8 commits into from
Mar 6, 2025
Merged

HDDS-12397. Persist putBlock for closed container. #7943

kerneltime merged 8 commits into from
Mar 6, 2025

Conversation

@aswinshakil
Copy link
Member

@aswinshakil aswinshakil commented Feb 21, 2025 •
edited
Loading

What changes were proposed in this pull request?

The putBlock call for OPEN container cannot be used for the CLOSED container and needs some modification to update bscId and remove changes that are only applicable to OPEN container such a pending put block cache, hsync support.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-12397

How was this patch tested?

Added Unit Test.

@aswinshakil aswinshakil self-assigned this Feb 21, 2025
errose28
errose28 reviewed Feb 25, 2025
View reviewed changes
...-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/BlockManagerImpl.java
}

@Override
public long putBlockForClosedContainer(Container container, BlockData data, boolean overwriteBcsId)
Copy link
Contributor

@errose28 errose28 Feb 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need a flag to overwrite the BCSID? Shouldn't it always be an automatic greater-than check?

hemantk-12
hemantk-12 reviewed Feb 27, 2025
View reviewed changes
...-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/impl/BlockManagerImpl.java
}

@Override
public long putBlockForClosedContainer(Container container, BlockData data, boolean overwriteBcsId)
Copy link
Contributor

@hemantk-12 hemantk-12 Feb 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: there are lots of warnings due to raw use Container. Maybe we change BlockManager to type T something like this BlockManager<T extends ContainerData>. It could be a separate PR if we agree.

Copy link
Member Author

@aswinshakil aswinshakil Mar 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. We can do that in the main branch as an improvement.

errose28
errose28 reviewed Mar 3, 2025
View reviewed changes
Copy link
Contributor

@errose28 errose28 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the updates @aswinshakil

errose28
errose28 approved these changes Mar 6, 2025
View reviewed changes
Copy link
Contributor

@errose28 errose28 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the continued work on this @aswinshakil

@kerneltime kerneltime merged commit be53ed7 into apache:HDDS-10239-container-reconciliation Mar 6, 2025
43 checks passed
@aswinshakil aswinshakil mentioned this pull request Mar 10, 2025
Merged
errose28 added a commit to errose28/ozone that referenced this pull request Apr 11, 2025
@errose28
Merge branch 'HDDS-10239-container-reconciliation' into HDDS-10374-sc…
15d6848 
…anner-builds-mt

* HDDS-10239-container-reconciliation: (646 commits)
  HDDS-11763. Implement container repair logic within datanodes. (apache#7474)
  HDDS-11887. Recon - Identify container replicas difference based on content checksums (apache#7942)
  HDDS-12397. Persist putBlock for closed container. (apache#7943)
  HDDS-12361. Mark testGetSnapshotDiffReportJob as flaky
  HDDS-12375. Random object created and used only once (apache#7933)
  HDDS-12335. Fix ozone admin namespace summary to give complete output (apache#7908)
  HDDS-12364. Require Override annotation for overridden methods (apache#7923)
  HDDS-11530. Support listMultipartUploads max uploads and markers (apache#7817)
  HDDS-11867. Remove code paths for non-Ratis SCM. (apache#7911)
  HDDS-12363. Add import options to IntelliJ IDEA style settings (apache#7921)
  HDDS-12215. Mark testContainerStateMachineRestartWithDNChangePipeline as flaky
  HDDS-12331. BlockOutputStream.failedServers is not thread-safe (apache#7885)
  HDDS-12188. Move server-only upgrade classes from hdds-common to hdds-server-framework (apache#7903)
  HDDS-12362. Remove temporary checkstyle suppression file (apache#7920)
  HDDS-12343. Fix spotbugs warnings in Recon (apache#7902)
  HDDS-12286. Fix license headers and imports for ozone-tools (apache#7919)
  HDDS-12284. Fix license headers and imports for ozone-s3-secret-store (apache#7917)
  HDDS-12275. Fix license headers and imports for ozone-integration-test (apache#7904)
  HDDS-12164. Rename and deprecate DFSConfigKeysLegacy config keys (apache#7803)
  HDDS-12283. Fix license headers and imports for ozone-recon-codegen (apache#7916)
  ...

Conflicts:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerChecksumTreeManager.java
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/checksum/ContainerMerkleTreeWriter.java
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainerCheck.java
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueHandler.java
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/checksum/ContainerMerkleTreeTestUtils.java
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/checksum/TestContainerMerkleTreeWriter.java
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/keyvalue/TestKeyValueContainerCheck.java
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestBackgroundContainerDataScanner.java
hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/ozoneimpl/TestOnDemandContainerDataScanner.java
hadoop-ozone/dist/src/main/smoketest/admincli/container.robot
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/dn/scanner/TestBackgroundContainerDataScannerIntegration.java
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/dn/scanner/TestBackgroundContainerMetadataScannerIntegration.java
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/dn/scanner/TestOnDemandContainerDataScannerIntegration.java
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hemantk-12 hemantk-12 hemantk-12 left review comments

@kerneltime kerneltime kerneltime approved these changes

@errose28 errose28 errose28 approved these changes

Assignees

@aswinshakil aswinshakil

Labels

container-reconciliation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@aswinshakil @kerneltime @hemantk-12 @errose28