HDDS-4123. Integrate OM Open Key Cleanup Service Into Existing Code #3319
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
Author
|
@errose28 @captainzmc PTAL. |
Member
Author
|
100x TestOpenKeyCleanupService: https://github.com/kaijchen/ozone/actions/runs/2452367006 |
captainzmc
reviewed
Jun 7, 2022
Member
captainzmc
left a comment
captainzmc
left a comment
There was a problem hiding this comment.
Thanks @kaijchen for the working, LGTM. Just have some minor comments here.
...-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/om/TestSecureOzoneManager.java
Show resolved
Hide resolved
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OpenKeyCleanupService.java
Show resolved
Hide resolved
Member
Member
Author
|
Thanks @captainzmc for the review. I think there is one more task left after this, HDDS-6769. |
captainzmc
approved these changes
Jun 8, 2022
Member
|
Hi @kaijchen , did you tested this patch in local cluster? For example, keep writing failed keys. The clean up service keep deleting in the background. Run a few days for stability test to see if there's anything problem. |
Member
Author
|
Yes, I have tested it in a local cluster. From the log we can see open key cleanup service works as expected. |
Member
Author
|
Limiting number of expired keys is working as expected. |
Member
The tests looked good, thanks @kaijchen working on this. Let's merge this PR first, if other modifications are needed, we can submit a new patch. |
Member
Author
|
Thanks @captainzmc for reviewing and merging this. |
errose28
added a commit
to errose28/ozone
that referenced
this pull request
Jun 23, 2022
* master: (34 commits) HDDS-6868 Add S3Auth information to thread local (apache#3527) HDDS-6877. Keep replication port unchanged when restarting datanode in MiniOzoneCluster (apache#3510) HDDS-6907. OFS should create buckets with FILE_SYSTEM_OPTIMIZED layout. (apache#3528) HDDS-6875. Migrate parameterized tests in hdds-common to JUnit5 (apache#3513) HDDS-6924. OBJECT_STORE isn't flat namespaced (apache#3533) HDDS-6899. [EC] Remove warnings and errors from console during online reconstruction of data. (apache#3522) HDDS-6695. Enable SCM Ratis by default for new clusters only (apache#3499) HDDS-4123. Integrate OM Open Key Cleanup Service Into Existing Code (apache#3319) HDDS-6882. Correct exit code for invalid arguments passed to command-line tools. (apache#3517) HDDS-6890. EC: Fix potential wrong replica read with over-replicated container. (apache#3523) HDDS-6902. Duplicate mockito-core entries in pom.xml (apache#3525) HDDS-6752. Migrate tests with rules in hdds-server-scm to JUnit5 (apache#3442) HDDS-6806. EC: Implement the EC Reconstruction coordinator. (apache#3504) HDDS-6829. Limit the no of inflight replication tasks in SCM. (apache#3482) HDDS-6898. [SCM HA finalization] Modify acceptance test configuration to speed up test finalization (apache#3521) HDDS-6577. Configurations to reserve HDDS volume space. (apache#3484) HDDS-6870 Clean up isTenantAdmin to use UGI (apache#3503) HDDS-6872. TestAuthorizationV4QueryParser should pass offline (apache#3506) HDDS-6840. Add MetaData volume information to the SCM and OM - UI (apache#3488) HDDS-6697. EC: ReplicationManager - create class to detect EC container health issues (apache#3512) ...
kaijchen
added a commit
to kaijchen/ozone
that referenced
this pull request
Jun 27, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
WIP: this PR depends on #3233 and #3226.This pull request completes the open key cleanup service outlined in the parent Jira HDDS-4120. It implements the
OpenKeyCleanupServiceclass, and starts and stops the service inKeyManagerImpl. The following configurations have been defined to specify the service's behavior:See
ozone-defaults.xmlfor their corresponding descriptions.What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-4123
How was this patch tested?
Integration test
TestOpenKeyCleanupService