HDDS-5821 Container cache management for closing RockDB #3426
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
Author
|
This can be merged post #3091 |
kerneltime
changed the title
Contributor
|
@kerneltime please enable the build-branch workflow in your fork to get CI feedback on this. |
jojochuang
reviewed
May 23, 2022
| public void batchDelete(WriteBatch writeBatch, byte[] key) | ||
| throws IOException { | ||
| try { | ||
| writeBatch.delete(getHandle(), key); |
Contributor
There was a problem hiding this comment.
WriteBatch needs to be closed if unused. See PR #3400
Contributor
Author
There was a problem hiding this comment.
@jojochuang I will be filling additional PRs to address remaining leaks. This PR is to get the RocksDatabase.java abstraction in and address the cache handling for it.
jojochuang
reviewed
May 23, 2022
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/TableConfig.java
Show resolved
Hide resolved
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/TableConfig.java
Show resolved
Hide resolved
jojochuang
reviewed
May 24, 2022
hadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/RocksDatabase.java
Show resolved
Hide resolved
...ainer-service/src/test/java/org/apache/hadoop/ozone/container/common/TestContainerCache.java
Outdated
Show resolved
Hide resolved
...-service/src/main/java/org/apache/hadoop/ozone/container/metadata/AbstractDatanodeStore.java
Outdated
Show resolved
Hide resolved
...-service/src/main/java/org/apache/hadoop/ozone/container/metadata/AbstractDatanodeStore.java
Show resolved
Hide resolved
Change-Id: I112b424fd280bb3ca9d370bef9dc611e2aef67e1
kerneltime
force-pushed
the
HDDS-5821
branch
2 times, most recently
from
9c98d54 to
d74b298
Compare
Change-Id: I239872c604da6eb107e61bae75695f942044fc15
Contributor
|
The last test failures don't look related. The test run before it passed the same set of tests. I'll merge it now. |
errose28
added a commit
to errose28/ozone
that referenced
this pull request
Jun 7, 2022
* master: (87 commits) HDDS-6686. Do Leadship check before SASL token verification. (apache#3382) HDDS-4364: [FSO]List FileStatus : startKey can be a non-existed path (apache#3481) HDDS-6091. Add file checksum to OmKeyInfo (apache#3201) HDDS-6706. Exposing Volume Information Metrics to the DataNode UI (apache#3478) HDDS-6759: Add listblock API in MockDatanodeStorage (apache#3452) HDDS-5821 Container cache management for closing RockDB (apache#3426) HDDS-6683. Refactor OM server bucket layout configuration usage (apache#3477) HDDS-6824. Revert changes made in proto.lock by HDDS-6768. (apache#3480) HDDS-6811. Bucket create message with layout type (apache#3479) HDDS-6810. Add a optional flag to trigger listStatus as part of listKeys for FSO buckets. (apache#3461) HDDS-6828. Revert RockDB version pending leak fixes (apache#3475) HDDS-6764: EC: DN ability to create RECOVERING containers for EC reconstruction. (apache#3458) HDDS-6795: EC: PipelineStateMap#addPipeline should not have precondition checks post db updates (apache#3453) HDDS-6823. Intermittent failure in TestOzoneECClient#testExcludeOnDNMixed (apache#3476) HDDS-6820. Bucket Layout Post-Finalization Validators for ACL Requests. (apache#3472) HDDS-6819. Add LEGACY to AllowedBucketLayouts in CreateBucketHandler (apache#3473) HDDS-4859. [FSO]ListKeys: seek all the files/dirs from startKey to keyPrefix (apache#3466) HDDS-6705 Add metrics for volume statistics including disk capacity, usage, Reserved (apache#3430) HDDS-6474. Add test to cover the FSO bucket list status with beyond batch boundary and cache. (apache#3379). Contributed by aswinshakil HDDS-6280. Support Container Balancer HA (apache#3423) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Add container cache management logic on top of #3091
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5821
How was this patch tested?
updated unit tests.