HDDS-7328. Improve Deletion of FSO Paths#3844
Conversation
|
@linyiqun @rakeshadr PTAL. This should be compatible with HDDS-5048. This aims to improve the efficiency of the deletion if not enough sub-files or sub-dirs are chosen. |
JacksonYao287
left a comment
JacksonYao287
left a comment
There was a problem hiding this comment.
thank @Xushaohong for the work! LGTM +1
| deletedDirsCount.addAndGet(dirNum); | ||
| movedDirsCount.addAndGet(subDirNum); | ||
| movedFilesCount.addAndGet(subFileNum); | ||
| LOG.info("Number of dirs deleted: {}, Number of sub-files moved:" + |
There was a problem hiding this comment.
NIT: maybe debug here is better than info
There was a problem hiding this comment.
Make sense! I have updated.
There was a problem hiding this comment.
Hi, Jackson! After some thought, and according to practice, such conclusion info should be logged instead of debugged, such msg will not fill too much, but useful!
|
Thx @JacksonYao287 for the review! |
|
@JacksonYao287 please don't forget to close the Jira issue and set the fix version based on where it was committed. |
|
Thanks @Xushaohong for the improvement and @JacksonYao287 for the review. I have backported this fix into 1.3 branch. |
|
|
||
| Table.KeyValue<String, OmKeyInfo> pendingDeletedDirInfo; | ||
| try { | ||
| TableIterator<String, ? extends KeyValue<String, OmKeyInfo>> |
There was a problem hiding this comment.
This introduces a TableIterator leak, which is fixed by this PR. #3922
Please help review it. @JacksonYao287
There was a problem hiding this comment.
thanks @duongkame for the fix, @Xushaohong could you also take a look?
There was a problem hiding this comment.
thanks @duongkame for the fix, @Xushaohong could you also take a look?
@duongkame Thx for the fix!
@JacksonYao287 That fix is merged. LTGM
(cherry picked from commit f6cad7c) Change-Id: I1c93e0a705b49e198b1332c7ce39321b2a77966e
5 participants
What changes were proposed in this pull request?
Currently, the deleting speed of paths in the FSO bucket is too slow.
Every time the DirectoryDeletingService executes(every 1m by default), it chooses only one path from DeletedDirectroyTable and tries adding its sub-files and sub-dirs into the PurgeDirectories Request. The real deletion happens in OMKeyDeleteResponseWithFSO, which moved all sub-dirs into DeletedDirectroyTable. Then these sub-dirs will be chosen by the future execution of DirectoryDeletingService.
In the real production environment, such deletion speed is not applicable. There could be lots of dirs deleted without sub-dirs or sub-files. The dir deletion speed would lag behind the dir creation speed.
In this PR, I propose to optimize the logic of the DirectoryDeletingService, we could consume all the quota (ozone.path.deleting.limit.per.task) to delete paths as much as possible. The good news is the PB is already designed to do so, we could add more PurgePathRequest in one PurgeDirectoriesRequest.
Originally: 1 min. At most 1 path from deletedDirectoryTable.
Now: 1min. At most 10000 paths from deletedDirectoryTable.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-7328
How was this patch tested?
UT and real env