Raptor background jobs#13849
Conversation
|
Given the last PR is still in process, this PR still have some commits from last PR. I will create a new PR after the last PR is merged. Thanks! |
fb2256b to
a23872b
Compare
presto-raptor/src/main/java/com/facebook/presto/raptor/metadata/DatabaseShardManager.java
Outdated
Show resolved
Hide resolved
...to-raptor/src/main/java/com/facebook/presto/raptor/storage/organization/OrganizationSet.java
Outdated
Show resolved
Hide resolved
...to-raptor/src/main/java/com/facebook/presto/raptor/storage/organization/OrganizationSet.java
Outdated
Show resolved
Hide resolved
presto-raptor/src/main/java/com/facebook/presto/raptor/storage/organization/ShardOrganizer.java
Outdated
Show resolved
Hide resolved
...to-raptor/src/main/java/com/facebook/presto/raptor/storage/organization/OrganizationSet.java
Outdated
Show resolved
Hide resolved
a23872b to
3a6009a
Compare
jessesleeping
left a comment
There was a problem hiding this comment.
Reviewed commit "Enable delta compaction and organization". Only minor comments.
...ptor/src/main/java/com/facebook/presto/raptor/storage/organization/CompactionSetCreator.java
Outdated
Show resolved
Hide resolved
...to-raptor/src/main/java/com/facebook/presto/raptor/storage/organization/OrganizationJob.java
Outdated
Show resolved
Hide resolved
...or/src/main/java/com/facebook/presto/raptor/storage/organization/ShardCompactionManager.java
Outdated
Show resolved
Hide resolved
presto-raptor/src/main/java/com/facebook/presto/raptor/storage/organization/ShardCompactor.java
Outdated
Show resolved
Hide resolved
presto-raptor/src/main/java/com/facebook/presto/raptor/storage/organization/ShardCompactor.java
Outdated
Show resolved
Hide resolved
.../src/main/java/com/facebook/presto/raptor/storage/organization/ShardOrganizationManager.java
Outdated
Show resolved
Hide resolved
.../src/test/java/com/facebook/presto/raptor/storage/organization/TestCompactionSetCreator.java
Outdated
Show resolved
Hide resolved
jessesleeping
left a comment
There was a problem hiding this comment.
Reviewed commit "Add priority to Compaction process". Some general comments besides the inline minor comments:
- Instead of using
intto indicate priority, I suggest we use anenumwhich hasDELTA_COMPACTIONandNORMAL_COMPACTIONfor now. It will be more readable. - Can you add test case to verify if the behavior of
ShardOrganizeris as expected?
...to-raptor/src/main/java/com/facebook/presto/raptor/storage/organization/OrganizationJob.java
Outdated
Show resolved
Hide resolved
presto-raptor/src/main/java/com/facebook/presto/raptor/storage/organization/ShardOrganizer.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Instead of adding 2 levels of callback, can you extract the for loop into a function and call it at onSuccess() and onFailure() below?
There was a problem hiding this comment.
But that would add duplicate code, Let's also confirm with James before we make this change
There was a problem hiding this comment.
Why would this create duplicate code?
jessesleeping
left a comment
There was a problem hiding this comment.
Commit "Add delta in reassign process" LGTM.
But currently I also add how many delta we have as the priority, so that the OrganizationSet that has the most delta will be compacted first, so the number couldn't be abstracted as ENUM for now |
3a6009a to
4ff6874
Compare
4ff6874 to
8934760
Compare
|
the third PR of #13248 |
042494c to
45c54c3
Compare
highker
left a comment
There was a problem hiding this comment.
nits only. Will leave to @jessesleeping for the final approval and merge.
presto-raptor/src/main/java/com/facebook/presto/raptor/metadata/DatabaseShardManager.java
Outdated
Show resolved
Hide resolved
presto-raptor/src/main/java/com/facebook/presto/raptor/metadata/DatabaseShardManager.java
Outdated
Show resolved
Hide resolved
presto-raptor/src/main/java/com/facebook/presto/raptor/storage/ShardEjector.java
Outdated
Show resolved
Hide resolved
...or/src/main/java/com/facebook/presto/raptor/storage/organization/ShardCompactionManager.java
Outdated
Show resolved
Hide resolved
...ptor/src/main/java/com/facebook/presto/raptor/storage/organization/CompactionSetCreator.java
Outdated
Show resolved
Hide resolved
presto-raptor/src/main/java/com/facebook/presto/raptor/storage/organization/ShardIndexInfo.java
Outdated
Show resolved
Hide resolved
.../src/main/java/com/facebook/presto/raptor/storage/organization/ShardOrganizationManager.java
Outdated
Show resolved
Hide resolved
45c54c3 to
f69836b
Compare
Uh oh!
There was an error while loading. Please reload this page.