[HUDI-3042] Refactoring clustering executors #4847
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
xushiyan
commented
Feb 18, 2022
Comment on lines
-116
to
-127
| protected Map<String, List<String>> getPartitionToReplacedFileIds(HoodieWriteMetadata<JavaRDD<WriteStatus>> writeMetadata) { | ||
| Set<HoodieFileGroupId> newFilesWritten = writeMetadata.getWriteStats().get().stream() | ||
| .map(s -> new HoodieFileGroupId(s.getPartitionPath(), s.getFileId())).collect(Collectors.toSet()); | ||
| // for the below execution strategy, new file group id would be same as old file group id | ||
| if (SparkSingleFileSortExecutionStrategy.class.getName().equals(config.getClusteringExecutionStrategyClass())) { | ||
| return ClusteringUtils.getFileGroupsFromClusteringPlan(clusteringPlan) | ||
| .collect(Collectors.groupingBy(fg -> fg.getPartitionPath(), Collectors.mapping(fg -> fg.getFileId(), Collectors.toList()))); | ||
| } | ||
| return ClusteringUtils.getFileGroupsFromClusteringPlan(clusteringPlan) | ||
| .filter(fg -> !newFilesWritten.contains(fg)) | ||
| .collect(Collectors.groupingBy(fg -> fg.getPartitionPath(), Collectors.mapping(fg -> fg.getFileId(), Collectors.toList()))); | ||
| } |
Member
Author
There was a problem hiding this comment.
extracted to BaseCommitActionExecutor.java
Comment on lines
-101
to
-107
| private void validateWriteResult(HoodieWriteMetadata<JavaRDD<WriteStatus>> writeMetadata) { | ||
| if (writeMetadata.getWriteStatuses().isEmpty()) { | ||
| throw new HoodieClusteringException("Clustering plan produced 0 WriteStatus for " + instantTime | ||
| + " #groups: " + clusteringPlan.getInputGroups().size() + " expected at least " | ||
| + clusteringPlan.getInputGroups().stream().mapToInt(HoodieClusteringGroup::getNumOutputFileGroups).sum() | ||
| + " write statuses"); | ||
| } |
Member
Author
There was a problem hiding this comment.
extracted to BaseCommitActionExecutor.java
Comment on lines
-73
to
-93
| HoodieInstant instant = HoodieTimeline.getReplaceCommitRequestedInstant(instantTime); | ||
| // Mark instant as clustering inflight | ||
| table.getActiveTimeline().transitionReplaceRequestedToInflight(instant, Option.empty()); | ||
| table.getMetaClient().reloadActiveTimeline(); | ||
|
|
||
| final Schema schema = HoodieAvroUtils.addMetadataFields(new Schema.Parser().parse(config.getSchema())); | ||
| HoodieWriteMetadata<JavaRDD<WriteStatus>> writeMetadata = ((ClusteringExecutionStrategy<T, JavaRDD<HoodieRecord<? extends HoodieRecordPayload>>, JavaRDD<HoodieKey>, JavaRDD<WriteStatus>>) | ||
| ReflectionUtils.loadClass(config.getClusteringExecutionStrategyClass(), | ||
| new Class<?>[] {HoodieTable.class, HoodieEngineContext.class, HoodieWriteConfig.class}, table, context, config)) | ||
| .performClustering(clusteringPlan, schema, instantTime); | ||
| JavaRDD<WriteStatus> writeStatusRDD = writeMetadata.getWriteStatuses(); | ||
| JavaRDD<WriteStatus> statuses = updateIndex(writeStatusRDD, writeMetadata); | ||
| writeMetadata.setWriteStats(statuses.map(WriteStatus::getStat).collect()); | ||
| writeMetadata.setPartitionToReplaceFileIds(getPartitionToReplacedFileIds(writeMetadata)); | ||
| commitOnAutoCommit(writeMetadata); | ||
| if (!writeMetadata.getCommitMetadata().isPresent()) { | ||
| HoodieCommitMetadata commitMetadata = CommitUtils.buildMetadata(writeMetadata.getWriteStats().get(), writeMetadata.getPartitionToReplaceFileIds(), | ||
| extraMetadata, operationType, getSchemaToStoreInCommit(), getCommitActionType()); | ||
| writeMetadata.setCommitMetadata(Option.of(commitMetadata)); | ||
| } | ||
| return writeMetadata; |
Member
Author
There was a problem hiding this comment.
extracted to executeClustering() in BaseCommitActionExecutor.java
xushiyan
force-pushed
the
HUDI-3042-refactor-clustering-executor
branch
from
c89db11 to
94c384e
Compare
xushiyan
force-pushed
the
HUDI-3042-refactor-clustering-executor
branch
from
94c384e to
7306cb4
Compare
danny0405
reviewed
Feb 21, 2022
| Iterator<HoodieRecord<T>> recordItr) throws IOException; | ||
|
|
||
| protected HoodieWriteMetadata<HoodieData<WriteStatus>> executeClustering(HoodieClusteringPlan clusteringPlan) { | ||
| HoodieInstant instant = HoodieTimeline.getReplaceCommitRequestedInstant(instantTime); |
Contributor
There was a problem hiding this comment.
The BaseCommitActionExecutor responsibilities are a bit confusing, it handles regular writing process such as insert, upsert and with this path clustering, then what about the compaction?
Should we make a new base class for table services then ?
Contributor
There was a problem hiding this comment.
I think the goal is to revamp the commit executors and write pipeline altogether later on, so the refactoring here is limited to code reuse. @xushiyan is that the case?
Member
Author
There was a problem hiding this comment.
@danny0405 agreed that it looks like some mixed responsibilities there. i'll make clearer separation in https://issues.apache.org/jira/browse/HUDI-2439
Collaborator
yihua
approved these changes
Feb 23, 2022
| */ | ||
| public abstract class JavaExecutionStrategy<T extends HoodieRecordPayload<T>> | ||
| extends ClusteringExecutionStrategy<T, List<HoodieRecord<T>>, List<HoodieKey>, List<WriteStatus>> { | ||
| extends ClusteringExecutionStrategy<T, HoodieData<HoodieRecord<T>>, HoodieData<HoodieKey>, HoodieData<WriteStatus>> { |
Contributor
There was a problem hiding this comment.
Is this Java-specific class going to be removed as a follow-up?
Member
Author
There was a problem hiding this comment.
@yihua yes i should make another PR to deal with ClusteringExecutionStrategy and subclasses, which can be a good separation.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Extract common code from
to BaseCommitActionExecutor.java
Remove redundant engine specific classes
use ClusteringPlanActionExecutor.java instead.
Committer checklist
Has a corresponding JIRA in PR title & commit
Commit message is descriptive of the change
CI is green
Necessary doc changes done or have another open PR
For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.