Skip to content

[HUDI-4255] Make the flink merge and replace handle intermediate file… #5866

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
danny0405 merged 1 commit into from
Jun 15, 2022
Merged

[HUDI-4255] Make the flink merge and replace handle intermediate file… #5866

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandle.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,8 +200,10 @@ private void init(String fileId, String partitionPath, HoodieBaseFile baseFileTo
writeStatus.getStat().setFileId(fileId);
setWriteStatusPath();

// Create Marker file
createMarkerFile(partitionPath, newFileName);
// Create Marker file,
// uses name of `newFilePath` instead of `newFileName`
// in case the sub-class may roll over the file handle name.
createMarkerFile(partitionPath, newFilePath.getName());

// Create the writer for writing the new version file
fileWriter = createNewFileWriter(instantTime, newFilePath, hoodieTable, config,
Expand Down
4 changes: 1 addition & 3 deletions ...client/hudi-flink-client/src/main/java/org/apache/hudi/io/FlinkMergeAndReplaceHandle.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,10 +137,8 @@ protected void makeOldAndNewFilePaths(String partitionPath, String oldFileName,
* Use the writeToken + "-" + rollNumber as the new writeToken of a mini-batch write.
*/
protected String newFileNameWithRollover(int rollNumber) {
// make the intermediate file as hidden
final String fileID = "." + this.fileId;
return FSUtils.makeBaseFileName(instantTime, writeToken + "-" + rollNumber,
fileID, hoodieTable.getBaseFileExtension());
this.fileId, hoodieTable.getBaseFileExtension());
}

@Override
Expand Down
8 changes: 7 additions & 1 deletion hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/io/FlinkMergeHandle.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,11 +158,17 @@ protected void makeOldAndNewFilePaths(String partitionPath, String oldFileName,
* Use the writeToken + "-" + rollNumber as the new writeToken of a mini-batch write.
*/
protected String newFileNameWithRollover(int rollNumber) {
// make the intermediate file as hidden
return FSUtils.makeBaseFileName(instantTime, writeToken + "-" + rollNumber,
this.fileId, hoodieTable.getBaseFileExtension());
}

@Override
protected void setWriteStatusPath() {
// if there was rollover, should set up the path as the initial new file path.
Path path = rolloverPaths.size() > 0 ? rolloverPaths.get(0) : newFilePath;
writeStatus.getStat().setPath(new Path(config.getBasePath()), path);
}

@Override
public List<WriteStatus> close() {
try {
Expand Down