Skip to content

[HUDI-2994] Add judgement to existed partitionPath in the catch code block for HU… #4294

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
danny0405 merged 2 commits into from
Dec 13, 2021
Merged

[HUDI-2994] Add judgement to existed partitionPath in the catch code block for HU… #4294

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
fcdfe25
[HUDI-2994] Add judgement to existed partitionPath in the catch code …
Dec 13, 2021
08a0713
[HUDI-2994] Add judgement to existed partitionPath in the catch code …
Dec 13, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -336,9 +336,11 @@ protected FileStatus[] listPartition(Path partitionPath) throws IOException {
if (!metaClient.getFs().exists(partitionPath)) {
Copy link
Contributor

@danny0405 danny0405 Dec 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the partition path was created by another process/thread, we should invoke the #listStatus again.

metaClient.getFs().mkdirs(partitionPath);
return new FileStatus[0];
} else {
// in case the partition path was created by another caller
return metaClient.getFs().listStatus(partitionPath);
Copy link
Contributor

@danny0405 danny0405 Dec 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add one line comment here:

// in case the partition path was created by another caller.

Copy link
Contributor

@nsivabalan nsivabalan Dec 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Copy link
Contributor

@nsivabalan nsivabalan Dec 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, you may want to add the fix to

hudi/hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java

Line 472 in 46de25d

if (!fs.exists(partitionPath)) {
as well.

Copy link
Contributor

@nsivabalan nsivabalan Dec 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in general, I thought list partitions for any given partition can't have concurrent callers. across partitions we will parallelize, but for a given partition, I wasn't aware that there could be concurrent calls. can you throw some light on when this could happen.

Copy link
Contributor

@danny0405 danny0405 Dec 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In flink pipeline, there are cases that we need to look into the partition files through the fs view with this method call, the bucket assigner would check the small files under the partition and the partition may be new.

Multiple bucket assigner may operate on same partitions at the same time.

}
}
throw new HoodieIOException(String.format("Failed to list partition path: %s", partitionPath));
}

/**
Expand Down