[HUDI-840]Clean blank file created by HoodieLogFormatWriter

[hddong]

Tips

• Thank you very much for contributing to Apache Hudi.
• Please review https://hudi.apache.org/contributing.html before opening a pull request.

What is the purpose of the pull request

When roll over is true, HoodieLogFormatWriter will create next version log file. But it always left a blank file when close.

Brief change log

(for example:)

• Clean blank file created by HoodieLogFormatWriter

Verify this pull request

This pull request is a trivial rework / code cleanup without any test coverage.

Committer checklist

• Has a corresponding JIRA in PR title & commit

• Commit message is descriptive of the change

• CI is green

• Necessary doc changes done or have another open PR

• For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

[hddong]

@yanghua please have a review when free.

[vinothchandar]

@n3nash could you also review this since it touches the common log format writer as well

[bvaradar]

@hddong : Any possible reasons why blank file is created in the first place ?

[hddong]

@hddong : Any possible reasons why blank file is created in the first place ?

Create blank file for appendBlock when new HoodieLogFormatWriter
https://github.com/apache/incubator-hudi/blob/f1592be629c3f9762f62d4e1dbf3be54f213d92d/hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFormatWriter.java#L105-L108
But there is a special case, when roll over is true(block size is past the threshold), we will close the old writer and create a new writer . And if we close a new writer created by rolloverIfNeeded , there will left a blank file.

[vinothchandar]

fair enough. but this is effectively doing an extra RPC every close, just to handle this corner case. what is the actual issue caused by the empty file? does the query/writing fail or get stuck?

[yanghua]

@hddong This PR has conflicts files.

[hddong]

@yanghua had rebase this.

[hddong]

@vinothchandar :

hudi/hudi-common/src/main/java/org/apache/hudi/common/table/TableSchemaResolver.java

Line 382 in 0cb24e4

public static MessageType readSchemaFromLogFile(FileSystem fs, Path path) throws IOException {

readSchemaFromLogFile may read the blank file and the blank file had been read before in HoodieLogFileCommand(modified to avoid reading the blank files)

[bvaradar]

@hddong : I went ahead and redid this change in the interest of time :)
Instead of deleting on close, I have made changes to lazily create the log file when appending next block. This should avoid creating empty files.
(cc @vinothchandar )

[vinothchandar]

LGTM