Fix hadoop 3 compile Error, use FSDataOutputStream(OutputStream out, FileSystem.Statistics stats) instead of FSDataOutputStream(OutputStream out) #5251
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…tream out) is deleted in hadoop 3, use FSDataOutputStream(OutputStream out, FileSystem.Statistics stats) instead.
danny0405
reviewed
Apr 7, 2022
|
|
||
| try (FSDataOutputStream outputStream = new FSDataOutputStream(baos)) { | ||
| try (FSDataOutputStream outputStream = new FSDataOutputStream(baos, null)) { | ||
| try (HoodieParquetStreamWriter<IndexedRecord> parquetWriter = new HoodieParquetStreamWriter<>(outputStream, avroParquetConfig)) { |
Contributor
There was a problem hiding this comment.
Does hadoop 2 has this method ?
Contributor
Author
There was a problem hiding this comment.
Yes, hadoop2 has three constructors, just like below.
@Deprecated
public FSDataOutputStream(OutputStream out) throws IOException {
this(out, null);
}
public FSDataOutputStream(OutputStream out, FileSystem.Statistics stats)
throws IOException {
this(out, stats, 0);
}
public FSDataOutputStream(OutputStream out, FileSystem.Statistics stats,
long startPosition) throws IOException {
super(new PositionCache(out, stats, startPosition));
wrappedStream = out;
}But hadoop3 delete the constructor of FSDataOutputStream(OutputStream out), the rest constructors is like below.
public FSDataOutputStream(OutputStream out, FileSystem.Statistics stats) {
this(out, stats, 0);
}
public FSDataOutputStream(OutputStream out, FileSystem.Statistics stats,
long startPosition) {
super(new PositionCache(out, stats, startPosition));
wrappedStream = out;
}
Collaborator
Contributor
|
This change has already been covered by #4286 which is going to be ready for review soon. I'm wondering if it still makes sense to have this PR independently. |
Contributor
Author
|
Sorry, I didn't nitice that PR, this PR can be closed. |
Contributor
No worries. Feel free to chime in on that PR regarding the concern @danny0405 has put up :) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Tips
What is the purpose of the pull request
Fix hadoop 3 compile error.
Compile hudi with hadoop3, there arouse the following errors:
The reason is that, Hadoop3 delete FSDataOutputStream(OutputStream out) constructor of FSDataOutputStream
Brief change log
use FSDataOutputStream(OutputStream out, FileSystem.Statistics stats) instead of FSDataOutputStream(OutputStream out) in HoodieParquetDataBlock
Verify this pull request
This pull request is a trivial rework / code cleanup without any test coverage.
Committer checklist