Skip to content

Fix hadoop 3 compile Error, use FSDataOutputStream(OutputStream out, FileSystem.Statistics stats) instead of FSDataOutputStream(OutputStream out) #5251

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
xiaozhch5 wants to merge 1 commit into from
Closed

Fix hadoop 3 compile Error, use FSDataOutputStream(OutputStream out, FileSystem.Statistics stats) instead of FSDataOutputStream(OutputStream out) #5251

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieParquetDataBlock.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ protected byte[] serializeRecords(List<IndexedRecord> records) throws IOExceptio

ByteArrayOutputStream baos = new ByteArrayOutputStream();

try (FSDataOutputStream outputStream = new FSDataOutputStream(baos)) {
try (FSDataOutputStream outputStream = new FSDataOutputStream(baos, null)) {
try (HoodieParquetStreamWriter<IndexedRecord> parquetWriter = new HoodieParquetStreamWriter<>(outputStream, avroParquetConfig)) {

@danny0405 danny0405 Apr 7, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does hadoop 2 has this method ?

@xiaozhch5 xiaozhch5 Apr 7, 2022

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, hadoop2 has three constructors, just like below.

  @Deprecated
  public FSDataOutputStream(OutputStream out) throws IOException {
    this(out, null);
  }

  public FSDataOutputStream(OutputStream out, FileSystem.Statistics stats)
    throws IOException {
    this(out, stats, 0);
  }

  public FSDataOutputStream(OutputStream out, FileSystem.Statistics stats,
                            long startPosition) throws IOException {
    super(new PositionCache(out, stats, startPosition));
    wrappedStream = out;
  }

But hadoop3 delete the constructor of FSDataOutputStream(OutputStream out), the rest constructors is like below.

  public FSDataOutputStream(OutputStream out, FileSystem.Statistics stats) {
    this(out, stats, 0);
  }

  public FSDataOutputStream(OutputStream out, FileSystem.Statistics stats,
                            long startPosition) {
    super(new PositionCache(out, stats, startPosition));
    wrappedStream = out;
  }

for (IndexedRecord record : records) {
String recordKey = getRecordKey(record).orElse(null);
Expand Down