Skip to content

Fix reading large SequenceFile, RCFile, or Avro files#18837

Merged
electrum merged 1 commit intotrinodb:masterfrom
electrum:hive-reader
Aug 30, 2023
Merged

Fix reading large SequenceFile, RCFile, or Avro files#18837
electrum merged 1 commit intotrinodb:masterfrom
electrum:hive-reader

Conversation

@electrum
Copy link
Copy Markdown
Member

@electrum electrum commented Aug 28, 2023

Release notes

(x) Release notes are required, with the following suggested text:

# Hive
* Fix reading large SequenceFile, RCFile, or Avro files. ({issue}`18837`)

Fixes #17890

@cla-bot cla-bot bot added the cla-signed label Aug 28, 2023
@electrum electrum requested a review from dain August 28, 2023 19:11
@wendigo
Copy link
Copy Markdown
Contributor

wendigo commented Aug 28, 2023

@electrum can we have a test for it?

@electrum
Copy link
Copy Markdown
Member Author

I'm not sure how to test this other than generating a 2GB+ file, which would negatively impact the CI, and doesn't seem to be worth the cost.

@electrum electrum merged commit 9926d4e into trinodb:master Aug 30, 2023
@electrum electrum deleted the hive-reader branch August 30, 2023 03:28
@github-actions github-actions bot added this to the 426 milestone Aug 30, 2023
throws IOException
{
if (length <= 0) {
return 0;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for length == 0, it's obvious.
why also for length < 0? is it worth explanatory comment?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the behavior specified in the Javadoc and in the default implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

IllegalArgumentException: Size is greater than maximum int value when reading big file

5 participants