Support native parquet writer in hive module and Misc fixes#3400
Merged
dain merged 12 commits intotrinodb:masterfrom Jun 24, 2020
Merged
Support native parquet writer in hive module and Misc fixes#3400dain merged 12 commits intotrinodb:masterfrom
dain merged 12 commits intotrinodb:masterfrom
Conversation
qqibrow
changed the title
qqibrow
changed the title
dain
requested changes
May 5, 2020
Member
dain
left a comment
dain
left a comment
There was a problem hiding this comment.
Looking good. Just some minor comments.
Member
There was a problem hiding this comment.
Consider caching this value if it is expensive to calculate. In the ORC writer, we update the cached value after each write operation is processed.
Contributor
Author
There was a problem hiding this comment.
I resolved this. Also, I don't quite understand why in OrcWriter:
@Override
public long getWrittenBytes()
{
return orcWriter.getWrittenBytes() + orcWriter.getBufferedBytes();
}
why bufferedBytes is part of written bytes?
presto-hive/src/main/java/io/prestosql/plugin/hive/HiveModule.java
Outdated
Show resolved
Hide resolved
presto-hive/src/main/java/io/prestosql/plugin/hive/parquet/ParquetFileWriter.java
Outdated
Show resolved
Hide resolved
presto-hive/src/main/java/io/prestosql/plugin/hive/parquet/ParquetFileWriterFactory.java
Outdated
Show resolved
Hide resolved
presto-hive/src/main/java/io/prestosql/plugin/hive/parquet/ParquetWriterConfig.java
Outdated
Show resolved
Hide resolved
presto-parquet/src/main/java/io/prestosql/parquet/writer/PrimitiveColumnWriter.java
Outdated
Show resolved
Hide resolved
Member
There was a problem hiding this comment.
Are there statistics for non-primitive types?
Contributor
Author
There was a problem hiding this comment.
No. This aims to fill the stats part for primitive columns https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L740 . There is no support for this on non-primitive types.
qqibrow
force-pushed
the
add_parquet_writer
branch
from
673eead to
5e46b04
Compare
resetDictionary in reset() cleaned dictionary regardless of the fact that dictionary page is null or not. resetDictionary should happen only after get dictionary page and the page is not null.
Set both setting to smaller value could force writer to write multiple row groups and multiple pages in each row group, which could better simulate real world examples.
qqibrow
force-pushed
the
add_parquet_writer
branch
from
5e46b04 to
a5f7df3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.