PARQUET-4: Use LRU caching for footers in ParquetInputFormat. #2

matt-martin · 2014-06-19T17:40:51Z

Reopening https://github.com/Parquet/parquet-mr/pull/403 against the new Apache repository.

matt-martin · 2014-06-19T17:47:25Z

@dvryaboy Please note this is effectively the same pull request as https://github.com/Parquet/parquet-mr/pull/403. It includes a change I made in response to your last comment (and some minor white space fixes).

dvryaboy · 2014-06-19T18:28:54Z

This looks ok to me. @julienledem could you take a quick look at the code?
Also I'm not super familiar with the new merge process (do we have a script to automate apache jira creation, pull, merge, etc?)

julienledem · 2014-06-20T03:18:46Z

please create a jira on https://issues.apache.org/jira/browse/PARQUET and refer to the pull request
the tool to merge pull requests is coming: https://github.com/apache/incubator-parquet-mr/pull/5

matt-martin · 2014-06-20T03:48:42Z

I've created the following JIRA for this pull request: https://issues.apache.org/jira/browse/PARQUET-4

julienledem · 2014-06-21T04:35:01Z

please prefix the PR description with "PARQUET-4: "

julienledem · 2014-06-21T04:37:20Z

parquet-hadoop/src/main/java/parquet/hadoop/LruCache.java

+  interface Entry<V> {
+    public boolean isCurrent();
+    public boolean isNewerThan(V otherEntry);
+  }


please add javadoc to this interface

julienledem · 2014-06-21T04:50:04Z

Thank you Matt for fixing the interaction with Hive.
I added a few comments

…che "values" to avoid confusion with java.util.Map.Entry (which contains key value pairs whereas our old "entries" really only refer to the values).

julienledem · 2014-06-23T17:55:38Z

Thanks Matt.
Here is my suggestion on how to finalize this: https://github.com/apache/incubator-parquet-mr/pull/2#discussion_r14089413

…n appended to/overwritten/deleted in order to minimize the number of namenode interactions.

matt-martin · 2014-06-23T21:38:00Z

Thanks for the feedback @julienledem. I've tried to incorporate your suggestion in the latest version of the code.

…r class to ParquetInputFormat to make sure that the debugging log statements print out meaningful paths.

1) Cascading write support (see: apache/incubator-parquet-mr@76bbf4a) 2) Caching fix (see: https://github.com/apache/incubator-parquet-mr/pull/2 and https://github.com/matt-martin/incubator-parquet-mr/tree/99bb5a3acdf6c998f5d344bbeaeb446031f47df0)

matt-martin · 2014-06-25T03:03:47Z

@julienledem I am not planning to make any further changes to the code. Please review when you get the chance. As mentioned in the previous comment, I believe the most recent changes address your performance-related concerns.

matt-martin · 2014-07-18T02:19:40Z

@dvryaboy @julienledem just checking-in to see what I might be able to do (if anything) to help close out this pull request

julienledem · 2014-07-18T22:51:04Z

LGTM +1

matt-martin · 2014-07-26T16:54:30Z

Just wanted to belatedly thank @julienledem and @dvryaboy for your help in closing this out! Thanks again for all the feedback!

Reopening https://github.com/Parquet/parquet-mr/pull/403 against the new Apache repository. Author: Matthieu Martin <[email protected]> Closes #2 from matt-martin/master and squashes the following commits: 99bb5a3 [Matthieu Martin] Minor javadoc and whitespace changes. Also added the FileStatusWrapper class to ParquetInputFormat to make sure that the debugging log statements print out meaningful paths. 250a398 [Matthieu Martin] Be less aggressive about checking whether the underlying file has been appended to/overwritten/deleted in order to minimize the number of namenode interactions. d946445 [Matthieu Martin] Add javadocs to parquet.hadoop.LruCache. Rename cache "entries" as cache "values" to avoid confusion with java.util.Map.Entry (which contains key value pairs whereas our old "entries" really only refer to the values). a363622 [Matthieu Martin] Use LRU caching for footers in ParquetInputFormat. Conflicts: parquet-hadoop/src/test/java/parquet/hadoop/TestInputFormat.java

…with_caching_fix_and_cascading_write_support branch: ae77413 Here is the original commit message (note that that commit was itself a cherry-pick): Merging/cherry-picking: 1) Cascading write support (see: apache/incubator-parquet-mr@76bbf4a) 2) Caching fix (see: https://github.com/apache/incubator-parquet-mr/pull/2 and https://github.com/matt-martin/incubator-parquet-mr/tree/99bb5a3acdf6c998f5d344bbeaeb446031f47df0) Conflicts: parquet-hadoop/src/main/java/parquet/hadoop/ParquetOutputFormat.java

Reopening https://github.com/Parquet/parquet-mr/pull/403 against the new Apache repository. Author: Matthieu Martin <[email protected]> Closes #2 from matt-martin/master and squashes the following commits: 99bb5a3 [Matthieu Martin] Minor javadoc and whitespace changes. Also added the FileStatusWrapper class to ParquetInputFormat to make sure that the debugging log statements print out meaningful paths. 250a398 [Matthieu Martin] Be less aggressive about checking whether the underlying file has been appended to/overwritten/deleted in order to minimize the number of namenode interactions. d946445 [Matthieu Martin] Add javadocs to parquet.hadoop.LruCache. Rename cache "entries" as cache "values" to avoid confusion with java.util.Map.Entry (which contains key value pairs whereas our old "entries" really only refer to the values). a363622 [Matthieu Martin] Use LRU caching for footers in ParquetInputFormat.

…2 api Currently for creating a user defined predicate using the new filter api, no value can be passed to create a dynamic filter at runtime. This reduces the usefulness of the user defined predicate, and meaningful predicates cannot be created. We can add a generic Object value that is passed through the api, which can internally be used in the keep function of the user defined predicate for creating many different types of filters. For example, in spark sql, we can pass in a list of filter values for a where IN clause query and filter the row values based on that list. Author: Yash Datta <[email protected]> Author: Alex Levenson <[email protected]> Author: Yash Datta <[email protected]> Closes #73 from saucam/master and squashes the following commits: 7231a3b [Yash Datta] Merge pull request #3 from isnotinvain/alexlevenson/fix-binary-compat dcc276b [Alex Levenson] Ignore binary incompatibility in private filter2 class 7bfa5ad [Yash Datta] Merge pull request #2 from isnotinvain/alexlevenson/simplify-udp-state 0187376 [Alex Levenson] Resolve merge conflicts 25aa716 [Alex Levenson] Simplify user defined predicates with state 51952f8 [Yash Datta] PARQUET-116: Fix whitespace d7b7159 [Yash Datta] PARQUET-116: Make UserDefined abstract, add two subclasses, one accepting udp class, other accepting serializable udp instance 40d394a [Yash Datta] PARQUET-116: Fix whitespace 9a63611 [Yash Datta] PARQUET-116: Fix whitespace 7caa4dc [Yash Datta] PARQUET-116: Add ConfiguredUserDefined that takes a serialiazble udp directly 0eaabf4 [Yash Datta] PARQUET-116: Move the config object from keep method to a configure method in udp predicate f51a431 [Yash Datta] PARQUET-116: Adding type safety for the filter object to be passed to user defined predicate d5a2b9e [Yash Datta] PARQUET-116: Enforce that the filter object to be passed must be Serializable dfd0478 [Yash Datta] PARQUET-116: Add a test case for passing a filter object to user defined predicate 4ab46ec [Yash Datta] PARQUET-116: Pass a filter object to user defined predicate in filter2 api

This buffer initializes itself to a default size when instantiated. This leads to a lot of unused small buffers when there are a lot of empty columns. Author: Alex Levenson <[email protected]> Author: julien <[email protected]> Author: Julien Le Dem <[email protected]> Closes #98 from julienledem/avoid_wasting_64K_per_empty_buffer and squashes the following commits: b0200dd [julien] add license a1b278e [julien] Merge branch 'master' into avoid_wasting_64K_per_empty_buffer 5304ee1 [julien] remove unused constant 81e399f [julien] Merge branch 'avoid_wasting_64K_per_empty_buffer' of github.com:julienledem/incubator-parquet-mr into avoid_wasting_64K_per_empty_buffer ccf677d [julien] Merge branch 'master' into avoid_wasting_64K_per_empty_buffer 37148d6 [Julien Le Dem] Merge pull request #2 from isnotinvain/PR-98 b9abab0 [Alex Levenson] Address Julien's comment 965af7f [Alex Levenson] one more typo 9939d8d [Alex Levenson] fix typos in comments 61c0100 [Alex Levenson] Make initial slab size heuristic into a helper method, apply in DictionaryValuesWriter as well a257ee4 [Alex Levenson] Improve IndexOutOfBoundsException message 64d6c7f [Alex Levenson] update comments 8b54667 [Alex Levenson] Don't use CapacityByteArrayOutputStream for writing page chunks 6a20e8b [Alex Levenson] Remove initialSlabSize decision from InternalParquetRecordReader, use a simpler heuristic in the column writers instead 3a0f8e4 [Alex Levenson] Use simpler settings for column chunk writer b2736a1 [Alex Levenson] Some cleanup in CapacityByteArrayOutputStream 1df4a71 [julien] refactor CapacityByteArray to be aware of page size 95c8fb6 [julien] avoid wasting 64K per empty buffer.

…2 api Currently for creating a user defined predicate using the new filter api, no value can be passed to create a dynamic filter at runtime. This reduces the usefulness of the user defined predicate, and meaningful predicates cannot be created. We can add a generic Object value that is passed through the api, which can internally be used in the keep function of the user defined predicate for creating many different types of filters. For example, in spark sql, we can pass in a list of filter values for a where IN clause query and filter the row values based on that list. Author: Yash Datta <[email protected]> Author: Alex Levenson <[email protected]> Author: Yash Datta <[email protected]> Closes apache#73 from saucam/master and squashes the following commits: 7231a3b [Yash Datta] Merge pull request #3 from isnotinvain/alexlevenson/fix-binary-compat dcc276b [Alex Levenson] Ignore binary incompatibility in private filter2 class 7bfa5ad [Yash Datta] Merge pull request #2 from isnotinvain/alexlevenson/simplify-udp-state 0187376 [Alex Levenson] Resolve merge conflicts 25aa716 [Alex Levenson] Simplify user defined predicates with state 51952f8 [Yash Datta] PARQUET-116: Fix whitespace d7b7159 [Yash Datta] PARQUET-116: Make UserDefined abstract, add two subclasses, one accepting udp class, other accepting serializable udp instance 40d394a [Yash Datta] PARQUET-116: Fix whitespace 9a63611 [Yash Datta] PARQUET-116: Fix whitespace 7caa4dc [Yash Datta] PARQUET-116: Add ConfiguredUserDefined that takes a serialiazble udp directly 0eaabf4 [Yash Datta] PARQUET-116: Move the config object from keep method to a configure method in udp predicate f51a431 [Yash Datta] PARQUET-116: Adding type safety for the filter object to be passed to user defined predicate d5a2b9e [Yash Datta] PARQUET-116: Enforce that the filter object to be passed must be Serializable dfd0478 [Yash Datta] PARQUET-116: Add a test case for passing a filter object to user defined predicate 4ab46ec [Yash Datta] PARQUET-116: Pass a filter object to user defined predicate in filter2 api

This buffer initializes itself to a default size when instantiated. This leads to a lot of unused small buffers when there are a lot of empty columns. Author: Alex Levenson <[email protected]> Author: julien <[email protected]> Author: Julien Le Dem <[email protected]> Closes apache#98 from julienledem/avoid_wasting_64K_per_empty_buffer and squashes the following commits: b0200dd [julien] add license a1b278e [julien] Merge branch 'master' into avoid_wasting_64K_per_empty_buffer 5304ee1 [julien] remove unused constant 81e399f [julien] Merge branch 'avoid_wasting_64K_per_empty_buffer' of github.com:julienledem/incubator-parquet-mr into avoid_wasting_64K_per_empty_buffer ccf677d [julien] Merge branch 'master' into avoid_wasting_64K_per_empty_buffer 37148d6 [Julien Le Dem] Merge pull request #2 from isnotinvain/PR-98 b9abab0 [Alex Levenson] Address Julien's comment 965af7f [Alex Levenson] one more typo 9939d8d [Alex Levenson] fix typos in comments 61c0100 [Alex Levenson] Make initial slab size heuristic into a helper method, apply in DictionaryValuesWriter as well a257ee4 [Alex Levenson] Improve IndexOutOfBoundsException message 64d6c7f [Alex Levenson] update comments 8b54667 [Alex Levenson] Don't use CapacityByteArrayOutputStream for writing page chunks 6a20e8b [Alex Levenson] Remove initialSlabSize decision from InternalParquetRecordReader, use a simpler heuristic in the column writers instead 3a0f8e4 [Alex Levenson] Use simpler settings for column chunk writer b2736a1 [Alex Levenson] Some cleanup in CapacityByteArrayOutputStream 1df4a71 [julien] refactor CapacityByteArray to be aware of page size 95c8fb6 [julien] avoid wasting 64K per empty buffer.

@andredasilvapinto

This PR adds Hive (https://github.com/apache/hive) and Presto (https://github.com/prestodb/presto) support for parquet messages written with ProtoParquetWriter. Hive and other tools, such as Presto (used by AWS Athena), rely on specific LIST/MAP wrappers (as defined in the parquet spec: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md). These wrappers are currently missing from the ProtoParquet schema. AvroParquet works just fine, because it adds these wrappers when it deals with arrays and maps. This PR brings these wrappers in parquet-proto, providing the same functionality that already exists in parquet-avro. This is backward compatible. Messages written without the extra LIST/MAP wrappers are still being read successfully using the updated ProtoParquetReader. Regarding the change. Given the following protobuf schema: ``` message ListOfPrimitives { repeated int64 my_repeated_id = 1; } ``` Old parquet schema was: ``` message ListOfPrimitives { repeated int64 my_repeated_id = 1; } ``` New parquet schema is: ``` message ListOfPrimitives { required group my_repeated_id (LIST) = 1 { repeated group list { required int64 element; } } } ``` --- For list of messages, the changes look like this: Protobuf schema: ``` message ListOfMessages { string top_field = 1; repeated MyInnerMessage first_array = 2; } message MyInnerMessage { int32 inner_field = 1; } ``` Old parquet schema was: ``` message TestProto3.ListOfMessages { optional binary top_field (UTF8) = 1; repeated group first_array = 2 { optional int32 inner_field = 1; } } ``` The expected parquet schema, compatible with Hive (and similar to parquet-avro) is the following (notice the LIST wrapper): ``` message TestProto3.ListOfMessages { optional binary top_field (UTF8) = 1; required group first_array (LIST) = 2 { repeated group list { optional group element { optional int32 inner_field = 1; } } } } ``` --- Similar for maps. Protobuf schema: ``` message TopMessage { map<int64, MyInnerMessage> myMap = 1; } message MyInnerMessage { int32 inner_field = 1; } ``` Old parquet schema: ``` message TestProto3.TopMessage { repeated group myMap = 1 { optional int64 key = 1; optional group value = 2 { optional int32 inner_field = 1; } } } ``` New parquet schema (notice the `MAP` wrapper): ``` message TestProto3.TopMessage { required group myMap (MAP) = 1 { repeated group key_value { required int64 key; optional group value { optional int32 inner_field = 1; } } } } ``` Jira: https://issues.apache.org/jira/browse/PARQUET-968 Author: Constantin Muraru <[email protected]> Author: Benoît Hanotte <[email protected]> Closes #411 from costimuraru/PARQUET-968 and squashes the following commits: 16eafcb [Benoît Hanotte] PARQUET-968 add proto flag to enable writing using specs-compliant schemas (#2) a8bd704 [Constantin Muraru] Pick up commit from @andredasilvapinto 5cf9248 [Constantin Muraru] PARQUET-968 Add Hive support in ProtoParquet

@andredasilvapinto

This PR adds Hive (https://github.com/apache/hive) and Presto (https://github.com/prestodb/presto) support for parquet messages written with ProtoParquetWriter. Hive and other tools, such as Presto (used by AWS Athena), rely on specific LIST/MAP wrappers (as defined in the parquet spec: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md). These wrappers are currently missing from the ProtoParquet schema. AvroParquet works just fine, because it adds these wrappers when it deals with arrays and maps. This PR brings these wrappers in parquet-proto, providing the same functionality that already exists in parquet-avro. This is backward compatible. Messages written without the extra LIST/MAP wrappers are still being read successfully using the updated ProtoParquetReader. Regarding the change. Given the following protobuf schema: ``` message ListOfPrimitives { repeated int64 my_repeated_id = 1; } ``` Old parquet schema was: ``` message ListOfPrimitives { repeated int64 my_repeated_id = 1; } ``` New parquet schema is: ``` message ListOfPrimitives { required group my_repeated_id (LIST) = 1 { repeated group list { required int64 element; } } } ``` --- For list of messages, the changes look like this: Protobuf schema: ``` message ListOfMessages { string top_field = 1; repeated MyInnerMessage first_array = 2; } message MyInnerMessage { int32 inner_field = 1; } ``` Old parquet schema was: ``` message TestProto3.ListOfMessages { optional binary top_field (UTF8) = 1; repeated group first_array = 2 { optional int32 inner_field = 1; } } ``` The expected parquet schema, compatible with Hive (and similar to parquet-avro) is the following (notice the LIST wrapper): ``` message TestProto3.ListOfMessages { optional binary top_field (UTF8) = 1; required group first_array (LIST) = 2 { repeated group list { optional group element { optional int32 inner_field = 1; } } } } ``` --- Similar for maps. Protobuf schema: ``` message TopMessage { map<int64, MyInnerMessage> myMap = 1; } message MyInnerMessage { int32 inner_field = 1; } ``` Old parquet schema: ``` message TestProto3.TopMessage { repeated group myMap = 1 { optional int64 key = 1; optional group value = 2 { optional int32 inner_field = 1; } } } ``` New parquet schema (notice the `MAP` wrapper): ``` message TestProto3.TopMessage { required group myMap (MAP) = 1 { repeated group key_value { required int64 key; optional group value { optional int32 inner_field = 1; } } } } ``` Jira: https://issues.apache.org/jira/browse/PARQUET-968 Author: Constantin Muraru <[email protected]> Author: Benoît Hanotte <[email protected]> Closes apache#411 from costimuraru/PARQUET-968 and squashes the following commits: 16eafcb [Benoît Hanotte] PARQUET-968 add proto flag to enable writing using specs-compliant schemas (#2) a8bd704 [Constantin Muraru] Pick up commit from @andredasilvapinto 5cf9248 [Constantin Muraru] PARQUET-968 Add Hive support in ProtoParquet

ColumnIndexValidator public, remove fault injection, add index valida…

Use LRU caching for footers in ParquetInputFormat.

a363622

julienledem reviewed Jun 21, 2014
View reviewed changes

Add javadocs to parquet.hadoop.LruCache. Rename cache "entries" as ca…

d946445

…che "values" to avoid confusion with java.util.Map.Entry (which contains key value pairs whereas our old "entries" really only refer to the values).

matt-martin changed the title ~~Use LRU caching for footers in ParquetInputFormat.~~ PARQUET-4: Use LRU caching for footers in ParquetInputFormat. Jun 22, 2014

Be less aggressive about checking whether the underlying file has bee…

250a398

…n appended to/overwritten/deleted in order to minimize the number of namenode interactions.

Minor javadoc and whitespace changes. Also added the FileStatusWrappe…

99bb5a3

…r class to ParquetInputFormat to make sure that the debugging log statements print out meaningful paths.

liancheng mentioned this pull request Jul 11, 2014

PARQUET-16: Avoid calling getFileStatus() on all part-files #17

Closed

asfgit closed this in 5dffe35 Jul 18, 2014

rdblue mentioned this pull request Aug 2, 2016

PARQUET-601: Add support to configure the encoding used by ValueWriters #342

Closed

costimuraru referenced this pull request in costimuraru/parquet-mr Apr 29, 2017

Simplify #2

1b21f35

costimuraru referenced this pull request in costimuraru/parquet-mr Apr 29, 2017

Simplify #2

2781f77

ggershinsky mentioned this pull request Apr 30, 2018

PARQUET-968 Add Hive/Presto support in ProtoParquet ggershinsky/parquet-mr#1

Merged

gszadovszky pushed a commit to gszadovszky/parquet-mr that referenced this pull request Dec 13, 2018

Merge pull request apache#2 from gszadovszky/index-validator

dea4a2b

ColumnIndexValidator public, remove fault injection, add index valida…

shangxinli mentioned this pull request Oct 31, 2019

PARQUET-1685: Truncate Min/Max for Statistics #696

Merged

1 task

shangxinli mentioned this pull request May 10, 2020

PARQUET-1827: UUID type currently not supported by parquet-mr #778

Merged

4 tasks

shangxinli mentioned this pull request Jun 20, 2020

Parquet-1872: Add TransCompression command to parquet-tools #796

Merged

4 tasks

asfimport mentioned this pull request Jun 23, 2024

String decode using 'new String' is slow #1532

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PARQUET-4: Use LRU caching for footers in ParquetInputFormat. #2

PARQUET-4: Use LRU caching for footers in ParquetInputFormat. #2

matt-martin commented Jun 19, 2014

matt-martin commented Jun 19, 2014

dvryaboy commented Jun 19, 2014

julienledem commented Jun 20, 2014

matt-martin commented Jun 20, 2014

julienledem commented Jun 21, 2014

julienledem Jun 21, 2014

julienledem commented Jun 21, 2014

julienledem commented Jun 23, 2014

matt-martin commented Jun 23, 2014

matt-martin commented Jun 25, 2014

matt-martin commented Jul 18, 2014

julienledem commented Jul 18, 2014

matt-martin commented Jul 26, 2014

PARQUET-4: Use LRU caching for footers in ParquetInputFormat. #2

PARQUET-4: Use LRU caching for footers in ParquetInputFormat. #2

Conversation

matt-martin commented Jun 19, 2014

matt-martin commented Jun 19, 2014

dvryaboy commented Jun 19, 2014

julienledem commented Jun 20, 2014

matt-martin commented Jun 20, 2014

julienledem commented Jun 21, 2014

julienledem Jun 21, 2014

Choose a reason for hiding this comment

julienledem commented Jun 21, 2014

julienledem commented Jun 23, 2014

matt-martin commented Jun 23, 2014

matt-martin commented Jun 25, 2014

matt-martin commented Jul 18, 2014

julienledem commented Jul 18, 2014

matt-martin commented Jul 26, 2014