[SPARK-4502][SQL] Parquet nested column pruning#16578

Closed

mallman wants to merge 25 commits intoapache:masterfrom

VideoAmp:spark-4502-parquet_column_pruning

Commits on Apr 24, 2018

[SPARK-4502][SQL] Parquet nested column pruning

Michael Allman
authored and
mallman
committed
Replace '* with "*" for better github syntax highlighting

Michael Allman
authored and
mallman
committed
Execute some refactorings in ParquetSchemaPruning.scala to simplify it

Michael Allman
authored and
mallman
committed
Fix SelectedField extractor to correctly handle the case of a nested

Michael Allman
authored and
mallman
committed
Add a ColumnCount metadata property to the FileSourceScanExec to count

Michael Allman
authored and
mallman
committed
WIP

Michael Allman
authored and
mallman
committed
WIP

Michael Allman
authored and
mallman
committed
Refine and (attempt to) clarify SelectedField.scala

Michael Allman
authored and
mallman
committed
"Fix" scalastyle errors in SelectedFieldSuite.scala

Michael Allman
authored and
mallman
committed
Adjust some code in ParquetSchemaPruning.scala to conform to external
mallman
committed
(Updates from PR review feedback)
mallman
committed
Document ColumnarFileFormat and make it private[sql]
mallman
committed
Update documentation of `QueryPlanConstraints.scala` to cover case of
mallman
committed
Compare the parquet file schema and pruned parquet schema in
mallman
committed
Update order of fields in expected schema in some test cases in
mallman
committed
Replace nonsensical example data in ParquetSchemaPruningSuite.scala with
mallman
committed
Fix Filter null check.

viirya
authored and
mallman
committed
Alter some code docs to change terminology from "proper field" to
mallman
committed
Forgo the use of `ParquetSchemaConverter` when counting the number of
mallman
committed
Enhance documentation and skip application of the
mallman
committed
Don't print the schema before running the tests in the SelectedFieldSuite
mallman
committed
Put the schema pruning optimizer rules behind a boolean SQLConf flag,
mallman
committed
Remark in the documentation of the spark.sql.nestedSchemaPruning.enabled
mallman
committed
Don't set catalogTable to None when copying an existing logical relation
mallman
committed
Add a SchemaPruningTest trait for running tests with schema pruning
mallman
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-4502][SQL] Parquet nested column pruning#16578

[SPARK-4502][SQL] Parquet nested column pruning#16578
mallman wants to merge 25 commits intoapache:masterfrom
VideoAmp:spark-4502-parquet_column_pruning

Commits on Apr 24, 2018

[SPARK-4502][SQL] Parquet nested column pruning

Replace '* with "*" for better github syntax highlighting

Execute some refactorings in ParquetSchemaPruning.scala to simplify it

Fix SelectedField extractor to correctly handle the case of a nested

Add a ColumnCount metadata property to the FileSourceScanExec to count

WIP

WIP

Refine and (attempt to) clarify SelectedField.scala

"Fix" scalastyle errors in SelectedFieldSuite.scala

Adjust some code in ParquetSchemaPruning.scala to conform to external

(Updates from PR review feedback)

Document ColumnarFileFormat and make it private[sql]

Update documentation of `QueryPlanConstraints.scala` to cover case of

Compare the parquet file schema and pruned parquet schema in

Update order of fields in expected schema in some test cases in

Replace nonsensical example data in ParquetSchemaPruningSuite.scala with

Fix Filter null check.

Alter some code docs to change terminology from "proper field" to

Forgo the use of `ParquetSchemaConverter` when counting the number of

Enhance documentation and skip application of the

Don't print the schema before running the tests in the SelectedFieldSuite

Put the schema pruning optimizer rules behind a boolean SQLConf flag,

Remark in the documentation of the spark.sql.nestedSchemaPruning.enabled

Don't set catalogTable to None when copying an existing logical relation

Add a SchemaPruningTest trait for running tests with schema pruning

[SPARK-4502][SQL] Parquet nested column pruning#16578mallman wants to merge 25 commits intoapache:masterapache/spark:masterfrom VideoAmp:spark-4502-parquet_column_pruningVideoAmp/spark-public:spark-4502-parquet_column_pruningCopy head branch name to clipboard

Commits

Commits on Apr 24, 2018

[SPARK-4502][SQL] Parquet nested column pruning#16578
mallman wants to merge 25 commits intoapache:masterfrom
VideoAmp:spark-4502-parquet_column_pruning