Dereference projections and predicate pushdown in Hive by phd3 · Pull Request #1720 · trinodb/trino

phd3 · 2019-10-11T02:31:06Z

(required by #1953)

This PR includes the following major changes:

1. Add support for projected columns in Hive

Page sources need to be supplied projected columns representing internal fields from a struct column. We modify HiveColumnHandle to include this information about column subfield access. Right now, the HiveColumnProjectionInfo simply stores a chain of dereferences, but that can later be extended to support other types of partial columns (eg. subscripts)
Internal page sources in Hive Connector may or may not be able to push down projected column reads, depending on the formats. Reader page sources or record cursor can provide the partial projections they'll apply, and the rest is "adapted" by the HivePageSource or HiveReaderProjectionsAdaptingRecordCursor.
By default, the readers only read base columns for projected columns. For example, column "a" will be read for a projection "a.x". This is implemented through ReaderProjections#projectBaseColumns

2. Pushdown for Dereferences in Hive:

The goal is to pushdown dereference expressions to Hive Connector for projections and predicates both.
As described in Pushdown of dereference expressions (with focus on Hive connector) #1953, creating partial column handles in HiveMetadata#applyProjection will give us both predicate and projection pushdown into Hive through iterative optimization.
HiveMetadata#applyProjection extracts longest simple dereference chains and variables from input expressions and provies new columnhandles for them. All the columnhandles returned by this method are unique.

3. Projection and Predicate pushdown of subfields in Columnar Sources:

In case of Columnar formats (eg. ORC, Parquet), the partial columns can be read without reading entire columns, which is the primary motive here.
We implement projectSufficientColumns for such page sources to leverage, which lets the pagesource only read the required superset columns. Say we have a column C of type RowType. The query asks for a set of subcolumns S from column C. Then the superset columns here is the smallest set of columns from S such that all columns in S can be derived using identity or dereference expressions. i.e when S = {"a.b.c", "a.d" and "a.b.c.x"}, the reader will read create columns {"a.b.c", "a.d"}. The HivePageSource will adapt for ".x" for the third column.
ORC and Parquet changes are extracted out in different PRs.

martint

A couple of initial comments that will affect the overall approach and scope. I'm reviewing the core logic now.

presto-spi/src/main/java/io/prestosql/spi/expression/ConnectorExpressionVisitor.java

presto-hive/src/main/java/io/prestosql/plugin/hive/HiveMetadata.java

wagnermarkd

Left a few comments. I don't think the DereferenceAdaption has the desired effect. More details inline.

W.r.t. sequencing, can we handle partial pushdown or unique pushdowns in a separate RB?

presto-hive/src/main/java/io/prestosql/plugin/hive/HiveColumnHandle.java

presto-hive/src/main/java/io/prestosql/plugin/hive/HiveMetadata.java

presto-hive/src/main/java/io/prestosql/plugin/hive/HivePageSourceProvider.java

phd3 · 2019-10-25T19:40:24Z

@wagnermarkd split the PR into two parts. #1868 deals with the engine side changes.

phd3 · 2019-11-13T20:07:36Z

@wagnermarkd @martint this PR is ready for review again.

martint

Just a few minor comments.

presto-hive/src/main/java/io/prestosql/plugin/hive/parquet/ParquetPageSourceFactory.java

presto-hive/src/main/java/io/prestosql/plugin/hive/rcfile/RcFilePageSourceFactory.java

presto-hive/src/main/java/io/prestosql/plugin/hive/s3select/S3SelectRecordCursorProvider.java

presto-hive/src/main/java/io/prestosql/plugin/hive/HiveApplyProjectionUtil.java

martint · 2020-03-18T19:17:03Z

...o-hive/src/main/java/io/prestosql/plugin/hive/HiveReaderProjectionsAdaptingRecordCursor.java

I think you forgot to address this comment.

presto-hive/src/main/java/io/prestosql/plugin/hive/ReaderProjections.java

martint · 2020-03-18T19:19:24Z

presto-hive/src/main/java/io/prestosql/plugin/hive/HiveApplyProjectionUtil.java


    /**
     * Returns the assignment key corresponding to the column represented by {@param projectedColumn} in the {@param assignments}, if one exists.
+     * The variable in the {@param projectedColumn} can itself be a representation of another projected column. For example,


Does this comment reflect new behavior? If not, move it to the commit that introduced the method.

resolved the mixup: 26b83a5#diff-537e8077e259dbc890e77332fc3fa6ceR105

presto-hive/src/main/java/io/prestosql/plugin/hive/HiveMetadata.java

phd3 · 2020-03-30T19:10:35Z

@martint Thanks for the review, addressed your comments. The test failure looks unrelated, and doesn't seem to reproduce locally.

There're two main other changes during the rebase:

Excluding test for applyProjection from alluxio metastore since it involves creating a table
Resolving conflicts to use TableToPartitionMapping introduced instead of coercions in HivePageSourceProvider::buildColumnMappings.

phd3 · 2020-02-11T22:30:55Z

presto-hive/src/main/java/io/prestosql/plugin/hive/HiveColumnHandle.java

should be the same fields as equals

phd3 · 2020-02-11T22:51:59Z

presto-hive/src/main/java/io/prestosql/plugin/hive/IonSqlQueryBuilder.java

Add a check for column being a base hive column

phd3 · 2020-03-30T17:52:29Z

presto-hive/src/main/java/io/prestosql/plugin/hive/HiveApplyProjectionUtil.java


    /**
     * Returns the assignment key corresponding to the column represented by {@param projectedColumn} in the {@param assignments}, if one exists.
+     * The variable in the {@param projectedColumn} can itself be a representation of another projected column. For example,


resolved the mixup: 26b83a5#diff-537e8077e259dbc890e77332fc3fa6ceR105

phd3 · 2020-03-30T17:53:58Z

...o-hive/src/main/java/io/prestosql/plugin/hive/HiveReaderProjectionsAdaptingRecordCursor.java

resolved the mixup: 8dbd476#diff-9008b1573bb475fa173bd5917b1f249aR183

presto-hive/src/main/java/io/prestosql/plugin/hive/HiveMetadata.java

This method enables creating only the minimally required columns for reading a set of column handles". e.g. for hive columns ["a.b", "a", "c"], this method will create projections using columns "a" and "c", since "a.b" can be projected from "a".

cla-bot bot added the cla-signed label Oct 11, 2019

phd3 mentioned this pull request Oct 11, 2019

Add equals and hashcode methods to ConnectorExpression #1712

Merged

martint self-requested a review October 11, 2019 16:30

martint reviewed Oct 16, 2019

View reviewed changes

presto-spi/src/main/java/io/prestosql/spi/expression/ConnectorExpressionVisitor.java Outdated Show resolved Hide resolved

presto-hive/src/main/java/io/prestosql/plugin/hive/HiveMetadata.java Outdated Show resolved Hide resolved

phd3 force-pushed the hive-deref-pushdown branch from 1787d4b to fb292f9 Compare October 24, 2019 00:31

phd3 changed the title ~~WIP dereference pushdown~~ Dereference pushdown Oct 24, 2019

phd3 changed the title ~~Dereference pushdown~~ Connector Dereference pushdown Oct 24, 2019

wagnermarkd reviewed Oct 24, 2019

View reviewed changes

phd3 mentioned this pull request Oct 25, 2019

Partial projections pushdown #1868

Merged

phd3 force-pushed the hive-deref-pushdown branch from fb292f9 to d6c5c76 Compare October 25, 2019 19:37

phd3 changed the title ~~Connector Dereference pushdown~~ Dereference projections pushdown in Hive Oct 25, 2019

phd3 mentioned this pull request Nov 4, 2019

Pushdown of dereference expressions (with focus on Hive connector) #1953

Closed

10 tasks

phd3 force-pushed the hive-deref-pushdown branch from d6c5c76 to 92a2dd9 Compare November 9, 2019 02:00

phd3 added the WIP label Nov 9, 2019

phd3 self-assigned this Nov 9, 2019

phd3 force-pushed the hive-deref-pushdown branch 7 times, most recently from d80cbb6 to c84be63 Compare November 13, 2019 20:00

phd3 removed the WIP label Nov 13, 2019

phd3 force-pushed the hive-deref-pushdown branch 4 times, most recently from 9725ec7 to 112af55 Compare November 18, 2019 23:20

phd3 changed the title ~~Dereference projections pushdown in Hive~~ Dereference projections and predicate pushdown in Hive Nov 19, 2019

phd3 force-pushed the hive-deref-pushdown branch from b65f374 to 6bd7a15 Compare February 26, 2020 18:04

JamesRTaylor mentioned this pull request Mar 17, 2020

Make Parquet changes related to #1720 #3116

Closed

JamesRTaylor pushed a commit to lyft/presto that referenced this pull request Mar 17, 2020

Dereference projections and predicate pushdown in Hive (trinodb#1720)

3d92cc5

martint reviewed Mar 18, 2020

View reviewed changes

JamesRTaylor pushed a commit to lyft/presto that referenced this pull request Mar 22, 2020

Dereference projections and predicate pushdown in Hive (trinodb#1720)

d33f832

phd3 force-pushed the hive-deref-pushdown branch 3 times, most recently from 826f84b to cfbbc51 Compare March 26, 2020 06:41

JamesRTaylor mentioned this pull request Mar 27, 2020

Dereference Projection Pushdown in Query Plan #2672

Merged

phd3 force-pushed the hive-deref-pushdown branch from cfbbc51 to 9769f3a Compare March 29, 2020 23:19

phd3 commented Mar 30, 2020

View reviewed changes

phd3 force-pushed the hive-deref-pushdown branch from 9769f3a to c8eb95f Compare March 31, 2020 23:54

phd3 added 2 commits March 31, 2020 23:35

Fix spelling in PlanOptimizers class

aa6c54e

Encode BigDecimal to Slice in StructuralTestUtil::appendToBlockBuilder

62a82f7

phd3 force-pushed the hive-deref-pushdown branch from c8eb95f to 8b7f573 Compare April 1, 2020 06:35

phd3 added 9 commits April 6, 2020 23:05

Support projected columns in Hive

de5d550

Implement pushdown of dereference projections into hive connector

29a3f67

Add dereference pushdown integration tests for all hive formats

4479caa

Create uniquely named test columns in TestOrcPageSourceMemoryTracking

77590e9

Add plan and rule test for Hive Projection Pushdown

3cbeec4

Add tests for schema mismatch with dereference projections

c80c121

Remove unused parameter in createTestFile method

90243f3

Fix inaccurate test assertion message

1dc3447

This was referenced Apr 16, 2020

Pushdown dereferences in ORC Reader #3445

Merged

Fix incorrect computation of reader columns for S3SelectRecordCursor #3446

Merged

Fire PushPredicateIntoTableScan rule with change in TableHandle #3470

Merged

phd3 mentioned this pull request Sep 15, 2020

Dereference Pushdown for Iceberg Connector #5179

Closed

findepi mentioned this pull request Mar 14, 2022

Make ConnectorExpression JSON serializable #11447

Closed

Conversation

phd3 commented Oct 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

martint left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

wagnermarkd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

phd3 commented Oct 25, 2019

Uh oh!

phd3 commented Nov 13, 2019

Uh oh!

martint left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

martint Mar 18, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

martint Mar 18, 2020

Choose a reason for hiding this comment

Uh oh!

phd3 Mar 30, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

phd3 commented Mar 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

phd3 Feb 11, 2020

Choose a reason for hiding this comment

Uh oh!

phd3 Feb 11, 2020

Choose a reason for hiding this comment

Uh oh!

phd3 Mar 30, 2020

Choose a reason for hiding this comment

Uh oh!

phd3 Mar 30, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

phd3 commented Oct 11, 2019 •

edited

Loading

phd3 commented Mar 30, 2020 •

edited

Loading