Support Parquet TupleDomain using ColumnDescriptor

[zhenxiao]

Currently Parquet TupleDomain is constructed based on HiveColumnHandle. This would not work if Nested predicate are pushed down, e.g.

select s.a
from t
where s.b > 10

This patch construct Parquet TupleDomain with Parquet's ColumnDescriptor, so that it could work with nested predicate pushdown

[zhenxiao]

@nezihyigitbasi @dain this is a first step to support nested predicate pushdown for Parquet in Presto. This PR is to make Parquet TupleDomain based on Parquet ColumnDescriptor, so that when nested predicates are pushed down, it could easily work

[zhenxiao]

@dain @nezihyigitbasi kindly ping

[zhenxiao]

@dain @nezihyigitbasi kindly ping

[zhenxiao]

@dain @nezihyigitbasi could you please review when you are free?

[zhenxiao]

@nezihyigitbasi @dain kindly ping

[zhenxiao]

@nezihyigitbasi @dain kindly ping

[dain]

In an offline discussion we decided that @zhenxiao was going to investigate using synthetic virtual-columns in the connector to enable this push-down feature.

[zhenxiao]

Hi @dain:
This PR is to make Parquet TupleDomain based on Parquet ColumnDescriptor, instead of index based. It is purely Parquet stuff. This is a preparation for nested predicate pushdown stuff. It is beneficial for existing Parquet stuff as well.

Could you please review this first?

[dain]

Ah ok. Will do.

[zhenxiao]

@dain @nezihyigitbasi kindly ping

[nezihyigitbasi]

@zhenxiao I will take a look at this shortly.

[nezihyigitbasi]

@zhenxiao I did a first pass, back to you.

[nezihyigitbasi]

This doesn't handle the decimal type correctly, please see Parquet spec and my recent related fix 37c57c9

[zhenxiao]

thank you, get it resolved

[nezihyigitbasi]

now that the typeManager is not used anymore in this method we can remove it.

[zhenxiao]

get it, resolved

[nezihyigitbasi]

static import Map.Entry

[zhenxiao]

yep, updated

[nezihyigitbasi]

This class already has access to the column descriptor so we shouldn't need to pass the Presto type as well since Presto type can be derived from that (like you do in getPrestoType()). So I guess we can move getPrestoType() logic to this class.

[zhenxiao]

get it. this class not needed at all. Will delete and use RichColumnDescriptor directly

[nezihyigitbasi]

@param statistics column statistics

[zhenxiao]

resolved

[nezihyigitbasi]

I don't think you need the paths variable, you can just pass Arrays.asList(columnMetaData.getPath().toArray()) below.

[zhenxiao]

get it. Updated

[nezihyigitbasi]

Although not entirely related to this patch, why are we ignoring the IOException here? If we get an exception we will silently skip populating dictionaries for that particular column.

[zhenxiao]

if hitting any problem when trying to read the dictionary page, we will just silently skip, and will not use that dictionary to try skip reading row groups

[nezihyigitbasi]

OK as long as we are not messing up while we read.

[nezihyigitbasi]

What will happen if the descriptor is not present and we return an empty map of dictionaries?

[zhenxiao]

only build dictionaries when descriptor exist, otherwise, dictionaries is empty, will scan file, dictionary predicate will not apply

[nezihyigitbasi]

getDomains() returns an Optional, are we sure it's non-empty? If yes, we should better assert that (com.google.common.base.Verify::verify).

[nezihyigitbasi]

throw new PrestoException(NOT_SUPPORTED, "Unsupported parquet type: " + descriptor.getType());

[zhenxiao]

yep, updated

[zhenxiao]

thank you @nezihyigitbasi
get comments addressed

[nezihyigitbasi]

LGTM except some minor comments. BTW do we need similar changes in the new reader for nested predicate pushdown support?

[nezihyigitbasi]

now that we have this method in ParquetTypeUtils we can update ParquetColumnReader to also call this one instead of maintaining two copies.

[zhenxiao]

yep, get it updated

[nezihyigitbasi]

"parquetTupleDomain is empty"

[zhenxiao]

thank you @nezihyigitbasi get comments addressed
Yes, need to do something like virtual column to push down nested columns. New Parquet Reader part support is there

[nezihyigitbasi]

@dain I think this looks good.

[dain]

Just one minor comment. Otherwise, Nezih, merge whenever you want.

[dain]

Can we rename i and j to something more descriptive? Maybe columnIndex and level (and rename level to maxLevel)?

[zhenxiao]

thank you @dain @nezihyigitbasi
get comments addressed

[nezihyigitbasi]

thanks @zhenxiao I will merge this once the tests all pass.

[shurvitz]

Hi @zhenxiao ,
I have a question regarding this change to getDictionaries(). It looks like the change folded a loop inside a loop into a single loop. What I'm noticing in Presto217 is a decrease in bytes-scanned during the predicate push-down process. This is the end result of that loop hard-existing via break after the first predicate descriptor is matched and a dictionary page is added. Before the change, the loop would iterate over all descriptors... Was leaving the break statement intentional?

[zhenxiao]

Hi @shurvitz thanks for reaching out.
I did not quite get ur question, this PR is actually a refactoring, leveraging ColumnDescriptor in Parquet, instead of integer index, for dictionary pushdown and predicate pushdown.
Could you please elaborate more, with file line number?

[zhenxiao]

Get it. Thank you, @shurvitz @pettyjamesm