Benchmark for Hive Projection Pushdown by phd3 · Pull Request #2307 · trinodb/trino

phd3 · 2019-12-18T18:35:32Z

This PR adds benchmarks for the Hive Projection pushdown changes in #1720. The change in #1720 enables reading only the projected subfields for row type columns. cc @martint

This benchmark can be run to validate the dereference projection pushdown performance for different file formats, as they are implemented. I think the following comparisons can help us be confident about the anticipated benefits of projection pushdown.

Dereference projection pushdown into readers results in performance gain proportional to the size of the row type column. i.e. if we have a column a of row type containing N varchar fields, then accessing the virtual column for "a.b" should be ~N times faster than accessing column a. With nested row type, the throughput increase should be even more drastic.
Performance comparison between accessing a dereferenced field v/s accessing a base column of the same type (eg. "a.b" of type VARCHAR vs "c" of type VARCHAR)

Benchmark Setup and parameters

The benchmark writes some data in the setup phase and then measures throughput of operations while reading pages. Here's some explanation of parameters used in the benchmark.

columnTypeString : type of STRUCT column to write.
writeStrategy: Decides the writing schema

STRUCT : the schema is a single column with the columnTypeString type
TOP_LEVEL : the schema consists of columns with types given by the fields of columnTypeString.

readStrategy: Decides whether we project columns or not. No difference in case of TOP_LEVEL write strategy

WITH_PUSHDOWN : read projected columns
WITHOUT_PUSHDOWN : read struct column

readColumnCount : Number of columns/fields to read

Some benchmark Results

Say a represents the struct, a.x represents the primitive field. b represents the primitive top-level column.

ORC

Parquet

Benchmark                                                                                                          (columnTypeString)    (fileFormat)  (readColumnCount)    (readStrategy)  (writeStrategy)   Mode  Cnt   Score   Error  Units
BenchmarkProjectionPushdownHive.readPages                                                     ROW(f0 VARCHAR, f1 VARCHAR, f2 VARCHAR)      PRESTO_ORC                  1  without_pushdown           struct  thrpt  150  16.931 ± 0.228  ops/s
BenchmarkProjectionPushdownHive.readPages                                                     ROW(f0 VARCHAR, f1 VARCHAR, f2 VARCHAR)      PRESTO_ORC                  1  without_pushdown         toplevel  thrpt  150  49.408 ± 0.736  ops/s
BenchmarkProjectionPushdownHive.readPages                                                     ROW(f0 VARCHAR, f1 VARCHAR, f2 VARCHAR)      PRESTO_ORC                  1     with_pushdown           struct  thrpt  150  45.070 ± 1.316  ops/s
BenchmarkProjectionPushdownHive.readPages                                                     ROW(f0 VARCHAR, f1 VARCHAR, f2 VARCHAR)      PRESTO_ORC                  1     with_pushdown         toplevel  thrpt  150  48.390 ± 0.524  ops/s
BenchmarkProjectionPushdownHive.readPages                                                     ROW(f0 VARCHAR, f1 VARCHAR, f2 VARCHAR)  PRESTO_PARQUET                  1  without_pushdown           struct  thrpt  150  21.359 ± 0.238  ops/s
BenchmarkProjectionPushdownHive.readPages                                                     ROW(f0 VARCHAR, f1 VARCHAR, f2 VARCHAR)  PRESTO_PARQUET                  1  without_pushdown         toplevel  thrpt  150  61.866 ± 0.774  ops/s
BenchmarkProjectionPushdownHive.readPages                                                     ROW(f0 VARCHAR, f1 VARCHAR, f2 VARCHAR)  PRESTO_PARQUET                  1     with_pushdown           struct  thrpt  150  59.709 ± 0.853  ops/s
BenchmarkProjectionPushdownHive.readPages                                                     ROW(f0 VARCHAR, f1 VARCHAR, f2 VARCHAR)  PRESTO_PARQUET                  1     with_pushdown         toplevel  thrpt  150  60.416 ± 0.826  ops/s
BenchmarkProjectionPushdownHive.readPages  ROW(f0 VARCHAR, f1 VARCHAR, f2 VARCHAR, f3 ARRAY(ROW(f0 VARCHAR, f1 VARCHAR, f2 VARCHAR)))      PRESTO_ORC                  1  without_pushdown           struct  thrpt  150   0.544 ± 0.006  ops/s
BenchmarkProjectionPushdownHive.readPages  ROW(f0 VARCHAR, f1 VARCHAR, f2 VARCHAR, f3 ARRAY(ROW(f0 VARCHAR, f1 VARCHAR, f2 VARCHAR)))      PRESTO_ORC                  1  without_pushdown         toplevel  thrpt  150  44.182 ± 0.551  ops/s
BenchmarkProjectionPushdownHive.readPages  ROW(f0 VARCHAR, f1 VARCHAR, f2 VARCHAR, f3 ARRAY(ROW(f0 VARCHAR, f1 VARCHAR, f2 VARCHAR)))      PRESTO_ORC                  1     with_pushdown           struct  thrpt  150  41.744 ± 0.474  ops/s
BenchmarkProjectionPushdownHive.readPages  ROW(f0 VARCHAR, f1 VARCHAR, f2 VARCHAR, f3 ARRAY(ROW(f0 VARCHAR, f1 VARCHAR, f2 VARCHAR)))      PRESTO_ORC                  1     with_pushdown         toplevel  thrpt  150  43.299 ± 0.575  ops/s
BenchmarkProjectionPushdownHive.readPages  ROW(f0 VARCHAR, f1 VARCHAR, f2 VARCHAR, f3 ARRAY(ROW(f0 VARCHAR, f1 VARCHAR, f2 VARCHAR)))  PRESTO_PARQUET                  1  without_pushdown           struct  thrpt  150   0.597 ± 0.008  ops/s
BenchmarkProjectionPushdownHive.readPages  ROW(f0 VARCHAR, f1 VARCHAR, f2 VARCHAR, f3 ARRAY(ROW(f0 VARCHAR, f1 VARCHAR, f2 VARCHAR)))  PRESTO_PARQUET                  1  without_pushdown         toplevel  thrpt  150  58.687 ± 0.491  ops/s
BenchmarkProjectionPushdownHive.readPages  ROW(f0 VARCHAR, f1 VARCHAR, f2 VARCHAR, f3 ARRAY(ROW(f0 VARCHAR, f1 VARCHAR, f2 VARCHAR)))  PRESTO_PARQUET                  1     with_pushdown           struct  thrpt  150  59.964 ± 0.590  ops/s
BenchmarkProjectionPushdownHive.readPages  ROW(f0 VARCHAR, f1 VARCHAR, f2 VARCHAR, f3 ARRAY(ROW(f0 VARCHAR, f1 VARCHAR, f2 VARCHAR)))  PRESTO_PARQUET                  1     with_pushdown         toplevel  thrpt  150  56.636 ± 1.347  ops/s

sopel39 · 2019-12-19T09:37:15Z

How does Hive projection pushdown relates to lazy nested blocks? From high-level perspective, they both seem to do the same.

phd3 · 2019-12-21T06:36:44Z

@sopel39 They both attempt to optimize the reads for nested types, but I believe there're some differences. We implement the hive dereference projection pushdown by creating virtual handles for "projected" columns, such as "a.b". This allows projections as well as predicates on such columns to be pushed down . They can be used as predicates in the readers. For example, files/stripes can be skipped in case of ORC format, based on the predicates on nested columns. As an example, consider the following query:

Filter: [x.a = 5 and f(y) = 2]
   Scan: Table T, [ x→column(x), y --> column(y)] , no_constraint

Optimizer rule application:

i. PushdownDereferences rule (in the plan, #1435)

Filter: [expr = 5 and f(y) = 2]
    Project: [expr := x.a, y := y]
        Scan: Table T, [x --> column(x), y --> column(y)], no_constraint

ii. PushProjectionIntoTableScan rule (creates virtual column handle for x.a)

Filter: [expr = 5 and f(y) = 2]
    Project: [expr:=expr_2, y:= y]	
        Scan: Table T, [expr_2 --> column(x_a), y --> column(y)], no_constraint

iii. PredicatePushdown optimizer

Project: [expr:=expr_2, y:= y]	
    Filter: [expr_2 = 5 and f(y) = 2]
        Scan: Table T, [expr_2 --> column(x_a), y --> column(y)], no_constraint

iv. PushPredicateIntoTableScan:

Project: [expr:=expr_2, y:= y]	
    Filter: [expr_2 = 5 and f(y) = 2]
        Scan: Table T, [expr_2 --> column(x_a), y --> column(y)], constraint : expr_2 = 5

The optimization around skipping a stripe/file has to done with the help of pushdowns during planning, and not something that can be achieved by loading blocks lazily. Another approach could be predicate pushdowns by supporting dereference projections in HiveMetadata::applyFilter, but then it would also require storing some representation of nested field in the tupledomains. The approach taken in #1720 achieves this by embedding the nested field representation inside hive column handle, which gives us both projection and predicate pushdown. In future, we might want to push down other projections in hive, and I think this would set up the basis for such work.

I looked into the lazy nested blocks some time back, but didn't feel that it's giving us the desired advantage, feel free to correct my understanding. Let's say we have a query SELECT col_2.field_1 from T. Here T has ORC fileformat and col_2 is of type ROW(field_1 BIGINT, field_2 BIGINT). So we'd want to read only the subfield block corresponding to col_2.field1. But that doesn't seem to be the case. blocks for field_1 and field_2 are both loaded since getLoadedBlock is invoked on the row's lazy block in DictionaryAwarePageProjection. I think for the lazy nested blocks to work, we'd need to apply the dereference operation at block level, using something like ColumnarRow.

sopel39 · 2019-12-23T09:50:41Z

@phd3 Thank you for throughout explanation. Indeed, ability to pushdown predicates for dereferenced expression is really valuable.

sopel39 · 2019-12-31T13:45:17Z

there are build failures

phd3 · 2020-01-01T02:28:38Z

@sopel39 this is dependent on #1720, we'd need to wait for that to get checked in first.

phd3 · 2020-07-22T18:40:02Z

@sopel39 the failures seem to be unrelated.

presto-hive/src/test/java/io/prestosql/plugin/hive/benchmark/FileFormat.java

...o-hive/src/test/java/io/prestosql/plugin/hive/benchmark/BenchmarkProjectionPushdownHive.java

presto-hive/src/test/java/io/prestosql/plugin/hive/benchmark/FileFormat.java

phd3 · 2020-09-02T21:19:14Z

@sopel39 thanks for the review, addressed your comments.

sopel39

lgtm % comments

sopel39 · 2020-09-07T14:49:10Z

presto-hive/src/test/java/io/prestosql/plugin/hive/benchmark/BenchmarkHiveFileFormatUtil.java

+{
+    private BenchmarkHiveFileFormatUtil() {}
+
+    protected static class TestData


make the class top level and call it BenchmarkHiveFileTestData. Also why it has to be top-level class?

Since we're using TestData in multiple independent classes, I thought it might be better to extract it out. I hadn't thought about making it top-level, but it makes sense with the name you suggested, rather than putting in a random util class.

presto-hive/src/test/java/io/prestosql/plugin/hive/benchmark/FileFormat.java

sopel39 · 2020-09-07T14:55:21Z

presto-hive/src/test/java/io/prestosql/plugin/hive/benchmark/FileFormat.java

    }

-    private static ConnectorPageSource createPageSource(
+    private static ConnectorPageSource createPageSourceForBaseColumns(


What are baseColumns? Undo this rename?

I had added ForBaseColumns to indicate that the provided column names to this method will always correspond to top-level (or base) columns, not projected ones. For example, createFileFormatReader should not be given a projected column (like "a.b"), since the createPageSource method is only capable of reading "a", and will leave up to the hive page source wrapper to extract out the field. createGenericReader uses the column handles constructed for such nested fields.

However, this method doesn't have a way to provide projected handles, so undoing the rename.

sopel39 · 2020-09-07T15:00:41Z

presto-hive/src/test/java/io/prestosql/plugin/hive/benchmark/FileFormat.java

+        return Optional.empty();
+    }
+
+    public final ConnectorPageSource createFileFormatReader(


Actually, do we need this method if we have createGenericReader? Would createGenericReader create page source differently than this method?

createHiveReader returns a format-specific reader with the hive connector's wrapper page source.

HivePageSourceProvider should first try to instantiate native reader

HivePageSourceProvider should first try to instantiate native reader

Currently HivePageSourceProvider::createPageSource does that implicitly, when provided with the factories. So simulating that flow requires providing a factory, without instantiating the reader. Not sure if I understand your point correctly.

Would createGenericReader create page source differently than this method?

The method is the same, the pagesource returned by createGenericReader may incur some overhead in addition to the processing done by the delegate readers returned in createFileFormatReader. I kept createFileFormatReader thinking that the existing benchmarks testing the file-formate specific readers would want numbers without the HivePageSource's overhead.

presto-hive/src/test/java/io/prestosql/plugin/hive/benchmark/FileFormat.java

...o-hive/src/test/java/io/prestosql/plugin/hive/benchmark/BenchmarkProjectionPushdownHive.java

sopel39 · 2020-09-07T15:09:19Z

...o-hive/src/test/java/io/prestosql/plugin/hive/benchmark/BenchmarkProjectionPushdownHive.java

+@Measurement(iterations = 50)
+@Warmup(iterations = 20)
+@Fork(3)
+public class BenchmarkProjectionPushdownHive


are there some parts of code we could unify with BenchmarkHiveFileFormat?

hmm, I thought about this, but feel that keeping the two classes separate would be cleaner. BenchmarkHiveFileFormat is more about looking at reader performances across different types/compressions/formats, whereas BenchmarkProjectionPushdownHive documents the performance of the adaptations.

code-wise, I don't think there is much duplication in the two classes.

sopel39 · 2020-09-09T11:35:26Z

presto-hive/src/test/java/io/prestosql/plugin/hive/benchmark/BenchmarkHiveFileFormat.java

@@ -504,44 +504,6 @@ private static <E extends TpchEntity> TestData createTpchDataSet(FileFormat form
        return new TestData(columnNames, columnTypes, pages.build());


change commit title. It now extracts TestData as top-level class

good catch, fixed.

sopel39 · 2020-09-09T11:47:38Z

small comment + CI fails

sopel39 · 2020-09-10T11:24:36Z

merged, thanks!

cla-bot bot added the cla-signed label Dec 18, 2019

phd3 mentioned this pull request Dec 18, 2019

Dereference projections and predicate pushdown in Hive #1720

Merged

phd3 mentioned this pull request Feb 18, 2020

Pushdown of dereference expressions (with focus on Hive connector) #1953

Closed

10 tasks

phd3 force-pushed the cherry-picked-benchmark-for-1720 branch 5 times, most recently from 0f877d4 to 7e06391 Compare July 21, 2020 05:23

phd3 requested a review from sopel39 July 21, 2020 05:23

phd3 force-pushed the cherry-picked-benchmark-for-1720 branch from 7e06391 to 5af2d5d Compare August 25, 2020 23:16

sopel39 reviewed Aug 31, 2020

View reviewed changes

phd3 force-pushed the cherry-picked-benchmark-for-1720 branch from 5af2d5d to 5d8bd0a Compare September 1, 2020 20:59

phd3 requested a review from sopel39 September 2, 2020 21:19

sopel39 reviewed Sep 7, 2020

View reviewed changes

phd3 force-pushed the cherry-picked-benchmark-for-1720 branch from 5d8bd0a to 8a53d6f Compare September 9, 2020 00:11

sopel39 approved these changes Sep 9, 2020

View reviewed changes

phd3 added 3 commits September 9, 2020 14:49

Extract TestData as a top-level class

92ad14b

Support benchmark reads through HivePageSource

4227a50

Add benchmark for projected column reads

0024e49

phd3 force-pushed the cherry-picked-benchmark-for-1720 branch from 8a53d6f to 0024e49 Compare September 9, 2020 21:50

sopel39 merged commit 6803205 into trinodb:master Sep 10, 2020

		@@ -504,44 +504,6 @@ private static <E extends TpchEntity> TestData createTpchDataSet(FileFormat form
		return new TestData(columnNames, columnTypes, pages.build());

Conversation

phd3 commented Dec 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmark Setup and parameters

Some benchmark Results

ORC

Parquet

Uh oh!

sopel39 commented Dec 19, 2019

Uh oh!

phd3 commented Dec 21, 2019

Uh oh!

sopel39 commented Dec 23, 2019

Uh oh!

sopel39 commented Dec 31, 2019

Uh oh!

phd3 commented Jan 1, 2020

Uh oh!

phd3 commented Jul 22, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

phd3 commented Sep 2, 2020

Uh oh!

sopel39 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

phd3 Sep 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

phd3 Sep 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sopel39 commented Sep 9, 2020

Uh oh!

sopel39 commented Sep 10, 2020

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

phd3 commented Dec 18, 2019 •

edited

Loading

phd3 Sep 8, 2020 •

edited

Loading

phd3 Sep 8, 2020 •

edited

Loading