Estimate cost of inequality comparision filters by raunaqmorarka · Pull Request #11518 · trinodb/trino

raunaqmorarka · 2022-03-16T15:36:36Z

On top of #11469

Description

When there is some overlap between the ranges of the
operands of an inequality expression, we approximate
it's cost assuming uniform distribution of values
within each range and that all pairs of distinct values
from both ranges exist.

Is this change a fix, improvement, new feature, refactoring, or other?

Improvement

Is this a change to the core query engine, a connector, client library, or the SPI interfaces? (be specific)

CBO

How would you describe this change to a non-technical end user or system administrator?

Improves query plans in the presence of inequality expressions

Related issues, pull requests, and links

Documentation

(x) No documentation is needed.
( ) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.

Release notes

( ) No release notes entries required.
(x) Release notes entries required with the following suggested text:

# Section
* Improve query plans in the presence of inequality expressions. ({issue}`11518`)

raunaqmorarka · 2022-03-17T06:05:51Z

TPC benchmarks
Inequality estimation sf1000 orc unpartitioned.pdf

Inequality estimation sf1000 orc partitioned.pdf

lukasz-stec

lgtm

core/trino-main/src/main/java/io/trino/cost/ComparisonStatsCalculator.java

core/trino-main/src/main/java/io/trino/cost/FilterProjectAggregationStatsRule.java

core/trino-main/src/main/java/io/trino/cost/ComparisonStatsCalculator.java

sopel39 · 2022-03-21T12:07:46Z

There are some regressions in partitioned tpcds (some lesser regressions in unpart tpcds). Why it's the case?

raunaqmorarka · 2022-03-21T12:45:17Z

There are some regressions in partitioned tpcds (some lesser regressions in unpart tpcds). Why it's the case?

The only actual regression is in tpch/q04 partitioned (which we already knew about, it's due to bad NDV estimate), the rest is benchmark flakiness.

core/trino-main/src/main/java/io/trino/cost/ComparisonStatsCalculator.java

sopel39 · 2022-03-21T12:00:16Z

core/trino-main/src/main/java/io/trino/cost/ComparisonStatsCalculator.java

nit: can it be empty?

core/trino-main/src/main/java/io/trino/cost/ComparisonStatsCalculator.java

sopel39 · 2022-03-21T13:48:13Z

The only actual regression is in tpch/q04 partitioned (which we already knew about, it's due to bad NDV estimate), the rest is benchmark flakiness.

What about tpcds/q93, q95, 97, 78? Did plans change for these?

raunaqmorarka · 2022-03-22T05:17:56Z

The only actual regression is in tpch/q04 partitioned (which we already knew about, it's due to bad NDV estimate), the rest is benchmark flakiness.

What about tpcds/q93, q95, 97, 78? Did plans change for these?

The inequality estimation code path is executed only for q72 in tpcds and there was no change to plan there.

sopel39 · 2022-03-22T10:47:14Z

The inequality estimation code path is executed only for q72 in tpcds and there was no change to plan there.

You mean partitioned q72 plan?

core/trino-main/src/main/java/io/trino/cost/ComparisonStatsCalculator.java

raunaqmorarka · 2022-03-22T18:41:30Z

Updated benchmarks
Inequality estimation 2 sf1000 orc partitioned.pdf
Inequality estimation 2 sf1000 orc unpartitioned.pdf

Ignore results from all the TPCDS queries other than q72 as the inequality estimation code path is encountered only in that TPCDS query.

sopel39

lgtm % comments

core/trino-main/src/main/java/io/trino/cost/ComparisonStatsCalculator.java

core/trino-main/src/test/java/io/trino/cost/TestFilterStatsCalculator.java

core/trino-main/src/main/java/io/trino/cost/ComparisonStatsCalculator.java

sopel39

lgtm % comments

core/trino-main/src/main/java/io/trino/cost/ComparisonStatsCalculator.java

core/trino-main/src/test/java/io/trino/cost/TestComparisonStatsCalculator.java

core/trino-main/src/main/java/io/trino/cost/ComparisonStatsCalculator.java

When there is some overlap between the ranges of the operands of an inequality expression, we approximate it's cost assuming uniform distribution of values within each range and that all pairs of distinct values from both ranges exist.

sopel39

mind automation (if it fails)

raunaqmorarka requested a review from sopel39 March 16, 2022 15:36

cla-bot bot added the cla-signed label Mar 16, 2022

sopel39 requested review from lukasz-stec and skrzypo987 March 16, 2022 16:09

raunaqmorarka mentioned this pull request Mar 17, 2022

Add estimates for filter nodes after an aggregation #11469

Merged

lukasz-stec approved these changes Mar 18, 2022

View reviewed changes

sopel39 reviewed Mar 21, 2022

View reviewed changes

raunaqmorarka force-pushed the exp-ineq branch 3 times, most recently from d543273 to 48b2735 Compare March 22, 2022 06:36

github-actions bot added the docs label Mar 22, 2022

raunaqmorarka force-pushed the exp-ineq branch from 48b2735 to e3055c0 Compare March 22, 2022 10:33

raunaqmorarka requested a review from sopel39 March 22, 2022 10:45

sopel39 reviewed Mar 22, 2022

View reviewed changes

raunaqmorarka force-pushed the exp-ineq branch from e3055c0 to 3886f00 Compare March 22, 2022 13:42

raunaqmorarka requested a review from sopel39 March 22, 2022 13:53

raunaqmorarka force-pushed the exp-ineq branch from 3886f00 to 54b5ff7 Compare March 22, 2022 18:06

raunaqmorarka force-pushed the exp-ineq branch 2 times, most recently from 81aa639 to 1d2e45f Compare March 23, 2022 15:07

sopel39 reviewed Mar 24, 2022

View reviewed changes

core/trino-main/src/main/java/io/trino/cost/ComparisonStatsCalculator.java Outdated Show resolved Hide resolved

raunaqmorarka force-pushed the exp-ineq branch from 1d2e45f to 0ada23f Compare March 25, 2022 07:53

raunaqmorarka requested a review from sopel39 March 25, 2022 07:58

sopel39 reviewed Mar 25, 2022

View reviewed changes

raunaqmorarka force-pushed the exp-ineq branch 3 times, most recently from 3851aad to 194059d Compare March 29, 2022 04:19

raunaqmorarka requested a review from sopel39 March 29, 2022 07:07

sopel39 reviewed Mar 29, 2022

View reviewed changes

core/trino-main/src/test/java/io/trino/cost/TestComparisonStatsCalculator.java Outdated Show resolved Hide resolved

core/trino-main/src/main/java/io/trino/cost/ComparisonStatsCalculator.java Outdated Show resolved Hide resolved

raunaqmorarka force-pushed the exp-ineq branch from 194059d to e79ea6b Compare March 31, 2022 07:46

raunaqmorarka requested a review from sopel39 March 31, 2022 07:49

sopel39 reviewed Mar 31, 2022

View reviewed changes

core/trino-main/src/main/java/io/trino/cost/ComparisonStatsCalculator.java Outdated Show resolved Hide resolved

core/trino-main/src/main/java/io/trino/cost/ComparisonStatsCalculator.java Outdated Show resolved Hide resolved

Estimate cost of inequality comparision filters

b17e418

When there is some overlap between the ranges of the operands of an inequality expression, we approximate it's cost assuming uniform distribution of values within each range and that all pairs of distinct values from both ranges exist.

raunaqmorarka force-pushed the exp-ineq branch from e79ea6b to b17e418 Compare March 31, 2022 10:41

sopel39 approved these changes Mar 31, 2022

View reviewed changes

sopel39 merged commit fbe6079 into trinodb:master Mar 31, 2022

sopel39 mentioned this pull request Mar 31, 2022

Release notes for 376 #11692

Closed

raunaqmorarka deleted the exp-ineq branch March 31, 2022 12:09

github-actions bot added this to the 376 milestone Mar 31, 2022

mosabua mentioned this pull request Apr 4, 2022

Add Trino 376 release notes #11691

Merged

Conversation

raunaqmorarka commented Mar 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related issues, pull requests, and links

Documentation

Release notes

Uh oh!

raunaqmorarka commented Mar 17, 2022

Uh oh!

lukasz-stec left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sopel39 commented Mar 21, 2022

Uh oh!

raunaqmorarka commented Mar 21, 2022

Uh oh!

Uh oh!

Uh oh!

sopel39 Mar 21, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sopel39 commented Mar 21, 2022

Uh oh!

raunaqmorarka commented Mar 22, 2022

Uh oh!

sopel39 commented Mar 22, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

raunaqmorarka commented Mar 22, 2022

Uh oh!

sopel39 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sopel39 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sopel39 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

raunaqmorarka commented Mar 16, 2022 •

edited

Loading