Track whether an optimization decision was cost-based by mlyublena · Pull Request #20990 · prestodb/presto

mlyublena · 2023-09-28T21:39:00Z

Some of Presto's optimizers are heuristic, while others are cost-based. This change allows tracking which optimizers were driven by a cost-based decision (independent of whether the cost was estimated or supplied by HBO). This information is added to PlanOptimizerInformation and can be seen in the explain plan when verbose_optimizer_info_enabled=true, for example:

presto:tpch> explain select lineitem.linenumber,count(*) from orders join lineitem on (lineitem.orderkey=orders.orderkey) group by linenumber;
                                                                                                                                                                                                                                        Query Plan
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 - Output[PlanNodeId 12][linenumber, _col1] => [linenumber:integer, count:bigint]
         _col1 := count (1:36)
...
 Triggered optimizers: [AddLocalExchanges, ApplyConnectorOptimization, HashGenerationOptimizer, PickTableLayoutWithoutPredicate, PruneJoinChildrenColumns, PruneJoinColumns, PruneProjectColumns, PruneTableScanColumns, PruneUnreferencedOutputs, PushPartialAggregationThroughExchange, RemoveRedundantDistinctAggregation, RemoveRedundantIdentityProjections, ReorderJoins, SetFlatteningOptimizer, SimplifyPlanWithEmptyInput, StatsRecordingPlanOptimizer, UnaliasSymbolReferences]
 Applicable optimizers: [AddNotNullFiltersToJoinNode, KeyBasedSampler, MergePartialAggregationsWithFilter, PushPartialAggregationThroughJoin]
 Cost-based optimizers: [PushPartialAggregationThroughExchange(CBO), ReorderJoins(CBO)]

Note: currently we don't track whether the cost came from HBO or through cost estimation, only that the decision was cost-driven. We already have a change that tracks what the source of the cost estimation is, so we can potentially intersect the two to find this information. I could potentially reconsider and track and log this information directly here.

Description

Motivation and Context

Impact

Test Plan

Contributor checklist

Please make sure your submission complies with our development, formatting, commit message, and attribution guidelines.
PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
Documented new properties (with its default value), SQL syntax, functions, or other functionality.
If release notes are required, they follow the release notes guidelines.
Adequate tests were added if applicable.
CI passed.

Release Notes

Please follow release notes guidelines and fill in the release notes below.

== RELEASE NOTES ==

General Changes
* Print information about cost-based optimizers and the source of stats they use (CBO/HBO) in explain plans when session property verbose_optimizer_info_enabled=true

mlyublena · 2023-10-02T18:24:42Z

presto-main/src/main/java/com/facebook/presto/sql/planner/planPrinter/TextRenderer.java

cleaned up the output to be sorted and de-duplicated

presto-main/src/main/java/com/facebook/presto/sql/planner/planPrinter/TextRenderer.java

...va/com/facebook/presto/sql/planner/iterative/rule/PushPartialAggregationThroughExchange.java

jainxrohit · 2023-10-03T06:05:31Z

.../main/java/com/facebook/presto/sql/planner/iterative/rule/DetermineJoinDistributionType.java

I wonder if caller can pass this information rather setting this here?

vivek-bharathan · 2023-10-03T18:33:34Z

presto-main/src/main/java/com/facebook/presto/sql/Optimizer.java

Wouldn't this disjunction cause all cost-based optimizers to be logged? Currently it does not matter since none of the PlanOptimizers are cost-based, but if that should change this would not be correct

I think you're right, I'll change it to only log isCostBased if the optimization triggered

vivek-bharathan · 2023-10-03T20:23:26Z

.../main/java/com/facebook/presto/sql/planner/iterative/rule/DetermineJoinDistributionType.java

vivek-bharathan · 2023-10-03T23:13:55Z

Add these as well?

DetermineSemiJoinDistributionType
TransformDistinctInnerJoinToRightEarlyOutJoin

pranjalssh

Looks good modulo the nits. Please fix the release notes as well

...va/com/facebook/presto/sql/planner/iterative/rule/PushPartialAggregationThroughExchange.java

Some of Presto's optimizers are heuristic, while others are cost-based. This change allows tracking which optimizers were driven by a cost-based decision (independent of whether the cost was estimated or supplied by HBO). This information is added to PlanOptimizerInformation and can be seen in the explain plan when verbose_optimizer_info_enabled=true, for example: presto:tpch> explain select lineitem.linenumber,count(*) from orders join lineitem on (lineitem.orderkey=orders.orderkey) group by linenumber; Query Plan ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ - Output[PlanNodeId 12][linenumber, _col1] => [linenumber:integer, count:bigint] _col1 := count (1:36) - RemoteStreamingExchange[PlanNodeId 297][GATHER] => [linenumber:integer, count:bigint] - Project[PlanNodeId 406][projectLocality = LOCAL] => [linenumber:integer, count:bigint] - Aggregate(FINAL)[linenumber][$hashvalue][PlanNodeId 7] => [linenumber:integer, $hashvalue:bigint, count:bigint] count := "presto.default.count"((count_15)) (1:36) - LocalExchange[PlanNodeId 355][HASH][$hashvalue] (linenumber) => [linenumber:integer, count_15:bigint, $hashvalue:bigint] - RemoteStreamingExchange[PlanNodeId 361][REPARTITION][$hashvalue_16] => [linenumber:integer, count_15:bigint, $hashvalue_16:bigint] - Aggregate(PARTIAL)[linenumber][$hashvalue_22][PlanNodeId 359] => [linenumber:integer, $hashvalue_22:bigint, count_15:bigint] count_15 := "presto.default.count"(*) (1:36) - Project[PlanNodeId 405][projectLocality = LOCAL] => [linenumber:integer, $hashvalue_22:bigint] Estimates: {source: CostBasedSourceInfo, rows: 58490 (799.67kB), cpu: 7320844.01, memory: 270000.00, network: 1654025.00} $hashvalue_22 := combine_hash(BIGINT'0', COALESCE($operator$hash_code(linenumber), BIGINT'0')) (1:119) - InnerJoin[PlanNodeId 271][("orderkey_0" = "orderkey")][$hashvalue_17, $hashvalue_19] => [linenumber:integer] Estimates: {source: CostBasedSourceInfo, rows: 58490 (799.67kB), cpu: 6501977.37, memory: 270000.00, network: 1654025.00} Distribution: PARTITIONED - RemoteStreamingExchange[PlanNodeId 294][REPARTITION][$hashvalue_17] => [orderkey_0:bigint, linenumber:integer, $hashvalue_17:bigint] Estimates: {source: CostBasedSourceInfo, rows: 60175 (822.71kB), cpu: 3610500.00, memory: 0.00, network: 1384025.00} - ScanProject[PlanNodeId 1,403][table = TableHandle {connectorId='hive', connectorHandle='HiveTableHandle{schemaName=tpch, tableName=lineitem, analyzePartitionValues=Optional.empty}', layout='Optional[tpch.lineitem{}]'}, projectLocality = LOCAL] => [orderkey_0:bigint, linenumber:integer, $hashvalue_18:bigint] Estimates: {source: CostBasedSourceInfo, rows: 60175 (822.71kB), cpu: 842450.00, memory: 0.00, network: 0.00}/{source: CostBasedSourceInfo, rows: 60175 (822.71kB), cpu: 2226475.00, memory: 0.00, network: 0.00} $hashvalue_18 := combine_hash(BIGINT'0', COALESCE($operator$hash_code(orderkey_0), BIGINT'0')) (1:62) LAYOUT: tpch.lineitem{} orderkey_0 := orderkey:bigint:0:REGULAR (1:62) linenumber := linenumber:int:3:REGULAR (1:62) - LocalExchange[PlanNodeId 338][HASH][$hashvalue_19] (orderkey) => [orderkey:bigint, $hashvalue_19:bigint] Estimates: {source: CostBasedSourceInfo, rows: 15000 (205.08kB), cpu: 945000.00, memory: 0.00, network: 270000.00} - RemoteStreamingExchange[PlanNodeId 295][REPARTITION][$hashvalue_20] => [orderkey:bigint, $hashvalue_20:bigint] Estimates: {source: CostBasedSourceInfo, rows: 15000 (205.08kB), cpu: 675000.00, memory: 0.00, network: 270000.00} - ScanProject[PlanNodeId 0,404][table = TableHandle {connectorId='hive', connectorHandle='HiveTableHandle{schemaName=tpch, tableName=orders, analyzePartitionValues=Optional.empty}', layout='Optional[tpch.orders{}]'}, projectLocality = LOCAL] => [orderkey:bigint, $hashvalue_21:bigint] Estimates: {source: CostBasedSourceInfo, rows: 15000 (205.08kB), cpu: 135000.00, memory: 0.00, network: 0.00}/{source: CostBasedSourceInfo, rows: 15000 (205.08kB), cpu: 405000.00, memory: 0.00, network: 0.00} $hashvalue_21 := combine_hash(BIGINT'0', COALESCE($operator$hash_code(orderkey), BIGINT'0')) (1:50) LAYOUT: tpch.orders{} orderkey := orderkey:bigint:0:REGULAR (1:50) Triggered optimizers: [AddLocalExchanges, ApplyConnectorOptimization, HashGenerationOptimizer, PickTableLayoutWithoutPredicate, PruneJoinChildrenColumns, PruneJoinColumns, PruneProjectColumns, PruneTableScanColumns, PruneUnreferencedOutputs, PushPartialAggregationThroughExchange, RemoveRedundantDistinctAggregation, RemoveRedundantIdentityProjections, ReorderJoins, SetFlatteningOptimizer, SimplifyPlanWithEmptyInput, StatsRecordingPlanOptimizer, UnaliasSymbolReferences] Applicable optimizers: [AddNotNullFiltersToJoinNode, KeyBasedSampler, MergePartialAggregationsWithFilter, PushPartialAggregationThroughJoin] Cost-based optimizers: [PushPartialAggregationThroughExchange(CBO), ReorderJoins(CBO)]

mlyublena requested review from feilong-liu and pranjalssh September 28, 2023 21:44

mlyublena marked this pull request as ready for review September 28, 2023 21:44

mlyublena requested a review from a team as a code owner September 28, 2023 21:44

mlyublena requested a review from presto-oss September 28, 2023 21:44

mlyublena commented Oct 2, 2023

View reviewed changes

jainxrohit reviewed Oct 3, 2023

View reviewed changes

vivek-bharathan reviewed Oct 3, 2023

View reviewed changes

pranjalssh approved these changes Oct 5, 2023

View reviewed changes

...va/com/facebook/presto/sql/planner/iterative/rule/PushPartialAggregationThroughExchange.java Outdated Show resolved Hide resolved

mlyublena force-pushed the log-hbo-optimizer-info branch 6 times, most recently from 2ba92de to 4eb770b Compare October 10, 2023 04:14

mlyublena force-pushed the log-hbo-optimizer-info branch from 4eb770b to 1108b01 Compare October 10, 2023 05:18

mlyublena merged commit 53e07a1 into prestodb:master Oct 10, 2023

mlyublena mentioned this pull request Oct 10, 2023

Set initial number of tasks for scaled writer with HBO #20901

Merged

6 tasks

feilong-liu mentioned this pull request Oct 11, 2023

Add cost based decision log for scaled writer optimizer #21120

Merged

6 tasks

wanglinsong mentioned this pull request Dec 8, 2023

Add release notes for 0.285 #21500

Closed

26 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Track whether an optimization decision was cost-based#20990

Track whether an optimization decision was cost-based#20990
mlyublena merged 1 commit intoprestodb:masterfrom
mlyublena:log-hbo-optimizer-info

mlyublena commented Sep 28, 2023 •

edited

Loading

Uh oh!

mlyublena Oct 2, 2023

Uh oh!

Uh oh!

Uh oh!

jainxrohit Oct 3, 2023

Uh oh!

vivek-bharathan Oct 3, 2023 •

edited

Loading

Uh oh!

mlyublena Oct 3, 2023

Uh oh!

vivek-bharathan Oct 3, 2023

Uh oh!

vivek-bharathan commented Oct 3, 2023

Uh oh!

pranjalssh left a comment •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

mlyublena commented Sep 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

Impact

Test Plan

Contributor checklist

Release Notes

Uh oh!

mlyublena Oct 2, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jainxrohit Oct 3, 2023

Choose a reason for hiding this comment

Uh oh!

vivek-bharathan Oct 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mlyublena Oct 3, 2023

Choose a reason for hiding this comment

Uh oh!

vivek-bharathan Oct 3, 2023

Choose a reason for hiding this comment

Uh oh!

vivek-bharathan commented Oct 3, 2023

Uh oh!

pranjalssh left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mlyublena commented Sep 28, 2023 •

edited

Loading

vivek-bharathan Oct 3, 2023 •

edited

Loading

pranjalssh left a comment •

edited

Loading