Add upper limit on number of tasks that take part in writing stages by radek-kondziolka · Pull Request #16238 · trinodb/trino

radek-kondziolka · 2023-02-23T14:49:36Z

Description

The option query.max-writer-node-count was added to limit number of nodes that take part in executing writer stages.
Roughly, it was implemented by some changes in ScaledWriterScheduler (for unpartitioned data) and by changing the AddExchanges rule (mostly for partitioned data).

Release notes

() This is not user-visible or docs only and no release notes are required.
() Release notes are required, please propose a release note for me.
(*) Release notes are required, with the following suggested text:

Add posibility to limit number of writing tasks.

Documentation

There is a change in this PR

radek-kondziolka · 2023-02-23T15:16:30Z

I will add more tests after we agree that this approach is ok.

.../trino-main/src/main/java/io/trino/execution/scheduler/PartitioningHandlePartitionCount.java

core/trino-main/src/main/java/io/trino/sql/planner/optimizations/AddExchanges.java

core/trino-main/src/main/java/io/trino/metadata/Metadata.java

core/trino-main/src/main/java/io/trino/execution/scheduler/PipelinedQueryScheduler.java

...rino-main/src/test/java/io/trino/sql/planner/optimizations/TestLimitMaxWriterNodesCount.java

core/trino-main/src/main/java/io/trino/metadata/Metadata.java

core/trino-main/src/main/java/io/trino/SystemSessionProperties.java

gaurav8297 · 2023-02-27T08:36:13Z

There are some relevant CI failures. Please fix them

gaurav8297 · 2023-02-27T10:13:51Z

Can we also have an integration test in BaseHiveConnectorTest? Something similar to testMultipleWriters where we check the number of files in the end

radek-kondziolka · 2023-02-27T10:18:32Z

Can we also have an integration test in BaseHiveConnectorTest? Something similar to testMultipleWriters where we check the number of files in the end

Sure, as I wrote here: #16238 (comment) I am going to add new tests after we agree this approach is ok.

wendigo · 2023-02-27T13:21:54Z

2023-02-27T12:44:11.0895418Z java.lang.AssertionError: io.trino.plugin.base.classloader.ClassLoaderSafeConnectorMetadata does not override [public default java.util.OptionalInt io.trino.spi.connector.ConnectorMetadata.getMaxWriterTasks(io.trino.spi.connector.ConnectorSession)]

core/trino-main/src/main/java/io/trino/SystemSessionProperties.java

core/trino-main/src/main/java/io/trino/execution/QueryManagerConfig.java

core/trino-main/src/main/java/io/trino/metadata/Metadata.java

core/trino-main/src/main/java/io/trino/execution/scheduler/PipelinedQueryScheduler.java

core/trino-main/src/main/java/io/trino/SystemSessionProperties.java

core/trino-main/src/main/java/io/trino/metadata/Metadata.java

core/trino-main/src/main/java/io/trino/sql/planner/LogicalPlanner.java

core/trino-main/src/main/java/io/trino/sql/planner/optimizations/AddExchanges.java

docs/src/main/sphinx/admin/properties-query-management.rst

Map's key in PipelinedQueryScheduler.partitioningCacheMap is built only from partitioningHandle parameter. If any two stages has same partitioning but different number of partitions the key does not reflect that. This is why key was extended with partitionCount.

sopel39

lgtm % comments

docs/src/main/sphinx/admin/properties-query-management.rst

core/trino-main/src/main/java/io/trino/SystemSessionProperties.java

core/trino-main/src/main/java/io/trino/execution/QueryManagerConfig.java

core/trino-main/src/main/java/io/trino/execution/scheduler/PipelinedQueryScheduler.java

core/trino-main/src/main/java/io/trino/sql/planner/LogicalPlanner.java

core/trino-main/src/main/java/io/trino/sql/planner/optimizations/AddExchanges.java

core/trino-main/src/main/java/io/trino/sql/planner/plan/TableWriterNode.java

sopel39

minor comments

core/trino-main/src/main/java/io/trino/sql/planner/optimizations/AddExchanges.java

core/trino-main/src/main/java/io/trino/SystemSessionProperties.java

Limit number of tasks that will be scheduled to execute writing stages basing on needs of specific connector or global limit - query.max-writer-nodes-count. Added an option query.max-writer-nodes-count to QueryManagerConfig and a session option that limits number of tasks that take part in writing nodes.

sopel39 · 2023-03-17T15:44:58Z

Thx!

cla-bot bot added the cla-signed label Feb 23, 2023

radek-kondziolka mentioned this pull request Feb 23, 2023

Limit number of nodes that execute writing stages #15987

Closed

radek-kondziolka requested review from gaurav8297, hashhar and sopel39 February 23, 2023 15:16

radek-kondziolka force-pushed the rk/add_upper_limit_on_writer_scaling_backup_with_connector branch from 713aae5 to 6906e11 Compare February 23, 2023 15:33

gaurav8297 reviewed Feb 24, 2023

View reviewed changes

.../trino-main/src/main/java/io/trino/execution/scheduler/PartitioningHandlePartitionCount.java Outdated Show resolved Hide resolved

gaurav8297 reviewed Feb 24, 2023

View reviewed changes

core/trino-main/src/main/java/io/trino/sql/planner/optimizations/AddExchanges.java Outdated Show resolved Hide resolved

gaurav8297 reviewed Feb 24, 2023

View reviewed changes

core/trino-main/src/main/java/io/trino/metadata/Metadata.java Outdated Show resolved Hide resolved

radek-kondziolka force-pushed the rk/add_upper_limit_on_writer_scaling_backup_with_connector branch from 6906e11 to 2bb3511 Compare February 24, 2023 09:40

radek-kondziolka requested a review from gaurav8297 February 24, 2023 10:01

gaurav8297 reviewed Feb 27, 2023

View reviewed changes

core/trino-main/src/main/java/io/trino/execution/scheduler/PipelinedQueryScheduler.java Outdated Show resolved Hide resolved

gaurav8297 reviewed Feb 27, 2023

View reviewed changes

...rino-main/src/test/java/io/trino/sql/planner/optimizations/TestLimitMaxWriterNodesCount.java Outdated Show resolved Hide resolved

...rino-main/src/test/java/io/trino/sql/planner/optimizations/TestLimitMaxWriterNodesCount.java Outdated Show resolved Hide resolved

gaurav8297 reviewed Feb 27, 2023

View reviewed changes

core/trino-main/src/main/java/io/trino/metadata/Metadata.java Outdated Show resolved Hide resolved

gaurav8297 reviewed Feb 27, 2023

View reviewed changes

core/trino-main/src/main/java/io/trino/SystemSessionProperties.java Outdated Show resolved Hide resolved

radek-kondziolka force-pushed the rk/add_upper_limit_on_writer_scaling_backup_with_connector branch from 2bb3511 to f3bd21e Compare February 27, 2023 10:15

radek-kondziolka changed the title ~~Rk/add upper limit on writer scaling backup with connector~~ Add upper limit on writer scaling backup with connector Feb 27, 2023

radek-kondziolka changed the title ~~Add upper limit on writer scaling backup with connector~~ Add upper limit on number of tasks that take part in writing stages Feb 27, 2023

radek-kondziolka force-pushed the rk/add_upper_limit_on_writer_scaling_backup_with_connector branch from f3bd21e to 16aded6 Compare February 27, 2023 12:12

wendigo mentioned this pull request Feb 27, 2023

Make JDBC write parallelism configurable #16280

Merged

radek-kondziolka force-pushed the rk/add_upper_limit_on_writer_scaling_backup_with_connector branch from 16aded6 to 203784d Compare February 27, 2023 13:59

gaurav8297 reviewed Feb 28, 2023

View reviewed changes

core/trino-main/src/main/java/io/trino/SystemSessionProperties.java Outdated Show resolved Hide resolved

core/trino-main/src/main/java/io/trino/execution/QueryManagerConfig.java Outdated Show resolved Hide resolved

gaurav8297 reviewed Feb 28, 2023

View reviewed changes

core/trino-main/src/main/java/io/trino/metadata/Metadata.java Outdated Show resolved Hide resolved

core/trino-main/src/main/java/io/trino/metadata/Metadata.java Outdated Show resolved Hide resolved

gaurav8297 reviewed Feb 28, 2023

View reviewed changes

core/trino-main/src/main/java/io/trino/execution/scheduler/PipelinedQueryScheduler.java Outdated Show resolved Hide resolved

radek-kondziolka force-pushed the rk/add_upper_limit_on_writer_scaling_backup_with_connector branch from 203784d to 7afb274 Compare February 28, 2023 15:33

sopel39 reviewed Mar 8, 2023

View reviewed changes

radek-kondziolka force-pushed the rk/add_upper_limit_on_writer_scaling_backup_with_connector branch from ae493bb to fdfca2a Compare March 9, 2023 07:36

radek-kondziolka requested a review from sopel39 March 9, 2023 07:38

radek-kondziolka force-pushed the rk/add_upper_limit_on_writer_scaling_backup_with_connector branch from fdfca2a to 7240ab1 Compare March 9, 2023 10:43

sopel39 approved these changes Mar 10, 2023

View reviewed changes

radek-kondziolka force-pushed the rk/add_upper_limit_on_writer_scaling_backup_with_connector branch 6 times, most recently from 1009e95 to c56bc02 Compare March 14, 2023 07:29

kokosing force-pushed the master branch from 3f05134 to 58d6356 Compare March 14, 2023 11:34

sopel39 approved these changes Mar 15, 2023

View reviewed changes

core/trino-main/src/main/java/io/trino/sql/planner/optimizations/AddExchanges.java Outdated Show resolved Hide resolved

core/trino-main/src/main/java/io/trino/SystemSessionProperties.java Outdated Show resolved Hide resolved

radek-kondziolka force-pushed the rk/add_upper_limit_on_writer_scaling_backup_with_connector branch 7 times, most recently from 884826e to 738e3dc Compare March 16, 2023 09:30

Add description for query.max-writer-tasks-count to docs

fa9a667

radek-kondziolka force-pushed the rk/add_upper_limit_on_writer_scaling_backup_with_connector branch from 738e3dc to fa9a667 Compare March 16, 2023 19:25

sopel39 merged commit 8750f07 into trinodb:master Mar 17, 2023

sopel39 mentioned this pull request Mar 17, 2023

Release notes for 411 #16453

Closed

github-actions bot added this to the 411 milestone Mar 17, 2023

colebow mentioned this pull request Mar 29, 2023

Add Trino 411 release notes #16552

Merged

ebyhr mentioned this pull request Nov 17, 2023

Fix typo in config #19784

Closed

Conversation

radek-kondziolka commented Feb 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Release notes

Documentation

Uh oh!

radek-kondziolka commented Feb 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gaurav8297 commented Feb 27, 2023

Uh oh!

gaurav8297 commented Feb 27, 2023

Uh oh!

radek-kondziolka commented Feb 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wendigo commented Feb 27, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sopel39 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sopel39 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sopel39 commented Mar 17, 2023

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

radek-kondziolka commented Feb 23, 2023 •

edited

Loading

radek-kondziolka commented Feb 23, 2023 •

edited

Loading

radek-kondziolka commented Feb 27, 2023 •

edited

Loading