Add scale-writers.max-nodes-count property to limit writer scaling#15877
Closed
radek-kondziolka wants to merge 89 commits intotrinodb:masterfrom
Closed
Add scale-writers.max-nodes-count property to limit writer scaling#15877radek-kondziolka wants to merge 89 commits intotrinodb:masterfrom
scale-writers.max-nodes-count property to limit writer scaling#15877radek-kondziolka wants to merge 89 commits intotrinodb:masterfrom
Conversation
radek-kondziolka
force-pushed
the
rk/add_upper_limit_on_writer_scaling
branch
2 times, most recently
from
1c426f6 to
0341a57
Compare
radek-kondziolka
changed the title
scale-writers.max-nodes-count property to limit writer scaling
radek-kondziolka
force-pushed
the
rk/add_upper_limit_on_writer_scaling
branch
4 times, most recently
from
6d86f08 to
13bb374
Compare
gaurav8297
reviewed
Jan 30, 2023
core/trino-main/src/main/java/io/trino/execution/TaskManagerConfig.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/execution/scheduler/ScaledWriterScheduler.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/test/java/io/trino/execution/scheduler/TestScaledWriterScheduler.java
Outdated
Show resolved
Hide resolved
Adaptively set the hash partition count for a query. It increases the concurrency on the large cluster when there are many small queries. Benchmarks: We ran concurrency benchmarks on a cluster size of 20 with queries from the TPC-DS sf10 set submitted concurrently from 64 threads. Before: 17578 queries/hour After: 22782 queries/hour (30% increase)
radek-kondziolka
force-pushed
the
rk/add_upper_limit_on_writer_scaling
branch
from
13bb374 to
cc1d7f1
Compare
sopel39
reviewed
Jan 30, 2023
Member
There was a problem hiding this comment.
Node limit should also apply task.scale-writers.enabled when preferred partitioning insert mode is used. Please refer to @gaurav8297 for help
sopel39
reviewed
Jan 30, 2023
core/trino-main/src/main/java/io/trino/execution/TaskManagerConfig.java
Outdated
Show resolved
Hide resolved
radek-kondziolka
force-pushed
the
rk/add_upper_limit_on_writer_scaling
branch
from
cc1d7f1 to
e5ae293
Compare
Distribution of http requests duration sent by HttpPageBufferClient was added and exposed by DirectExchangeClientStatus.
Add methods in SystemSecurityMetadata to provide notification of adding, dropping and renaming columns.
Usually "a dispatcher" is the entity defining tasks to be done rather than the entity doing an individual task. The change has the additional benefit that the matrix task checking individual commits has a shorter name, so the matrix parameter is slightly more visible in the GitHub actions UI.
Upcoming 1.2.0 version doesn't delete data in RESTSessionCatalog.dropTable method.
Similar to `TimeType` in 3d0af3b.
The unsupported types are handled at the end of the `toWriteMapping` method.
This isn't dissimilar to how we use constants in other temporal types.
Previously `date` would coerce to `timestamp(3) with time zone`, now it gets coerced to `timestamp(0) with time zone`.
This is sometimes useful, so let's always do it.
This is for code consistency reasons. Most of the places where `ExpressionMatcher` is constructed already goes through `PlanMatchPattern.expression` factory method.
`UnwrapCastInComparison` works on any nested comparison expressions.
`StatementAnalyzer.Visitor#analyzeRowFilter` captures necessary coercions for filters that are not of boolean type (e.g. `null` literal being of `unknown` type). Before the change, the coercions where not getting applied though.
When predicate is not of boolean type (e.g. `null` literal being of `unknown` type), the `Analysis` holds a coercion that should be applied to it. Before the change, the coercion was not getting applied though. Co-authored-by: Assaf Bern <assaf.bern@starburstdata.com>
radek-kondziolka
force-pushed
the
rk/add_upper_limit_on_writer_scaling
branch
from
e5ae293 to
3a64db9
Compare
github-actions
bot
added
docs
jdbc
Added option task.scale-writers.max-nodes-count to TaskManagerConfig and a session option to make it possible to limit writer scaling by maximal number of nodes that can be used by ScaledWriterScheduler.
radek-kondziolka
force-pushed
the
rk/add_upper_limit_on_writer_scaling
branch
from
3a64db9 to
ddcbc48
Compare
Member
|
@radek-starburst please rebase |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Added option
task.scale-writers.max-nodes-counttoTaskManagerConfigand a session option to make it possible to limit writer scaling by number of nodes that can be used byScaledWriterScheduler.Release notes
( ) This is not user-visible or docs only and no release notes are required.
(x ) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text: