Decrease a lock contention in PipelinedStageExecution#14395
Merged
sopel39 merged 2 commits intotrinodb:masterfrom Oct 10, 2022
Merged
Conversation
radek-kondziolka
force-pushed
the
rk/decrease_lock_contention_PipelinedStageExecution
branch
2 times, most recently
from
bf65968 to
c42071b
Compare
radek-kondziolka
requested review from
arhimondr,
losipiuk,
lukasz-stec and
sopel39
radek-kondziolka
force-pushed
the
rk/decrease_lock_contention_PipelinedStageExecution
branch
from
c42071b to
261f3ac
Compare
sopel39
reviewed
Oct 3, 2022
core/trino-main/src/main/java/io/trino/execution/scheduler/PipelinedStageExecution.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/execution/scheduler/PipelinedStageExecution.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/execution/scheduler/PipelinedStageExecution.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/execution/scheduler/PipelinedStageExecution.java
Outdated
Show resolved
Hide resolved
Dith3r
reviewed
Oct 4, 2022
core/trino-main/src/main/java/io/trino/execution/scheduler/PipelinedStageExecution.java
Outdated
Show resolved
Hide resolved
radek-kondziolka
force-pushed
the
rk/decrease_lock_contention_PipelinedStageExecution
branch
from
261f3ac to
9510d8d
Compare
Dith3r
reviewed
Oct 4, 2022
core/trino-main/src/main/java/io/trino/execution/scheduler/PipelinedStageExecution.java
Outdated
Show resolved
Hide resolved
sopel39
reviewed
Oct 4, 2022
core/trino-main/src/main/java/io/trino/execution/scheduler/PipelinedStageExecution.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/execution/scheduler/PipelinedStageExecution.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/execution/scheduler/PipelinedStageExecution.java
Outdated
Show resolved
Hide resolved
radek-kondziolka
force-pushed
the
rk/decrease_lock_contention_PipelinedStageExecution
branch
from
9510d8d to
46798c3
Compare
lukasz-stec
reviewed
Oct 5, 2022
core/trino-main/src/main/java/io/trino/execution/scheduler/PipelinedStageExecution.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/execution/scheduler/PipelinedStageExecution.java
Outdated
Show resolved
Hide resolved
sopel39
reviewed
Oct 5, 2022
core/trino-main/src/main/java/io/trino/execution/scheduler/PipelinedStageExecution.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/execution/scheduler/PipelinedStageExecution.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/execution/scheduler/PipelinedStageExecution.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/execution/scheduler/PipelinedStageExecution.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/execution/scheduler/PipelinedStageExecution.java
Outdated
Show resolved
Hide resolved
core/trino-main/src/main/java/io/trino/execution/scheduler/PipelinedStageExecution.java
Outdated
Show resolved
Hide resolved
sopel39
approved these changes
Oct 5, 2022
radek-kondziolka
force-pushed
the
rk/decrease_lock_contention_PipelinedStageExecution
branch
2 times, most recently
from
2f4fde7 to
a117151
Compare
radek-kondziolka
force-pushed
the
rk/decrease_lock_contention_PipelinedStageExecution
branch
from
a117151 to
1cba3c6
Compare
sopel39
approved these changes
Oct 6, 2022
Member
|
There are some test failures. Could you rerun? |
Contributor
Author
Yes, these failures are not relevant. I am going to rerun but before I want to see result of product tests. After that, I rerun. |
radek-kondziolka
force-pushed
the
rk/decrease_lock_contention_PipelinedStageExecution
branch
2 times, most recently
from
ed1a002 to
200a20c
Compare
radek-kondziolka
force-pushed
the
rk/decrease_lock_contention_PipelinedStageExecution
branch
2 times, most recently
from
36dbf6c to
64a8e46
Compare
Taking a monitor of io.trino.execution.scheduler.PipelinedStageExection in the updateTaskStatus method causes a high lock contention. Make this method lock-less.
radek-kondziolka
force-pushed
the
rk/decrease_lock_contention_PipelinedStageExecution
branch
from
64a8e46 to
348ea11
Compare
Closed
arhimondr
reviewed
Oct 13, 2022
| if (isFlushing()) { | ||
| stateMachine.transitionToFlushing(); | ||
| // avoid extra synchronization if no new flushing or finished task was observed | ||
| if (newFlushingOrFinishedTaskObserved) { |
Contributor
There was a problem hiding this comment.
The stageState is read before a task is added to the finishedTasks set. I think this breaks happens-before assumptions. It is possible that State stageState = stateMachine.getState(); is run when the state is not yet SCHEDULED so the stateMachine.transitionToFinished(); is not run while at the same time the schedulingComplete() transitions the stage to SCHEDULED but doesn't see all tasks being finished.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
We observed that there is a high lock contention on
io.trino.execution.scheduler.PipelinedStageExecution's monitor.Lock contention before:
Lock contention after:
We measured the throughput (80 trino workers, 40 r5.4xlarge nodes, 64 concurrent queries) before/after
Throughput (queries/h) before:
6000 queries / h
Throughput (queries/h) after:
6600 queries / h (10% difference)
Non-technical explanation
The Trino is able to process more queries within one hour.
Release notes
( ) This is not user-visible and no release notes are required.
( ) Release notes are required, please propose a release note for me.
(* ) Release notes are required, with the following suggested text:
Increase throughput when running highly concurrent workloads on big Trino clusters