Skip to content

[WIP] ajay opec #21135

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
ajaygeorge wants to merge 30 commits into
base: master
Choose a base branch
from
Draft

[WIP] ajay opec #21135

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
3d446bc
Prepare for next development iteration - 0.284-edge19-SNAPSHOT
presto-release-bot Sep 16, 2023
8e5c638
Prepare for next development iteration - 0.284-edge19.1-SNAPSHOT
presto-release-bot Sep 16, 2023
c99a5e3
Prepare for next development iteration - 0.284-edge19.2-SNAPSHOT
presto-release-bot Sep 28, 2023
95ea2a7
Add split retry feature
tdcmeehan Oct 21, 2022
e38076c
Worker Isolation
abhiseksaikia Oct 21, 2022
a8ba20d
Add Thrift serializer for LongSet
tdcmeehan Oct 27, 2022
04d56d8
Fix worker isolation
abhiseksaikia Jan 25, 2023
5b6131c
Fix multiple issues with retry
MnO2 Nov 16, 2022
dbe80a7
Fix query runner
abhiseksaikia Jan 27, 2023
3b946d6
Fix stuck query on intermediate workers
MnO2 Jan 28, 2023
ec3d939
Recovering opec worker
abhiseksaikia Mar 2, 2023
a48684e
Schedule queries with LocalExchange to intermediate nodes
MnO2 May 18, 2023
77c2e9f
reduce no_more_retry_delay to 5 sec
abhiseksaikia Sep 6, 2023
d2bebee
setNoMorePages for shutting down task's outputbuffer
abhiseksaikia Sep 8, 2023
c2d623c
Handle FixedSourcePartitionedScheduler for splitting leaf and interme…
abhiseksaikia Sep 11, 2023
8efc854
Logging, metrics and handle race conditions
abhiseksaikia Sep 13, 2023
ba66981
Handle race condition #2
abhiseksaikia Sep 22, 2023
c8d14ae
Change the test to make the pageCount randomly generated
MnO2 Sep 25, 2023
445c0a0
Add complex case for graceful shutdown
MnO2 Sep 25, 2023
0f16ad0
Fix the corner case where the outputBuffer has not received noMoreBuf…
MnO2 Sep 26, 2023
270d8e7
Fix compilation errors
MnO2 Sep 26, 2023
6a85dd8
Use the latest retried logic
MnO2 Sep 27, 2023
bc3bdd4
Add a new GRACEFUL_FAILED stsate for task
MnO2 Sep 28, 2023
49d90ee
Attempt to fix TOO_MANY_REQUEST_FAILED in the verifier results
MnO2 Sep 28, 2023
79fa396
Using callback to decide whether to unblock the SqlStageExecution
MnO2 Sep 29, 2023
0fe719d
Revert the cacnellation conditions
MnO2 Sep 30, 2023
5bcd2bb
Add SplitRetrySourcePartitionedScheduler to support noMoreRetry
MnO2 Oct 1, 2023
b44813b
Commented out line for checking coordinator count
MnO2 Oct 2, 2023
11f19f5
Add debug log
MnO2 Oct 3, 2023
9dfd827
Fix noMoreSplits error on the intermediate stage
MnO2 Oct 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@

<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
<packaging>pom</packaging>

<name>presto-root</name>
Expand Down Expand Up @@ -1148,7 +1148,7 @@
<dependency>
<groupId>it.unimi.dsi</groupId>
<artifactId>fastutil</artifactId>
<version>8.5.2</version>
<version>8.5.9</version>
</dependency>

<dependency>
Expand Down
2 changes: 1 addition & 1 deletion presto-accumulo/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-accumulo</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-analyzer/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-analyzer</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-atop/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-atop</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-base-jdbc/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-base-jdbc</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-benchmark-driver/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-benchmark-driver</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-benchmark-runner/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<artifactId>presto-root</artifactId>
<groupId>com.facebook.presto</groupId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-benchmark-runner</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-benchmark/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<artifactId>presto-root</artifactId>
<groupId>com.facebook.presto</groupId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-benchmark</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-benchto-benchmarks/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-benchto-benchmarks</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-bigquery/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-bigquery</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-blackhole/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-blackhole</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-bytecode/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-bytecode</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-cache/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-cache</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-cassandra/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-cassandra</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-cli/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-cli</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-clickhouse/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<artifactId>presto-root</artifactId>
<groupId>com.facebook.presto</groupId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-clickhouse</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-client/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-client</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-cluster-ttl-providers/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<parent>
<artifactId>presto-root</artifactId>
<groupId>com.facebook.presto</groupId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>

Expand Down
2 changes: 1 addition & 1 deletion presto-common/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-common</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-delta/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-delta</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-docs/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-docs</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-druid/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-druid</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-elasticsearch/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>
<artifactId>presto-elasticsearch</artifactId>
<description>Presto - Elasticsearch Connector</description>
Expand Down
2 changes: 1 addition & 1 deletion presto-example-http/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-example-http</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-expressions/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-expressions</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-function-namespace-managers/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<parent>
<artifactId>presto-root</artifactId>
<groupId>com.facebook.presto</groupId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>

Expand Down
2 changes: 1 addition & 1 deletion presto-geospatial-toolkit/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-geospatial-toolkit</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-google-sheets/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-google-sheets</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-grpc-api/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<parent>
<artifactId>presto-root</artifactId>
<groupId>com.facebook.presto</groupId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>

Expand Down
2 changes: 1 addition & 1 deletion presto-grpc-testing-udf-server/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<parent>
<artifactId>presto-root</artifactId>
<groupId>com.facebook.presto</groupId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>

Expand Down
2 changes: 1 addition & 1 deletion presto-hive-common/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<artifactId>presto-root</artifactId>
<groupId>com.facebook.presto</groupId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<properties>
Expand Down
2 changes: 1 addition & 1 deletion presto-hive-function-namespace/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<artifactId>presto-root</artifactId>
<groupId>com.facebook.presto</groupId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-hive-function-namespace</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-hive-hadoop2/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-hive-hadoop2</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-hive-metastore/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-hive-metastore</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-hive/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-hive</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-hudi/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<artifactId>presto-root</artifactId>
<groupId>com.facebook.presto</groupId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>
<artifactId>presto-hudi</artifactId>
<description>Presto - Hudi Connector</description>
Expand Down
2 changes: 1 addition & 1 deletion presto-i18n-functions/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-i18n-functions</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-iceberg/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>
<artifactId>presto-iceberg</artifactId>
<description>Presto - Iceberg Connector</description>
Expand Down
2 changes: 1 addition & 1 deletion presto-jdbc/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-jdbc</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-jmx/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-jmx</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-kafka/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-kafka</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-kudu/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>

<artifactId>presto-kudu</artifactId>
Expand Down
2 changes: 1 addition & 1 deletion presto-lark-sheets/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<parent>
<groupId>com.facebook.presto</groupId>
<artifactId>presto-root</artifactId>
<version>0.284-SNAPSHOT</version>
<version>0.284-edge19.2-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>

Expand Down
Loading