Skip to content

[BUILD] [GA] Support JDK 11 and enable it in GitHub Action #693

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
pan3793 wants to merge 16 commits into from
Closed

[BUILD] [GA] Support JDK 11 and enable it in GitHub Action #693

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
c5b0e58
[GA] Enable JDK 11 in GA
pan3793 Jun 13, 2021
4283777
Add dependency jaxb-api
pan3793 Jun 14, 2021
b29aa2f
license for jaxb
pan3793 Jun 14, 2021
a7e4bdf
GA matrix
pan3793 Jun 14, 2021
7baeb96
minicluster
pan3793 Jun 14, 2021
51604d7
enlarge ENGINE_INIT_TIMEOUT
pan3793 Jun 27, 2021
a945ade
use in-memory catalog in EngineRefSuite
pan3793 Jul 3, 2021
38d2775
enlarge engine crash timeout
pan3793 Jul 3, 2021
c880edd
minor
pan3793 Jul 3, 2021
e86893c
enlarge YarnCluster timeout
pan3793 Jul 3, 2021
801e4cd
jdk11 in GA
pan3793 Jul 3, 2021
9f070a4
maven profile for jdk versions
pan3793 Jul 4, 2021
5fa74aa
disable test in kyuubi-extension-spark_3.1
pan3793 Jul 4, 2021
2b47939
address comments
pan3793 Jul 4, 2021
6ebac68
address comments
pan3793 Jul 4, 2021
5706daf
shrink scalatest memory
pan3793 Jul 4, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 21 additions & 7 deletions .github/workflows/master.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,18 +32,30 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
profiles:
- ''
- '-Pspark-3.0 -Dspark.archive.mirror=https://archive.apache.org/dist/spark/spark-3.1.2 -Dspark.archive.name=spark-3.1.2-bin-hadoop2.7.tgz -Dmaven.plugin.scalatest.exclude.tags=org.apache.kyuubi.tags.ExtendedSQLTest,org.apache.kyuubi.tags.DeltaTest,org.apache.kyuubi.tags.IcebergTest'
- '-Pspark-3.1 -Pkyuubi-extension-spark_3.1'
include:
- java: 8
spark: '3.0'
spark-hadoop: '2.7'
- java: 8
spark: '3.0'
spark-hadoop: '2.7'
profiles: '-Dspark.archive.mirror=https://archive.apache.org/dist/spark/spark-3.1.2 -Dspark.archive.name=spark-3.1.2-bin-hadoop2.7.tgz -Dmaven.plugin.scalatest.exclude.tags=org.apache.kyuubi.tags.ExtendedSQLTest,org.apache.kyuubi.tags.DeltaTest,org.apache.kyuubi.tags.IcebergTest'
- java: 8
spark: '3.1'
spark-hadoop: '3.2'
profiles: '-Pkyuubi-extension-spark_3.1'
- java: 11
spark: '3.1'
spark-hadoop: '3.2'

env:
SPARK_LOCAL_IP: localhost
steps:
- uses: actions/checkout@v2
- name: Setup JDK 1.8
- name: Setup JDK ${{ matrix.java }}
uses: actions/setup-java@v1
with:
java-version: '1.8'
java-version: ${{ matrix.java }}
- uses: actions/cache@v1
with:
path: ~/.m2/repository/com
Expand All @@ -69,7 +81,9 @@ jobs:
restore-keys: |
${{ runner.os }}-maven-io-
- name: Build with Maven
run: ./build/mvn clean install ${{ matrix.profiles }} -Dmaven.javadoc.skip=true -V
run: >-
./build/mvn clean install -Dmaven.javadoc.skip=true -V
-Pspark-${{ matrix.spark }} -Pspark-hadoop-${{ matrix.spark-hadoop }} ${{ matrix.profiles }}
- name: Code coverage
if: ${{ matrix.profiles == '' }}
run: bash <(curl -s https://codecov.io/bash)
Expand Down
4 changes: 4 additions & 0 deletions LICENSE-binary
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -249,6 +249,10 @@ Eclipse Public License (EPL) 2.0
--------------------------------
jakarta.servlet:jakarta.servlet-api https://projects.eclipse.org/projects/ee4j.servlet

Common Development and Distribution License (CDDL) 1.1
------------------------------------------------------
javax.xml.bind:jaxb-api https://github.com/javaee/jaxb-v2

MIT License
-----------
org.slf4j:slf4j-api
Expand Down
1 change: 1 addition & 0 deletions dev/dependencyList
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ jackson-core/2.11.4//jackson-core-2.11.4.jar
jackson-databind/2.11.4//jackson-databind-2.11.4.jar
jakarta.servlet-api/4.0.4//jakarta.servlet-api-4.0.4.jar
javassist/3.18.1-GA//javassist-3.18.1-GA.jar
jaxb-api/2.2.11//jaxb-api-2.2.11.jar
jcl-over-slf4j/1.7.30//jcl-over-slf4j-1.7.30.jar
jetty-http/9.4.41.v20210516//jetty-http-9.4.41.v20210516.jar
jetty-io/9.4.41.v20210516//jetty-io-9.4.41.v20210516.jar
Expand Down
6 changes: 6 additions & 0 deletions dev/kyuubi-extension-spark_3.1/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,12 @@
<scope>provided</scope>
</dependency>

<dependency>
<groupId>javax.xml.bind</groupId>
<artifactId>jaxb-api</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_${scala.binary.version}</artifactId>
Expand Down
5 changes: 5 additions & 0 deletions kyuubi-common/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,11 @@
<artifactId>hive-service-rpc</artifactId>
</dependency>

<dependency>
Copy link
Member

@yaooqinn yaooqinn Jul 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

who/where use this in the common module？

Copy link
Member Author

@pan3793 pan3793 Jul 4, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't use it. jaxb-api should be considered as a transitive dep of hadoop-client-runtime.
If we move jaxb-api to test scope, any accident invocation of Hadoop API which direct or indirect requires jaxb-api, the ut pass, but will cause runtime failures in production.

<groupId>javax.xml.bind</groupId>
<artifactId>jaxb-api</artifactId>
</dependency>

<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-minikdc</artifactId>
Expand Down
6 changes: 6 additions & 0 deletions kyuubi-main/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,12 @@
<artifactId>hadoop-client-minicluster</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>javax.activation</groupId>
<artifactId>activation</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

<build>
Expand Down
3 changes: 2 additions & 1 deletion kyuubi-main/src/test/scala/org/apache/kyuubi/WithKyuubiServer.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,8 @@ trait WithKyuubiServer extends KyuubiFunSuite {
conf.set(HA_ZK_QUORUM, zkServer.getConnectString)
conf.set(HA_ZK_ACL_ENABLED, false)

conf.setIfMissing(ENGINE_INIT_TIMEOUT, 120000L)
// TODO KYUUBI #745
conf.setIfMissing(ENGINE_INIT_TIMEOUT, 300000L)
server = KyuubiServer.startServer(conf)
super.beforeAll()
Thread.sleep(1500)
Expand Down
1 change: 1 addition & 0 deletions kyuubi-main/src/test/scala/org/apache/kyuubi/engine/EngineRefSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ class EngineRefSuite extends KyuubiFunSuite {
val zkData = Utils.createTempDir()
conf.set(ZookeeperConf.ZK_DATA_DIR, zkData.toString)
.set(ZookeeperConf.ZK_CLIENT_PORT, 0)
.set("spark.sql.catalogImplementation", "in-memory")
zkServer.initialize(conf)
zkServer.start()
super.beforeAll()
Expand Down
3 changes: 2 additions & 1 deletion ...i-main/src/test/scala/org/apache/kyuubi/operation/KyuubiOperationPerConnectionSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,8 @@ class KyuubiOperationPerConnectionSuite extends WithKyuubiServer with JDBCTestUt
executeStmtReq.setRunAsync(true)
val executeStmtResp = client.ExecuteStatement(executeStmtReq)

eventually(timeout(10.seconds), interval(500.milliseconds)) {
// TODO KYUUBI #745
eventually(timeout(60.seconds), interval(500.milliseconds)) {
val getOpStatusReq = new TGetOperationStatusReq(executeStmtResp.getOperationHandle)
val getOpStatusResp = client.GetOperationStatus(getOpStatusReq)
assert(getOpStatusResp.getStatus.getStatusCode === TStatusCode.SUCCESS_STATUS)
Expand Down
3 changes: 2 additions & 1 deletion kyuubi-main/src/test/scala/org/apache/kyuubi/operation/KyuubiOperationYarnClusterSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,8 @@ class KyuubiOperationYarnClusterSuite extends WithKyuubiServerOnYarn with JDBCTe
override protected def jdbcUrl: String = getJdbcUrl

override protected val kyuubiServerConf: KyuubiConf = {
KyuubiConf().set(ENGINE_INIT_TIMEOUT, 300000L)
// TODO KYUUBI #745
KyuubiConf().set(ENGINE_INIT_TIMEOUT, 600000L)
}

override protected val connectionConf: Map[String, String] = Map(
Expand Down
6 changes: 2 additions & 4 deletions kyuubi-main/src/test/scala/org/apache/kyuubi/server/MiniYarnService.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ class MiniYarnService(name: String) extends AbstractService(name) {

override def start(): Unit = {
yarnCluster.start()
val config = yarnCluster.getConfig()
val config = yarnCluster.getConfig
eventually(timeout(10.seconds), interval(100.milliseconds)) {
config.get(YarnConfiguration.RM_ADDRESS).split(":")(1) != "0"
}
Expand All @@ -112,7 +112,5 @@ class MiniYarnService(name: String) extends AbstractService(name) {
writer.close()
}

def getHadoopConfDir(): String = {
hadoopConfDir.getAbsolutePath
}
def getHadoopConfDir: String = hadoopConfDir.getAbsolutePath
}
Loading