[DO-NOT-MERGE][KYUUBI #2619] Add profile spark-3.3 #2620

cfmcgrady · 2022-05-10T09:46:30Z

Why are the changes needed?

How was this patch tested?

Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before make a pull request

cfmcgrady · 2022-05-10T09:48:25Z

...ension-spark-common/src/main/scala/org/apache/kyuubi/sql/watchdog/MaxPartitionStrategy.scala

              _,
              _,
-              _)) if fsRelation.partitionSchemaOption.isDefined =>
+              _)) if fsRelation.partitionSchema.nonEmpty =>


HadoopFsRelation.partitionSchemaOption was removed by apache/spark#34582

cfmcgrady · 2022-05-10T09:49:51Z

...k/kyuubi-extension-spark-3-3/src/main/scala/org/apache/kyuubi/sql/KyuubiSparkSQLParser.scala

+    delegate.parseDataType(sqlText)
+  }
+
+  override def parseQuery(sqlText: String): LogicalPlan = {


the new abstract function parseQuery was added by apache/spark#34543

codecov-commenter · 2022-05-11T06:57:24Z

Codecov Report

Merging #2620 (4380f4c) into master (8de2f5f) will decrease coverage by 3.14%.
The diff coverage is 75.00%.

@@             Coverage Diff              @@
##             master    #2620      +/-   ##
============================================
- Coverage     64.31%   61.16%   -3.15%     
+ Complexity       80       39      -41     
============================================
  Files           385      382       -3     
  Lines         18628    18564      -64     
  Branches       2523     2514       -9     
============================================
- Hits          11980    11355     -625     
- Misses         5507     6123     +616     
+ Partials       1141     1086      -55

Impacted Files	Coverage Δ
...che/kyuubi/sql/watchdog/MaxPartitionStrategy.scala	`0.00% <0.00%> (ø)`
...in/scala/org/apache/kyuubi/config/KyuubiConf.scala	`96.75% <100.00%> (+0.13%)`	⬆️
...pache/kyuubi/sql/KyuubiSQLExtensionException.scala	`0.00% <0.00%> (-100.00%)`	⬇️
...rg/apache/kyuubi/sql/zorder/ZorderBytesUtils.scala	`0.00% <0.00%> (-94.74%)`	⬇️
...ubi/sql/zorder/InsertZorderBeforeWritingBase.scala	`0.00% <0.00%> (-89.16%)`	⬇️
...cala/org/apache/kyuubi/sql/zorder/ZorderBase.scala	`0.00% <0.00%> (-87.50%)`	⬇️
.../kyuubi/sql/zorder/OptimizeZorderCommandBase.scala	`0.00% <0.00%> (-81.82%)`	⬇️
...yuubi/sql/zorder/OptimizeZorderStatementBase.scala	`0.00% <0.00%> (-71.43%)`	⬇️
...g/apache/kyuubi/sql/zorder/ResolveZorderBase.scala	`0.00% <0.00%> (-69.57%)`	⬇️
...pache/kyuubi/sql/KyuubiQueryStagePreparation.scala	`0.00% <0.00%> (-32.36%)`	⬇️
... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8de2f5f...4380f4c. Read the comment docs.

cfmcgrady · 2022-05-13T09:08:22Z

GA passed.

Ignore flaky test JpsApplicationOperation with spark local mode. Flaky test: JpsApplicationOperation with spark local mode #2642
Disable module kyuubi-spark-authz for spark-3.3, we can add support in a separate PR.
add maven profile spark-3.3, and also add the GitHub action job.
all existing spark extensions have passed CI.

cc @pan3793 @ulysses-you @turboFei

ulysses-you · 2022-05-13T09:13:18Z

Disable module kyuubi-spark-authz for spark-3.3, we can add support in a separate PR.

cc @yaooqinn

ulysses-you · 2022-05-13T09:28:09Z

@cfmcgrady thank you for the investigation !

yaooqinn · 2022-05-13T09:35:12Z

Disable module kyuubi-spark-authz for spark-3.3, we can add support in a separate PR.

it breaks ?

cfmcgrady · 2022-05-13T09:53:46Z

Disable module kyuubi-spark-authz for spark-3.3, we can add support in a separate PR.

it breaks ?

yes, it also happen in the master branch. https://github.com/apache/incubator-kyuubi/runs/6416448132?check_suite_focus=true

yaooqinn · 2022-05-13T09:56:58Z

shall we fix the failure first? spark release process for major one take quite a long time

cfmcgrady · 2022-05-13T11:30:48Z

shall we fix the failure first? spark release process for major one take quite a long time

Ignore the flaky test to pass CI, to see whether we have other compatibility issues. I will enable the unit test after we fix the issue.

yaooqinn · 2022-05-13T11:40:48Z

seems we are not discussing the same thing，I only mean removing authz module is unacceptable as we have PRs for it now.

cfmcgrady · 2022-05-13T12:06:31Z

seems we are not discussing the same thing，I only mean removing authz module is unacceptable as we have PRs for it now.

Oops, I misread before.

we have 31 failed unit tests in the module kyuubi-spark-authz, I think we can raise a separate pr to fix it based on this one.

FYI: https://github.com/apache/incubator-kyuubi/runs/6418487737?check_suite_focus=true

### _Why are the changes needed?_ to close #2619 add maven profile `spark-3.3`, part of #2620 ### _How was this patch tested?_ Pass CI. Closes #2692 from cfmcgrady/kyuubi-2619-1. Closes #2619 fbd336f [Fu Chen] refine SparkProcessBuilderSuite - `engine log truncation` 039ef82 [Fu Chen] move module kyuubi-spark-connector-kudu to spark-3.3 03d799e [Fu Chen] update kyuubi.operation.plan.only.excludes 724fc7a [Fu Chen] GA 53b97a1 [Fu Chen] add profile `spark-3.3` Authored-by: Fu Chen <[email protected]> Signed-off-by: Cheng Pan <[email protected]>

### _Why are the changes needed?_ to close #2706 Spark extensions support Spark-3.3, part of #2620 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes #2707 from cfmcgrady/kyuubi-2706. Closes #2706 0b07b6e [Fu Chen] spark extensions support spark 3.3 Authored-by: Fu Chen <[email protected]> Signed-off-by: Cheng Pan <[email protected]>

github-actions bot added kind:build module:spark labels May 10, 2022

cfmcgrady commented May 10, 2022

View reviewed changes

github-actions bot added kind:infra license, community building, project builds, asf infra related, etc. module:common kind:documentation Documentation is a feature! labels May 11, 2022

cfmcgrady added 10 commits May 13, 2022 11:00

add profile spark-3.3

0460a0d

move ZorderSuite

c0210c2

move InsertShuffleNodeBeforeJoinSuite

918fcee

fixt DropIgnoreNonexistent

566dfe7

add ci

0380d69

update spark-3.3.0 rc1 archive mirror

9c44cff

fix PlanOnlyOperationSuite

0bfcfb7

fix style

f4678fc

update docs

da161b4

ignore JpsApplicationOperation with spark local mode

7ca2d72

cfmcgrady force-pushed the kyuubi-2619 branch from 8bcabdb to 7ca2d72 Compare May 13, 2022 05:10

github-actions bot added the module:server label May 13, 2022

diabbled authz for spark-3.3

4380f4c

typo

f915fb6

cfmcgrady changed the title ~~[WIP][KYUUBI #2619] Add profile spark-3.3~~ [KYUUBI #2619] Add profile spark-3.3 May 13, 2022

cfmcgrady changed the title ~~[KYUUBI #2619] Add profile spark-3.3~~ [DO-NOT-MERGE][KYUUBI #2619] Add profile spark-3.3 May 18, 2022

cfmcgrady mentioned this pull request May 18, 2022

[KYUUBI #2619] Add profile spark-3.3 #2692

Closed

cfmcgrady mentioned this pull request May 19, 2022

[KYUUBI #2706] Spark extensions support Spark-3.3 #2707

Closed

3 tasks

cfmcgrady closed this May 23, 2022

[DO-NOT-MERGE][KYUUBI #2619] Add profile spark-3.3 #2620

[DO-NOT-MERGE][KYUUBI #2619] Add profile spark-3.3 #2620

Uh oh!

Conversation

cfmcgrady commented May 10, 2022

Why are the changes needed?

How was this patch tested?

Uh oh!

cfmcgrady May 10, 2022

Choose a reason for hiding this comment

Uh oh!

cfmcgrady May 10, 2022

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented May 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

cfmcgrady commented May 13, 2022

Uh oh!

ulysses-you commented May 13, 2022

Uh oh!

ulysses-you commented May 13, 2022

Uh oh!

yaooqinn commented May 13, 2022

Uh oh!

cfmcgrady commented May 13, 2022

Uh oh!

yaooqinn commented May 13, 2022

Uh oh!

cfmcgrady commented May 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yaooqinn commented May 13, 2022

Uh oh!

cfmcgrady commented May 13, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov-commenter commented May 11, 2022 •

edited

Loading

cfmcgrady commented May 13, 2022 •

edited

Loading

cfmcgrady commented May 13, 2022 •

edited

Loading