Skip to content

Conversation

@davidyuan1223
Copy link
Contributor

@davidyuan1223 davidyuan1223 commented Oct 20, 2023

Why are the changes needed?

Current now, in spark-engine module, some session-level configurations are ignored due to the complexity of get session-level configurations in kyuubi spark engine, so As discussed in #5410 (comment). If we need unit test use withSessionConf method, we need make the code get configuration from the right session

The PR is unfinished, it need wait the pr #5410 success so that i can use the new change in unit test

closes #5438

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before make a pull request

Was this patch authored or co-authored using generative AI tooling?

No

@codecov-commenter
Copy link

codecov-commenter commented Oct 20, 2023

Codecov Report

Merging #5487 (e1ded36) into master (03d6223) will increase coverage by 61.60%.
Report is 55 commits behind head on master.
The diff coverage is 91.27%.

@@              Coverage Diff              @@
##             master    #5487       +/-   ##
=============================================
+ Coverage      0.00%   61.60%   +61.60%     
- Complexity        0       23       +23     
=============================================
  Files           588      603       +15     
  Lines         33480    35657     +2177     
  Branches       4405     4866      +461     
=============================================
+ Hits              0    21967    +21967     
+ Misses        33480    11307    -22173     
- Partials          0     2383     +2383     
Files Coverage Δ
...ache/kyuubi/plugin/spark/authz/OperationType.scala 100.00% <100.00%> (+100.00%) ⬆️
.../kyuubi/plugin/spark/authz/PrivilegesBuilder.scala 90.69% <100.00%> (+90.69%) ⬆️
...g/apache/kyuubi/engine/spark/KyuubiSparkUtil.scala 86.48% <100.00%> (+86.48%) ⬆️
...uubi/engine/spark/operation/ExecuteStatement.scala 79.59% <100.00%> (+79.59%) ⬆️
...ache/kyuubi/engine/spark/operation/GetTables.scala 97.77% <100.00%> (+97.77%) ⬆️
...ubi/engine/spark/operation/PlanOnlyStatement.scala 47.11% <100.00%> (+47.11%) ⬆️
...kyuubi/engine/spark/operation/SparkOperation.scala 75.67% <100.00%> (+75.67%) ⬆️
...in/scala/org/apache/spark/kyuubi/StageStatus.scala 75.00% <100.00%> (+75.00%) ⬆️
...in/scala/org/apache/kyuubi/config/KyuubiConf.scala 97.25% <ø> (+97.25%) ⬆️
...ache/kyuubi/server/KyuubiRestFrontendService.scala 78.64% <100.00%> (+78.64%) ⬆️
... and 11 more

... and 558 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@github-actions github-actions bot added kind:documentation Documentation is a feature! kind:infra license, community building, project builds, asf infra related, etc. module:server kind:build module:ui labels Oct 26, 2023
davidyuan1223 and others added 14 commits October 26, 2023 10:53
…t_session_config' into 5438_add_common_method_to_support_session_config
…t_session_config' into 5438_add_common_method_to_support_session_config
### _Why are the changes needed?_

This PR aims to shade the kyuubi spark authz plugin to simplify the user's use.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [x] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No

Closes #5427 from Yikf/shade-authz.

Closes #5427

d2f7ea8 [yikaifei] fix
695133d [Kent Yao] Update docs/security/authorization/spark/install.md
f3a6531 [Kent Yao] Update docs/security/authorization/spark/build.md
963cab3 [yikaifei] bundle
2068c98 [yikaifei] relocation
6c6e50e [yikaifei] Shade spark authz plugin

Lead-authored-by: yikaifei <[email protected]>
Co-authored-by: Kent Yao <[email protected]>
Signed-off-by: yikaifei <[email protected]>
…dieTableCommand/MergeIntoHoodieTableCommand

### _Why are the changes needed?_
To close #5447. Kyuubi authz Support hudi DeleteHoodieTableCommand/UpdateHoodieTableCommand/MergeIntoHoodieTableCommand

- DeleteHoodieTableCommand: https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/DeleteHoodieTableCommand.scala
- UpdateHoodieTableCommand: https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/UpdateHoodieTableCommand.scala
- MergeIntoHoodieTableCommand: https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/MergeIntoHoodieTableCommand.scala

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_
No

Closes #5482 from AngersZhuuuu/KYUUBI-5447.

Closes #5447

2598af2 [Angerszhuuuu] Update HudiCatalogRangerSparkExtensionSuite.scala
08be589 [Angerszhuuuu] Update org.apache.kyuubi.plugin.spark.authz.serde.QueryExtractor
19497d1 [Angerszhuuuu] Update tableExtractors.scala
df6e244 [Angerszhuuuu] update
1a72f13 [Angerszhuuuu] update
f7ca684 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5447
3700686 [Angerszhuuuu] [KYUUBI #5447][AUTHZ] Support hudi DeleteHoodieTableCommand/UpdateHoodieTableCommand/MergeIntoHoodieTableCommand

Authored-by: Angerszhuuuu <[email protected]>
Signed-off-by: liangbowen <[email protected]>
…fierTableExtractor

### _Why are the changes needed?_
To close #5447
Remove unrelated debug change in  tableExtractor

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_
No

Closes #5494 from AngersZhuuuu/KYUUBI-5447-FOLLOWUP.

Closes #5447

90a0bcd [Bowen Liang] Update extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/serde/tableExtractors.scala
85941e2 [Angerszhuuuu] [KYUUBI #5447][AUTHZ] Remove unrelated debug change in  tableExtractor

Lead-authored-by: Angerszhuuuu <[email protected]>
Co-authored-by: Bowen Liang <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
### _Why are the changes needed?_
To fix #5492
When we use saveAsTable and write as a DataSource table, since CreateTableAsDataSource command's catalogTable was directly constructed by identifier and only will check when executing, so here authz will miss db information as below case
![image](https://github.com/apache/kyuubi/assets/46485123/f6135598-489a-47db-89eb-0dba3843b90d)

This fixes this issue by following the Spark's code
<img width="1256" alt="截屏2023-10-21 下午3 37 37" src="https://github.com/apache/kyuubi/assets/46485123/d7c66db4-400b-4637-b75a-973a8a8a9968">

### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_
No

Closes #5493 from AngersZhuuuu/KYUUBI-5492.

Closes #5492

46867af [Angerszhuuuu] update
7a4c86f [Angerszhuuuu] Merge branch 'master' into KYUUBI-5492
f56f299 [Angerszhuuuu] Update RangerSparkExtensionSuite.scala
7bcfc23 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5492
6f24728 [Angerszhuuuu] [KYUUBI #5492] saveAsTable create DataSource table miss db info

Authored-by: Angerszhuuuu <[email protected]>
Signed-off-by: Kent Yao <[email protected]>
### _Why are the changes needed?_

Add Kyuubi Code Program to Doc for long-term management and sponsorship.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

no

Closes #5500 from yaooqinn/program.

Closes #5500

14f1447 [Kent Yao] Add Kyuubi Code Program to Doc

Authored-by: Kent Yao <[email protected]>
Signed-off-by: Kent Yao <[email protected]>
… check view's correct privilege

### _Why are the changes needed?_
To fix #5475

In issue #5417 we fixed the problem that AUTHZ will still check scalar-subquery/in-subquery in permanent will.
But we just ignore the check, the subquery still will run, in this PR,  we record the permanent view's visited column to check the permanent view's privilege to avoid extra execution effort.

For the test `[KYUUBI #5417] should not check scalar-subquery in permanent view` I print all the plan that pass to privilege builder as below
<img width="1398" alt="截屏2023-10-19 下午4 05 46" src="https://github.com/apache/kyuubi/assets/46485123/b136bb47-816c-4066-aba7-a74cbe323f7d">

before this pr
<img width="1310" alt="截屏2023-10-19 下午4 15 29" src="https://github.com/apache/kyuubi/assets/46485123/aa2e3cfe-bca7-493d-a364-b2c196c76c3a">

This two graph shows this pr deny the execution of subquery when we don't have  the veiw's privilege

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

Closes #5476 from AngersZhuuuu/KYUUBI-5475.

Closes #5475

e1f7920 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5475
3bfd9e6 [Angerszhuuuu] update
6b8c0e6 [Angerszhuuuu] Merge branch 'master' into KYUUBI-5475
f7585a4 [Angerszhuuuu] Update PrivilegesBuilder.scala
faea9c6 [Angerszhuuuu] [KYUUBI #5475] Authz check permanent view's subquery should check view's correct privilege

Authored-by: Angerszhuuuu <[email protected]>
Signed-off-by: Kent Yao <[email protected]>
…d in ReflectUtils

### _Why are the changes needed?_

It's easy for developers to check a member or method from a Java class using a code viewer or online Java doc.

The current debug msg is kinda noisy for them to locate the key information they want, which is
Java class and field name. It makes our debug log unreadable.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

Closes #5497 from yaooqinn/debug.

Closes #5497

3ad42d5 [Kent Yao] addr comments
23e8e7a [Kent Yao] [AuthZ] Simplify debug message for missing field/methond in ReflectUtils
b7a7fbb [Kent Yao] [AuthZ] Simplify debug message for missing field/methond in ReflectUtils
9f94c62 [Kent Yao] [AuthZ] Simplify debug message for missing field/methond in ReflectUtils
78a66a3 [Kent Yao] [AuthZ] Simplify debug message for missing field/methond in ReflectUtils

Authored-by: Kent Yao <[email protected]>
Signed-off-by: liangbowen <[email protected]>
…nd schema helpers

### _Why are the changes needed?_

To close #5382.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No

Closes #5490 from zhuyaogai/issue-5382.

Closes #5382

4757445 [Fantasy-Jay] Remove unrelated comment.
f68c7aa [Fantasy-Jay] Refactor JDBC engine to reduce to code duplication.
4ad6b3c [Fantasy-Jay] Refactor JDBC engine to reduce to code duplication.

Authored-by: Fantasy-Jay <[email protected]>
Signed-off-by: liangbowen <[email protected]>
@davidyuan1223 davidyuan1223 reopened this Oct 26, 2023
@github-actions github-actions bot removed kind:documentation Documentation is a feature! kind:infra license, community building, project builds, asf infra related, etc. module:server kind:build module:ui labels Oct 26, 2023
@davidyuan1223 davidyuan1223 changed the title [WIP][KYUUBI #5438] add common method to get session level config [KYUUBI #5438] add common method to get session level config Oct 26, 2023
@davidyuan1223
Copy link
Contributor Author

@wForget could you review the pr?

@wForget
Copy link
Member

wForget commented Oct 27, 2023

@davidyuan1223 Can you use this method for more session level configurations.

@davidyuan1223
Copy link
Contributor Author

@davidyuan1223 Can you change this method for more session level configurations.

ok

@wForget
Copy link
Member

wForget commented Oct 27, 2023

Most of them like:

sparkSession.conf.getOption(XXX)
  .orElse(session.sessionManager.getConf(XXX))

@davidyuan1223
Copy link
Contributor Author

Most of them like:

sparkSession.conf.getOption(XXX)
  .orElse(session.sessionManager.getConf(XXX))

i have fix this problem, i think 3 session level is enough for this module, they are SparkConf,SparkSession.KyuubiConf,Kyuubi's Session, what do you think?

@wForget
Copy link
Member

wForget commented Oct 30, 2023

i have fix this problem, i think 3 session level is enough for this module, they are SparkConf,SparkSession.KyuubiConf,Kyuubi's Session, what do you think?

Sorry, I may not have described it clearly enough. What I mean is changing the way to get the old configurations, not changing this method.

@wForget
Copy link
Member

wForget commented Oct 30, 2023

BTW, session.sessionManager.getConf and SparkSQLEngine.kyuubiConf are the same object.

@davidyuan1223
Copy link
Contributor Author

can we merge? @wForget

Copy link
Member

@wForget wForget left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, Thanks.

@pan3793 pan3793 changed the title [KYUUBI #5438] add common method to get session level config [KYUUBI #5438] Add common method to get session level config Nov 7, 2023
@pan3793 pan3793 closed this in 9615db5 Nov 8, 2023
pan3793 added a commit that referenced this pull request Nov 8, 2023
### _Why are the changes needed?_

Current now, in spark-engine module, some session-level configurations are ignored due to the complexity of get session-level configurations in kyuubi spark engine, so As discussed in #5410 (comment). If we need unit test use withSessionConf method, we need make the code get configuration from the right session

The PR is unfinished, it need wait the pr #5410 success so that i can use the new change in unit test

closes #5438
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request

### _Was this patch authored or co-authored using generative AI tooling?_

No

Closes #5487 from davidyuan1223/5438_add_common_method_to_support_session_config.

Closes #5438

e1ded36 [davidyuan] add more optional session level to get conf
84c4568 [davidyuan] add more optional session level to get conf
4d70902 [davidyuan] add more optional session level to get conf
96d7cde [davidyuan] Revert "add more optional session level to get conf"
940f8f8 [davidyuan] add more optional session level to get conf
15641e8 [davidyuan] add more optional session level to get conf
d838931 [davidyuan] Merge branch '5438_add_common_method_to_support_session_config' of https://github.com/davidyuan1223/kyuubi into 5438_add_common_method_to_support_session_config
2de96b5 [davidyuan] add common method to get session level config
3ec73ad [liangbowen] [KYUUBI #5522] [BATCH] Ignore main class for PySpark batch job submission
d8b808d [Cheng Pan] [KYUUBI #5523] [DOC] Update the Kyuubi supported components version
c7d15ae [Cheng Pan] [KYUUBI #5483] Release Spark TPC-H/DS Connectors with Scala 2.13
4a1db42 [zwangsheng] [KYUUBI #5513][BATCH] Always redirect delete batch request to Kyuubi instance that owns batch session
b06e044 [labbomb] [KYUUBI #5517] [UI] Initial implement the SQL Lab page
88bb6b4 [liangbowen] [KYUUBI #5486] Bump Kafka client version from 3.4.0 to 3.5.1
538a648 [davidyuan] [KYUUBI #4186] Spark showProgress with JobInfo
682e5b5 [Xianxun Ye] [KYUUBI #5405] [FLINK] Support Flink 1.18
c71528e [Cheng Pan] [KYUUBI #5484] Remove legacy Web UI
ee52b2a [Angerszhuuuu] [KYUUBI #5446][AUTHZ] Support Create/Drop/Show/Reresh index command for Hudi
6a5bb10 [weixi] [KYUUBI #5380][UT] Create PySpark batch jobs tests for RESTful API
86f692d [Kent Yao] [KYUUBI #5512] [AuthZ] Remove the non-existent query specs in Deletes and Updates
dfdd7a3 [fwang12] [KYUUBI #5499][KYUUBI #2503] Catch any exception when closing idle session
b7b3544 [伟程] [KYUUBI #5212] Fix configuration errors causing by helm charts of prometheus services
d123a5a [liupeiyue] [KYUUBI #5282] Support configure Trino session conf in `kyuubi-default.conf`
0750437 [yangming] [KYUUBI #5294] [DOC] Update supported dialects for JDBC engine
9c75d82 [zwangsheng] [KYUUBI #5435][INFRA][TEST] Improve Kyuubi On Kubernetes IT
1dc264a [Angerszhuuuu] [KYUUBI #5479][AUTHZ] Support Hudi CallProcedureHoodieCommand for stored procedures
bc3fcbb [Angerszhuuuu] [KYUUBI #5472] Permanent View should pass column when child plan no output
a67b824 [Fantasy-Jay] [KYUUBI #5382][JDBC] Duplication cleanup improvement in JdbcDialect and schema helpers
c039e1b [Kent Yao] [KYUUBI #5497] [AuthZ] Simplify debug message for missing field/method in ReflectUtils
0c8be79 [Angerszhuuuu] [KYUUBI #5475][FOLLOWUP] Authz check permanent view's subquery should check view's correct privilege
1293cf2 [Kent Yao] [KYUUBI #5500] Add Kyuubi Code Program to Doc
e2754fe [Angerszhuuuu] [KYUUBI #5492][AUTHZ] saveAsTable create DataSource table miss db info
0c53d00 [Angerszhuuuu] [KYUUBI #5447][FOLLOWUP] Remove unrelated debug prints in TableIdentifierTableExtractor
119c393 [Angerszhuuuu] [KYUUBI #5447][AUTHZ] Support Hudi DeleteHoodieTableCommand/UpdateHoodieTableCommand/MergeIntoHoodieTableCommand
3af5ed1 [yikaifei] [KYUUBI #5427] [AUTHZ] Shade spark authz plugin
503c3f7 [davidyuan] Merge remote-tracking branch 'origin/5438_add_common_method_to_support_session_config' into 5438_add_common_method_to_support_session_config
7a67ace [davidyuan] add common method to get session level config
3f42317 [davidyuan] add common method to get session level config
bb5d5ce [davidyuan] add common method to get session level config
623200f [davidyuan] Merge remote-tracking branch 'origin/5438_add_common_method_to_support_session_config' into 5438_add_common_method_to_support_session_config
8011959 [davidyuan] add common method to get session level config
605ef16 [davidyuan] Merge remote-tracking branch 'origin/5438_add_common_method_to_support_session_config' into 5438_add_common_method_to_support_session_config
bb63ed8 [davidyuan] add common method to get session level config
d9cf248 [davidyuan] add common method to get session level config
c8647ef [davidyuan] add common method to get session level config
618c0f6 [david yuan] Merge branch 'apache:master' into 5438_add_common_method_to_support_session_config
c1024bd [david yuan] Merge branch 'apache:master' into 5438_add_common_method_to_support_session_config
32028f9 [davidyuan] add common method to get session level config
03e2887 [davidyuan] add common method to get session level config

Lead-authored-by: David Yuan <[email protected]>
Co-authored-by: davidyuan <[email protected]>
Co-authored-by: Angerszhuuuu <[email protected]>
Co-authored-by: Cheng Pan <[email protected]>
Co-authored-by: Kent Yao <[email protected]>
Co-authored-by: liangbowen <[email protected]>
Co-authored-by: david yuan <[email protected]>
Co-authored-by: zwangsheng <[email protected]>
Co-authored-by: yangming <[email protected]>
Co-authored-by: 伟程 <[email protected]>
Co-authored-by: weixi <[email protected]>
Co-authored-by: fwang12 <[email protected]>
Co-authored-by: Xianxun Ye <[email protected]>
Co-authored-by: liupeiyue <[email protected]>
Co-authored-by: Fantasy-Jay <[email protected]>
Co-authored-by: yikaifei <[email protected]>
Co-authored-by: labbomb <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
(cherry picked from commit 9615db5)
Signed-off-by: Cheng Pan <[email protected]>
@pan3793 pan3793 added this to the v1.8.1 milestone Nov 8, 2023
@pan3793
Copy link
Member

pan3793 commented Nov 8, 2023

Thanks, merged to master/1.8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[TASK][EASY] Add a common getSessionConf method in spark engine module