[SPARK] Support pyspark batch job by restful api #3836
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
3 tasks
turboFei
reviewed
Nov 22, 2022
kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/BatchesResource.scala
Outdated
Show resolved
Hide resolved
Codecov Report
@@ Coverage Diff @@
## master #3836 +/- ##
============================================
- Coverage 52.97% 52.94% -0.04%
Complexity 13 13
============================================
Files 497 497
Lines 28022 28029 +7
Branches 3863 3867 +4
============================================
- Hits 14845 14839 -6
- Misses 11781 11784 +3
- Partials 1396 1406 +10
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
leoluan2009
changed the title
leoluan2009
changed the title
pan3793
reviewed
Nov 23, 2022
kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
Outdated
Show resolved
Hide resolved
pan3793
approved these changes
Nov 23, 2022
Member
|
Do we have a chance to add a test? |
pan3793
reviewed
Nov 24, 2022
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KyuubiApplicationManager.scala
Outdated
Show resolved
Hide resolved
…plicationManager.scala
Member
|
Thanks, merged to master. (It's a new feature, I changed the milestone from 1.6.2 to 1.7.0) |
minyk
pushed a commit
to nexr/kyuubi
that referenced
this pull request
Mar 9, 2023
### _Why are the changes needed?_ Submit pyspark batch job by restful api ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request Closes apache#3836 from leoluan2009/pyspark-1. Closes apache#3836 550021a [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KyuubiApplicationManager.scala 357691c [Xuedong Luan] fix comment 7dfdbe2 [Xuedong Luan] fix comment 31bda17 [Xuedong Luan] [WIP] Support pyspark batch job by restful api Lead-authored-by: Xuedong Luan <[email protected]> Co-authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]> (cherry picked from commit 00d2d2e)
3 tasks
bowenliang123
added a commit
that referenced
this pull request
Oct 24, 2023
### _Why are the changes needed?_ To close #5380. As PySpark jobs become popular approach for data exploring and processing, we need to create tests for creating PySpark jobs. According the existing Spark Jar unit tests, two PySpark job unit test were added, they are all simple PI computing jobs from Spark examples. #### case1, "pyspark submit - basic batch rest client with existing resource file" It's almost same with the spark jar job test case, except the following two points: 1. param `batchType` should be set to `PYSPARK`, not `SPARK`. please refer to #3836 for detailed information. 2. For PySpark job，param `className` is useless， should be set to null #### case2, "pyspark submit - basic batch rest client with uploading resource file" Through the two test cases, simple PySpark jobs can be submitted normally. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes #5498 from weixi62961/unittest-batchapi-pyspark-simple. Closes #5380 b693efc [Bowen Liang] simplify sparkBatchTestResource 72a92b5 [Bowen Liang] Update kyuubi-server/src/test/scala/org/apache/kyuubi/server/rest/client/PySparkBatchRestApiSuite.scala b2035a3 [weixi] remove no necessary wrapper object "PySparkJobPI" 27d12e8 [weixi] rename from BatchRestApiPySparkSuite to PySparkBatchRestApiSuite e680e60 [weixi] Create a dedicated batch API suite for PySpark jobs. dc8b6bf [weixi] add 2 test cases for pyspark batch job submit. Lead-authored-by: weixi <[email protected]> Co-authored-by: Bowen Liang <[email protected]> Co-authored-by: Bowen Liang <[email protected]> Signed-off-by: liangbowen <[email protected]>
bowenliang123
added a commit
that referenced
this pull request
Oct 25, 2023
### _Why are the changes needed?_ To close #5380. As PySpark jobs become popular approach for data exploring and processing, we need to create tests for creating PySpark jobs. According the existing Spark Jar unit tests, two PySpark job unit test were added, they are all simple PI computing jobs from Spark examples. #### case1, "pyspark submit - basic batch rest client with existing resource file" It's almost same with the spark jar job test case, except the following two points: 1. param `batchType` should be set to `PYSPARK`, not `SPARK`. please refer to #3836 for detailed information. 2. For PySpark job，param `className` is useless， should be set to null #### case2, "pyspark submit - basic batch rest client with uploading resource file" Through the two test cases, simple PySpark jobs can be submitted normally. ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes #5498 from weixi62961/unittest-batchapi-pyspark-simple. Closes #5380 b693efc [Bowen Liang] simplify sparkBatchTestResource 72a92b5 [Bowen Liang] Update kyuubi-server/src/test/scala/org/apache/kyuubi/server/rest/client/PySparkBatchRestApiSuite.scala b2035a3 [weixi] remove no necessary wrapper object "PySparkJobPI" 27d12e8 [weixi] rename from BatchRestApiPySparkSuite to PySparkBatchRestApiSuite e680e60 [weixi] Create a dedicated batch API suite for PySpark jobs. dc8b6bf [weixi] add 2 test cases for pyspark batch job submit. Lead-authored-by: weixi <[email protected]> Co-authored-by: Bowen Liang <[email protected]> Co-authored-by: Bowen Liang <[email protected]> Signed-off-by: liangbowen <[email protected]> (cherry picked from commit 5cff4fb) Signed-off-by: liangbowen <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are the changes needed?
Submit pyspark batch job by restful api
How was this patch tested?
Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before make a pull request