[KYUUBI #4186] Spark showProgress with JobInfo #5410
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
davidyuan1223
changed the title
Member
|
@davidyuan1223 thanks for your contribution, it's not necessary to put everything with screenshots, for example, it's clear to the reviewer to check your unit test code via GitHub. And would you mind filling in "Why are the changes needed?" in the PR description? Generally, for most of the PR, it's clear for the reviewer to check the code to understand "what you changed", but not "why are the changes needed". In addition, we provided some contributor guidance in #5357, please read and fix the code style issue. |
Contributor
Author
ok, i will update the pr comment |
cxzl25
changed the title
wForget
reviewed
Oct 12, 2023
...ls/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SQLOperationListener.scala
Outdated
Show resolved
Hide resolved
wForget
reviewed
Oct 12, 2023
...ls/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SQLOperationListener.scala
Outdated
Show resolved
Hide resolved
wForget
reviewed
Oct 12, 2023
...ls/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SQLOperationListener.scala
Outdated
Show resolved
Hide resolved
wForget
reviewed
Oct 12, 2023
...kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SparkConsoleProgressBar.scala
Outdated
Show resolved
Hide resolved
wForget
reviewed
Oct 12, 2023
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/StageStatus.scala
Outdated
Show resolved
Hide resolved
wForget
reviewed
Oct 12, 2023
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/StageStatus.scala
Outdated
Show resolved
Hide resolved
wForget
reviewed
Oct 12, 2023
...uubi-spark-sql-engine/src/test/scala/org/apache/spark/kyuubi/SQLOperationListenerSuite.scala
Outdated
Show resolved
Hide resolved
1. remove duplicate synchronized 2. because the activeJobs is ConcurrentHashMap, so reduce synchronized 3. fix scala code style 4. change forEach to asScala code style 5. change conf str to KyuubiConf.XXX.key
wForget
reviewed
Oct 13, 2023
externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/StageStatus.scala
Outdated
Show resolved
Hide resolved
...kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SparkConsoleProgressBar.scala
Outdated
Show resolved
Hide resolved
1. fix new end line 2. provide Option[Int] with JobId
1. provide Option[Int] with JobId
1. provide Option[Int] with JobId
1. provide Option[Int] with JobId
wForget
requested changes
Oct 18, 2023
...ls/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SQLOperationListener.scala
Outdated
Show resolved
Hide resolved
wForget
reviewed
Oct 18, 2023
...kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SparkConsoleProgressBar.scala
Outdated
Show resolved
Hide resolved
update desc about findJob
wForget
approved these changes
Oct 19, 2023
3 tasks
cxzl25
reviewed
Oct 20, 2023
...kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SparkConsoleProgressBar.scala
Show resolved
Hide resolved
wForget
approved these changes
Oct 25, 2023
cxzl25
approved these changes
Oct 25, 2023
pan3793
reviewed
Oct 25, 2023
...ls/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SQLOperationListener.scala
Outdated
Show resolved
Hide resolved
…ark/kyuubi/SQLOperationListener.scala
pan3793
approved these changes
Oct 25, 2023
pan3793
added a commit
that referenced
this pull request
Oct 25, 2023
### _Why are the changes needed?_ current version, when set `kyuubi.session.engine.spark.showProgress=true`, it will show stage's progress info,but the info only show stage's detail, now we need to add job info in this, just like ``` [Stage 1:> (0 + 1) / 2] ``` to ``` [Job 1 (0 / 1) Stages] [Stage 1:> (0 + 1) / 2] ``` **this update is useful when user want know their sql execute detail** closes #4186 ### _How was this patch tested?_ - [x] Add screenshots for manual tests if appropriate **The photo show match log**  - [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes #5410 from davidyuan1223/improvement_add_job_log. Closes #4186 d8d03c4 [Cheng Pan] Update externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/SQLOperationListener.scala a06e9a1 [david yuan] Update SparkConsoleProgressBar.scala 8544084 [david yuan] Merge branch 'apache:master' into improvement_add_job_log 963ff18 [david yuan] Update SparkConsoleProgressBar.scala 9e46356 [david yuan] Update SparkConsoleProgressBar.scala 8c04dca [david yuan] Update SQLOperationListener.scala 39751bf [davidyuan] fix 4f657e7 [davidyuan] fix deleted files 86756eb [david yuan] Merge branch 'apache:master' into improvement_add_job_log 0c9ac27 [davidyuan] add showProgress with jobInfo Unit Test d4434a0 [davidyuan] Revert "add showProgress with jobInfo Unit Test" 84b1aa0 [davidyuan] Revert "improvement_add_job_log fix" 66126f9 [davidyuan] Merge remote-tracking branch 'origin/improvement_add_job_log' into improvement_add_job_log 228fd9c [davidyuan] add showProgress with jobInfo Unit Test 055e0ac [davidyuan] add showProgress with jobInfo Unit Test e4aac34 [davidyuan] Merge remote-tracking branch 'origin/improvement_add_job_log' into improvement_add_job_log b226ada [davidyuan] Merge remote-tracking branch 'origin/improvement_add_job_log' into improvement_add_job_log a08799c [david yuan] Update externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/StageStatus.scala a991b68 [david yuan] Update externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/spark/kyuubi/StageStatus.scala d12046d [davidyuan] add showProgress with jobInfo Unit Test 10a56b1 [davidyuan] add showProgress with jobInfo Unit Test a973cdd [davidyuan] improvement_add_job_log fix 1. provide Option[Int] with JobId e8a5108 [davidyuan] improvement_add_job_log fix 1. provide Option[Int] with JobId 7b9e874 [davidyuan] improvement_add_job_log fix 1. provide Option[Int] with JobId 5b4aaa8 [davidyuan] improvement_add_job_log fix 1. fix new end line 2. provide Option[Int] with JobId 780f9d1 [davidyuan] improvement_add_job_log fix 1. remove duplicate synchronized 2. because the activeJobs is ConcurrentHashMap, so reduce synchronized 3. fix scala code style 4. change forEach to asScala code style 5. change conf str to KyuubiConf.XXX.key 59340b7 [davidyuan] add showProgress with jobInfo Unit Test af05089 [davidyuan] add showProgress with jobInfo Unit Test c07535a [davidyuan] [Improvement] spark showProgress can briefly output info of the job #4186 d4bdec7 [yuanfuyuan] fix_4186 9fa8e73 [davidyuan] add showProgress with jobInfo Unit Test 49debfb [davidyuan] improvement_add_job_log fix 1. provide Option[Int] with JobId 5cf8714 [davidyuan] improvement_add_job_log fix 1. provide Option[Int] with JobId 249a422 [davidyuan] improvement_add_job_log fix 1. provide Option[Int] with JobId e15fc71 [davidyuan] improvement_add_job_log fix 1. fix new end line 2. provide Option[Int] with JobId 4564ef9 [davidyuan] improvement_add_job_log fix 1. remove duplicate synchronized 2. because the activeJobs is ConcurrentHashMap, so reduce synchronized 3. fix scala code style 4. change forEach to asScala code style 5. change conf str to KyuubiConf.XXX.key 32ad075 [davidyuan] add showProgress with jobInfo Unit Test d30492e [davidyuan] add showProgress with jobInfo Unit Test 6209c34 [davidyuan] [Improvement] spark showProgress can briefly output info of the job #4186 56b91a3 [yuanfuyuan] fix_4186 Lead-authored-by: davidyuan <[email protected]> Co-authored-by: davidyuan <[email protected]> Co-authored-by: david yuan <[email protected]> Co-authored-by: yuanfuyuan <[email protected]> Co-authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]> (cherry picked from commit ed0d997) Signed-off-by: Cheng Pan <[email protected]>
Member
|
thanks @davidyuan1223 and all reviewers, merged to master/1.8 |
pan3793
added a commit
that referenced
this pull request
Nov 8, 2023
### _Why are the changes needed?_ Current now, in spark-engine module, some session-level configurations are ignored due to the complexity of get session-level configurations in kyuubi spark engine, so As discussed in #5410 (comment). If we need unit test use withSessionConf method, we need make the code get configuration from the right session The PR is unfinished, it need wait the pr #5410 success so that i can use the new change in unit test closes #5438 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes #5487 from davidyuan1223/5438_add_common_method_to_support_session_config. Closes #5438 e1ded36 [davidyuan] add more optional session level to get conf 84c4568 [davidyuan] add more optional session level to get conf 4d70902 [davidyuan] add more optional session level to get conf 96d7cde [davidyuan] Revert "add more optional session level to get conf" 940f8f8 [davidyuan] add more optional session level to get conf 15641e8 [davidyuan] add more optional session level to get conf d838931 [davidyuan] Merge branch '5438_add_common_method_to_support_session_config' of https://github.com/davidyuan1223/kyuubi into 5438_add_common_method_to_support_session_config 2de96b5 [davidyuan] add common method to get session level config 3ec73ad [liangbowen] [KYUUBI #5522] [BATCH] Ignore main class for PySpark batch job submission d8b808d [Cheng Pan] [KYUUBI #5523] [DOC] Update the Kyuubi supported components version c7d15ae [Cheng Pan] [KYUUBI #5483] Release Spark TPC-H/DS Connectors with Scala 2.13 4a1db42 [zwangsheng] [KYUUBI #5513][BATCH] Always redirect delete batch request to Kyuubi instance that owns batch session b06e044 [labbomb] [KYUUBI #5517] [UI] Initial implement the SQL Lab page 88bb6b4 [liangbowen] [KYUUBI #5486] Bump Kafka client version from 3.4.0 to 3.5.1 538a648 [davidyuan] [KYUUBI #4186] Spark showProgress with JobInfo 682e5b5 [Xianxun Ye] [KYUUBI #5405] [FLINK] Support Flink 1.18 c71528e [Cheng Pan] [KYUUBI #5484] Remove legacy Web UI ee52b2a [Angerszhuuuu] [KYUUBI #5446][AUTHZ] Support Create/Drop/Show/Reresh index command for Hudi 6a5bb10 [weixi] [KYUUBI #5380][UT] Create PySpark batch jobs tests for RESTful API 86f692d [Kent Yao] [KYUUBI #5512] [AuthZ] Remove the non-existent query specs in Deletes and Updates dfdd7a3 [fwang12] [KYUUBI #5499][KYUUBI #2503] Catch any exception when closing idle session b7b3544 [伟程] [KYUUBI #5212] Fix configuration errors causing by helm charts of prometheus services d123a5a [liupeiyue] [KYUUBI #5282] Support configure Trino session conf in `kyuubi-default.conf` 0750437 [yangming] [KYUUBI #5294] [DOC] Update supported dialects for JDBC engine 9c75d82 [zwangsheng] [KYUUBI #5435][INFRA][TEST] Improve Kyuubi On Kubernetes IT 1dc264a [Angerszhuuuu] [KYUUBI #5479][AUTHZ] Support Hudi CallProcedureHoodieCommand for stored procedures bc3fcbb [Angerszhuuuu] [KYUUBI #5472] Permanent View should pass column when child plan no output a67b824 [Fantasy-Jay] [KYUUBI #5382][JDBC] Duplication cleanup improvement in JdbcDialect and schema helpers c039e1b [Kent Yao] [KYUUBI #5497] [AuthZ] Simplify debug message for missing field/method in ReflectUtils 0c8be79 [Angerszhuuuu] [KYUUBI #5475][FOLLOWUP] Authz check permanent view's subquery should check view's correct privilege 1293cf2 [Kent Yao] [KYUUBI #5500] Add Kyuubi Code Program to Doc e2754fe [Angerszhuuuu] [KYUUBI #5492][AUTHZ] saveAsTable create DataSource table miss db info 0c53d00 [Angerszhuuuu] [KYUUBI #5447][FOLLOWUP] Remove unrelated debug prints in TableIdentifierTableExtractor 119c393 [Angerszhuuuu] [KYUUBI #5447][AUTHZ] Support Hudi DeleteHoodieTableCommand/UpdateHoodieTableCommand/MergeIntoHoodieTableCommand 3af5ed1 [yikaifei] [KYUUBI #5427] [AUTHZ] Shade spark authz plugin 503c3f7 [davidyuan] Merge remote-tracking branch 'origin/5438_add_common_method_to_support_session_config' into 5438_add_common_method_to_support_session_config 7a67ace [davidyuan] add common method to get session level config 3f42317 [davidyuan] add common method to get session level config bb5d5ce [davidyuan] add common method to get session level config 623200f [davidyuan] Merge remote-tracking branch 'origin/5438_add_common_method_to_support_session_config' into 5438_add_common_method_to_support_session_config 8011959 [davidyuan] add common method to get session level config 605ef16 [davidyuan] Merge remote-tracking branch 'origin/5438_add_common_method_to_support_session_config' into 5438_add_common_method_to_support_session_config bb63ed8 [davidyuan] add common method to get session level config d9cf248 [davidyuan] add common method to get session level config c8647ef [davidyuan] add common method to get session level config 618c0f6 [david yuan] Merge branch 'apache:master' into 5438_add_common_method_to_support_session_config c1024bd [david yuan] Merge branch 'apache:master' into 5438_add_common_method_to_support_session_config 32028f9 [davidyuan] add common method to get session level config 03e2887 [davidyuan] add common method to get session level config Lead-authored-by: David Yuan <[email protected]> Co-authored-by: davidyuan <[email protected]> Co-authored-by: Angerszhuuuu <[email protected]> Co-authored-by: Cheng Pan <[email protected]> Co-authored-by: Kent Yao <[email protected]> Co-authored-by: liangbowen <[email protected]> Co-authored-by: david yuan <[email protected]> Co-authored-by: zwangsheng <[email protected]> Co-authored-by: yangming <[email protected]> Co-authored-by: 伟程 <[email protected]> Co-authored-by: weixi <[email protected]> Co-authored-by: fwang12 <[email protected]> Co-authored-by: Xianxun Ye <[email protected]> Co-authored-by: liupeiyue <[email protected]> Co-authored-by: Fantasy-Jay <[email protected]> Co-authored-by: yikaifei <[email protected]> Co-authored-by: labbomb <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
pan3793
added a commit
that referenced
this pull request
Nov 8, 2023
### _Why are the changes needed?_ Current now, in spark-engine module, some session-level configurations are ignored due to the complexity of get session-level configurations in kyuubi spark engine, so As discussed in #5410 (comment). If we need unit test use withSessionConf method, we need make the code get configuration from the right session The PR is unfinished, it need wait the pr #5410 success so that i can use the new change in unit test closes #5438 ### _How was this patch tested?_ - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible - [ ] Add screenshots for manual tests if appropriate - [x] [Run test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests) locally before make a pull request ### _Was this patch authored or co-authored using generative AI tooling?_ No Closes #5487 from davidyuan1223/5438_add_common_method_to_support_session_config. Closes #5438 e1ded36 [davidyuan] add more optional session level to get conf 84c4568 [davidyuan] add more optional session level to get conf 4d70902 [davidyuan] add more optional session level to get conf 96d7cde [davidyuan] Revert "add more optional session level to get conf" 940f8f8 [davidyuan] add more optional session level to get conf 15641e8 [davidyuan] add more optional session level to get conf d838931 [davidyuan] Merge branch '5438_add_common_method_to_support_session_config' of https://github.com/davidyuan1223/kyuubi into 5438_add_common_method_to_support_session_config 2de96b5 [davidyuan] add common method to get session level config 3ec73ad [liangbowen] [KYUUBI #5522] [BATCH] Ignore main class for PySpark batch job submission d8b808d [Cheng Pan] [KYUUBI #5523] [DOC] Update the Kyuubi supported components version c7d15ae [Cheng Pan] [KYUUBI #5483] Release Spark TPC-H/DS Connectors with Scala 2.13 4a1db42 [zwangsheng] [KYUUBI #5513][BATCH] Always redirect delete batch request to Kyuubi instance that owns batch session b06e044 [labbomb] [KYUUBI #5517] [UI] Initial implement the SQL Lab page 88bb6b4 [liangbowen] [KYUUBI #5486] Bump Kafka client version from 3.4.0 to 3.5.1 538a648 [davidyuan] [KYUUBI #4186] Spark showProgress with JobInfo 682e5b5 [Xianxun Ye] [KYUUBI #5405] [FLINK] Support Flink 1.18 c71528e [Cheng Pan] [KYUUBI #5484] Remove legacy Web UI ee52b2a [Angerszhuuuu] [KYUUBI #5446][AUTHZ] Support Create/Drop/Show/Reresh index command for Hudi 6a5bb10 [weixi] [KYUUBI #5380][UT] Create PySpark batch jobs tests for RESTful API 86f692d [Kent Yao] [KYUUBI #5512] [AuthZ] Remove the non-existent query specs in Deletes and Updates dfdd7a3 [fwang12] [KYUUBI #5499][KYUUBI #2503] Catch any exception when closing idle session b7b3544 [伟程] [KYUUBI #5212] Fix configuration errors causing by helm charts of prometheus services d123a5a [liupeiyue] [KYUUBI #5282] Support configure Trino session conf in `kyuubi-default.conf` 0750437 [yangming] [KYUUBI #5294] [DOC] Update supported dialects for JDBC engine 9c75d82 [zwangsheng] [KYUUBI #5435][INFRA][TEST] Improve Kyuubi On Kubernetes IT 1dc264a [Angerszhuuuu] [KYUUBI #5479][AUTHZ] Support Hudi CallProcedureHoodieCommand for stored procedures bc3fcbb [Angerszhuuuu] [KYUUBI #5472] Permanent View should pass column when child plan no output a67b824 [Fantasy-Jay] [KYUUBI #5382][JDBC] Duplication cleanup improvement in JdbcDialect and schema helpers c039e1b [Kent Yao] [KYUUBI #5497] [AuthZ] Simplify debug message for missing field/method in ReflectUtils 0c8be79 [Angerszhuuuu] [KYUUBI #5475][FOLLOWUP] Authz check permanent view's subquery should check view's correct privilege 1293cf2 [Kent Yao] [KYUUBI #5500] Add Kyuubi Code Program to Doc e2754fe [Angerszhuuuu] [KYUUBI #5492][AUTHZ] saveAsTable create DataSource table miss db info 0c53d00 [Angerszhuuuu] [KYUUBI #5447][FOLLOWUP] Remove unrelated debug prints in TableIdentifierTableExtractor 119c393 [Angerszhuuuu] [KYUUBI #5447][AUTHZ] Support Hudi DeleteHoodieTableCommand/UpdateHoodieTableCommand/MergeIntoHoodieTableCommand 3af5ed1 [yikaifei] [KYUUBI #5427] [AUTHZ] Shade spark authz plugin 503c3f7 [davidyuan] Merge remote-tracking branch 'origin/5438_add_common_method_to_support_session_config' into 5438_add_common_method_to_support_session_config 7a67ace [davidyuan] add common method to get session level config 3f42317 [davidyuan] add common method to get session level config bb5d5ce [davidyuan] add common method to get session level config 623200f [davidyuan] Merge remote-tracking branch 'origin/5438_add_common_method_to_support_session_config' into 5438_add_common_method_to_support_session_config 8011959 [davidyuan] add common method to get session level config 605ef16 [davidyuan] Merge remote-tracking branch 'origin/5438_add_common_method_to_support_session_config' into 5438_add_common_method_to_support_session_config bb63ed8 [davidyuan] add common method to get session level config d9cf248 [davidyuan] add common method to get session level config c8647ef [davidyuan] add common method to get session level config 618c0f6 [david yuan] Merge branch 'apache:master' into 5438_add_common_method_to_support_session_config c1024bd [david yuan] Merge branch 'apache:master' into 5438_add_common_method_to_support_session_config 32028f9 [davidyuan] add common method to get session level config 03e2887 [davidyuan] add common method to get session level config Lead-authored-by: David Yuan <[email protected]> Co-authored-by: davidyuan <[email protected]> Co-authored-by: Angerszhuuuu <[email protected]> Co-authored-by: Cheng Pan <[email protected]> Co-authored-by: Kent Yao <[email protected]> Co-authored-by: liangbowen <[email protected]> Co-authored-by: david yuan <[email protected]> Co-authored-by: zwangsheng <[email protected]> Co-authored-by: yangming <[email protected]> Co-authored-by: 伟程 <[email protected]> Co-authored-by: weixi <[email protected]> Co-authored-by: fwang12 <[email protected]> Co-authored-by: Xianxun Ye <[email protected]> Co-authored-by: liupeiyue <[email protected]> Co-authored-by: Fantasy-Jay <[email protected]> Co-authored-by: yikaifei <[email protected]> Co-authored-by: labbomb <[email protected]> Signed-off-by: Cheng Pan <[email protected]> (cherry picked from commit 9615db5) Signed-off-by: Cheng Pan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are the changes needed?
current version, when set
kyuubi.session.engine.spark.showProgress=true, it will show stage's progress info,but the info only show stage's detail, now we need to add job info in this, just liketo
this update is useful when user want know their sql execute detail
closes #4186
How was this patch tested?
Add screenshots for manual tests if appropriate
The photo show match log
Run test locally before make a pull request
Was this patch authored or co-authored using generative AI tooling?
No