[SPARK-31792][SS][DOCS] Introduce the structured streaming UI in the Web UI doc #28609
Conversation
|
@gatorsmile, please take a look when it's convenient for you. |
|
ok to test |
|
cc @xuanyuanking FYI |
|
Test build #123046 has finished for PR 28609 at commit
|
xuanyuanking
left a comment
xuanyuanking
left a comment
There was a problem hiding this comment.
Thanks Xingcan for the document! Also cc @uncleGen
docs/web-ui.md
Outdated
| </p> | ||
|
|
||
| The statistics page displays some useful metrics (e.g., how many records received per second and | ||
| the time used for different stages of a micro-batch execution) for insight into the status of your |
There was a problem hiding this comment.
How about adding a section for the metrics in operation duration histogram?
There was a problem hiding this comment.
+1, we can also give some information for each metrics, like:
- Input Rate: The aggregate (across all sources) rate of data arriving.
- Process Rate: The aggregate (across all sources) rate at which Spark is processing data.
- Input Rows: The aggregate (across all sources) number of records processed in a trigger.
- Batch Duration: The process duration of each batch.
There was a problem hiding this comment.
Sounds good. I'll add a list of metrics.
docs/web-ui.md
Outdated
| ## Structured Streaming Tab | ||
| When running Structured Streaming jobs in micro-batch mode, a Structured Streaming tab will be | ||
| available on the Web UI. You can check the statistics of a running query or a completed query | ||
| via clicking a run id on the Streaming Query overview page. |
There was a problem hiding this comment.
We can also check the latest exception for the failed query.
There was a problem hiding this comment.
Oh, sorry for missing this part. I'll add this.
|
@xuanyuanking @uncleGen Thanks for your feedback. I've added the requested parts. Please take a look. Thanks.
|
|
Test build #123100 has finished for PR 28609 at commit
|
xuanyuanking
left a comment
xuanyuanking
left a comment
There was a problem hiding this comment.
LGTM, thanks Xingcan for the changes! cc @HyukjinKwon @gatorsmile
Let's change the title to "[SPARK-31792][SS][DOCS] ..."
xccui
changed the title
|
Merged to master and branch-3.0. |
…Web UI doc ### What changes were proposed in this pull request? This PR adds the structured streaming UI introduction to the Web UI doc.  ### Why are the changes needed? The structured streaming web UI introduced before was missing from the Web UI documentation. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? N.A. Closes #28609 from xccui/ss-ui-doc. Authored-by: Xingcan Cui <[email protected]> Signed-off-by: HyukjinKwon <[email protected]> (cherry picked from commit 8ba2b47) Signed-off-by: HyukjinKwon <[email protected]>
6 participants
What changes were proposed in this pull request?
This PR adds the structured streaming UI introduction to the Web UI doc.
Why are the changes needed?
The structured streaming web UI introduced before was missing from the Web UI documentation.
Does this PR introduce any user-facing change?
No
How was this patch tested?
N.A.