[MINOR][DOCS] Fix few typos in structured streaming doc #17037
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Author
tdas
reviewed
Feb 23, 2017
| see how this model handles event-time based processing and late arriving data. | ||
|
|
||
| ## Handling Event-time and Late Data | ||
| Event-time is the time embedded in the data itself. For many applications, you may want to operate on this event-time. For example, if you want to get the number of events generated by IoT devices every minute, then you probably want to use the time when the data was generated (that is, event-time in the data), rather than the time Spark receives them. This event-time is very naturally expressed in this model -- each event from the devices is a row in the table, and event-time is a column value in the row. This allows window-based aggregations (e.g. number of events every minute) to be just a special type of grouping and aggregation on the even-time column -- each time window is a group and each row can belong to multiple windows/groups. Therefore, such event-time-window-based aggregation queries can be defined consistently on both a static dataset (e.g. from collected device events logs) as well as on a data stream, making the life of the user much easier. |
Contributor
There was a problem hiding this comment.
can you point out what changed here? github doesnt seeming to showing the difference clearly like the other diffs.
Author
There was a problem hiding this comment.
and aggregation on the even-time column
is changed to
and aggregation on the event-time column
srowen
reviewed
Feb 23, 2017
Member
srowen
left a comment
srowen
left a comment
There was a problem hiding this comment.
We'd generally ask you scan the rest of the docs and similar docs for typos too, in order to cut down on the number of typo fix PRs we need to review.
|
|
||
| ### Window Operations on Event Time | ||
| Aggregations over a sliding event-time window are straightforward with Structured Streaming. The key idea to understand about window-based aggregations are very similar to grouped aggregations. In a grouped aggregation, aggregate values (e.g. counts) are maintained for each unique value in the user-specified grouping column. In case of window-based aggregations, aggregate values are maintained for each window the event-time of a row falls into. Let's understand this with an illustration. | ||
| Aggregations over a sliding event-time window are straightforward with Structured Streaming. The key idea to understand window-based aggregations is very similar to grouped aggregations. In a grouped aggregation, aggregate values (e.g. counts) are maintained for each unique value in the user-specified grouping column. In case of window-based aggregations, aggregate values are maintained for each window the event-time of a row falls into. Let's understand this with an illustration. |
Member
There was a problem hiding this comment.
This still needs a fix -- I would just say "Window-based aggregations are very similar to grouped aggregations"
Contributor
There was a problem hiding this comment.
agreed.
"The key idea to understand is that window-based aggregations are very similar to grouped aggregations."
|
Test build #3583 has finished for PR 17037 at commit
|
Member
|
Merged to master |
Yunni
pushed a commit
to Yunni/spark
that referenced
this pull request
Feb 27, 2017
## What changes were proposed in this pull request? Minor typo in `even-time`, which is changed to `event-time` and a couple of grammatical errors fix. ## How was this patch tested? N/A - since this is a doc fix. I did a jekyll build locally though. Author: Ramkumar Venkataraman <[email protected]> Closes apache#17037 from ramkumarvenkat/doc-fix.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Minor typo in
even-time, which is changed toevent-timeand a couple of grammatical errors fix.How was this patch tested?
N/A - since this is a doc fix. I did a jekyll build locally though.