Skip to content

[SPARK-19499][SS]Add more notes in the comments of Sink.addBatch() #16840

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
CodingCat wants to merge 8 commits into from
Closed

[SPARK-19499][SS]Add more notes in the comments of Sink.addBatch() #16840

Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
24bfa38
improve the doc for "spark.memory.offHeap.size"
CodingCat Mar 7, 2016
2209e34
fix
CodingCat Mar 7, 2016
f2d9db1
Merge branch 'master' of https://github.com/apache/spark
CodingCat Dec 28, 2016
dd409b4
Merge branch 'master' of https://github.com/apache/spark
CodingCat Jan 13, 2017
f026ade
Merge branch 'master' of https://github.com/apache/spark
CodingCat Jan 25, 2017
535b60e
Merge branch 'master' of https://github.com/apache/spark
CodingCat Feb 7, 2017
d9222d7
add notes
CodingCat Feb 7, 2017
a58e1c8
address the comments
CodingCat Feb 7, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/Sink.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,11 @@ trait Sink {
* this method is called more than once with the same batchId (which will happen in the case of
* failures), then `data` should only be added once.
*
* Note: You cannot apply any operators on `data` except consuming it (e.g., `collect/foreach`).
* Note 1: You cannot apply any operators on `data` except consuming it (e.g., `collect/foreach`).
* Otherwise, you may get a wrong result.
*
* Note 2: The method is supposed to be executed synchronously, i.e. the method should only return
* after data is consumed by sink successfully.
*/
def addBatch(batchId: Long, data: DataFrame): Unit
}