Skip to content

[MINOR][DOC] Add note regarding proper usage of QueryExecution.toRdd #23822

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HeartSaVioR wants to merge 3 commits into from
Closed

[MINOR][DOC] Add note regarding proper usage of QueryExecution.toRdd #23822

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
493b2cb
[MINOR][DOC] Add note regarding proper usage of QueryExecution.toRdd
HeartSaVioR Feb 18, 2019
2f224f2
Add more guide for end users
HeartSaVioR Feb 18, 2019
d68dcf2
Minor correction on typo/grammar
HeartSaVioR Feb 18, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,16 @@ class QueryExecution(
prepareForExecution(sparkPlan)
}

/** Internal version of the RDD. Avoids copies and has no schema */
/**
* Internal version of the RDD. Avoids copies and has no schema.
* Note for callers: Spark may apply various optimization including reusing object: this means
* the row is valid only for the iteration it is retrieved. You should avoid storing row and
* accessing after iteration. (Calling `collect()` is one of known bad usage.)
* If you want to store these rows into collection, please apply some converter or copy row
* which produces new object per iteration.
* Given QueryExecution is not a public class, end users are discouraged to use this: please
* use `Dataset.rdd` instead where conversion will be applied.
*/
Copy link
Contributor

@dilipbiswal dilipbiswal Feb 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@HeartSaVioR Should we point the users to dataset.rdd method where the conversion is already applied ?

Copy link
Contributor Author

@HeartSaVioR HeartSaVioR Feb 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that's good suggestion for end users (not Spark developers). Will add.

Copy link
Member

@HyukjinKwon HyukjinKwon Feb 19, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, I don't think it's an API though .. technically we don't have to worry about end users.

lazy val toRdd: RDD[InternalRow] = executedPlan.execute()

/**
Expand Down