[MINOR][DOC] Add note regarding proper usage of QueryExecution.toRdd #23822
Conversation
|
Test build #102462 has finished for PR 23822 at commit
|
|
Retest this, please |
|
cc. @cloud-fan |
|
LGTM |
|
Test build #102469 has finished for PR 23822 at commit
|
|
retest this please |
| * accessing after iteration. (Calling `collect()` is one of known bad usage.) | ||
| * If you want to store these rows into collection, please apply some converter or copy row | ||
| * which produces new object per iteration. | ||
| */ |
There was a problem hiding this comment.
@HeartSaVioR Should we point the users to dataset.rdd method where the conversion is already applied ?
There was a problem hiding this comment.
Yeah that's good suggestion for end users (not Spark developers). Will add.
There was a problem hiding this comment.
BTW, I don't think it's an API though .. technically we don't have to worry about end users.
|
Test build #102484 has finished for PR 23822 at commit
|
| * If you want to store these rows into collection, please apply some converter or copy row | ||
| * which produces new object per iteration. | ||
| * Given QueryExecution is not a public class, end users are discouraged to use this: please | ||
| * user `Dataset.rdd` instead which conversion will be applied. |
There was a problem hiding this comment.
user -> use
which -> in which
or
which -> where ?
There was a problem hiding this comment.
Nice finding. Applied.
|
LGTM with a very minor comment. |
|
Test build #102490 has finished for PR 23822 at commit
|
|
Merged to master. |
6 participants
What changes were proposed in this pull request?
This proposes adding a note on
QueryExecution.toRddregarding Spark's internal optimization callers would need to indicate.How was this patch tested?
This patch is a documentation change.