[SPARK-6608] [SQL] Makes DataFrame.rdd a lazy val

[liancheng]

Before 1.3.0, SchemaRDD.id works as a unique identifier of each SchemaRDD. In 1.3.0, unlike SchemaRDD, DataFrame is no longer an RDD, and DataFrame.rdd is actually a function which always returns a new RDD instance. Making DataFrame.rdd a lazy val should bring the unique identifier back.

[liancheng]

@rxin Is there a good reason that makes DataFrame.rdd have to be a function?

[SparkQA]

Test build #29399 has started for PR 5265 at commit 7f37d21.

[petro-rudenko]

+1 for this, since for example the caching logic from ml package doesn't work properly.

[SparkQA]

Test build #29399 has finished for PR 5265 at commit 7f37d21.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29399/
Test PASSed.

[liancheng]

@petro-rudenko Oops, that's a good catch!

[rxin]

can you update the doc to say that the RDD is memoized, i.e. once called, even if you change the spark sql configuration, it won't change the plan anymore?

[liancheng]

Thanks, that's a good point.

[SparkQA]

Test build #29484 has started for PR 5265 at commit 7500968.

[SparkQA]

Test build #29484 has finished for PR 5265 at commit 7500968.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29484/
Test PASSed.

[rxin]

LGTM.

[liancheng]

Merged to master. Thanks for the review!