Skip to content

Conversation

@viirya
Copy link
Member

@viirya viirya commented May 4, 2016

What changes were proposed in this pull request?

We will eliminate the pair of DeserializeToObject and SerializeFromObject in optimizer and add extra Project. However, when DeserializeToObject's outputObjectType is ObjectType and its cls can't be processed by unsafe project, the projection will be failed.

As the extra Project is used to preserve output expr id, if the DeserializeToObject is at the top, we can avoid this extra Project.

How was this patch tested?

DatasetSuite.

@viirya viirya changed the title [SPARK-15094][SQL] Do not eliminate DeserializeToObject at top of logical plan [SPARK-15094][SQL][WIP] Do not eliminate DeserializeToObject at top of logical plan May 4, 2016
@viirya viirya changed the title [SPARK-15094][SQL][WIP] Do not eliminate DeserializeToObject at top of logical plan [SPARK-15094][SQL][WIP] Avoid extra Project when DeserializeToObject at top of logical plan May 4, 2016
@SparkQA
Copy link

SparkQA commented May 4, 2016

Test build #57761 has finished for PR 12898 at commit 697679e.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented May 4, 2016

Test build #57764 has finished for PR 12898 at commit 7108c3a.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@viirya viirya changed the title [SPARK-15094][SQL][WIP] Avoid extra Project when DeserializeToObject at top of logical plan [SPARK-15094][SQL] Avoid extra Project when DeserializeToObject at top of logical plan May 5, 2016
@SparkQA
Copy link

SparkQA commented May 5, 2016

Test build #57837 has finished for PR 12898 at commit 94cad73.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented May 5, 2016

Test build #57850 has finished for PR 12898 at commit 039a717.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Conflicts:
	sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
@SparkQA
Copy link

SparkQA commented May 6, 2016

Test build #57954 has finished for PR 12898 at commit 08c614a.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@viirya
Copy link
Member Author

viirya commented May 6, 2016

cc @cloud-fan @yhuai @rxin

@viirya viirya closed this May 12, 2016
@viirya viirya deleted the fix-ds-generic branch December 27, 2023 18:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants