Skip to content

Conversation

@davies
Copy link
Contributor

@davies davies commented Apr 2, 2016

What changes were proposed in this pull request?

Currently we extract Python UDFs into a special logical plan EvaluatePython in analyzer, But EvaluatePython is not part of catalyst, many rules have no knowledge of it , which will break many things (for example, filter push down or column pruning).

We should treat Python UDFs as normal expressions, until we want to evaluate in physical plan, we could extract them in end of optimizer, or physical plan.

This PR extract Python UDFs in physical plan.

Closes #10935

How was this patch tested?

Added regression tests.

@davies
Copy link
Contributor Author

davies commented Apr 2, 2016

cc @marmbrus

@SparkQA
Copy link

SparkQA commented Apr 2, 2016

Test build #54765 has finished for PR 12127 at commit c10d80d.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Apr 2, 2016

Test build #54766 has finished for PR 12127 at commit cf97922.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Apr 2, 2016

Test build #2732 has finished for PR 12127 at commit cf97922.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@marmbrus
Copy link
Contributor

marmbrus commented Apr 4, 2016

LGTM

@davies
Copy link
Contributor Author

davies commented Apr 4, 2016

Merging this into master, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants