Skip to content

[SPARK-22699][SQL] GenerateSafeProjection should not use global variables for struct#19914

Closed
mgaido91 wants to merge 1 commit intoapache:masterfrom
mgaido91:SPARK-22699
Closed

[SPARK-22699][SQL] GenerateSafeProjection should not use global variables for struct#19914
mgaido91 wants to merge 1 commit intoapache:masterfrom
mgaido91:SPARK-22699

Conversation

@mgaido91
Copy link
Copy Markdown
Contributor

@mgaido91 mgaido91 commented Dec 6, 2017

What changes were proposed in this pull request?

GenerateSafeProjection is defining a mutable state for each struct, which is not needed. This is bad for the well known issues related to constant pool limits.
The PR replace the global variable with a local one.

How was this patch tested?

added UT

@mgaido91
Copy link
Copy Markdown
Contributor Author

mgaido91 commented Dec 6, 2017

@cloud-fan @kiszk @viirya may you please review this? Thanks

@SparkQA
Copy link
Copy Markdown

SparkQA commented Dec 6, 2017

Test build #84569 has finished for PR 19914 at commit 6d2ce8e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@viirya
Copy link
Copy Markdown
Member

viirya commented Dec 7, 2017

Since GenerateSafeProjection produces a class, so I think it doesn't face such high pressure from constant pool limit. But it also looks no harm to replace this global variable with a local one.

@mgaido91
Copy link
Copy Markdown
Contributor Author

mgaido91 commented Dec 7, 2017

@viirya it generates a class, but the changed code and therefore the additional variable is added for each struct object. It might be a problem especially for very complex schemas with nested structs.

@cloud-fan
Copy link
Copy Markdown
Contributor

thanks, merging to master!

@asfgit asfgit closed this in fc29446 Dec 7, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants