[SPARK-5287][SQL] Add defaultSizeOf to every data type.

[yhuai]

JIRA: https://issues.apache.org/jira/browse/SPARK-5287

This PR only add defaultSizeOf to data types and make those internal type classes protected[sql]. I will use another PR to cleanup the type hierarchy of data types.

[SparkQA]

Test build #25687 has started for PR 4081 at commit 64bb05d.

• This patch merges cleanly.

[SparkQA]

Test build #25687 has finished for PR 4081 at commit 64bb05d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25687/
Test PASSed.

[rxin]

How come we don't want it for all data types?

[marmbrus]

Yeah, will this still fail for limits with struct types for example?

[rxin]

I'd say we can just put a defaultSize method to datatype trait, and then implement it for all data types.

[yhuai]

For now, it will not fail since we will check if the data type is a NativeType before calling defaultSize. I thought we will implement it for all types when we remove NativeType.

I can move defaultSize to datatype trait in this PR.

[jesman]

A of a or the reader t AD
On 2015-01-16 6:22 PM, "Yin Huai" notifications@github.com wrote:

JIRA: https://issues.apache.org/jira/browse/SPARK-5287

You can view, comment on, or merge this pull request online at:

#4081
Commit Summary

• Add all native types to NativeType.defaultSizeOf.

File Changes

• M
  sql/catalyst/src/main/scala/org/apache/spark/sql/types/dataTypes.scala
  https://github.com/apache/spark/pull/4081/files#diff-0 (23)
• M sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
  https://github.com/apache/spark/pull/4081/files#diff-1 (34)

Patch Links:

• https://github.com/apache/spark/pull/4081.patch
• https://github.com/apache/spark/pull/4081.diff

—
Reply to this email directly or view it on GitHub
#4081.

[rxin]

Sorry @jesman what did you mean?

[SparkQA]

Test build #25829 has started for PR 4081 at commit e1c600c.

• This patch merges cleanly.

[SparkQA]

Test build #25829 has finished for PR 4081 at commit e1c600c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • protected[sql] trait PrimitiveType extends DataType
  • protected[sql] abstract class NativeType extends DataType
  • protected[sql] abstract class NumericType extends NativeType with PrimitiveType
  • protected[sql] sealed abstract class IntegralType extends NumericType
  • protected[sql] sealed abstract class FractionalType extends NumericType

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25829/
Test PASSed.

[SparkQA]

Test build #25836 has started for PR 4081 at commit 90cec75.

• This patch merges cleanly.

[SparkQA]

Test build #25836 has finished for PR 4081 at commit 90cec75.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • protected[sql] trait PrimitiveType extends DataType
  • protected[sql] abstract class NativeType extends DataType
  • protected[sql] abstract class NumericType extends NativeType with PrimitiveType
  • protected[sql] sealed abstract class IntegralType extends NumericType
  • protected[sql] sealed abstract class FractionalType extends NumericType

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25836/
Test PASSed.