[SPARK-16288][SQL] Implement inline table generating function #13976

dongjoon-hyun · 2016-06-29T18:09:23Z

What changes were proposed in this pull request?

This PR implements inline table generating function.

How was this patch tested?

Pass the Jenkins tests with new testcase.

SparkQA · 2016-06-29T19:48:14Z

Test build #61479 has finished for PR 13976 at commit 1add31a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Inline(child: Expression) extends UnaryExpression with Generator with CodegenFallback

dongjoon-hyun · 2016-06-29T19:59:58Z

The failure seems to be irrelevant to this PR.

Exception encountered when attempting to run a suite with class name: org.apache.spark.sql.jdbc.JDBCWriteSuite *** ABORTED *** (886 milliseconds)
[info]   org.h2.jdbc.JdbcSQLException: Schema "TEST" already exists; SQL statement:

dongjoon-hyun · 2016-06-29T20:00:05Z

Retest this please.

SparkQA · 2016-06-29T22:32:28Z

Test build #61494 has finished for PR 13976 at commit 1add31a.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Inline(child: Expression) extends UnaryExpression with Generator with CodegenFallback

rxin · 2016-06-30T07:27:01Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala

give an example in extended?

SparkQA · 2016-06-30T13:50:08Z

Test build #61542 has finished for PR 13976 at commit e8d284c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2016-06-30T20:00:49Z

Hi, @rxin .

I rebased and updated according to the previous PosExplode PR.
By the way, I didn't register into sql/Dataset.scala and python/R since I was confused at map_keys/map_values in #13967.

If you give an advice, I will proceed this PR (inline) and #13967 (map_keys/map_values) further.

rxin · 2016-06-30T20:29:24Z

looks pretty good. I will let @cloud-fan review and do the merge.

dongjoon-hyun · 2016-06-30T20:33:17Z

Yep. Thank you, @rxin .

SparkQA · 2016-06-30T21:52:15Z

Test build #61562 has finished for PR 13976 at commit be408ac.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2016-06-30T23:47:43Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala

how about

StructType(et.fields.zipWithIndex.map { case (field, index) => StructField(s"col{$index + 1}", field.dataType, nullable = field.nullable) })

actually that's the best !

Sure. Thanks, @cloud-fan !

dongjoon-hyun · 2016-07-01T11:35:26Z

Thank you, @cloud-fan . :) I've learn a lot in this PR again.
The followings are applied.

Use inputArray.getStruct(i, ncol)
Keep the original field name
Fix elementSchema generation style
Add a column-based test, Array() expression-level test, add empty row test.

SparkQA · 2016-07-01T13:22:35Z

Test build #61615 has finished for PR 13976 at commit 9382f64.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Inline(child: Expression) extends UnaryExpression with Generator with CodegenFallback

cloud-fan · 2016-07-01T13:39:04Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala

+
+  override def elementSchema: StructType = child.dataType match {
+    case ArrayType(et : StructType, _) =>
+      StructType(et.fields.zipWithIndex.map {


hmm, so it's just et now?

Yep. Currently, our type checker ensures that homogeneous StructType array.

why not return et directly?

Oh, my god. I was too naive, here.
Thank you!

dongjoon-hyun · 2016-07-01T16:26:47Z

Thank you, @cloud-fan .
I renovated ugly testcases with create_row()!

SparkQA · 2016-07-01T16:27:36Z

Test build #61622 has started for PR 13976 at commit c43a187.

shaneknapp · 2016-07-01T17:27:50Z

jenkins, test this please

dongjoon-hyun · 2016-07-01T17:39:43Z

Thank you, @shaneknapp ! :)

shaneknapp · 2016-07-01T17:41:55Z

no problem... sorry i had to kill it, but i wanted to get a quick jenkins restart in. :)

SparkQA · 2016-07-01T19:20:43Z

Test build #61624 has finished for PR 13976 at commit c43a187.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2016-07-02T02:55:28Z

sql/core/src/test/scala/org/apache/spark/sql/GeneratorFunctionSuite.scala

      Row(3) :: Nil)
  }
+
+  test("inline raises exception on empty array") {


on array of null type?

Yep. That's more clear.

dongjoon-hyun · 2016-07-02T04:17:26Z

Thank you, @cloud-fan .
elementSchema is simplified and the testcase name is updated.

SparkQA · 2016-07-02T06:05:45Z

Test build #61647 has finished for PR 13976 at commit ebd9cfa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2016-07-03T01:53:29Z

Oh, I found a bug and am working on this.

dongjoon-hyun · 2016-07-03T03:20:26Z

Actually, that is not a bug, but I found that there exists a little difference between Spark and Hive with the following query.

SELECT inline(array(struct(a), struct(b))) FROM (SELECT 1 a, 2 b) T

In short, Spark does more strict type-checking, e.g., [struct<a:int>, struct<b:int>] is considered heterogeneous due to name difference.

I only add more tests to clarify the cases. We cannot touch that because it depends on many things.

The following query is a workaround which both Spark/Hive work.

SELECT inline(array(struct(a), named_struct('a', b))) FROM (SELECT 1 a, 2 b) T

SparkQA · 2016-07-03T05:01:00Z

Test build #61669 has finished for PR 13976 at commit 31ffa75.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-07-03T05:06:04Z

Test build #61670 has finished for PR 13976 at commit fed3ba2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-07-03T07:21:54Z

Test build #61676 has finished for PR 13976 at commit e260359.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2016-07-03T09:13:03Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/generators.scala

+      Nil
+    } else {
+      for (i <- 0 until inputArray.numElements())
+        yield inputArray.getStruct(i, numFields)


I'm not sure how is the performance of for-yield, maybe it's safe to create an array manually and use while loop here?

Thank you, @cloud-fan . By the way, for about this, @rxin gave me an advice at the first commit of this PR.

we don't need to materialize the array, do we? We can create an iterator to return the results.

ah i see, for-yield returns an iterator.

cloud-fan · 2016-07-03T17:59:08Z

merging to master, thanks!

dongjoon-hyun · 2016-07-03T18:52:40Z

Thank you, @cloud-fan and @rxin ! :)

This PR implements `inline` table generating function. Pass the Jenkins tests with new testcase. Author: Dongjoon Hyun <[email protected]> Closes #13976 from dongjoon-hyun/SPARK-16288. (cherry picked from commit 88134e7) Signed-off-by: Reynold Xin <[email protected]>

### What changes were proposed in this pull request? The pr aims to upgrade `netty` from `4.1.109.Final` to `4.1.110.Final`. ### Why are the changes needed? - https://netty.io/news/2024/05/22/4-1-110-Final.html This version has brought some bug fixes and improvements, such as: Fix Zstd throws Exception on read-only volumes (netty/netty#13982) Add unix domain socket transport in netty 4.x via JDK16+ ([#13965](netty/netty#13965)) Backport #13075: Add the AdaptivePoolingAllocator ([#13976](netty/netty#13976)) Add no-value key handling only for form body ([#13998](netty/netty#13998)) Add support for specifying SecureRandom in SSLContext initialization ([#14058](netty/netty#14058)) - https://github.com/netty/netty/issues?q=milestone%3A4.1.110.Final+is%3Aclosed ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #46744 from panbingkun/SPARK-48420. Authored-by: panbingkun <[email protected]> Signed-off-by: yangjie01 <[email protected]>

dongjoon-hyun mentioned this pull request Jun 29, 2016

[SPARK-16267][TEST] Replace deprecated CREATE TEMPORARY TABLE ... USING from testsuites. #13956

Closed

rxin reviewed Jun 30, 2016
View reviewed changes

cloud-fan reviewed Jun 30, 2016
View reviewed changes

[SPARK-16288][SQL] Implement inline table generating function

9382f64

cloud-fan reviewed Jul 1, 2016
View reviewed changes

Remove redundant match and use create_row() in testsuites.

c43a187

cloud-fan reviewed Jul 2, 2016
View reviewed changes

Simplify elementSchema and update a testcase name.

ebd9cfa

Add more testcases. Spark array type verification is more strict.

31ffa75

Remove redundant import statement.

fed3ba2

Fix detail example description.

e260359

cloud-fan reviewed Jul 3, 2016
View reviewed changes

asfgit closed this in 88134e7 Jul 3, 2016

dongjoon-hyun deleted the SPARK-16288 branch July 20, 2016 07:42

[SPARK-16288][SQL] Implement inline table generating function #13976

[SPARK-16288][SQL] Implement inline table generating function #13976

Uh oh!

Conversation

dongjoon-hyun commented Jun 29, 2016

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Jun 29, 2016

Uh oh!

dongjoon-hyun commented Jun 29, 2016

Uh oh!

dongjoon-hyun commented Jun 29, 2016

Uh oh!

SparkQA commented Jun 29, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jun 30, 2016

Uh oh!

dongjoon-hyun commented Jun 30, 2016

Uh oh!

rxin commented Jun 30, 2016

Uh oh!

dongjoon-hyun commented Jun 30, 2016

Uh oh!

SparkQA commented Jun 30, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jul 1, 2016

Uh oh!

SparkQA commented Jul 1, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jul 1, 2016

Uh oh!

SparkQA commented Jul 1, 2016

Uh oh!

shaneknapp commented Jul 1, 2016

Uh oh!

dongjoon-hyun commented Jul 1, 2016

Uh oh!

shaneknapp commented Jul 1, 2016

Uh oh!

SparkQA commented Jul 1, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jul 2, 2016

Uh oh!

SparkQA commented Jul 2, 2016

Uh oh!

dongjoon-hyun commented Jul 3, 2016

Uh oh!

dongjoon-hyun commented Jul 3, 2016

Uh oh!

SparkQA commented Jul 3, 2016

Uh oh!

SparkQA commented Jul 3, 2016

Uh oh!

SparkQA commented Jul 3, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!