[SPARK-21354] [SQL] INPUT FILE related functions do not support more than one sources by gatorsmile · Pull Request #18580 · apache/spark

gatorsmile · 2017-07-09T23:03:02Z

What changes were proposed in this pull request?

The build-in functions input_file_name, input_file_block_start, input_file_block_length do not support more than one sources, like what Hive does. Currently, Spark does not block it and the outputs are ambiguous/non-deterministic. It could be from any side.

hive> select *, INPUT__FILE__NAME FROM t1, t2;
FAILED: SemanticException Column INPUT__FILE__NAME Found in more than One Tables/Subqueries

This PR blocks it and issues an error.

How was this patch tested?

Added a test case

SparkQA · 2017-07-10T01:29:54Z

Test build #79426 has finished for PR 18580 at commit 596ea17.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-07-10T01:47:25Z


+          case e @ (_: InputFileName | _: InputFileBlockLength | _: InputFileBlockStart)
+              if getNumLeafNodes(operator) > 1 =>
+            e.failAnalysis(s"'${e.prettyName}' does not support more than one sources")


nit: one sources -> one source.

dongjoon-hyun · 2017-07-10T01:47:34Z

+        val e = intercept[AnalysisException] {
+          sql(s"SELECT *, $func() FROM tab1 JOIN tab2 ON tab1.id = tab2.id")
+        }.getMessage
+        assert(e.contains(s"'$func' does not support more than one sources"))


dongjoon-hyun

.

dongjoon-hyun

Opps. Sorry. It seems we need to consider UNION ALL. UNION ALL works correctly before.

sql("select *, input_file_name() from ((select * from tab1) union all (select * from tab2))").show(false)

gatorsmile · 2017-07-10T07:28:41Z

@dongjoon-hyun What is the output of Hive for your case?

dongjoon-hyun · 2017-07-10T07:37:11Z

The following is the output of the current Spark.

scala> spark.range(10).write.saveAsTable("t1")

scala> spark.range(100,110).write.saveAsTable("t2")

scala> sql("select *, input_file_name() from t1").show(false)
+---+-------------------------------------------------------------------------------------------------------------------+
|id |input_file_name()                                                                                                  |
+---+-------------------------------------------------------------------------------------------------------------------+
|3  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00003-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|4  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00003-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|8  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00007-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|9  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00007-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|0  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00000-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|1  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00001-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|2  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00002-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|5  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00004-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|6  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00005-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|7  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00006-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
+---+-------------------------------------------------------------------------------------------------------------------+


scala> sql("select *, input_file_name() from t2").show(false)
+---+-------------------------------------------------------------------------------------------------------------------+
|id |input_file_name()                                                                                                  |
+---+-------------------------------------------------------------------------------------------------------------------+
|103|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00003-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|104|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00003-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|108|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00007-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|109|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00007-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|100|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00000-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|101|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00001-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|102|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00002-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|105|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00004-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|106|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00005-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|107|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00006-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
+---+-------------------------------------------------------------------------------------------------------------------+


scala> sql("select *, input_file_name() from ((select * from t1) union all (select * from t2))").show(false)
+---+-------------------------------------------------------------------------------------------------------------------+
|id |input_file_name()                                                                                                  |
+---+-------------------------------------------------------------------------------------------------------------------+
|3  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00003-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|4  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00003-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|8  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00007-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|9  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00007-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|0  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00000-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|1  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00001-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|2  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00002-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|5  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00004-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|6  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00005-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|7  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00006-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|103|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00003-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|104|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00003-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|108|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00007-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|109|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00007-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|100|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00000-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|101|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00001-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|102|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00002-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|105|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00004-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|106|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00005-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
|107|file:///Users/dongjoon/spark/spark-warehouse/t2/part-00006-76ea547d-0187-40f0-b5dd-f9f1fffeeabf-c000.snappy.parquet|
+---+-------------------------------------------------------------------------------------------------------------------+

dongjoon-hyun · 2017-07-10T07:38:06Z

In case of UNION ALL, everything looks normal. So, it would be a case of regression of Spark.

gatorsmile · 2017-07-10T17:58:58Z

Does Hive report an error?

dongjoon-hyun · 2017-07-10T18:12:22Z

Let me check that. BTW, I think Spark is better than Hive. :)

dongjoon-hyun · 2017-07-10T18:24:00Z

This is the error and difference.

HIVE

hive> select *, INPUT__FILE__NAME from (select * from t1) T;
FAILED: SemanticException [Error 10004]: Line 1:10 Invalid table alias or column reference 'INPUT__FILE__NAME': (possible column names are: _c0)

SPARK

scala> sql("select *, input_file_name() from (select * from t1)").show(false)
+---+-------------------------------------------------------------------------------------------------------------------+
|id |input_file_name()                                                                                                  |
+---+-------------------------------------------------------------------------------------------------------------------+
|3  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00003-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|4  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00003-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|8  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00007-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|9  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00007-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|0  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00000-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|1  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00001-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|2  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00002-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|5  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00004-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|6  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00005-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
|7  |file:///Users/dongjoon/spark/spark-warehouse/t1/part-00006-b0ca8fa4-03ae-4e3a-b4b4-a13d601cd155-c000.snappy.parquet|
+---+-------------------------------------------------------------------------------------------------------------------+

gatorsmile · 2017-07-10T18:55:08Z

For union, does Hive output an error?

dongjoon-hyun · 2017-07-10T19:07:47Z

It was the same error, Invalid table alias or column reference 'INPUT__FILE__NAME': (possible column names are: _c0). So, I reduce that into a simple above example.

hive> select *, INPUT__FILE__NAME from (select * from t1 union all select * from t1) T;
FAILED: SemanticException [Error 10004]: Line 1:10 Invalid table alias or column reference 'INPUT__FILE__NAME': (possible column names are: _c0)

gatorsmile · 2017-07-11T04:17:42Z

cc @cloud-fan

cloud-fan · 2017-07-11T05:33:15Z


+  private def getNumInputFileBlockSources(operator: LogicalPlan): Int = {
+    operator match {
+      case _: LeafNode => 1


shall we only consider file data source leaf node?

Unable to check it in CheckAnalysis. Both HadoopRDD and FileScanRDD have the same issues. To block both, we need to add the check as another rule.

SparkQA · 2017-07-11T05:36:40Z

Test build #79496 has finished for PR 18580 at commit 6b48a9e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-07-13T03:50:14Z

Retest this please.

SparkQA · 2017-07-13T06:19:12Z

Test build #79576 has finished for PR 18580 at commit 6b48a9e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-07-17T02:32:45Z

+      case o =>
+        val numInputFileBlockSources = o.children.map(checkNumInputFileBlockSources(e, _)).sum
+        if (numInputFileBlockSources > 1) {
+          e.failAnalysis(s"'${e.prettyName}' does not support more than one sources")


Need to check it as early as possible; otherwise, Union might eat it.

cloud-fan · 2017-07-17T04:56:52Z

LGTM, pending jenkins

SparkQA · 2017-07-17T05:07:25Z

Test build #79651 has finished for PR 18580 at commit c4de2b8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2017-07-17T06:58:28Z

thanks, merging to master!

gatorsmile added 2 commits July 9, 2017 15:52

fix.

04ff406

fix.

596ea17

dongjoon-hyun reviewed Jul 10, 2017

View reviewed changes

dongjoon-hyun approved these changes Jul 10, 2017

View reviewed changes

dongjoon-hyun reviewed Jul 10, 2017

View reviewed changes

fix.

6b48a9e

cloud-fan reviewed Jul 11, 2017

View reviewed changes

fix.

c4de2b8

gatorsmile commented Jul 17, 2017

View reviewed changes

asfgit closed this in e398c28 Jul 17, 2017

Conversation

gatorsmile commented Jul 9, 2017

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Jul 10, 2017

Uh oh!

dongjoon-hyun Jul 10, 2017

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Jul 10, 2017

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gatorsmile commented Jul 10, 2017

Uh oh!

dongjoon-hyun commented Jul 10, 2017

Uh oh!

dongjoon-hyun commented Jul 10, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gatorsmile commented Jul 10, 2017

Uh oh!

dongjoon-hyun commented Jul 10, 2017

Uh oh!

dongjoon-hyun commented Jul 10, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gatorsmile commented Jul 10, 2017

Uh oh!

dongjoon-hyun commented Jul 10, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gatorsmile commented Jul 11, 2017

Uh oh!

cloud-fan Jul 11, 2017

Choose a reason for hiding this comment

Uh oh!

gatorsmile Jul 17, 2017

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 11, 2017

Uh oh!

dongjoon-hyun commented Jul 13, 2017

Uh oh!

SparkQA commented Jul 13, 2017

Uh oh!

gatorsmile Jul 17, 2017

Choose a reason for hiding this comment

Uh oh!

cloud-fan commented Jul 17, 2017

Uh oh!

SparkQA commented Jul 17, 2017

Uh oh!

cloud-fan commented Jul 17, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dongjoon-hyun left a comment •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading

dongjoon-hyun commented Jul 10, 2017 •

edited

Loading

dongjoon-hyun commented Jul 10, 2017 •

edited

Loading

dongjoon-hyun commented Jul 10, 2017 •

edited

Loading