[SPARK-30703][SQL][DOCS] Add a document for the ANSI mode #27489

maropu · 2020-02-07T09:05:21Z

What changes were proposed in this pull request?

This pr intends to add a document for the ANSI mode;

Why are the changes needed?

For better document coverage and usability.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

N/A

maropu · 2020-02-07T09:05:28Z

WIP

SparkQA · 2020-02-07T09:21:23Z

Test build #118026 has finished for PR 27489 at commit ed76c3e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

kiszk · 2020-02-07T10:09:48Z

docs/sql-ref-ansi-compliance.md

+### Arithmetic Operations
+
+In Spark SQL, arithmetic operations performed on numeric types (with the exception of decimal) are not checked for overflow by default.
+This means that in case an operation causes an overflow, the result is the same that the same operation returns in a Java/Scala program (eg. if the sum of 2 integers is higher than the maximum value representable, the result is a negative number).


nit: eg. -> e.g.

kiszk · 2020-02-07T10:14:09Z

docs/sql-ref-ansi-compliance.md

+When `spark.sql.ansi.enabled` is set to `true` (`false` by default), Spark SQL follows the standard in basic behaviours (e.g., arithmetic operations, type conversion, and SQL parsing).
+Moreover, Spark SQL has an independent option to control implicit casting behaviours when inserting rows in a table.
+The casting behaviours are defined as store assignment rules in the standard.
+When `spark.sql.storeAssignmentPolicy` is set to `ANSI`, Spark SQL complies with the ANSI store assignment rules and this setting is enabled by default.


Is it better to write default value?

SparkQA · 2020-02-07T13:10:24Z

Test build #118034 has finished for PR 27489 at commit a8681de.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2020-02-07T17:26:30Z

cc @gatorsmile

maropu · 2020-02-08T00:34:13Z

also cc: @gengliangwang @cloud-fan

maropu · 2020-02-08T00:37:04Z

I wrote a basic part of the ANSI mode. Please tell me if you have more we should explicitly describe here for the mode.

dongjoon-hyun · 2020-02-08T22:41:28Z

docs/sql-ref-ansi-compliance.md

+When `spark.sql.ansi.enabled` is set to `true` (`false` by default), Spark SQL follows the standard in basic behaviours (e.g., arithmetic operations, type conversion, and SQL parsing).
+Moreover, Spark SQL has an independent option to control implicit casting behaviours when inserting rows in a table.
+The casting behaviours are defined as store assignment rules in the standard.
+When `spark.sql.storeAssignmentPolicy` is set to `ANSI`, Spark SQL complies with the ANSI store assignment rules and this setting is a default value.


Shall we summarize as a table like https://spark.apache.org/docs/3.0.0-preview2/configuration.html#application-properties ? That will be more easier to explain and to understand and to maintain(add new conf later or change the default value).

For example, the default value of spark.sql.storeAssignmentPolicy is not mentioned in this paragraph.

Ah, I see. But, when I was writing this doc, I assumed the basic info about spark.sql.storeAssignmentPolicy would appear in the SQL Conf doc: #27459
Better to describe the two options in a tabular format here, too?

I am +1 with showing a table here, which can be more clear.

dongjoon-hyun · 2020-02-08T22:44:03Z

docs/sql-ref-ansi-compliance.md

+In Spark SQL, arithmetic operations performed on numeric types (with the exception of decimal) are not checked for overflow by default.
+This means that in case an operation causes an overflow, the result is the same that the same operation returns in a Java/Scala program (e.g., if the sum of 2 integers is higher than the maximum value representable, the result is a negative number).
+On the other hand, Spark SQL returns null for decimal overflow.
+When `spark.sql.ansi.enabled` is set to `true` and overflow occurs in numeric and interval arithmetic operations, it throws an arithmetic exception at runtime.


overflow -> an overflow?

dongjoon-hyun · 2020-02-08T22:44:46Z

docs/sql-ref-ansi-compliance.md

+### Type Conversion
+
+Spark SQL has three kinds of type conversions: explicit casting, type coercion, and store assignment casting.
+When `spark.sql.ansi.enabled` is set to `true`, explicit castings by `CAST` syntax throws a number-format exception at runtime for illegal cast patterns defined in the standard, e.g. casts from a string to an integer.


throws -> throw?

dongjoon-hyun · 2020-02-08T22:45:28Z

docs/sql-ref-ansi-compliance.md

+
+Spark SQL has three kinds of type conversions: explicit casting, type coercion, and store assignment casting.
+When `spark.sql.ansi.enabled` is set to `true`, explicit castings by `CAST` syntax throws a number-format exception at runtime for illegal cast patterns defined in the standard, e.g. casts from a string to an integer.
+On the other hand, `INSERT INTO` syntax throws an analysis exception when the ANSI mode enabled via `spark.sql.storeAssignmentPolicy=ANSI`.


Just a question, shall we mention CTAS(CREATE TABLE AS SELECT) together?

Ah, good suggestion! I totally missed that syntax. I'll check.

I've checked CTAS (CTAS cannot exist with schema definition);

scala> sql("create table t1 (v string)") scala> sql("create table t2 (v int) as select * from t1") org.apache.spark.sql.catalyst.parser.ParseException: Operation not allowed: Schema may not be specified in a Create Table As Select (CTAS) statement(line 1, pos 0)

Any other concern?

dongjoon-hyun · 2020-02-08T22:47:01Z

docs/sql-ref-ansi-compliance.md

+
+  java.lang.NumberFormatException: invalid input syntax for type numeric: a
+
+-- `spark.sql.ansi.enabled=false` (This is a legacy behaviour until Spark 2.x)


legacy can mislead the users. We may need to point out again this is the default behavior in 3.0.0, too.

Ur, I see. I'll update.

Changed it to This is a default behaviour. I dropped the in 3.0.0 phrase in the statement because to I thought update it in future releases (4.0, 5.0, ...) looks a bit annoying.

dongjoon-hyun

Thank you, @maropu . Only minor comments from me.

SparkQA · 2020-02-09T01:17:12Z

Test build #118083 has finished for PR 27489 at commit 6b0d5d6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gengliangwang · 2020-02-12T18:59:49Z

docs/sql-ref-ansi-compliance.md

+### Type Conversion
+
+Spark SQL has three kinds of type conversions: explicit casting, type coercion, and store assignment casting.
+When `spark.sql.ansi.enabled` is set to `true`, explicit casting by `CAST` syntax throws a number-format exception at runtime for illegal cast patterns defined in the standard, e.g. casts from a string to an integer.


number-format exception is not the only runtime exception for ANSI mode

spark.sql("select cast(2147483648L as int)").show() java.lang.ArithmeticException: Casting 2147483648 to int causes overflow

Oh, right. I'll update.

gengliangwang · 2020-02-12T19:06:10Z

@maropu Thanks for the work!

SparkQA · 2020-02-12T23:13:31Z

Test build #118327 has finished for PR 27489 at commit 64b42b4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gengliangwang

LGTM

maropu · 2020-02-13T12:32:12Z

Thanks for the check, @gengliangwang !

### What changes were proposed in this pull request? This pr intends to add a document for the ANSI mode; <img width="600" alt="Screen Shot 2020-02-13 at 8 08 52" src="https://user-images.githubusercontent.com/692303/74386041-5934f780-4e38-11ea-8162-26e524e11c65.png"> <img width="600" alt="Screen Shot 2020-02-13 at 8 09 13" src="https://user-images.githubusercontent.com/692303/74386040-589c6100-4e38-11ea-8a64-899788eaf55f.png"> <img width="600" alt="Screen Shot 2020-02-13 at 8 09 26" src="https://user-images.githubusercontent.com/692303/74386039-5803ca80-4e38-11ea-949f-049208d2203d.png"> <img width="600" alt="Screen Shot 2020-02-13 at 8 09 38" src="https://user-images.githubusercontent.com/692303/74386036-563a0700-4e38-11ea-9ec3-87a8f6771cf0.png"> ### Why are the changes needed? For better document coverage and usability. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? N/A Closes #27489 from maropu/SPARK-30703. Authored-by: Takeshi Yamamuro <[email protected]> Signed-off-by: Gengliang Wang <[email protected]> (cherry picked from commit 3c4044e) Signed-off-by: Gengliang Wang <[email protected]>

gengliangwang · 2020-02-13T18:55:13Z

Merged to master/branch-3.0

dongjoon-hyun · 2020-02-13T19:39:48Z

Thank you, @maropu and @gengliangwang !

maropu · 2020-02-13T21:11:50Z

Thanks, again, @gengliangwang and @dongjoon-hyun ~

…tions as experimental ### What changes were proposed in this pull request? This is a follow-up of #27489. It declares the ANSI SQL compliance options as experimental in the documentation. ### Why are the changes needed? The options are experimental. There can be new features/behaviors in future releases. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Generating doc Closes #27590 from gengliangwang/ExperimentalAnsi. Authored-by: Gengliang Wang <[email protected]> Signed-off-by: HyukjinKwon <[email protected]> (cherry picked from commit da2ca85) Signed-off-by: HyukjinKwon <[email protected]>

…tions as experimental ### What changes were proposed in this pull request? This is a follow-up of #27489. It declares the ANSI SQL compliance options as experimental in the documentation. ### Why are the changes needed? The options are experimental. There can be new features/behaviors in future releases. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Generating doc Closes #27590 from gengliangwang/ExperimentalAnsi. Authored-by: Gengliang Wang <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>

cloud-fan

late LGTM!

cloud-fan · 2020-02-17T07:41:56Z

docs/sql-ref-ansi-compliance.md

+### Arithmetic Operations
+
+In Spark SQL, arithmetic operations performed on numeric types (with the exception of decimal) are not checked for overflows by default.
+This means that in case an operation causes overflows, the result is the same that the same operation returns in a Java/Scala program (e.g., if the sum of 2 integers is higher than the maximum value representable, the result is a negative number).


the result is the same that -> the result is the same with

Ah, yes. I'll fix later. (If other developers plan to open a PR for typo fixes, it would be helpful to include this fix as well.)

cloud-fan · 2020-02-17T07:44:08Z

docs/sql-ref-ansi-compliance.md

+    With legacy policy, Spark allows the type coercion as long as it is a valid Cast, which is very loose.
+    e.g. converting string to int or double to boolean is allowed.
+    It is also the only behavior in Spark 2.x and it is compatible with Hive.
+    With strict policy, Spark doesn't allow any possible precision loss or data truncation in type coercion,


I'm wondering if we should remove the STRICT mode. It's not ANSI compliant and no other SQL system has this behavior.

cc @rdblue @brkyvz @rxin

HyukjinKwon · 2020-02-17T09:54:14Z

Looks good. Thanks, @maropu.

### What changes were proposed in this pull request? This pr intends to add a document for the ANSI mode; <img width="600" alt="Screen Shot 2020-02-13 at 8 08 52" src="https://user-images.githubusercontent.com/692303/74386041-5934f780-4e38-11ea-8162-26e524e11c65.png"> <img width="600" alt="Screen Shot 2020-02-13 at 8 09 13" src="https://user-images.githubusercontent.com/692303/74386040-589c6100-4e38-11ea-8a64-899788eaf55f.png"> <img width="600" alt="Screen Shot 2020-02-13 at 8 09 26" src="https://user-images.githubusercontent.com/692303/74386039-5803ca80-4e38-11ea-949f-049208d2203d.png"> <img width="600" alt="Screen Shot 2020-02-13 at 8 09 38" src="https://user-images.githubusercontent.com/692303/74386036-563a0700-4e38-11ea-9ec3-87a8f6771cf0.png"> ### Why are the changes needed? For better document coverage and usability. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? N/A Closes apache#27489 from maropu/SPARK-30703. Authored-by: Takeshi Yamamuro <[email protected]> Signed-off-by: Gengliang Wang <[email protected]>

…tions as experimental ### What changes were proposed in this pull request? This is a follow-up of apache#27489. It declares the ANSI SQL compliance options as experimental in the documentation. ### Why are the changes needed? The options are experimental. There can be new features/behaviors in future releases. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Generating doc Closes apache#27590 from gengliangwang/ExperimentalAnsi. Authored-by: Gengliang Wang <[email protected]> Signed-off-by: HyukjinKwon <[email protected]>

Fix

ed76c3e

kiszk reviewed Feb 7, 2020

View reviewed changes

Update

a8681de

maropu changed the title ~~[WIP][SPARK-30703][SQL][DOCS] Add a document for the ANSI mode~~ [SPARK-30703][SQL][DOCS] Add a document for the ANSI mode Feb 7, 2020

dongjoon-hyun added the DOCUMENTATION label Feb 7, 2020

dongjoon-hyun reviewed Feb 8, 2020

View reviewed changes

Fix

6b0d5d6

dongjoon-hyun added SQL and removed DOCUMENTATION labels Feb 10, 2020

gengliangwang reviewed Feb 12, 2020

View reviewed changes

Fix

64b42b4

gengliangwang approved these changes Feb 13, 2020

View reviewed changes

gengliangwang closed this in 3c4044e Feb 13, 2020

dongjoon-hyun added the DOCUMENTATION label Feb 13, 2020

gengliangwang mentioned this pull request Feb 14, 2020

[SPARK-30703][SQL][DOCS][FollowUp] Declare the ANSI SQL compliance options as experimental #27590

Closed

cloud-fan reviewed Feb 17, 2020

View reviewed changes

maropu mentioned this pull request Mar 6, 2020

[SQL][DOCS][MINOR] Fix typos and wrong phrases in docs #27819

Closed


		java.lang.NumberFormatException: invalid input syntax for type numeric: a

		-- `spark.sql.ansi.enabled=false` (This is a legacy behaviour until Spark 2.x)

[SPARK-30703][SQL][DOCS] Add a document for the ANSI mode #27489

[SPARK-30703][SQL][DOCS] Add a document for the ANSI mode #27489

Uh oh!

Conversation

maropu commented Feb 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

maropu commented Feb 7, 2020

Uh oh!

SparkQA commented Feb 7, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Feb 7, 2020

Uh oh!

dongjoon-hyun commented Feb 7, 2020

Uh oh!

maropu commented Feb 8, 2020

Uh oh!

maropu commented Feb 8, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Feb 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Feb 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Feb 9, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gengliangwang commented Feb 12, 2020

Uh oh!

SparkQA commented Feb 12, 2020

Uh oh!

gengliangwang left a comment

Choose a reason for hiding this comment

Uh oh!

maropu commented Feb 13, 2020

Uh oh!

gengliangwang commented Feb 13, 2020

Uh oh!

dongjoon-hyun commented Feb 13, 2020

Uh oh!

maropu commented Feb 13, 2020

maropu commented Feb 7, 2020 •

edited

Loading

dongjoon-hyun Feb 8, 2020 •

edited

Loading

dongjoon-hyun Feb 8, 2020 •

edited

Loading