Skip to content

[SPARK-32207][SQL] Support 'F'-suffixed Float Literals #29022

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yaooqinn wants to merge 5 commits into from
Closed

[SPARK-32207][SQL] Support 'F'-suffixed Float Literals #29022

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f260e41
[SPARK-32207][SQL] Support 'F'-suffixed Float Literals
yaooqinn Jul 7, 2020
90bb546
nit
yaooqinn Jul 7, 2020
0f9feb6
doc
yaooqinn Jul 8, 2020
4439bb6
nit
yaooqinn Jul 8, 2020
6e422ff
addr comments
yaooqinn Jul 8, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions docs/sql-ref-literals.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -219,6 +219,11 @@ double literals:
decimal_digits { D | exponent [ D ] } | digit [ ... ] { exponent [ D ] | [ exponent ] D }
```

float literals:
```sql
decimal_digits { F | exponent [ F ] } | digit [ ... ] { exponent [ F ] | [ exponent ] F }
```

While decimal_digits is defined as
```sql
[ + | - ] { digit [ ... ] . [ digit [ ... ] ] | . digit [ ... ] }
Expand All @@ -239,6 +244,10 @@ E [ + | - ] digit [ ... ]

Case insensitive, indicates `DOUBLE`, which is an 8-byte double-precision floating point number.

* **F**

Case insensitive, indicates `FLOAT`, which is a 4-byte single-precision floating point number.

* **BD**

Case insensitive, indicates `DECIMAL`, with the total number of digits as precision and the number of digits to right of decimal point as scale.
Expand Down
6 changes: 6 additions & 0 deletions sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -988,6 +988,7 @@ number
| MINUS? SMALLINT_LITERAL #smallIntLiteral
| MINUS? TINYINT_LITERAL #tinyIntLiteral
| MINUS? DOUBLE_LITERAL #doubleLiteral
| MINUS? FLOAT_LITERAL #floatLiteral
| MINUS? BIGDECIMAL_LITERAL #bigDecimalLiteral
;

Expand Down Expand Up @@ -1776,6 +1777,11 @@ DECIMAL_VALUE
: DECIMAL_DIGITS {isValidDecimal()}?
;

FLOAT_LITERAL
: DIGIT+ EXPONENT? 'F'
| DECIMAL_DIGITS EXPONENT? 'F' {isValidDecimal()}?
;

DOUBLE_LITERAL
: DIGIT+ EXPONENT? 'D'
| DECIMAL_DIGITS EXPONENT? 'D' {isValidDecimal()}?
Expand Down
9 changes: 9 additions & 0 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2030,6 +2030,15 @@ class AstBuilder(conf: SQLConf) extends SqlBaseBaseVisitor[AnyRef] with Logging
Long.MinValue, Long.MaxValue, LongType.simpleString)(_.toLong)
}

/**
* Create a Float Literal expression.
*/
override def visitFloatLiteral(ctx: FloatLiteralContext): Literal = {
Copy link
Member

@dongjoon-hyun dongjoon-hyun Jul 8, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add a function description like the other functions around here, @yaooqinn ?

Copy link
Member Author

@yaooqinn yaooqinn Jul 8, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done, thanks!

val rawStrippedQualifier = ctx.getText.substring(0, ctx.getText.length - 1)
numericLiteral(ctx, rawStrippedQualifier,
Float.MinValue, Float.MaxValue, FloatType.simpleString)(_.toFloat)
}

/**
* Create a Double Literal expression.
*/
Expand Down
6 changes: 6 additions & 0 deletions sql/core/src/test/resources/sql-tests/inputs/literals.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,11 @@ select 9223372036854775808, -9223372036854775809;
select 1234567890123456789012345678901234567890;
select 1234567890123456789012345678901234567890.0;

-- float
select 1F, 1.2F, .10f, 0.10f;
select -1F, -1.2F, -.10F, -0.10F;
select -3.4028235E39f;

-- double
select 1D, 1.2D, 1e10, 1.5e5, .10D, 0.10D, .1e5, .9e+2, 0.9e+2, 900e-1, 9.e+1;
select -1D, -1.2D, -1e10, -1.5e5, -.10D, -0.10D, -.1e5;
Expand All @@ -55,6 +60,7 @@ select 1E309, -1E309;

-- decimal parsing
select 0.3, -0.8, .5, -.18, 0.1111, .1111;
select 0.3 F, 0.4 D, 0.5 BD;

-- super large scientific notation double literals should still be valid doubles
select 123456789012345678901234567890123456789e10d, 123456789012345678901234567890123456789.1e10d;
Expand Down
40 changes: 39 additions & 1 deletion sql/core/src/test/resources/sql-tests/results/ansi/literals.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
-- Automatically generated by SQLQueryTestSuite
-- Number of queries: 50
-- Number of queries: 54


-- !query
Expand Down Expand Up @@ -164,6 +164,36 @@ decimal can only support precision up to 38
select 1234567890123456789012345678901234567890.0


-- !query
select 1F, 1.2F, .10f, 0.10f
-- !query schema
struct<1.0:float,1.2:float,0.1:float,0.1:float>
-- !query output
1.0 1.2 0.1 0.1


-- !query
select -1F, -1.2F, -.10F, -0.10F
-- !query schema
struct<-1.0:float,-1.2:float,-0.1:float,-0.1:float>
-- !query output
-1.0 -1.2 -0.1 -0.1


-- !query
select -3.4028235E39f
-- !query schema
struct<>
-- !query output
org.apache.spark.sql.catalyst.parser.ParseException

Numeric literal -3.4028235E39 does not fit in range [-3.4028234663852886E+38, 3.4028234663852886E+38] for type float(line 1, pos 7)

== SQL ==
select -3.4028235E39f
-------^^^


-- !query
select 1D, 1.2D, 1e10, 1.5e5, .10D, 0.10D, .1e5, .9e+2, 0.9e+2, 900e-1, 9.e+1
-- !query schema
Expand Down Expand Up @@ -216,6 +246,14 @@ struct<0.3:decimal(1,1),-0.8:decimal(1,1),0.5:decimal(1,1),-0.18:decimal(2,2),0.
0.3 -0.8 0.5 -0.18 0.1111 0.1111


-- !query
select 0.3 F, 0.4 D, 0.5 BD
-- !query schema
struct<F:decimal(1,1),D:decimal(1,1),BD:decimal(1,1)>
-- !query output
0.3 0.4 0.5


-- !query
select 123456789012345678901234567890123456789e10d, 123456789012345678901234567890123456789.1e10d
-- !query schema
Expand Down
40 changes: 39 additions & 1 deletion sql/core/src/test/resources/sql-tests/results/literals.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
-- Automatically generated by SQLQueryTestSuite
-- Number of queries: 50
-- Number of queries: 54


-- !query
Expand Down Expand Up @@ -164,6 +164,36 @@ decimal can only support precision up to 38
select 1234567890123456789012345678901234567890.0


-- !query
select 1F, 1.2F, .10f, 0.10f
-- !query schema
struct<1.0:float,1.2:float,0.1:float,0.1:float>
-- !query output
1.0 1.2 0.1 0.1


-- !query
select -1F, -1.2F, -.10F, -0.10F
-- !query schema
struct<-1.0:float,-1.2:float,-0.1:float,-0.1:float>
-- !query output
-1.0 -1.2 -0.1 -0.1


-- !query
select -3.4028235E39f
-- !query schema
struct<>
-- !query output
org.apache.spark.sql.catalyst.parser.ParseException

Numeric literal -3.4028235E39 does not fit in range [-3.4028234663852886E+38, 3.4028234663852886E+38] for type float(line 1, pos 7)

== SQL ==
select -3.4028235E39f
-------^^^


-- !query
select 1D, 1.2D, 1e10, 1.5e5, .10D, 0.10D, .1e5, .9e+2, 0.9e+2, 900e-1, 9.e+1
-- !query schema
Expand Down Expand Up @@ -216,6 +246,14 @@ struct<0.3:decimal(1,1),-0.8:decimal(1,1),0.5:decimal(1,1),-0.18:decimal(2,2),0.
0.3 -0.8 0.5 -0.18 0.1111 0.1111


-- !query
select 0.3 F, 0.4 D, 0.5 BD
-- !query schema
struct<F:decimal(1,1),D:decimal(1,1),BD:decimal(1,1)>
-- !query output
0.3 0.4 0.5


-- !query
select 123456789012345678901234567890123456789e10d, 123456789012345678901234567890123456789.1e10d
-- !query schema
Expand Down