Skip to content

[SPARK-20211][SQL] Fix the Precision and Scale of Decimal Values when the Input is BigDecimal between -1.0 and 1.0 #18244

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed

[SPARK-20211][SQL] Fix the Precision and Scale of Decimal Values when the Input is BigDecimal between -1.0 and 1.0 #18244

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a7396be
fix.
gatorsmile Jun 8, 2017
8e654bc
fix.
gatorsmile Jun 8, 2017
d36be12
fix.
gatorsmile Jun 9, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,15 @@ final class Decimal extends Ordered[Decimal] with Serializable {
def set(decimal: BigDecimal): Decimal = {
this.decimalVal = decimal
this.longVal = 0L
this._precision = decimal.precision
if (decimal.precision <= decimal.scale) {
Copy link
Contributor

@stanzhai stanzhai Jun 9, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But the comment is // For Decimal, we expect the precision is equal to or large than the scale.

= has been processed within the function floor and ceil.
https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/types/Decimal.scala#L387

This is reason that I think we should use if (decimal.precision < decimal.scale), and it works fine for 0.90.

Copy link
Member Author

@gatorsmile gatorsmile Jun 9, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fix is not for floor and ceil only. We need to correct the precision if the value is from BigDecimal.

Copy link
Contributor

@stanzhai stanzhai Jun 9, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it, thanks for the fix!

// For Decimal, we expect the precision is equal to or large than the scale, however,
// in BigDecimal, the digit count starts from the leftmost nonzero digit of the exact
// result. For example, the precision of 0.01 equals to 1 based on the definition, but
// the scale is 2. The expected precision should be 3.
this._precision = decimal.scale + 1
} else {
this._precision = decimal.precision
}
this._scale = decimal.scale
this
}
Expand Down
10 changes: 10 additions & 0 deletions sql/catalyst/src/test/scala/org/apache/spark/sql/types/DecimalSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,16 @@ class DecimalSuite extends SparkFunSuite with PrivateMethodTester {

test("creating decimals") {
checkDecimal(new Decimal(), "0", 1, 0)
checkDecimal(Decimal(BigDecimal("0.09")), "0.09", 3, 2)
checkDecimal(Decimal(BigDecimal("0.9")), "0.9", 2, 1)
checkDecimal(Decimal(BigDecimal("0.90")), "0.90", 3, 2)
checkDecimal(Decimal(BigDecimal("0.0")), "0.0", 2, 1)
checkDecimal(Decimal(BigDecimal("0")), "0", 1, 0)
checkDecimal(Decimal(BigDecimal("1.0")), "1.0", 2, 1)
checkDecimal(Decimal(BigDecimal("-0.09")), "-0.09", 3, 2)
checkDecimal(Decimal(BigDecimal("-0.9")), "-0.9", 2, 1)
checkDecimal(Decimal(BigDecimal("-0.90")), "-0.90", 3, 2)
checkDecimal(Decimal(BigDecimal("-1.0")), "-1.0", 2, 1)
checkDecimal(Decimal(BigDecimal("10.030")), "10.030", 5, 3)
checkDecimal(Decimal(BigDecimal("10.030"), 4, 1), "10.0", 4, 1)
checkDecimal(Decimal(BigDecimal("-9.95"), 4, 1), "-10.0", 4, 1)
Expand Down
7 changes: 7 additions & 0 deletions sql/core/src/test/resources/sql-tests/inputs/operators.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,8 +65,15 @@ select ceiling(0);
select ceiling(1);
select ceil(1234567890123456);
select ceiling(1234567890123456);
select ceil(0.01);
select ceiling(-0.10);

-- floor
select floor(0);
select floor(1);
select floor(1234567890123456);
select floor(0.01);
select floor(-0.10);

-- comparison operator
select 1 > 0.00001
58 changes: 49 additions & 9 deletions sql/core/src/test/resources/sql-tests/results/operators.sql.out
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
-- Automatically generated by SQLQueryTestSuite
-- Number of queries: 45
-- Number of queries: 50


-- !query 0
Expand Down Expand Up @@ -351,24 +351,64 @@ struct<CEIL(1234567890123456):bigint>


-- !query 42
select floor(0)
select ceil(0.01)
-- !query 42 schema
struct<FLOOR(CAST(0 AS DOUBLE)):bigint>
struct<CEIL(0.01):decimal(1,0)>
-- !query 42 output
0
1


-- !query 43
select floor(1)
select ceiling(-0.10)
-- !query 43 schema
struct<FLOOR(CAST(1 AS DOUBLE)):bigint>
struct<CEIL(-0.10):decimal(1,0)>
-- !query 43 output
1
0


-- !query 44
select floor(1234567890123456)
select floor(0)
-- !query 44 schema
struct<FLOOR(1234567890123456):bigint>
struct<FLOOR(CAST(0 AS DOUBLE)):bigint>
-- !query 44 output
0


-- !query 45
select floor(1)
-- !query 45 schema
struct<FLOOR(CAST(1 AS DOUBLE)):bigint>
-- !query 45 output
1


-- !query 46
select floor(1234567890123456)
-- !query 46 schema
struct<FLOOR(1234567890123456):bigint>
-- !query 46 output
1234567890123456


-- !query 47
select floor(0.01)
-- !query 47 schema
struct<FLOOR(0.01):decimal(1,0)>
-- !query 47 output
0


-- !query 48
select floor(-0.10)
-- !query 48 schema
struct<FLOOR(-0.10):decimal(1,0)>
-- !query 48 output
-1


-- !query 49
select 1 > 0.00001
-- !query 49 schema
struct<(CAST(1 AS BIGINT) > 0):boolean>
-- !query 49 output
true