Skip to content

[SPARK-29371][SQL] Fractional representation for interval string #26592

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yaooqinn wants to merge 15 commits into from
Closed

[SPARK-29371][SQL] Fractional representation for interval string #26592

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
62560cf
[SPARK-29371][SQL] Frational representation for interval string
yaooqinn Nov 19, 2019
7ee83f5
use double directly
yaooqinn Nov 19, 2019
4018278
benchmark
yaooqinn Nov 19, 2019
dd40ea0
jdk11 benchmark
yaooqinn Nov 19, 2019
4dc8f02
round for microseconds for each unit
yaooqinn Nov 19, 2019
393ad77
Merge branch 'master' into SPARK-29371-2
yaooqinn Nov 19, 2019
a5e2811
regen golden file
yaooqinn Nov 19, 2019
5b4f518
add negative value test
yaooqinn Nov 19, 2019
854b6f8
fix tests
yaooqinn Nov 19, 2019
5af9a93
Merge branch 'master' into SPARK-29371-2
yaooqinn Nov 20, 2019
5604b83
add . check back
yaooqinn Nov 20, 2019
1684d3a
Merge branch 'master' into SPARK-29371-2
yaooqinn Nov 25, 2019
4dffac9
regen golden file
yaooqinn Nov 25, 2019
1ccfbbf
Merge branch 'master' into SPARK-29371-2
yaooqinn Nov 27, 2019
fe3e6ba
regen golden file
yaooqinn Nov 27, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 27 additions & 42 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -478,9 +478,8 @@ object IntervalUtils {
var months: Int = 0
var days: Int = 0
var microseconds: Long = 0
var fractionScale: Int = 0
val initialFractionScale = (NANOS_PER_SECOND / 10).toInt
var fraction: Int = 0
var fractionScale: Double = 0
var fraction: Double = 0
var pointPrefixed: Boolean = false

def trimToNextState(b: Byte, next: ParseState): Unit = {
Expand Down Expand Up @@ -515,29 +514,23 @@ object IntervalUtils {
case TRIM_BEFORE_SIGN => trimToNextState(b, SIGN)
case SIGN =>
currentValue = 0
isNegative = false
fractionScale = 10
fraction = 0
// We preset next state from SIGN to TRIM_BEFORE_VALUE. If we meet '.' in the SIGN state,
// it means that the interval value we deal with here is a numeric with only fractional
// part, such as '.11 second', which can be parsed to 0.11 seconds. In this case, we need
// to reset next state to `VALUE_FRACTIONAL_PART` to go parse the fraction part of the
// interval value.
state = TRIM_BEFORE_VALUE
// We preset the scale to an invalid value to track fraction presence in the UNIT_BEGIN
// state. If we meet '.', the scale become valid for the VALUE_FRACTIONAL_PART state.
fractionScale = -1
pointPrefixed = false
b match {
case '-' =>
isNegative = true
i += 1
case '+' =>
isNegative = false
i += 1
case '+' => i += 1
case _ if '0' <= b && b <= '9' =>
isNegative = false
case '.' =>
isNegative = false
fractionScale = initialFractionScale
pointPrefixed = true
i += 1
state = VALUE_FRACTIONAL_PART
Expand All @@ -553,73 +546,65 @@ object IntervalUtils {
case e: ArithmeticException => throwIAE(e.getMessage, e)
}
case _ if b <= ' ' => state = TRIM_BEFORE_UNIT
case '.' =>
fractionScale = initialFractionScale
state = VALUE_FRACTIONAL_PART
case '.' => state = VALUE_FRACTIONAL_PART
case _ => throwIAE(s"invalid value '$currentWord'")
}
i += 1
case VALUE_FRACTIONAL_PART =>
if ('0' <= b && b <= '9' && fractionScale > 0) {
fraction += (b - '0') * fractionScale
fractionScale /= 10
} else if (b <= ' ' && (!pointPrefixed || fractionScale < initialFractionScale)) {
fraction /= NANOS_PER_MICROS.toInt
if ('0' <= b && b <= '9') {
fraction += (b - '0') / fractionScale
fractionScale *= 10
} else if (b <= ' ' && (!pointPrefixed || fractionScale > 10)) {
state = TRIM_BEFORE_UNIT
} else if ('0' <= b && b <= '9') {
throwIAE(s"interval can only support nanosecond precision, '$currentWord' is out" +
s" of range")
} else {
throwIAE(s"invalid value '$currentWord'")
}
i += 1
case TRIM_BEFORE_UNIT => trimToNextState(b, UNIT_BEGIN)
case UNIT_BEGIN =>
// Checks that only seconds can have the fractional part
if (b != 's' && fractionScale >= 0) {
throwIAE(s"'$currentWord' cannot have fractional part")
}
if (isNegative) {
currentValue = -currentValue
fraction = -fraction
}
try {
b match {
case 'y' if s.matchAt(yearStr, i) =>
val monthsInYears = Math.multiplyExact(MONTHS_PER_YEAR, currentValue)
months = Math.toIntExact(Math.addExact(months, monthsInYears))
val monthsInYears =
Math.toIntExact(((currentValue + fraction) * MONTHS_PER_YEAR).toLong)
months = Math.addExact(months, monthsInYears)
i += yearStr.numBytes()
case 'w' if s.matchAt(weekStr, i) =>
val daysInWeeks = Math.multiplyExact(DAYS_PER_WEEK, currentValue)
days = Math.toIntExact(Math.addExact(days, daysInWeeks))
val daysInWeeks = (currentValue + fraction) * DAYS_PER_WEEK
val daysTrimmed = Math.toIntExact(daysInWeeks.toLong)
days = Math.addExact(days, daysTrimmed)
microseconds += ((daysInWeeks - daysTrimmed) * MICROS_PER_DAY).round
i += weekStr.numBytes()
case 'd' if s.matchAt(dayStr, i) =>
days = Math.addExact(days, Math.toIntExact(currentValue))
microseconds += (fraction * MICROS_PER_DAY).round
i += dayStr.numBytes()
case 'h' if s.matchAt(hourStr, i) =>
val hoursUs = Math.multiplyExact(currentValue, MICROS_PER_HOUR)
microseconds = Math.addExact(microseconds, hoursUs)
microseconds += ((currentValue + fraction) * MICROS_PER_HOUR).round
i += hourStr.numBytes()
case 's' if s.matchAt(secondStr, i) =>
val secondsUs = Math.multiplyExact(currentValue, MICROS_PER_SECOND)
microseconds = Math.addExact(Math.addExact(microseconds, secondsUs), fraction)
microseconds += ((currentValue + fraction) * MICROS_PER_SECOND).round
i += secondStr.numBytes()
case 'm' =>
if (s.matchAt(monthStr, i)) {
months = Math.addExact(months, Math.toIntExact(currentValue))
val dayLeftover = fraction * DAYS_PER_MONTH
val daysTrimmed = Math.toIntExact(dayLeftover.toLong)
days = Math.addExact(days, daysTrimmed)
microseconds += ((dayLeftover - daysTrimmed) * MICROS_PER_DAY).round
i += monthStr.numBytes()
} else if (s.matchAt(minuteStr, i)) {
val minutesUs = Math.multiplyExact(currentValue, MICROS_PER_MINUTE)
microseconds = Math.addExact(microseconds, minutesUs)
microseconds += ((currentValue + fraction) * MICROS_PER_MINUTE).round
i += minuteStr.numBytes()
} else if (s.matchAt(millisStr, i)) {
val millisUs = Math.multiplyExact(
currentValue,
MICROS_PER_MILLIS)
microseconds = Math.addExact(microseconds, millisUs)
microseconds += ((currentValue + fraction) * MICROS_PER_MILLIS).round
i += millisStr.numBytes()
} else if (s.matchAt(microsStr, i)) {
microseconds = Math.addExact(microseconds, currentValue)
microseconds += (currentValue + fraction).round
i += microsStr.numBytes()
} else throwIAE(s"invalid unit '$currentWord'")
case _ => throwIAE(s"invalid unit '$currentWord'")
Expand Down
2 changes: 1 addition & 1 deletion sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -645,7 +645,7 @@ class ExpressionParserSuite extends AnalysisTest {
Literal(new CalendarInterval(
0,
0,
-13 * MICROS_PER_SECOND - 123 * MICROS_PER_MILLIS - 456)))
-13 * MICROS_PER_SECOND - 123 * MICROS_PER_MILLIS - 457)))
Copy link
Member Author

@yaooqinn yaooqinn Nov 19, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

postgres=# select interval '-13.123456789 second';
     interval
------------------
 -00:00:13.123457
(1 row)

checkIntervals(
"13.123456 second",
Literal(new CalendarInterval(
Expand Down
11 changes: 5 additions & 6 deletions sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/IntervalUtilsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,8 +104,6 @@ class IntervalUtilsSuite extends SparkFunSuite {
// Allow duplicated units and summarize their values
checkFromString("1 day 10 day", new CalendarInterval(0, 11, 0))
// Only the seconds units can have the fractional part
checkFromInvalidString("1.5 days", "'days' cannot have fractional part")
checkFromInvalidString("1. hour", "'hour' cannot have fractional part")
checkFromInvalidString("1 hourX", "invalid unit 'hourx'")
checkFromInvalidString("~1 hour", "unrecognized number '~1'")
checkFromInvalidString("1 Mour", "invalid unit 'mour'")
Expand All @@ -125,17 +123,18 @@ class IntervalUtilsSuite extends SparkFunSuite {
checkFromString("interval\r1\tday", new CalendarInterval(0, 1, 0))
}

test("string to interval: seconds with fractional part") {
test("string to interval: interval with fractional part") {
checkFromString("0.1 seconds", new CalendarInterval(0, 0, 100000))
checkFromString("1. seconds", new CalendarInterval(0, 0, 1000000))
checkFromString("123.001 seconds", new CalendarInterval(0, 0, 123001000))
checkFromString("1.001001 seconds", new CalendarInterval(0, 0, 1001001))
checkFromString("1 minute 1.001001 seconds", new CalendarInterval(0, 0, 61001001))
checkFromString("-1.5 seconds", new CalendarInterval(0, 0, -1500000))
// truncate nanoseconds to microseconds
checkFromString("0.999999999 seconds", new CalendarInterval(0, 0, 999999))
checkFromString(".999999999 seconds", new CalendarInterval(0, 0, 999999))
checkFromInvalidString("0.123456789123 seconds", "'0.123456789123' is out of range")
checkFromString("0.999999999 seconds", new CalendarInterval(0, 0, 1000000))
checkFromString(".999999 seconds", new CalendarInterval(0, 0, 999999))
checkFromString("0.123456789123 seconds", new CalendarInterval(0, 0, 123457))
checkFromString("1.5 days", new CalendarInterval(0, 1, 12 * MICROS_PER_HOUR))
}

test("from year-month string") {
Expand Down
44 changes: 19 additions & 25 deletions sql/core/benchmarks/IntervalBenchmark-jdk11-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,29 +1,23 @@
Java HotSpot(TM) 64-Bit Server VM 11.0.5+10-LTS on Mac OS X 10.14.6
Java HotSpot(TM) 64-Bit Server VM 11.0.5+10-LTS on Mac OS X 10.15.1
Intel(R) Core(TM) i5-5287U CPU @ 2.90GHz
cast strings to intervals: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
prepare string w/ interval 574 610 45 1.7 573.9 1.0X
prepare string w/o interval 518 538 27 1.9 517.7 1.1X
1 units w/ interval 425 439 16 2.4 425.3 1.3X
1 units w/o interval 385 393 10 2.6 385.2 1.5X
2 units w/ interval 553 561 11 1.8 553.1 1.0X
2 units w/o interval 531 543 11 1.9 531.0 1.1X
3 units w/ interval 1134 1159 32 0.9 1134.0 0.5X
3 units w/o interval 1121 1126 6 0.9 1121.3 0.5X
4 units w/ interval 1226 1250 21 0.8 1226.1 0.5X
4 units w/o interval 1227 1239 11 0.8 1227.1 0.5X
5 units w/ interval 1375 1447 93 0.7 1374.7 0.4X
5 units w/o interval 1335 1346 19 0.7 1335.1 0.4X
6 units w/ interval 1530 1556 24 0.7 1529.5 0.4X
6 units w/o interval 1481 1492 17 0.7 1480.7 0.4X
7 units w/ interval 1730 1745 14 0.6 1729.9 0.3X
7 units w/o interval 1788 1859 112 0.6 1788.1 0.3X
8 units w/ interval 1952 2087 117 0.5 1951.7 0.3X
8 units w/o interval 2083 2207 209 0.5 2082.5 0.3X
9 units w/ interval 2228 2291 60 0.4 2227.5 0.3X
9 units w/o interval 2130 2184 75 0.5 2130.1 0.3X
10 units w/ interval 2414 2502 81 0.4 2413.8 0.2X
10 units w/o interval 2463 2488 35 0.4 2463.1 0.2X
11 units w/ interval 2717 2755 42 0.4 2716.8 0.2X
11 units w/o interval 2578 2661 77 0.4 2577.7 0.2X
prepare string w/ interval 644 698 84 1.6 644.1 1.0X
prepare string w/o interval 599 625 39 1.7 598.7 1.1X
1 units w/ interval 441 448 9 2.3 441.0 1.5X
1 units w/o interval 389 394 6 2.6 388.6 1.7X
2 units w/ interval 559 575 22 1.8 559.1 1.2X
2 units w/o interval 542 548 5 1.8 542.2 1.2X
4 units w/ interval 1249 1263 15 0.8 1249.4 0.5X
4 units w/o interval 1237 1252 15 0.8 1237.3 0.5X
6 units w/ interval 1585 1596 12 0.6 1585.1 0.4X
6 units w/o interval 1573 1585 17 0.6 1573.3 0.4X
8 units w/ interval 2029 2032 4 0.5 2029.2 0.3X
8 units w/o interval 2015 2026 18 0.5 2015.1 0.3X
10 units w/ interval 2367 2370 2 0.4 2367.5 0.3X
10 units w/o interval 2356 2357 2 0.4 2356.1 0.3X
12 units w/ interval 2785 2821 43 0.4 2784.9 0.2X
12 units w/o interval 2817 2856 37 0.4 2816.9 0.2X
14 units w/ interval 3303 3319 28 0.3 3302.7 0.2X
14 units w/o interval 3293 3310 21 0.3 3292.6 0.2X

Loading