Skip to content

[SPARK-10522] [SQL] Nanoseconds of Timestamp in Parquet should be positive #8674

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
davies wants to merge 1 commit into from
Closed

[SPARK-10522] [SQL] Nanoseconds of Timestamp in Parquet should be positive #8674

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ object DateTimeUtils {
final val SECONDS_PER_DAY = 60 * 60 * 24L
final val MICROS_PER_SECOND = 1000L * 1000L
final val NANOS_PER_SECOND = MICROS_PER_SECOND * 1000L
final val MICROS_PER_DAY = MICROS_PER_SECOND * SECONDS_PER_DAY

final val MILLIS_PER_DAY = SECONDS_PER_DAY * 1000L

Expand Down Expand Up @@ -192,11 +193,10 @@ object DateTimeUtils {
* Returns Julian day and nanoseconds in a day from the number of microseconds
*/
def toJulianDay(us: SQLTimestamp): (Int, Long) = {
val seconds = us / MICROS_PER_SECOND
val day = seconds / SECONDS_PER_DAY + JULIAN_DAY_OF_EPOCH
val secondsInDay = seconds % SECONDS_PER_DAY
val nanos = (us % MICROS_PER_SECOND) * 1000L
(day.toInt, secondsInDay * NANOS_PER_SECOND + nanos)
val julian_us = us + JULIAN_DAY_OF_EPOCH * MICROS_PER_DAY
Copy link
Contributor

@vanzin vanzin Sep 9, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without the checks I'm adding in #8606 this can overflow and write garbage to the file...

Copy link
Contributor Author

@davies davies Sep 9, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We haven't handle overflow in many places, the behavior is undefined, it means will be garbage in the file, I think it's fine for now.

Copy link
Contributor Author

@davies davies Sep 9, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we throw an exception, the job will crash if there are some garbage rows in it.

Copy link
Contributor

@vanzin vanzin Sep 9, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok. I'll close my PR and leave the bug open.

Copy link
Contributor Author

@davies davies Sep 9, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Long (2^63) is enough for 584942 years before overflow, so don't worry about it.

Copy link
Contributor

@yhuai yhuai Sep 14, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about adding some comments at here?

val day = julian_us / MICROS_PER_DAY
val micros = julian_us % MICROS_PER_DAY
(day.toInt, micros * 1000L)
}

/**
Expand Down
17 changes: 8 additions & 9 deletions sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,15 +52,14 @@ class DateTimeUtilsSuite extends SparkFunSuite {
assert(ns === 0)
assert(fromJulianDay(d, ns) == 0L)

val t = Timestamp.valueOf("2015-06-11 10:10:10.100")
val (d1, ns1) = toJulianDay(fromJavaTimestamp(t))
val t1 = toJavaTimestamp(fromJulianDay(d1, ns1))
assert(t.equals(t1))

val t2 = Timestamp.valueOf("2015-06-11 20:10:10.100")
val (d2, ns2) = toJulianDay(fromJavaTimestamp(t2))
val t22 = toJavaTimestamp(fromJulianDay(d2, ns2))
assert(t2.equals(t22))
Seq(Timestamp.valueOf("2015-06-11 10:10:10.100"),
Timestamp.valueOf("2015-06-11 20:10:10.100"),
Timestamp.valueOf("1900-06-11 20:10:10.100")).foreach { t =>
val (d, ns) = toJulianDay(fromJavaTimestamp(t))
assert(ns > 0)
val t1 = toJavaTimestamp(fromJulianDay(d, ns))
assert(t.equals(t1))
}
}

test("SPARK-6785: java date conversion before and after epoch") {
Expand Down