INT96 should be marked as deprecated

[asfimport]

As discussed in the mailing list, INT96 is only used to represent nanosec timestamp in Impala for some historical reasons, and should be deprecated. Since nanosec precision is rarely a real requirement, one possible and simple solution would be replacing INT96 with INT64 (TIMESTAMP_MILLIS) or INT64 (TIMESTAMP_MICROS).

Several projects (Impala, Hive, Spark, ...) support INT96.
We need a clear spec of the replacement and the path to deprecation.

Reporter: Cheng Lian / @liancheng
Assignee: Lars Volker / @lekv

Related issues:

• Interpret Parquet INT96 type as FIXED[12] AVRO Schema (relates to)
• Define INT96 ordering (relates to)
• Support Parquet time related logical types (relates to)

PRs and other links:

• GitHub Pull Request #86
• IMPALA-5049 - Switch Parquet timestamp format to INT64

_{Note: This issue was originally created as PARQUET-323. Please see the migration documentation for further details.}

[asfimport]

Ryan Blue / @rdblue:
While we don't want to encourage anyone to use int96 as a timestamp, I'm not sure if it is necessary to get rid of the type entirely. If int96 is a JVM and hardware-supported type, then it makes sense to keep it. But if everything jumps to int128 instead, then maybe we should go with that instead. Do we have enough information to make that decision yet?

[asfimport]

Cheng Lian / @liancheng:
AFAIK JVM doesn't have native support for INT96.

[asfimport]

Julien Le Dem / @julienledem:
I think we should deprecate it and discourage its use. For backward compatibility, it has to stay.
https://github.com/apache/parquet-format/blob/master/LogicalTypes.md doesn't even refer to it.

[asfimport]

Ryan Blue / @rdblue:
+1

[asfimport]

Lars Volker / @lekv:
We discussed this issue in today's Parquet sync and agreed to deprecate INT96. As a replacement to store timestamps (the most common use for INT96) we will encourage all projects who currently use INT96 to switch to INT64 and either use the TIMESTAMP_MILLIS or TIMESTAMP_MICROS logical types.

We will not fix the ordering issues around INT96 that resulted in parquet-mr writing wrong min/max statistics.

[asfimport]

Lars Volker / @lekv:
I created IMPALA-5049 to track the required work on Impala's side.

What other projects need to transition away from INT96? Who should create JIRAs for them?

[asfimport]

Jeff Knupp / @jeffknupp:
I'm reasonably sure it's the only timestamp format that Spark currently supports (see apache/spark#3820).

[asfimport]

Uwe Korn / @xhochy:
The "offending code" that blocks the usage of TIMESTAMP_MILLIS in Spark is at https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala#L161

[asfimport]

Jeff Knupp / @jeffknupp:
There is a PR against Spark that adds support for TIMESTAMP_MILLIS (apache/spark#15332) but after a good deal of commentary and a working patch, it has sat untouched since October. Perhaps simply asking in a Spark JIRA ticket to re-review that PR would be the quickest way for Spark to support reading int64 TIMESTAMP_MILLIS?

[asfimport]

Songzhi Liu:
@lekv Hi Lars, is it the direction of the community to deprecate INT96 support? In that case, what will be used to support nanosec precision? In some financial services use cases, support for nanosecond precision is critical. Could you please advise the potential plan of the deprecation plan?

 

[asfimport]

Wes McKinney / @wesm:
[~liusztc09] support for nanoseconds in the INT64 type has been added

b879065

This is the recommended path forward