int96 support in parquet

[thesquelched]

When writing parquet, Spark wants to write timestamps as int96 for compatibility with hive/presto/impala/etc. Iceberg writes timestamps as the parquet timestamp logical type over int64. While this is less than ideal, the real problem is that int96 data is not supported at all, making it impossible to use iceberg with existing parquet data files without first rewriting the data.

Ideally, iceberg would add support for int96-as-timestamp, similar to how spark handles them. However, at minimum, iceberg should be able to understand the int96 type.

[gustavoatt]

It seems like the current spec does require timestamps to be stored as epoch microseconds in an int64.

However I do think that supporting int96 timestamps on the read path would be really handy when migrating tables to iceberg to avoid having to rewrite data.

[rdblue]

To be clear, when you use Iceberg with Spark, Iceberg's timestamp type works just fine. So this is only necessary when working with other systems. And I believe Spark can already write the INT64 timestamps that Iceberg uses.

Iceberg is strict about the files that it writes and uses the timestamps specified in Parquet. The INT96 timestamps have a ton of problems, so it's really unlikely that it would ever make sense for Iceberg to write them.

Iceberg could read data files with those timestamps, but they would have to be existing data files imported by converting a Parquet/Hive table over to Iceberg. I guess if someone needed to implement that, we'd review it.

[gustavoatt]

Thanks @rdblue! We would most likely need this to migrate some of our biggest tables to Iceberg since we were writing timestamps using int96.

I can take this on as only read-support, do you have any pointers where the change would need to be made? I'm looking at /parquet/ColumnIterator and /parquet/PageIterator since that is where we have parquet read support.

[aokolnychyi]

Is the INT96 timestamp representation in query engines like Hive, Impala, Spark identical? In other words, will we have to understand which system wrote that INT96 timestamp in order to read it in Iceberg? I remember different query engines did different workarounds.

[rdblue]

Off the top of my head, we'd need to support:

1. A value reader for INT96 that converts values to micros
2. Type conversion from Parquet schemas to Iceberg (timestamp or timestamptz?)

We would not need to modify any of the Parquet filter classes because INT96 timestamps have broken min/max stats. We would probably want to ignore them entirely in the Parquet metrics code as well.

[rdblue]

Also, we should find out more about differences between engines so we can document how our implementation works. I think there were some bugs there. Thanks for pointing it out, @aokolnychyi.

[gustavoatt]

Here is the spark code that writes int96 timestamps.

There is some discussion on SPARK-12297 about having Spark int96 timestamps to work correctly with Hive & Impala.

[gustavoatt]

Sent a PR to add int96 timestamp support in #1184. I was able to verify the behavior for spark.

I think Hive writes them in the exact same way, would need to verify that to be sure.

[thesquelched]

I think there's a little bit more work to be done. I get the following when using org.apache.iceberg.parquet.ParquetUtil.fileMetrics:

java.lang.UnsupportedOperationException: Cannot convert unknown primitive type: optional int96 created_at
  at org.apache.iceberg.parquet.MessageTypeToType.primitive(MessageTypeToType.java:163)
  at org.apache.iceberg.parquet.MessageTypeToType.primitive(MessageTypeToType.java:41)
  at org.apache.iceberg.parquet.ParquetTypeVisitor.visit(ParquetTypeVisitor.java:41)
  at org.apache.iceberg.parquet.ParquetTypeVisitor.visitFields(ParquetTypeVisitor.java:168)
  at org.apache.iceberg.parquet.ParquetTypeVisitor.visit(ParquetTypeVisitor.java:38)
  at org.apache.iceberg.parquet.ParquetSchemaUtil.convert(ParquetSchemaUtil.java:47)
  at org.apache.iceberg.parquet.ParquetUtil.footerMetrics(ParquetUtil.java:90)
  at org.apache.iceberg.parquet.ParquetUtil.fileMetrics(ParquetUtil.java:74)
  at getMetrics(<console>:43)
  ... 53 elided

[aokolnychyi]

I think we will need a test when we create a Parquet table with int96 timestamps, migrate it to Iceberg, and read back to make sure this works end-to-end.

[gustavoatt]

Agreed! I'll add a test for that and make sure that end to end icerberg-parquet can read tables with int96 timestamps.

Current tests only verifies that we can read the parquet file but not whether it can be read once it is migrated to Iceberg.

[aokolnychyi]

We will have to be careful with stats as well. I think Parquet does not keep stats for int96, though.

[rdblue]

Yeah, looks like a problem with import. @aokolnychyi is right that stats for int96 columns are unreliable. That makes the fix easy: we just ignore int96 columns in the import path.