Support for TIMESTAMP WITH TIME ZONE with nanosecond precision in Parquet

[zielmicha]

Description

Adds support for reading of Parquet nanosecond timestamp column into TIMESTAMP WITH TIME ZONE Trino type.

Is this change a fix, improvement, new feature, refactoring, or other?

This is a fix or a new feature, depending on how you look at it - the support for TIMESTAMP WITH TIME ZONE already existed for millis and micros precision.

Is this a change to the core query engine, a connector, client library, or the SPI interfaces? (be specific)

This is change to a library used by some connectors (Hive, Delta Lake, Iceberg), but in practice only changes anything for Hive connector (other connectors don't support nanosecond TIMESTAMP WITH TIME ZONE)

How would you describe this change to a non-technical end user or system administrator?

For Hive connector, the change adds support for reading into TIMESTAMP WITH TIME ZONE columns from Parquet files that contain timestamp columns formatted in certain way (storing the timestamp as number of nanoseconds).

Related issues, pull requests, and links

• PR Support for reading TIMESTAMP WITH LOCAL TIME ZONE in Hive connector #13595 that adds support for TIMESTAMP WITH TIME ZONE to Hive connector

Documentation

(I think no documentation is needed)

( ) No documentation is needed.
( ) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.

Release notes

( ) No release notes entries required.
( ) Release notes entries required with the following suggested text:

# Section
* Support for TIMESTAMP WITH TIME ZONE with nanosecond precision in Parquet

[findepi]

What is the end-user visible effect of these changes?

[zielmicha]

What is the end-user visible effect of these changes?

It's now possible to read Parquet files that use NANOS format (link to spec) with columns of type TIMESTAMP WITH TIME ZONE. Previously Trino simply raised exception when attempting to query them.

I have only tested this on Hive connector (with #13595), but this presumably applies to Iceberg and Delta Lake, which already have support for TIMESTAMP WITH TIME ZONE.

[findepi]

@zielmicha thanks for explaining. What's the schema of the attached parquet file?

[zielmicha]

I'm not sure what's the best way to get Parquet schema, but:

import pyarrow.parquet as pq
pq.read_table('issue-5483-nanos.parquet').schema

prints

created: timestamp[ns]

[findepi]

$ parquet meta plugin/trino-hive/src/test/resources/issue-5483-nanos.parquet

File path:  plugin/trino-hive/src/test/resources/issue-5483-nanos.parquet
Created by: parquet-cpp-arrow version 5.0.0
Properties: (none)
Schema:
message schema {
  optional int64 created (TIMESTAMP(NANOS,false));
}


Row group 0:  count: 1  96.00 B records  start: 4  total(compressed): 96 B total(uncompressed):92 B
--------------------------------------------------------------------------------
         type      encodings count     avg size   nulls   min / max
created  INT64     S _ R     1         96.00 B    0       "2020-10-12T16:26:02.90666..." / "2020-10-12T16:26:02.90666..."

[skrzypo987]

The PR name is sort of misleading, as the TIMESTAMP WITH TIME ZONE is the Tirno type, not Parquet.
Try something like "Support read of Parquet nanosecond timestamp column into TIMESTAMP WITH TIME ZONE Trino type"

[zielmicha]

@raunaqmorarka can you take a look? If I'm not mistaken @skrzypo987 has requested your review.

[zielmicha]

@raunaqmorarka will you have time to review this PR? Let me know if I should find another reviewer.

[raunaqmorarka]

@raunaqmorarka will you have time to review this PR? Let me know if I should find another reviewer.

I'll take a look at it early next week

[raunaqmorarka]

It looks like we don't have a way to write this data via trino-parquet, we should add support for that.
There should be a test in AbstractTestParquetReader similar to testTimestamp but with time zone. It'll need update to ParquetTester#writeValue to add support for timestamp with timezone type.
There should also be a test in TestHiveCompatibility once the write path is working.

[zielmicha]

Writing TIMESTAMP WITH TIME ZONE isn't supported by Hive connector anyway yet - I'm planning to implement this in the next PR.

[arhimondr]

I may not be the best person to review this. @raunaqmorarka could you please take an another look?

[mosabua]

@zielmicha could you rebase?

@martint @findepi and @raunaqmorarka .. could you help moving this forward?

[raunaqmorarka]

Please rebase to latest master

[Fokko]

@raunaqmorarka @zielmicha Any progress on this? At Iceberg we're adding support for Nanosecond precision: apache/iceberg#8683

[mosabua]

@zielmicha @martint @raunaqmorarka could we rebase this and move towards merge?

[github-actions]

This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua

[mosabua]

@zielmicha @martint @raunaqmorarka I know this PR has been around a long time. From what I understand it would still be great to get this feature in for @zielmicha .. what are next steps to enable that? Rebase probably.. anything else?

[github-actions]

This pull request has gone a while without any activity. Tagging the Trino developer relations team: @bitsondatadev @colebow @mosabua

[github-actions]

Closing this pull request, as it has been stale for six weeks. Feel free to re-open at any time.

[mosabua]

Reopening since I believe we want this feature added based on past discussions. I will leave @martint to decide on next steps.

[raunaqmorarka]

Superseded by #27835 and #22781