Skip to content

Conversation

@waterlx
Copy link
Contributor

@waterlx waterlx commented Apr 24, 2020

The PR is to address the following exception when trying to write a type of TimestampType.withoutZone() by GenericParquetWriter:

java.time.LocalDateTime cannot be cast to java.time.OffsetDateTime
java.lang.ClassCastException: java.time.LocalDateTime cannot be cast to java.time.OffsetDateTime
	at org.apache.iceberg.data.parquet.GenericParquetWriter$TimestamptzWriter.write(GenericParquetWriter.java:213)
	at org.apache.iceberg.parquet.ParquetValueWriters$StructWriter.write(ParquetValueWriters.java:445)
	at org.apache.iceberg.parquet.ParquetWriter.add(ParquetWriter.java:129)

The root cause is that it is hardcoded to new TimestamptzWriter when meeting TIMESTAMP_MICROS.
The exception above could be triggered by running the test case provided in the PR.

@waterlx
Copy link
Contributor Author

waterlx commented Apr 24, 2020

@rdblue would you please help to review it at your most convenience?

@rdblue
Copy link
Contributor

rdblue commented Apr 24, 2020

Looks like this is a subset of #891. I think we should work on that PR instead. That's currently blocked because it requires Parquet 1.11.0 or later, but we currently support 1.10.0.

@waterlx
Copy link
Contributor Author

waterlx commented Apr 26, 2020

This PR is a dup of (or trying to address) the issue of #475 , and it is a subset of #891 as mentioned by Ryan. Ready to close this PR.

@waterlx
Copy link
Contributor Author

waterlx commented Apr 26, 2020

Hi @rdblue
Regarding:

That's currently blocked because it requires Parquet 1.11.0 or later, but we currently support 1.10.0.

A little confused. Iceberg is already built on Parquet 1.11 (I checked versions.props and versions.lock). Do you mean that we would like to keep backward compatible with Parquet 1.10(when there is no logcial type annotation) ?

@waterlx waterlx closed this Apr 26, 2020
@rdblue
Copy link
Contributor

rdblue commented Apr 26, 2020

Yes, we want to keep Iceberg compatible with 1.10 for the next release, to give some time to vet the new Parquet version and roll back to 1.10 if needed without also rolling back Iceberg.

@waterlx waterlx deleted the withoutzone branch May 13, 2020 01:31
simonsssu pushed a commit to simonsssu/iceberg that referenced this pull request Aug 26, 2020
…ck-port apache#965 from open source. To be reverted when apache#891 is merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants