[SPARK-24772][SQL] Avro: support logical date type#21984
[SPARK-24772][SQL] Avro: support logical date type#21984gengliangwang wants to merge 2 commits intoapache:masterfrom
Conversation
|
Test build #94132 has finished for PR 21984 at commit
|
| (getter, ordinal) => ByteBuffer.wrap(getter.getBinary(ordinal)) | ||
| case DateType => | ||
| (getter, ordinal) => getter.getInt(ordinal) * DateTimeUtils.MILLIS_PER_DAY | ||
| (getter, ordinal) => getter.getInt(ordinal) |
There was a problem hiding this comment.
For the write path, let's drop the previous conversion to Long
There was a problem hiding this comment.
Does this cause a behaviour change comparing to the third party one?
There was a problem hiding this comment.
I don't think it is a real behavior change. The only concern is that the Avro file with date type column is written with this built-in package, and read by third party one with user specify schema. The case should be very trivial and we can ignore that.
There was a problem hiding this comment.
There are 2 kinds of compatibilities:
- the file written by old avro data source can be read by the new avro data source
- the file written by new avro data source can be read by the old avro data source
I think we should focus on 1) and ignore 2)
| s"Cannot convert Avro logical type ${other} to Catalyst Timestamp type.") | ||
| } | ||
|
|
||
| // Before we upgrade Avro to 1.8 for logical type support, spark-avo converts Long to Date. |
|
Test build #94260 has finished for PR 21984 at commit
|
|
LGTM, merging to master! |
|
LGTM too |
In PR apache#21984 and apache#21935 , the related test cases are using binary files created by Python scripts. Generate the binary files in test suite to make it more transparent. Also we can Also move the related test cases to a new file `AvroLogicalTypeSuite.scala`. Unit test. Closes apache#22091 from gengliangwang/logicalType_suite. Authored-by: Gengliang Wang <gengliang.wang@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> RB=2651977 BUG=LIHADOOP-59243 G=spark-reviewers R=ekrogen A=ekrogen
5 participants
What changes were proposed in this pull request?
Support Avro logical date type:
https://avro.apache.org/docs/1.8.2/spec.html#Date
How was this patch tested?
Unit test