Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Built-in conversions for nested logical types fail on Avro <= 1.8 #3114

Closed
clairemcginty opened this issue Jan 6, 2025 · 0 comments · Fixed by #3111
Closed

Built-in conversions for nested logical types fail on Avro <= 1.8 #3114

clairemcginty opened this issue Jan 6, 2025 · 0 comments · Fixed by #3111

Comments

@clairemcginty
Copy link
Contributor

Describe the bug, including details regarding any error messages, version, and platform.

When an Avro record contains nested logical types but no top-level logical types, i.e.

{
  "type": "record",
  "name": "NestedLogicalTypes",
  "fields": [
    {
      "name": "innerType",
      "type": {
        "type": "record",
        "name": "InnerType",
        "fields": [
          {
            "name": "ts",
            "type": {
              "type": "long",
              "logicalType": "timestamp-millis"
            }
          }
        ]
      }
    }
  ]
}

Writes fails to convert the logical types from in-memory representation to underlying primitive:

Cause: java.lang.ClassCastException: class org.joda.time.DateTime cannot be cast to class java.lang.Number (org.joda.time.DateTime is in unnamed module of loader 'app'; java.lang.Number is in module java.base of loader 'bootstrap')
at org.apache.parquet.avro.AvroWriteSupport.writeValueWithoutConversion(AvroWriteSupport.java:345)
at org.apache.parquet.avro.AvroWriteSupport.writeValue(AvroWriteSupport.java:282)
at org.apache.parquet.avro.AvroWriteSupport.writeRecordFields(AvroWriteSupport.java:202)
at org.apache.parquet.avro.AvroWriteSupport.writeRecord(AvroWriteSupport.java:185)
at org.apache.parquet.avro.AvroWriteSupport.writeValueWithoutConversion(AvroWriteSupport.java:371)
at org.apache.parquet.avro.AvroWriteSupport.writeValue(AvroWriteSupport.java:285)
at org.apache.parquet.avro.AvroWriteSupport.writeRecordFields(AvroWriteSupport.java:202)
at org.apache.parquet.avro.AvroWriteSupport.write(AvroWriteSupport.java:178)
at org.apache.parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:152)
at org.apache.parquet.hadoop.ParquetWriter.write(ParquetWriter.java:428)

Component(s)

Avro

Fokko pushed a commit that referenced this issue Jan 14, 2025
#3111)

* Fix LogicalType conversions for nested records on Avro <= 1.8

* Move inner field traversal outside finally block
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant