[HUDI-7995] MercifulJsonConverter support convert number to fixed #11637

waitingF · 2024-07-16T13:39:51Z

Change Logs

When using JsonKafkaSource to dump source, for type decimal, it will be convert to arvo fixed type.
But the source can be Number instead of Fixed, it will fail the conversion. The MR is to fix the small bug.

Impact

none

Risk level (write none, low medium or high below)

low

Documentation Update

no doc update

Contributor's checklist

Read through contributor's guide
Change Logs and Impact were stated clearly
Adequate tests were added if applicable
CI passed

hudi-bot · 2024-07-16T15:15:41Z

CI report:

cbe55e8 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

KnightChess · 2024-07-17T12:12:36Z

hudi-common/src/main/java/org/apache/hudi/avro/MercifulJsonConverter.java

+          GenericFixed genericFixed = DECIMAL_CONVERSION.toFixed(bigDecimal, schema, schema.getLogicalType());
+          return Pair.of(true, new GenericData.Fixed(schema, genericFixed.bytes()));
+        }
+
        // The ObjectMapper use List to represent FixedType
        // eg: "decimal_val": [0, 0, 14, -63, -52] will convert to ArrayList<Integer>
        List<Integer> converval = (List<Integer>) value;


hi, do you know why this value is a list?

The Decimal type uses a byte array to store data internally

thanks, can you provide a JSON example? and add it to ut for make ut more complete

already added a json example in ut

in ut, you directly specified the list, is should be meaningless, List should use Array Type. I don't know why here value can be List type? I am not sure, does mapper.readValue(json, Map.class) will create List type expect Json Array.

KnightChess · 2024-07-17T13:55:11Z

hudi-common/src/main/java/org/apache/hudi/avro/MercifulJsonConverter.java

@@ -301,6 +306,14 @@ private static JsonToAvroFieldProcessor generateFixedTypeHandler() {
    return new JsonToAvroFieldProcessor() {
      @Override
      public Pair<Boolean, Object> convert(Object value, String name, Schema schema, boolean shouldSanitize, String invalidCharMask) {
+        // the value can be Number
+        if (value instanceof Number) {


Why number type will use Fixed type schema?

In my case, I dump mysql binlog to kafka via debezium.
There is a Decimal(20, 0) type field and a Decimal(20, 2) field in the mysql source table, when dumping to kafka, the 2 fields are both stored as number instead of list of Integer, eg 1234567 for Decimal(20, 0), 1234567.89 for Decimal(20, 2).

yes, so I think you avro schema decimal_field should not be fixed, should be float or double

Yeah, in my case, the decimal(20, 0) is for bigint and decimal(20, 2) for double, So the debezium decode as I expected.
And the hudi JdbcSchemaProvider convert the mysql table schema as expected to Decimal(20, 0) and Decimal(20, 2).

But when hudi try decoding the bigint and double field from kafka, it throw exception which is unexpected.

the source table schema like

CREATE TABLE `xxxx` ( `push_id` bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT '', `plan_id` bigint(20) unsigned NOT NULL COMMENT '', `to_station_id` bigint(20) unsigned NOT NULL, `plan_status` tinyint(3) NOT NULL, `api_type` tinyint(3) NOT NULL COMMENT '' PRIMARY KEY (`push_id`), )

tranlated avro schema like

{ "type": "record", "name": "sample_table", "namespace": "sample_namespace", "fields": [ { "name": "push_id", "type": [ "null", { "type": "fixed", "name": "fixed", "namespace": "sample_namespace.sample_table.push_id", "size": 9, "logicalType": "decimal", "precision": 20, "scale": 0 } ], "default": null }, { "name": "plan_id", "type": [ "null", { "type": "fixed", "name": "fixed", "namespace": "sample_namespace.sample_table.plan_id", "size": 9, "logicalType": "decimal", "precision": 20, "scale": 0 } ], "default": null }, { "name": "to_station_id", "type": [ "null", { "type": "fixed", "name": "fixed", "namespace": "sample_namespace.sample_table.to_station_id", "size": 9, "logicalType": "decimal", "precision": 20, "scale": 0 } ], "default": null }, { "name": "plan_status", "type": [ "null", "int" ], "default": null }, { "name": "api_type", "type": [ "null", "int" ], "default": null } ] }

The bigint(20) is tranlated to Decimal(20, 0), but the data in kafka is just long, not the list of Integer, so the json convert throw below exception.

Caused by: org.apache.hudi.internal.schema.HoodieSchemaException: Failed to convert schema from json to avro: {"_event":{"old":null,"type":"insert","ts_ms":1720447722000,"version":"1.9.5.Final","connector":"mysql","name":"debezium.xx","ts":1720447722,"snapshot":"false","database":"xxx","sequence":null,"table":"xxx","server_id":241149,"gtid":"xxx","file":"mysql-bin.000416","position":161676701,"row":0,"thread":1459349,"query":null},"push_id":11193065,"plan_id":8928834333,"to_station_id":2028,"plan_status":5,"api_type":4} at org.apache.hudi.utilities.sources.helpers.AvroConvertor.fromJson(AvroConvertor.java:121) at org.apache.spark.api.java.JavaPairRDD$.$anonfun$toScalaFunction$1(JavaPairRDD.scala:1070) at scala.collection.Iterator$$anon$10.next(Iterator.scala:461) at scala.collection.convert.Wrappers$IteratorWrapper.next(Wrappers.scala:33) at org.apache.hudi.utilities.schema.LazyCastingIterator.computeNext(LazyCastingIterator.java:40) at org.apache.hudi.utilities.schema.LazyCastingIterator.computeNext(LazyCastingIterator.java:30) at org.apache.hudi.client.utils.LazyIterableIterator.next(LazyIterableIterator.java:119) ... 18 more Caused by: java.lang.ClassCastException: java.lang.Integer cannot be cast to java.util.List at org.apache.hudi.avro.MercifulJsonConverter$9.convert(MercifulJsonConverter.java:306) at org.apache.hudi.avro.MercifulJsonConverter$JsonToAvroFieldProcessor.convertToAvro(MercifulJsonConverter.java:203) at org.apache.hudi.avro.MercifulJsonConverter.convertJsonToAvroField(MercifulJsonConverter.java:192) at org.apache.hudi.avro.MercifulJsonConverter.convertJsonToAvro(MercifulJsonConverter.java:134) at org.apache.hudi.avro.MercifulJsonConverter.convert(MercifulJsonConverter.java:115) at org.apache.hudi.utilities.sources.helpers.AvroConvertor.fromJson(AvroConvertor.java:118) ... 24 more

KnightChess · 2024-07-17T13:56:25Z

hudi-common/src/test/java/org/apache/hudi/avro/TestMercifulJsonConverter.java

+    Schema schema = Schema.parse(testSchemaStr);
+
+    String testValueStr = "{\n"
+        + "    \"decimal_field\": 1720623716,\n"


the schema type is Fixed, why use Number value for test?

Because avro store Decimal type as Fixed type internally

the decimal type is logicalType, data type is Fixed, which is bytes, the logical type decimal I think in fixed is used to conver bytes to decimal, not conver number to decimal.

KnightChess

Strictly speaking, I think Fixed type data should not handle numeric data types, although it supports the logical type Decimal. It should ideally refer to handling data types like bytes or strings. However, in a broader sense, enhancing support for numeric types can be considered. Yet, when converting it to JSON, there is ambiguity about what type it should be converted to—based on the actual Avro type or the logical type?
what do you think about? I also consulted the official Avro tools for converting JSON to Avro, which directly throws errors when handling numeric types.

KnightChess · 2024-07-18T04:16:14Z

hudi-common/src/main/java/org/apache/hudi/avro/MercifulJsonConverter.java

+          GenericFixed genericFixed = DECIMAL_CONVERSION.toFixed(bigDecimal, schema, schema.getLogicalType());
+          return Pair.of(true, new GenericData.Fixed(schema, genericFixed.bytes()));
+        }
+
        // The ObjectMapper use List to represent FixedType
        // eg: "decimal_val": [0, 0, 14, -63, -52] will convert to ArrayList<Integer>
        List<Integer> converval = (List<Integer>) value;


thanks, can you provide a JSON example? and add it to ut for make ut more complete

KnightChess · 2024-07-18T06:05:59Z

hudi-common/src/test/java/org/apache/hudi/avro/TestMercifulJsonConverter.java

+    Schema schema = Schema.parse(testSchemaStr);
+
+    String testValueStr = "{\n"
+        + "    \"decimal_field\": 1720623716,\n"


the decimal type is logicalType, data type is Fixed, which is bytes, the logical type decimal I think in fixed is used to conver bytes to decimal, not conver number to decimal.

KnightChess · 2024-07-18T06:08:42Z

hudi-common/src/main/java/org/apache/hudi/avro/MercifulJsonConverter.java

@@ -301,6 +306,14 @@ private static JsonToAvroFieldProcessor generateFixedTypeHandler() {
    return new JsonToAvroFieldProcessor() {
      @Override
      public Pair<Boolean, Object> convert(Object value, String name, Schema schema, boolean shouldSanitize, String invalidCharMask) {
+        // the value can be Number
+        if (value instanceof Number) {


yes, so I think you avro schema decimal_field should not be fixed, should be float or double

waitingF · 2024-07-18T13:26:07Z

Strictly speaking, I think Fixed type data should not handle numeric data types, although it supports the logical type Decimal. It should ideally refer to handling data types like bytes or strings. However, in a broader sense, enhancing support for numeric types can be considered. Yet, when converting it to JSON, there is ambiguity about what type it should be converted to—based on the actual Avro type or the logical type? what do you think about? I also consulted the official Avro tools for converting JSON to Avro, which directly throws errors when handling numeric types.

Yeah, I think support for numeric types should be considered.
In my case, it should be the debezium's responsibility to encode the bigint(20) to fixed(as a byte array). But hudi as a data platform, should be compatible with most of cases for convenience.

KnightChess

@bvaradar can you help review it.

KnightChess · 2024-07-19T09:22:33Z

hudi-common/src/main/java/org/apache/hudi/avro/MercifulJsonConverter.java

@@ -301,6 +306,14 @@ private static JsonToAvroFieldProcessor generateFixedTypeHandler() {
    return new JsonToAvroFieldProcessor() {
      @Override
      public Pair<Boolean, Object> convert(Object value, String name, Schema schema, boolean shouldSanitize, String invalidCharMask) {
+        // the value can be Number
+        if (value instanceof Number) {


KnightChess · 2024-07-19T09:29:55Z

hudi-common/src/main/java/org/apache/hudi/avro/MercifulJsonConverter.java

+          GenericFixed genericFixed = DECIMAL_CONVERSION.toFixed(bigDecimal, schema, schema.getLogicalType());
+          return Pair.of(true, new GenericData.Fixed(schema, genericFixed.bytes()));
+        }
+
        // The ObjectMapper use List to represent FixedType
        // eg: "decimal_val": [0, 0, 14, -63, -52] will convert to ArrayList<Integer>
        List<Integer> converval = (List<Integer>) value;


in ut, you directly specified the list, is should be meaningless, List should use Array Type. I don't know why here value can be List type? I am not sure, does mapper.readValue(json, Map.class) will create List type expect Json Array.

yihua · 2024-07-24T20:16:49Z

This PR covers the decimal support: #11265. There will be more changes on top of it to support different decimal input format.

github-actions bot added the size:S PR with lines of changes in (10, 100] label Jul 16, 2024

MercifulJsonConverter support convert number to fixed

cbe55e8

waitingF force-pushed the HUDI-7995 branch from 0f01a4d to cbe55e8 Compare July 16, 2024 13:47

KnightChess reviewed Jul 17, 2024

View reviewed changes

KnightChess reviewed Jul 18, 2024

View reviewed changes

KnightChess reviewed Jul 19, 2024

View reviewed changes

waitingF closed this Aug 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HUDI-7995] MercifulJsonConverter support convert number to fixed #11637

[HUDI-7995] MercifulJsonConverter support convert number to fixed #11637

waitingF commented Jul 16, 2024

hudi-bot commented Jul 16, 2024

KnightChess Jul 17, 2024

waitingF Jul 18, 2024

KnightChess Jul 18, 2024

waitingF Jul 18, 2024

KnightChess Jul 19, 2024

KnightChess Jul 17, 2024

waitingF Jul 18, 2024

KnightChess Jul 18, 2024

waitingF Jul 18, 2024

KnightChess Jul 19, 2024

KnightChess Jul 17, 2024

waitingF Jul 18, 2024

KnightChess Jul 18, 2024

KnightChess left a comment

KnightChess Jul 18, 2024

KnightChess Jul 18, 2024

KnightChess Jul 18, 2024

waitingF commented Jul 18, 2024

KnightChess left a comment

KnightChess Jul 19, 2024

KnightChess Jul 19, 2024

yihua commented Jul 24, 2024

[HUDI-7995] MercifulJsonConverter support convert number to fixed #11637

[HUDI-7995] MercifulJsonConverter support convert number to fixed #11637

Conversation

waitingF commented Jul 16, 2024

Change Logs

Impact

Risk level (write none, low medium or high below)

Documentation Update

Contributor's checklist

hudi-bot commented Jul 16, 2024

CI report:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KnightChess left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

waitingF commented Jul 18, 2024

KnightChess left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yihua commented Jul 24, 2024