HIVE-24436: Fix Avro NULL_DEFAULT_VALUE compatibility issue

[wangyum]

What changes were proposed in this pull request?

This pr replace null with JsonProperties.NULL_VALUE to fix compatibility issue:

1. java.lang.NoSuchMethodError: 'void org.apache.avro.Schema$Field.(java.lang.String, org.apache.avro.Schema, java.lang.String, org.codehaus.jackson.JsonNode)'

   - create hive serde table with Catalog
   *** RUN ABORTED ***
     java.lang.NoSuchMethodError: 'void org.apache.avro.Schema$Field.<init>(java.lang.String, org.apache.avro.Schema, 
   java.lang.String, org.codehaus.jackson.JsonNode)'
     at org.apache.hadoop.hive.serde2.avro.TypeInfoToSchema.createAvroField(TypeInfoToSchema.java:76)
     at org.apache.hadoop.hive.serde2.avro.TypeInfoToSchema.convert(TypeInfoToSchema.java:61)
     at org.apache.hadoop.hive.serde2.avro.AvroSerDe.getSchemaFromCols(AvroSerDe.java:170)
     at org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:114)
     at org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:83)
     at org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:533)
     at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:450)
     at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:437)
     at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:281)
     at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:263)
   

2. org.apache.avro.AvroRuntimeException: Unknown datum class: class org.codehaus.jackson.node.NullNode

   - alter hive serde table add columns -- partitioned - AVRO *** FAILED ***
     org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: 
   org.apache.avro.AvroRuntimeException: Unknown datum class: class org.codehaus.jackson.node.NullNode;
     at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:112)
     at org.apache.spark.sql.hive.HiveExternalCatalog.createTable(HiveExternalCatalog.scala:245)
     at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.createTable(ExternalCatalogWithListener.scala:94)
     at org.apache.spark.sql.catalyst.catalog.SessionCatalog.createTable(SessionCatalog.scala:346)
     at org.apache.spark.sql.execution.command.CreateTableCommand.run(tables.scala:166)
     at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
     at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
     at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
     at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:228)
     at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3680)
   

Why are the changes needed?

For compatibility with Avro 1.9.x and Avro 1.10.0.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Apply this patch to branch-2.3 and build Hive 2.3.8-SNAPSHOT and run Spark test:

mvn -Dtest=none -DwildcardSuites=org.apache.spark.sql.hive.execution.HiveDDLSuite test -pl sql/hive

[sunchao]

LGTM

[sunchao]

Thanks @wangyum ! Merged to master, branch-2.3, and branch-3.1. Sorry I had to revert & retitle it because the first commit misses JIRA title.

[wangyum]

Thank you @sunchao.

[dongjoon-hyun]

Thank you, @sunchao and @wangyum .

[iemejia]

Excellent! For info vote for Avro 1.10.1 (that fixes the null default fix) is almost over and artifacts should be published tomorrow or the day after I will update my PR once it is out.