[#2039] Support Default Value Semantis: use default values reading AVRO files

[shenodaguirguis]

Last change of 3 (see description: apache/iceberg#2496 (comment))
The main changes here are in BuildAvroProjection.java and PruneColumns.java which makes sure that default value is copied over and used while reading projected columns with default values.
Other changes are utils, and testing changes.

There will be a followup PR to handle default values in serialization/deserializaiton, but this can go on parallel with ORC and Parquet changes.

[Update: during integration testing, I had to implement the default value serialization/deserialization. It works now on spark shell to select missing columns with default values and return the default value:

// required with non-null default
scala> spark.sql(s"select * from u_sguirgui.testi6").show
+---------+-----------------+                                                    
|firstname |     lastname          |
+---------+-----------------+
|        f        |               l.             |
|     Adam  | default lastname | <-- omitting the required col altogether
+---------+-----------------+
// optional with default value
scala> DaliSpark.createDataFrame("dalids:///u_sguirgui.test5", Map(DaliSpark.PROJECT_COLS -> "firstname, lastname")).show
+---------+----------------+
| firstname|        lastname      |
+---------+----------------+
|     Abel     |            Adam       |
|     Adam   |            null           | <-- using the null option
|     Adam|default lastname   |  <-- omitting the col altogether
+---------+----------------+

Also, during integration testing, it turns out that Avro default values for records are Maps, not Lists, so had to change that in the NestedField class.

[shenodaguirguis]

@wmoustafa @funcheetah @HotSushi please take a look

[wmoustafa]

This branch seems to have common code with a the out if branch. Do you see room to combine and simplify?

[shenodaguirguis]

we have 4 cases (for 2 conditions), for three of them we create a new field and add to updatedFileds, but different fields. The only thing that can be done is to create a shallow 3-lines function with 5 argument. I prefer to keep as is

[wmoustafa]

How about if the newSchema is an optional schema or has a default value that is null, then make sure the NULL option is the first.

[shenodaguirguis]

(optional_no_default || optional_null_default) != (optional || has_default)

[wmoustafa]

Refactoring as recommended above may help here.

[wmoustafa]

What is the value of calling isNonNullDefault here? It just another way to do the same check as before !(field.defaultVal() instanceof JsonProperties.Null)?

[wmoustafa]

For FIXED and BINARY, should do something like Base64.getEncoder().encodeToString(bytes)? (also the opposite transformation on the deserialization side).

[shenodaguirguis]

good catch! on it

[shenodaguirguis]

wrapped into ByteBuffer

[funcheetah]

Thanks @shenodaguirguis ! This PR looks good to me in general and leave a few minor comments. One question I have: is default value for complex union supported?

[funcheetah]

Thanks a lot @shenodaguirguis for this feature!