[SPARK-32501][SQL] Convert null to "null" in structs, maps and arrays while casting to strings

[MaxGekk]

What changes were proposed in this pull request?

Convert NULL elements of maps, structs and arrays to the "null" string while converting maps/struct/array values to strings. The SQL config spark.sql.legacy.omitNestedNullInCast.enabled controls the behaviour. When it is true, NULL elements of structs/maps/arrays will be omitted otherwise, when it is false, NULL elements will be converted to "null".

Why are the changes needed?

1. It is impossible to distinguish empty string and null, for instance:

scala> Seq(Seq(""), Seq(null)).toDF().show
+-----+
|value|
+-----+
|   []|
|   []|
+-----+

2. Inconsistent NULL conversions for top-level values and nested columns, for instance:

scala> sql("select named_struct('c', null), null").show
+---------------------+----+
|named_struct(c, NULL)|NULL|
+---------------------+----+
|                   []|null|
+---------------------+----+

3. .show() is different from conversions to Hive strings, and as a consequence its output is different from spark-sql (sql tests):

spark-sql> select named_struct('c', null) as struct;
{"c":null}

scala> sql("select named_struct('c', null) as struct").show
+------+
|struct|
+------+
|    []|
+------+

4. It is impossible to distinguish empty struct/array from struct/array with null in the current implementation:

scala> Seq[Seq[String]](Seq(), Seq(null)).toDF.show()
+-----+
|value|
+-----+
|   []|
|   []|
+-----+

Does this PR introduce any user-facing change?

Yes, before:

scala> Seq(Seq(""), Seq(null)).toDF().show
+-----+
|value|
+-----+
|   []|
|   []|
+-----+

After:

scala> Seq(Seq(""), Seq(null)).toDF().show
+------+
| value|
+------+
|    []|
|[null]|
+------+

How was this patch tested?

By existing test suite CastSuite.

[SparkQA]

Test build #126804 has finished for PR 29311 at commit 8e3b910.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[MaxGekk]

@cloud-fan @HyukjinKwon @maropu Please, take a look at this.

[maropu]

Looks a reasonable change. Actually, hive does so;

hive> select named_struct('a', 1, 'b', null);
{"a":1,"b":null}

hive> select array(1, 2, null);
[1,2,null]

hive> select map('a', 1, 'b', null);
{"a":1,"b":null}

[SparkQA]

Test build #126989 has finished for PR 29311 at commit 6347cda.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #126992 has finished for PR 29311 at commit 2325c7f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #127047 has finished for PR 29311 at commit 2a711dc.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #127064 has finished for PR 29311 at commit a686b99.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[MaxGekk]

@cloud-fan @maropu @HyukjinKwon Please, review this PR.

[cloud-fan]

In Spark 3.1, NULL elements of structures, arrays and maps are converted to "null" in casting them to strings.

[MaxGekk]

The passive stuff was not recommended by IntelliJ IDEA ;-) Native speakers don't like it.

[MaxGekk]

[cloud-fan]

OK, but let's highlight it's for "casting to string", not for general CAST

[MaxGekk]

I have replaced it by your sentence already.

[SparkQA]

Test build #127083 has finished for PR 29311 at commit 9547682.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #127088 has finished for PR 29311 at commit 39daa90.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

thanks, merging to master!