[SPARK-36702][SQL] ArrayUnion handle duplicated Double.NaN and Float.Nan

[AngersZhuuuu]

What changes were proposed in this pull request?

For query

select array_union(array(cast('nan' as double), cast('nan' as double)), array())

This returns [NaN, NaN], but it should return [NaN].
This issue is caused by OpenHashSet can't handle Double.NaN and Float.NaN too.
In this pr we handle NaN like Null value

Why are the changes needed?

Fix bug

Does this PR introduce any user-facing change?

ArrayUnion won't show duplicated NaN value

How was this patch tested?

Added UT

[AngersZhuuuu]

ping @cloud-fan

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47640/

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47641/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47640/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47641/

[SparkQA]

Test build #143136 has finished for PR 33951 at commit 5c515b8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #143137 has finished for PR 33951 at commit cb928ff.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

How does this differ from #33955 ?

[AngersZhuuuu]

How does this differ from #33955 ?

#33955 make all flag in SQLOpenHashSet,

[HyukjinKwon]

Mind elabourating #33951 (comment)? Also let's keep one PR only for the issue if we're clear on which approach to go ahead for.

[AngersZhuuuu]

Mind elabourating #33951 (comment)? Also let's keep one PR only for the issue if we're clear on which approach to go ahead for.

Close this, you can go ahead on that pr.