update to spark3.1

[aviatesk]

AFAIU the only requirement is update for apache/spark#29983.
In order to be consistent with the previous behavior and pass the
existing test suite, this PR is essentially equavalent to setting
spark.sql.legacy.statisticalAggregate to true.

Now the code is incompatible with spark-2.x or spark-3.0, and so I'd
like to recommend only supporting spark 3.1 and higher and scala 2.12
from now on.

Issue #, if available:

Description of changes:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[twollnik]

Hi @aviatesk,
Thanks for submitting this PR! Unfortunately we can't make changes that are incompatible with Spark-2.4. Is it possible to keep the backwards compatibility to your knowledge?

[chethanuk]

@aviatesk When will support for 3.1 gonna get released?

[twollnik]

Sadly, we can't drop 2.x compatibility. We have no immediate plan to support spark 3.1. thanks you anyways for introducing this PR!

[chethanuk]

Now the code is incompatible with spark-2.x or spark-3.0, and so I'd
like to recommend only supporting spark 3.1 and higher and scala 2.12
from now on.

Let's do this instead, create a separate folder for spark3.1 and keep the required changes in there.

Also, Just ignoring Spark 3.1 bcoz we can't drop 2.x compatibility will not help Spark 3X users.
I mean if support for 3.1X is not available, users like me will have to use ge or other data quality tool if no support

[apython1998]

If anybody has suggestions for scala spark alternatives to Deequ that supports 3.1.x out of the box, please update this thread.