Spark 3.1: Remove module

[aokolnychyi]

This PR removes support for Spark 3.1 as discussed on the dev list here.

[aokolnychyi]

cc @Fokko @nastra @amogh-jahagirdar @RussellSpitzer @flyrain @rdblue

[ajantha-bhat]

LGTM.

Some more followups may be needed.

• We can remove spark 3.1 mention from documents also. (spark-writes.md, nessie.md)
• Some table properties can be removed which was only used in spark-3.1
  

  iceberg/core/src/main/java/org/apache/iceberg/TableProperties.java

  Lines 356 to 367 in 1e52f2e

  	/**
  	* @deprecated will be removed once Spark 3.1 support is dropped, the cardinality check is always
  	* performed starting from 0.13.0.
  	*/
  	@Deprecated
  	public static final String MERGE_CARDINALITY_CHECK_ENABLED =
  	"write.merge.cardinality-check.enabled";
  	/**
  	* @deprecated will be removed once Spark 3.1 support is dropped, the cardinality check is always
  	* performed starting from 0.13.0.
  	*/
  	@Deprecated public static final boolean MERGE_CARDINALITY_CHECK_ENABLED_DEFAULT = true;

I can raise the followup PR if you are busy (If you can't handle in this PR maybe)

[Fokko]

I agree that we should remove 3.1, but why not do it after 1.4? Am I missing any downsides besides having to upload a few more artifacts?

Some folks went through the trouble of backporting features to 3.1 (eg. #8217). Also, as @ajantha-bhat mentioned, we need to clean up the docs as well: #7390 (review)

[aokolnychyi]

@Fokko, you are right that we did cherry-pick some changes to 3.1 but it is heavily behind in capabilities. There are some known limitations that may affect performance (e.g. the way we do cardinality check and distribute data on write, for example). Someone may test the brand new Iceberg jar with Spark 3.1 and think that's the best we offer.

That said, it is minor. I am debating this too and have no problem doing 1.4 with this module in.

[aokolnychyi]

@ajantha-bhat, I'd appreciate if you could help with the changes in core and docs in a follow-up PR.

[aokolnychyi]

@danielcweeks @RussellSpitzer @rdblue, any thoughts on removing or keeping Spark 3.1 before 1.4?

[RussellSpitzer]

+1 On removing it now, I think any work on 3.1 is really just sunk costs. We don't gain that much and we are basically saying we will continue supporting 3.1 for any 1.4.x releases that may need to occur.

[danielcweeks]

I'm +1 on removing it as well (agree with @RussellSpitzer points).

[aokolnychyi]

Agreed, let's remove now.

[aokolnychyi]

@ajantha-bhat, I've submitted a PR to quickly drop the table props to cut an RC asap. Could you look into other places that I missed?

[ajantha-bhat]

@ajantha-bhat, I've submitted a PR to quickly drop the table props to cut an RC asap. Could you look into other places that I missed?

Sorry for the delay. I live in India timezone.
Here is the PR #8671
We can merge this and use it if there is another RC for some reason.