[SPARK-29931][SQL] Declare removing of legacy SQL configs in Spark 4.0 or a later release

[MaxGekk]

What changes were proposed in this pull request?

Added the sentence "This config will be removed in Spark 4.0 or a later release." to doc for all legacy SQL configs existed before Spark 3.0.

Why are the changes needed?

To inform users that legacy configs are temporary, and have limited life time up to the next major release (or maybe later).

Does this PR introduce any user-facing change?

No

How was this patch tested?

Just checking coding style via ./dev/scalastyle.

[SparkQA]

Test build #113999 has finished for PR 26571 at commit f2219d7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Do we want to just deprecated these already? I don't think we have to tell people they may be deprecated in the future, nor commit to a release to remove them. It can say "removed in a future release" or just not at all. That's kind of implied by deprecation.

[MaxGekk]

Do we want to just deprecated these already?

We want to remove the legacy configs existed before Spark 3.0. So, users have time up to the next major release 4.0 to rewrite their apps, and get rid off the config settings. After Spark 4.0, we cannot guarantee that the configs will exist.

It can say "removed in a future release" or just not at all.

What do you mean by "a future release"? 3.0.1, 3.1, 4.0?

[MaxGekk]

@srowen Let's look at the recently removed legacy configs: #26559 . If they didn't have clear sentence about removing in Spark 3.0, what should happen that triggers you to remove them? There is a risk that the legacy code will exist forever.

[srowen]

I'm suggesting actually deprecating them as the signal that they'll be removed at some point. I agree that it's unlikely to be removed before 4.0, but we don't necessarily have to say that; it's kind of implied by general policy. I don't mind the mention in each of them, it's just a little more noise. Maybe it helps call attention to it, sure.

[HyukjinKwon]

I agree with @srowen's last comment. Ideally ALL of deprecated items should be removed before 4.0 or at most at 4.0.

Plus, I personally think we should remove those things out even in Spark 3.0. E.g., configurations deprecated before and at Spark 2.3 should likely be removed out in 3.0. If we do it more aggressively, we can remove out deprecated ones from 2.4.

[dongjoon-hyun]

I'll close this PR, @MaxGekk . According to the above advices, this seems to limit our options without giving us much benefit. I don't think 4.0 will happen in the next 3 years practically.

Thank you all!