[SPARK-30025][Core] Continuous shuffle block fetching should be disabled by default when the old fetch protocol is used

[xuanyuanking]

What changes were proposed in this pull request?

Disable continuous shuffle block fetching when the old fetch protocol in use.

Why are the changes needed?

The new feature of continuous shuffle block fetching depends on the latest version of the shuffle fetch protocol. We should keep this constraint in BlockStoreShuffleReader.fetchContinuousBlocksInBatch.

Does this PR introduce any user-facing change?

Users will not get the exception related to continuous shuffle block fetching when old version of the external shuffle service is used.

How was this patch tested?

Existing UT.

[cloud-fan]

just for curiosity, how things can go wrong if we do batch fetch with old shuffle protocol? the shuffle server can't recognize the new request and fail?

[SparkQA]

Test build #114403 has finished for PR 26663 at commit c05adaa.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[xuanyuanking]

how things can go wrong if we do batch fetch with old shuffle protocol?

That's the scenario Yuming met at https://github.com/apache/spark/pull/26147/files#r348943508.
For the old shuffle service user, only spark.shuffle.useOldFetchProtocol set to true can job run. But if we batch fetch here, only shuffle block fetcher will merge the request because, in the old code, we do the continuous fetching without checking old fetch protocol. While the server-side would use the old protocol and use OpenBlocks in OneForOneBlockFetcher, which still fetches blocks one by one.
So the size number between fetcher and iterator will be inconsistent.

[SparkQA]

Test build #114450 has finished for PR 26663 at commit ba84e6a.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #114620 has finished for PR 26663 at commit 892da74.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #114703 has finished for PR 26663 at commit 9c53b4a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

thanks, merging to master!

[xuanyuanking]

Thanks!