[SPARK-29609][SQL] DataSourceV2: Support DROP NAMESPACE

[imback82]

What changes were proposed in this pull request?

This PR adds DROP NAMESPACE support for V2 catalogs.

Why are the changes needed?

Currently, you cannot drop namespaces for v2 catalogs.

Does this PR introduce any user-facing change?

The user can now perform the following:

CREATE NAMESPACE mycatalog.ns
DROP NAMESPACE mycatalog.ns
SHOW NAMESPACES IN mycatalog # Will show no namespaces 

to drop a namespace ns inside mycatalog V2 catalog.

How was this patch tested?

Added unit tests.

[imback82]

cc: @cloud-fan @rdblue

[imback82]

@rdblue / @cloud-fan did we discuss how we want to drop namespaces if cascade is set for v2 catalogs?

[rdblue]

In the v2 API, namespaces that are not empty can be rejected by throwing IllegalStateException. I'd say catch that if cascade is set and throw an error that cascade isn't supported.

[cloud-fan]

yea this is a good workaround now. I'm wondering if we should support cascade before 3.0, as this may break DS v2 APIs, e.g. we may need to add a cascade parameter to the dropNamespace API. DROP TABLE has the same problem.

[imback82]

Thanks for the suggestion. I updated with a workaround solution. And +1 for adding cascade option to dropNamespace API. I initially tried to implement cascading option with existing APIs, but it was a bit cumbersome.

[SparkQA]

Test build #112691 has finished for PR 26262 at commit 588b2ce.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Could you resolve the conflicts, @imback82 ?

[SparkQA]

Test build #112788 has finished for PR 26262 at commit 634952e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• case class Version() extends LeafExpression with CodegenFallback
• case class LoadDataStatement(
• case class LocalShuffleReaderExec(child: QueryStageExec) extends UnaryExecNode

[dongjoon-hyun]

+1, LGTM. Thank you, @imback82 , @cloud-fan , @rdblue .
Merged to master.

[imback82]

Thanks everyone for help!