[SPARK-28612][SQL] Add DataFrameWriterV2 API

[rdblue]

What changes were proposed in this pull request?

This adds a new write API as proposed in the SPIP to standardize logical plans. This new API:

• Uses clear verbs to execute writes, like append, overwrite, create, and replace that correspond to the new logical plans.
• Only creates v2 logical plans so the behavior is always consistent.
• Does not allow table configuration options for operations that cannot change table configuration. For example, partitionedBy can only be called when the writer executes create or replace.

Here are a few example uses of the new API:

df.writeTo("catalog.db.table").append()
df.writeTo("catalog.db.table").overwrite($"date" === "2019-06-01")
df.writeTo("catalog.db.table").overwritePartitions()
df.writeTo("catalog.db.table").asParquet.create()
df.writeTo("catalog.db.table").partitionedBy(days($"ts")).createOrReplace()
df.writeTo("catalog.db.table").using("abc").replace()

How was this patch tested?

Added DataFrameWriterV2Suite that tests the new write API. Existing tests for v2 plans.

[SparkQA]

Test build #110143 has finished for PR 25681 at commit f7ecd17.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #110144 has finished for PR 25681 at commit 8d5b7fe.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #110147 has finished for PR 25681 at commit b36c156.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #110262 has finished for PR 25681 at commit 9472474.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rdblue]

@brkyvz. this is passing tests. Anything else we need to change?

[brkyvz]

nit: if we're only going to support Literals, I feel it may be better to leave this out initially

[brkyvz]

do you want to implement ExpectsInputTypes for these classes for auto analysis checks?

[brkyvz]

This LGTM like last time, do we have JIRAs for Python API support as well as Java API tests?

[brkyvz]

@rdblue Can you please rebase this?

[rdblue]

@brkyvz, I rebased, added Java API tests, and opened SPARK-29157 to track the PySpark API.

[SparkQA]

Test build #110933 has finished for PR 25681 at commit b45593d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• abstract class PartitionTransformExpression extends Expression with Unevaluable
• case class Years(child: Expression) extends PartitionTransformExpression
• case class Months(child: Expression) extends PartitionTransformExpression
• case class Days(child: Expression) extends PartitionTransformExpression
• case class Hours(child: Expression) extends PartitionTransformExpression
• case class Bucket(numBuckets: Literal, child: Expression) extends PartitionTransformExpression
• implicit class OptionsHelper(options: Map[String, String])
• trait WriteConfigMethods[R]
• trait CreateTableWriter[T] extends WriteConfigMethods[CreateTableWriter[T]]

[SparkQA]

Test build #110936 has finished for PR 25681 at commit ab7c3e8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• public class JavaDataFrameWriterV2Suite

[brkyvz]

+1. Merging to master!

[rdblue]

Thanks, @brkyvz!

[cloud-fan]

This should return CreateTableWriter. It doesn't make sense to specify table properties when inserting to an existing table.

[rdblue]

I agree.

[rdblue]

Opened SPARK-29249 for this. Should have a PR posted soon.