[WIP][SPARK-42578][CONNECT] Add JDBC to DataFrameWriter #40291
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -17,7 +17,7 @@ | |
|
|
||
| package org.apache.spark.sql | ||
|
|
||
| import java.util.{Locale, Properties} | ||
|
|
||
| import scala.collection.JavaConverters._ | ||
|
|
||
|
|
@@ -345,6 +345,44 @@ final class DataFrameWriter[T] private[sql] (ds: Dataset[T]) { | |
| }) | ||
| } | ||
|
|
||
| /** | ||
| * Saves the content of the `DataFrame` to an external database table via JDBC. In the case the | ||
| * table already exists in the external database, behavior of this function depends on the save | ||
| * mode, specified by the `mode` function (default to throwing an exception). | ||
| * | ||
| * Don't create too many partitions in parallel on a large cluster; otherwise Spark might crash | ||
| * your external database systems. | ||
| * | ||
| * JDBC-specific option and parameter documentation for storing tables via JDBC in <a | ||
| * href="https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html#data-source-option"> | ||
| * Data Source Option</a> in the version you use. | ||
| * | ||
| * @param table | ||
| * Name of the table in the external database. | ||
| * @param connectionProperties | ||
| * JDBC database connection arguments, a list of arbitrary string tag/value. Normally at least | ||
| * a "user" and "password" property should be included. "batchsize" can be used to control the | ||
| * number of rows per insert. "isolationLevel" can be one of "NONE", "READ_COMMITTED", | ||
| * "READ_UNCOMMITTED", "REPEATABLE_READ", or "SERIALIZABLE", corresponding to standard | ||
| * transaction isolation levels defined by JDBC's Connection object, with default of | ||
| * "READ_UNCOMMITTED". | ||
| * @since 3.4.0 | ||
| */ | ||
| def jdbc(url: String, table: String, connectionProperties: Properties): Unit = { | ||
| // connectionProperties should override settings in extraOptions. | ||
|
||
| this.extraOptions ++= connectionProperties.asScala | ||
| // explicit url and dbtable should override all | ||
| this.extraOptions ++= Seq("url" -> url, "dbtable" -> table) | ||
| format("jdbc") | ||
| executeWriteOperation(builder => { | ||
| builder.setTable( | ||
| proto.WriteOperation.SaveTable | ||
| .newBuilder() | ||
| .setTableName(table) | ||
| .setSaveMethod(proto.WriteOperation.SaveTable.TableSaveMethod.TABLE_SAVE_METHOD_SAVE)) | ||
| }) | ||
| } | ||
|
|
||
| /** | ||
| * Saves the content of the `DataFrame` in JSON format (<a href="http://jsonlines.org/"> JSON | ||
| * Lines text format or newline-delimited JSON</a>) at the specified path. This is equivalent | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can remove
ProblemFilters.exclude[Problem]("org.apache.spark.sql.DataFrameWriter.jdbc")fromCheckConnectJvmClientCompatibilityin this pr`
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the reminder.