Skip to content

[SPARK-34455][SQL] Deprecate spark.sql.legacy.replaceDatabricksSparkAvro.enabled #31578

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MaxGekk wants to merge 2 commits into from
Closed

[SPARK-34455][SQL] Deprecate spark.sql.legacy.replaceDatabricksSparkAvro.enabled #31578

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9de0183
Update the doc
MaxGekk Feb 17, 2021
660db04
Deprecate LEGACY_REPLACE_DATABRICKS_SPARK_AVRO_ENABLED
MaxGekk Feb 17, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/sql-data-sources-avro.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -295,6 +295,7 @@ Configuration of Avro can be done using the `setConf` method on SparkSession or
<td>
If it is set to true, the data source provider <code>com.databricks.spark.avro</code> is mapped
to the built-in but external Avro data source module for backward compatibility.
<br><b>Note:</b> the SQL config has been deprecated in Spark 3.2 and might be removed in the future.
</td>
<td>2.4.0</td>
</tr>
Expand Down
4 changes: 3 additions & 1 deletion sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3145,7 +3145,9 @@ object SQLConf {
DeprecatedConfig(AVRO_REBASE_MODE_IN_WRITE.alternatives.head, "3.2",
s"Use '${AVRO_REBASE_MODE_IN_WRITE.key}' instead."),
DeprecatedConfig(AVRO_REBASE_MODE_IN_READ.alternatives.head, "3.2",
s"Use '${AVRO_REBASE_MODE_IN_READ.key}' instead.")
s"Use '${AVRO_REBASE_MODE_IN_READ.key}' instead."),
DeprecatedConfig(LEGACY_REPLACE_DATABRICKS_SPARK_AVRO_ENABLED.key, "3.2",
"""Use `.format("avro")` in `DataFrameWriter` or `DataFrameReader` instead.""")
Copy link
Member

@dongjoon-hyun dongjoon-hyun Feb 17, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ur, is this guide correct? I guess the users are using .format("avro") already.

Copy link
Member Author

@MaxGekk MaxGekk Feb 18, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The configs is about automatically mapping of .format("com.databricks.spark.avro") to .format("avro"), right? If we remove the config in the future, "com.databricks.spark.avro" will be not mapped to built in avro. So, in the guide, we recommend to change users code, and use the avro format directly.

Copy link
Member

@dongjoon-hyun dongjoon-hyun Feb 18, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I was confused at that.

)

Map(configs.map { cfg => cfg.key -> cfg } : _*)
Expand Down