[SPARK-34455][SQL] Deprecate spark.sql.legacy.replaceDatabricksSparkAvro.enabled
#31578
Conversation
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
+1, LGTM. Thank you, @MaxGekk .
| s"Use '${AVRO_REBASE_MODE_IN_READ.key}' instead.") | ||
| s"Use '${AVRO_REBASE_MODE_IN_READ.key}' instead."), | ||
| DeprecatedConfig(LEGACY_REPLACE_DATABRICKS_SPARK_AVRO_ENABLED.key, "3.2", | ||
| """Use `.format("avro")` in `DataFrameWriter` or `DataFrameReader` instead.""") |
There was a problem hiding this comment.
Ur, is this guide correct? I guess the users are using .format("avro") already.
There was a problem hiding this comment.
The configs is about automatically mapping of .format("com.databricks.spark.avro") to .format("avro"), right? If we remove the config in the future, "com.databricks.spark.avro" will be not mapped to built in avro. So, in the guide, we recommend to change users code, and use the avro format directly.
There was a problem hiding this comment.
Thanks. I was confused at that.
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
I have a question on the deprecation message.
|
Test build #135202 has finished for PR 31578 at commit
|
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
+1, LGTM. Merged to master.
3 participants
What changes were proposed in this pull request?
spark.sql.legacy.replaceDatabricksSparkAvro.enabledto the list of deprecated configsdeprecatedSQLConfigsWhy are the changes needed?
The config exists for enough time. We can deprecate it, and recommend users to use
.format("avro")instead.Does this PR introduce any user-facing change?
Should not except of the warning with the recommendation to use the
avroformat.How was this patch tested?