Skip to content

Conversation

@MaxGekk
Copy link
Member

@MaxGekk MaxGekk commented Feb 17, 2021

What changes were proposed in this pull request?

Move the datetime rebase SQL configs from the legacy namespace by:

  1. Renaming of the existing rebase configs like spark.sql.legacy.parquet.datetimeRebaseModeInRead -> spark.sql.parquet.datetimeRebaseModeInRead.
  2. Add the legacy configs as alternatives
  3. Deprecate the legacy rebase configs.

Why are the changes needed?

The rebasing SQL configs like spark.sql.legacy.parquet.datetimeRebaseModeInRead can be used not only for migration from previous Spark versions but also to read/write datatime columns saved by other systems/frameworks/libs. So, the configs shouldn't be considered as legacy configs.

Does this PR introduce any user-facing change?

Should not. Users will see a warning if they still use one of the legacy configs.

How was this patch tested?

  1. Manually checking new configs:
scala> spark.conf.get("spark.sql.parquet.datetimeRebaseModeInRead")
res0: String = EXCEPTION

scala> spark.conf.set("spark.sql.legacy.parquet.datetimeRebaseModeInRead", "LEGACY")
21/02/17 14:57:10 WARN SQLConf: The SQL config 'spark.sql.legacy.parquet.datetimeRebaseModeInRead' has been deprecated in Spark v3.2 and may be removed in the future. Use 'spark.sql.parquet.datetimeRebaseModeInRead' instead.

scala> spark.conf.get("spark.sql.parquet.datetimeRebaseModeInRead")
res2: String = LEGACY
  1. By running a datetime rebasing test suite:
$ build/sbt "test:testOnly *ParquetRebaseDatetimeV1Suite"

@MaxGekk
Copy link
Member Author

MaxGekk commented Feb 17, 2021

@cloud-fan @HyukjinKwon @mswit-databricks @tomvanbussel Could you review this PR, please.

@SparkQA
Copy link

SparkQA commented Feb 17, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39778/

@SparkQA
Copy link

SparkQA commented Feb 17, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39778/

@cloud-fan
Copy link
Contributor

thanks, merging to master!

@cloud-fan cloud-fan closed this in 5957bc1 Feb 17, 2021
@SparkQA
Copy link

SparkQA commented Feb 17, 2021

Test build #135197 has finished for PR 31576 at commit 4fd6630.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants