Skip to content

Conversation

@MaxGekk
Copy link
Member

@MaxGekk MaxGekk commented Jun 14, 2020

What changes were proposed in this pull request?

  1. Change the max step from 1 week to 30 minutes in the tests RebaseDateTimeSuite.generate 'gregorian-julian-rebase-micros.json' and generate 'julian-gregorian-rebase-micros.json'.
  2. Parallelise JSON files generation in the function generateRebaseJson by using ThreadUtils.parmap.

Why are the changes needed?

  1. To prevent the bugs that are fixed by [SPARK-31959][SQL][test-java11] Fix Gregorian-Julian micros rebasing while switching standard time zone offset #28787 and [SPARK-31986][SQL] Fix Julian-Gregorian micros rebasing of overlapping local timestamps #28816.
  2. The parallelisation speeds up JSON file generation.

Does this PR introduce any user-facing change?

Yes

How was this patch tested?

By generating the JSON file julian-gregorian-rebase-micros.json.

@MaxGekk
Copy link
Member Author

MaxGekk commented Jun 14, 2020

@cloud-fan @HyukjinKwon After merging this bug fix #28816 and make the rebase functions pure by #28824, the JSON file will have no diffs.

@SparkQA
Copy link

SparkQA commented Jun 14, 2020

Test build #124010 has finished for PR 28827 at commit 900fe2c.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@MaxGekk
Copy link
Member Author

MaxGekk commented Jun 16, 2020

@cloud-fan @HyukjinKwon I re-generated JSON files on my local laptop. The JSON files are the same as the committed files to master currently.

@SparkQA
Copy link

SparkQA commented Jun 16, 2020

Test build #124133 has finished for PR 28827 at commit 69fd414.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon
Copy link
Member

Merged to master.

@MaxGekk MaxGekk deleted the rebase-30-min branch December 11, 2020 20:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants