[SPARK-40851][INFRA ][SQL][TESTS] Make GA run successfully with the latest Java 8/11/17 #38317

LuciferYang · 2022-10-20T09:00:43Z

What changes were proposed in this pull request?

The main change of this pr as follows:

Replace Antarctica/Vostok to Asia/Urumqi in Spark code
Replace Europe/Amsterdam to Europe/Brussels in Spark code
Regenerate gregorian-julian-rebase-micros.json using generate 'gregorian-julian-rebase-micros.json' in RebaseDateTimeSuite with Java 8u352
Regenerate julian-gregorian-rebase-micros.json using generate 'julian-gregorian-rebase-micros.json' in RebaseDateTimeSuite with Java 8u352
Revert change of SPARK-40846

Why are the changes needed?

Make GA run successfully with the latest Java 8/11/17:

Java 8u352
Java 11.0.17
Java 17.0.5

Does this PR introduce any user-facing change?

No

How was this patch tested?

Pass GitHub Actions
Manual test: the following commands can test pass with Java 8u345, 8u352, 11.0.16, 11.0.17, 17.0.4 and 17.0.5
- build/sbt "catalyst/test"
- build/sbt "sql/testOnly *SQLQueryTestSuite -- -t \"timestamp-ltz.sql\""
- build/sbt "sql/testOnly *SQLQueryTestSuite -- -t \"timestamp-ntz.sql\""

LuciferYang · 2022-10-20T09:16:29Z

The replacement of Zone refers to the aliases.txt of Java's own test:

Antarctica/Vostok -> Asia/Urumqi: https://github.com/openjdk/jdk8u-dev/blob/46da2c31efef73b7c2914c8d252554556145b00b/jdk/test/java/util/TimeZone/TimeZoneData/aliases.txt#L40

Europe/Amsterdam -> Europe/Brussels: https://github.com/openjdk/jdk8u-dev/blob/46da2c31efef73b7c2914c8d252554556145b00b/jdk/test/java/util/TimeZone/TimeZoneData/aliases.txt#L68

…pass GA first" This reverts commit 1311976.

LuciferYang · 2022-10-20T09:20:22Z

sql/catalyst/src/main/resources/gregorian-julian-rebase-micros.json

@@ -4,15 +4,15 @@
  "diffs" : [ -173768, -87368, -968, 85432, 171832, 258232, 344632, 431032, 517432, 603832, 690232, 776632, 863032, 776632, 690232, 603832, 517432, 431032, 344632, 258232, 171832, 85432, -968, 0 ]
 }, {
  "tz" : "Africa/Accra",
-  "switches" : [ -62135596748, -59006361548, -55850687948, -52695014348, -46383580748, -43227907148, -40072233548, -33760799948, -30605126348, -27449452748, -21138019148, -17982345548, -14826671948, -12220070348, -12219983948, -12219897548, -12219811148, -12219724748, -12219638348, -12219551948, -12219465548, -12219379148, -12219292748, -2208988800 ],
-  "diffs" : [ -172852, -86452, -52, 86348, 172748, 259148, 345548, 431948, 518348, 604748, 691148, 777548, 863948, 777548, 691148, 604748, 518348, 431948, 345548, 259148, 172748, 86348, -52, 0 ]
+  "switches" : [ -62135595832, -59006360632, -55850687032, -52695013432, -46383579832, -43227906232, -40072232632, -33760799032, -30605125432, -27449451832, -21138018232, -17982344632, -14826671032, -12220069432, -12219983032, -12219896632, -12219810232, -12219723832, -12219637432, -12219551032, -12219464632, -12219378232, -12219291832, -2208988800 ],


Regenerate gregorian-julian-rebase-micros.json using generate 'gregorian-julian-rebase-micros.json' in RebaseDateTimeSuite with Java 8u352

LuciferYang · 2022-10-20T09:21:36Z

sql/catalyst/src/main/resources/julian-gregorian-rebase-micros.json

  "tz" : "Africa/Accra",
  "switches" : [ -62135769600, -59006448000, -55850688000, -52694928000, -46383408000, -43227648000, -40071888000, -33760368000, -30604608000, -27448848000, -21137328000, -17981568000, -14825808000, -12219292800, -2208988800 ],
-  "diffs" : [ 172852, 86452, 52, -86348, -172748, -259148, -345548, -431948, -518348, -604748, -691148, -777548, -863948, 52, 0 ]
+  "diffs" : [ 173768, 87368, 968, -85432, -171832, -258232, -344632, -431032, -517432, -603832, -690232, -776632, -863032, 968, 0 ]


Regenerate julian-gregorian-rebase-micros.json using generate 'julian-gregorian-rebase-micros.json' in RebaseDateTimeSuite with Java 8u352

allisonwang-db · 2022-10-20T20:39:13Z

Thanks for fixing this. This is the cause of the test failures in #38260. cc @cloud-fan

HyukjinKwon · 2022-10-21T00:20:22Z

cc @MaxGekk

MaxGekk · 2022-10-21T04:57:04Z

All GAs passed at least once:

Merging to master. Thank you, @LuciferYang and @cloud-fan @HyukjinKwon for review.

LuciferYang · 2022-10-21T05:00:55Z

Thanks @MaxGekk @cloud-fan @HyukjinKwon and @wangyum

srowen · 2022-10-23T18:34:31Z

@LuciferYang and everyone, we need to port this back to 3.3 and 3.2 right? b/c it will affect those branches, which do not pin Java 8. For example I think this backport is failing with the same issue: https://github.com/bjornjorgensen/spark/actions/runs/3304235420/jobs/5455772005

HyukjinKwon · 2022-10-24T01:59:17Z

sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala

      ("2021-12-05T22:00:00", "Europe/Moscow") -> ("2021-12-06T00:00:00", "Asia/Yekaterinburg"),
      ("2021-12-06T00:01:02.123456", "Asia/Yekaterinburg") ->
-        ("2021-12-05T20:01:02.123456", "Europe/Amsterdam"),
+        ("2021-12-05T20:01:02.123456", "Europe/Brussels"),


@LuciferYang mind creating a cherry-pick to other branches too?

Will do this today @srowen @HyukjinKwon

branch-3.3: [SPARK-40851][INFRA ][SQL][TESTS][3.3] Make GA run successfully with the latest Java 8/11/17 #38363

branch-3.2: [SPARK-40851][INFRA ][SQL][TESTS][3.2] Make GA run successfully with the latest Java 8/11/17 #38365

…atest Java 8/11/17 The main change of this pr as follows: - Replace `Antarctica/Vostok` to `Asia/Urumqi` in Spark code - Replace `Europe/Amsterdam` to `Europe/Brussels` in Spark code - Regenerate `gregorian-julian-rebase-micros.json` using generate 'gregorian-julian-rebase-micros.json' in `RebaseDateTimeSuite` with Java 8u352 - Regenerate `julian-gregorian-rebase-micros.json` using generate 'julian-gregorian-rebase-micros.json' in RebaseDateTimeSuite with Java 8u352 - Revert change of SPARK-40846 Make GA run successfully with the latest Java 8/11/17: - Java 8u352 - Java 11.0.17 - Java 17.0.5 No - Pass GitHub Actions - Manual test: the following commands can test pass with Java 8u345, 8u352, 11.0.16, 11.0.17, 17.0.4 and 17.0.5 - `build/sbt "catalyst/test"` - `build/sbt "sql/testOnly *SQLQueryTestSuite -- -t \"timestamp-ltz.sql\""` - `build/sbt "sql/testOnly *SQLQueryTestSuite -- -t \"timestamp-ntz.sql\""` Closes apache#38317 from LuciferYang/SPARK-40851. Lead-authored-by: yangjie01 <[email protected]> Co-authored-by: YangJie <[email protected]> Signed-off-by: Max Gekk <[email protected]>

…atest Java 8/11/17 ### What changes were proposed in this pull request? The main change of this pr as follows: - Replace `Antarctica/Vostok` to `Asia/Urumqi` in Spark code - Replace `Europe/Amsterdam` to `Europe/Brussels` in Spark code - Regenerate `gregorian-julian-rebase-micros.json` using generate 'gregorian-julian-rebase-micros.json' in `RebaseDateTimeSuite` with Java 8u352 - Regenerate `julian-gregorian-rebase-micros.json` using generate 'julian-gregorian-rebase-micros.json' in RebaseDateTimeSuite with Java 8u352 - Revert change of SPARK-40846 ### Why are the changes needed? Make GA run successfully with the latest Java 8/11/17: - Java 8u352 - Java 11.0.17 - Java 17.0.5 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass GitHub Actions - Manual test: the following commands can test pass with Java 8u345, 8u352, 11.0.16, 11.0.17, 17.0.4 and 17.0.5 - `build/sbt "catalyst/test"` - `build/sbt "sql/testOnly *SQLQueryTestSuite -- -t \"timestamp-ltz.sql\""` - `build/sbt "sql/testOnly *SQLQueryTestSuite -- -t \"timestamp-ntz.sql\""` Closes apache#38317 from LuciferYang/SPARK-40851. Lead-authored-by: yangjie01 <[email protected]> Co-authored-by: YangJie <[email protected]> Signed-off-by: Max Gekk <[email protected]>

Antarctica/Vostok -> Asia/Urumqi

6c26b24

github-actions bot added the SQL label Oct 20, 2022

LuciferYang added 2 commits October 20, 2022 17:02

Europe/Amsterdam -> Europe/Brussels

4f1f8db

update json files

07be340

LuciferYang changed the title ~~[WIP][SPARK-40851][SQL]~~ [WIP][SPARK-40851][SQL] Fix test failed with latest Java 8/11/17 Oct 20, 2022

LuciferYang changed the title ~~[WIP][SPARK-40851][SQL] Fix test failed with latest Java 8/11/17~~ [SPARK-40851][SQL] Fix test failed with latest Java 8/11/17 Oct 20, 2022

Revert "[SPARK-40846][INFRA] Temporarily pin GA used Java version to …

f0b896f

…pass GA first" This reverts commit 1311976.

github-actions bot added the INFRA label Oct 20, 2022

LuciferYang commented Oct 20, 2022

View reviewed changes

LuciferYang changed the title ~~[SPARK-40851][SQL] Fix test failed with latest Java 8/11/17~~ [SPARK-40851][SQL] Support GA run with the latest Java 8/11/17 Oct 20, 2022

LuciferYang changed the title ~~[SPARK-40851][SQL] Support GA run with the latest Java 8/11/17~~ [SPARK-40851][SQL] Make GA run successfully with the latest Java 8/11/17 Oct 20, 2022

LuciferYang changed the title ~~[SPARK-40851][SQL] Make GA run successfully with the latest Java 8/11/17~~ [SPARK-40851][INFRA ][SQL][TESTS] Make GA run successfully with the latest Java 8/11/17 Oct 20, 2022

Merge branch 'apache:master' into SPARK-40851

afe23e6

LuciferYang mentioned this pull request Oct 21, 2022

[SPARK-40800][SQL] Always inline expressions in OptimizeOneRowRelationSubquery #38260

Closed

cloud-fan approved these changes Oct 21, 2022

View reviewed changes

MaxGekk approved these changes Oct 21, 2022

View reviewed changes

MaxGekk closed this in 7ab8b8e Oct 21, 2022

HyukjinKwon reviewed Oct 24, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-40851][INFRA ][SQL][TESTS] Make GA run successfully with the latest Java 8/11/17 #38317

[SPARK-40851][INFRA ][SQL][TESTS] Make GA run successfully with the latest Java 8/11/17 #38317

Uh oh!

LuciferYang commented Oct 20, 2022 •

edited

Loading

Uh oh!

LuciferYang commented Oct 20, 2022 •

edited

Loading

Uh oh!

LuciferYang Oct 20, 2022 •

edited

Loading

Uh oh!

LuciferYang Oct 20, 2022

Uh oh!

allisonwang-db commented Oct 20, 2022

Uh oh!

HyukjinKwon commented Oct 21, 2022

Uh oh!

MaxGekk commented Oct 21, 2022

Uh oh!

LuciferYang commented Oct 21, 2022

Uh oh!

srowen commented Oct 23, 2022

Uh oh!

HyukjinKwon Oct 24, 2022

Uh oh!

LuciferYang Oct 24, 2022

Uh oh!

LuciferYang Oct 24, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

[SPARK-40851][INFRA ][SQL][TESTS] Make GA run successfully with the latest Java 8/11/17 #38317

[SPARK-40851][INFRA ][SQL][TESTS] Make GA run successfully with the latest Java 8/11/17 #38317

Uh oh!

Conversation

LuciferYang commented Oct 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

LuciferYang commented Oct 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LuciferYang Oct 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LuciferYang Oct 20, 2022

Choose a reason for hiding this comment

Uh oh!

allisonwang-db commented Oct 20, 2022

Uh oh!

HyukjinKwon commented Oct 21, 2022

Uh oh!

MaxGekk commented Oct 21, 2022

Uh oh!

LuciferYang commented Oct 21, 2022

Uh oh!

srowen commented Oct 23, 2022

Uh oh!

HyukjinKwon Oct 24, 2022

Choose a reason for hiding this comment

Uh oh!

LuciferYang Oct 24, 2022

Choose a reason for hiding this comment

Uh oh!

LuciferYang Oct 24, 2022

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

LuciferYang commented Oct 20, 2022 •

edited

Loading

LuciferYang commented Oct 20, 2022 •

edited

Loading

LuciferYang Oct 20, 2022 •

edited

Loading