Skip to content

Conversation

@srowen
Copy link
Member

@srowen srowen commented Oct 8, 2018

What changes were proposed in this pull request?

Try testing timezones in parallel instead in CastSuite, instead of random sampling.
See also #22631

How was this patch tested?

Existing test.

@SparkQA
Copy link

SparkQA commented Oct 8, 2018

Test build #97113 has finished for PR 22672 at commit 4565859.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@srowen
Copy link
Member Author

srowen commented Oct 8, 2018

The original change in #22631 made the test time go down from about 2:30 to 0:17. See build 96945:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96945/testReport/junit/org.apache.spark.sql.catalyst.expressions/CastSuite/cast_string_to_timestamp/history/?start=175

This change clocks in at 0:24 seconds instead of 0:17:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/97113/testReport/org.apache.spark.sql.catalyst.expressions/CastSuite/cast_string_to_timestamp/history/

That's still a pretty big win, and keeps all tests. I'd argue this is a step forward. However we could further, I'm sure, trim the number of cases it tests if desired. This could be a case where we do want to be pretty exhaustive though?

And: I suspect there are a lot of other tests we could speed up this way, that loop over some local bit of test code and could easily be run in parallel with a small change like this. It's the Spark-based jobs that are hard to run concurrently.

Copy link
Member

@felixcheung felixcheung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

like this

@asfgit asfgit closed this in eaafcd8 Oct 10, 2018
@srowen srowen deleted the SPARK-25605.2 branch October 10, 2018 22:30
jackylee-ch pushed a commit to jackylee-ch/spark that referenced this pull request Feb 18, 2019
…ts for a subset of timezones

## What changes were proposed in this pull request?

Try testing timezones in parallel instead in CastSuite, instead of random sampling.
See also apache#22631

## How was this patch tested?

Existing test.

Closes apache#22672 from srowen/SPARK-25605.2.

Authored-by: Sean Owen <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants