[SPARK-25610][SQL][TEST] Improve execution time of DatasetCacheSuite: cache UDF result correctly #22638
Conversation
…e UDF result correctly
|
cc @gatorsmile |
| test("cache UDF result correctly") { | ||
| val expensiveUDF = udf({x: Int => Thread.sleep(5000); x}) | ||
| val df = spark.range(0, 10).toDF("a").withColumn("b", expensiveUDF($"a")) | ||
| val expensiveUDF = udf({x: Int => Thread.sleep(2000); x}) |
There was a problem hiding this comment.
mmh...since we fail after 2 seconds we may pass this even in case it doesn't work. Shall we put it to 3? or 2500 at least?
There was a problem hiding this comment.
@mgaido91 OK, please correct me on this one. So we insert 2 rows .. i.e two invocation of the UDF amounting to 2 * 2sec = 4 secs of execution. So wouldn't a 2 sec fail time be ok ? Also marco, i did run this 10 times back to back without a problem - fyi.
There was a problem hiding this comment.
well, I do think this will pass 100% times, my concern was that in case of a regression we might fail detecting it. But yes, with the repartition to 1 you're right, I haven't considered it, otherwise they may have run in parallel. So this seems enough.
|
Test build #96980 has finished for PR 22638 at commit
|
gatorsmile
left a comment
gatorsmile
left a comment
There was a problem hiding this comment.
LGTM
Thanks! Merged to master.
|
Thanks a lot @gatorsmile @mgaido91 |
… cache UDF result correctly ## What changes were proposed in this pull request? In this test case, we are verifying that the result of an UDF is cached when the underlying data frame is cached and that the udf is not evaluated again when the cached data frame is used. To reduce the runtime we do : 1) Use a single partition dataframe, so the total execution time of UDF is more deterministic. 2) Cut down the size of the dataframe from 10 to 2. 3) Reduce the sleep time in the UDF from 5secs to 2secs. 4) Reduce the failafter condition from 3 to 2. With the above change, it takes about 4 secs to cache the first dataframe. And subsequent check takes a few hundred milliseconds. The new runtime for 5 consecutive runs of this test is as follows : ``` [info] - cache UDF result correctly (4 seconds, 906 milliseconds) [info] - cache UDF result correctly (4 seconds, 281 milliseconds) [info] - cache UDF result correctly (4 seconds, 288 milliseconds) [info] - cache UDF result correctly (4 seconds, 355 milliseconds) [info] - cache UDF result correctly (4 seconds, 280 milliseconds) ``` ## How was this patch tested? This is s test fix. Closes apache#22638 from dilipbiswal/SPARK-25610. Authored-by: Dilip Biswal <[email protected]> Signed-off-by: gatorsmile <[email protected]>
4 participants
What changes were proposed in this pull request?
In this test case, we are verifying that the result of an UDF is cached when the underlying data frame is cached and that the udf is not evaluated again when the cached data frame is used.
To reduce the runtime we do :
With the above change, it takes about 4 secs to cache the first dataframe. And subsequent check takes a few hundred milliseconds.
The new runtime for 5 consecutive runs of this test is as follows :
How was this patch tested?
This is s test fix.