Skip to content

[SPARK-29533][SQL][TEST] Benchmark casting strings to intervals #26189

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MaxGekk wants to merge 6 commits into from
Closed

[SPARK-29533][SQL][TEST] Benchmark casting strings to intervals #26189

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
66b4288
Add IntervalBenchmark
MaxGekk Oct 21, 2019
4eaae97
Generate results
MaxGekk Oct 21, 2019
b35439a
Use string interpolation
MaxGekk Oct 21, 2019
836eb9f
Use Scala API
MaxGekk Oct 21, 2019
165ee36
Generate results for jdk11
MaxGekk Oct 21, 2019
1772543
remove s"
MaxGekk Oct 21, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions sql/core/benchmarks/IntervalBenchmark-jdk11-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
OpenJDK 64-Bit Server VM 11.0.2+9 on Mac OS X 10.15
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
cast strings to intervals: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
string w/ interval 471 513 57 2.1 470.7 1.0X
string w/o interval 437 444 8 2.3 436.9 1.1X
1 units w/ interval 726 758 45 1.4 726.3 0.6X
1 units w/o interval 712 717 5 1.4 711.7 0.7X
2 units w/ interval 926 935 12 1.1 925.9 0.5X
2 units w/o interval 943 947 3 1.1 943.4 0.5X
3 units w/ interval 1089 1116 31 0.9 1089.0 0.4X
3 units w/o interval 1105 1108 3 0.9 1105.1 0.4X
4 units w/ interval 1260 1261 1 0.8 1260.4 0.4X
4 units w/o interval 1276 1277 1 0.8 1275.9 0.4X
5 units w/ interval 1436 1445 11 0.7 1435.6 0.3X
5 units w/o interval 1455 1463 6 0.7 1455.5 0.3X
6 units w/ interval 1634 1639 4 0.6 1634.4 0.3X
6 units w/o interval 1642 1644 3 0.6 1641.7 0.3X
7 units w/ interval 1829 1838 8 0.5 1828.6 0.3X
7 units w/o interval 1850 1853 4 0.5 1849.5 0.3X
8 units w/ interval 2065 2070 5 0.5 2065.4 0.2X
8 units w/o interval 2070 2090 21 0.5 2070.0 0.2X
9 units w/ interval 2279 2290 10 0.4 2278.7 0.2X
9 units w/o interval 2276 2285 8 0.4 2275.7 0.2X

25 changes: 25 additions & 0 deletions sql/core/benchmarks/IntervalBenchmark-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
Java HotSpot(TM) 64-Bit Server VM 1.8.0_202-b08 on Mac OS X 10.15
Intel(R) Core(TM) i7-4850HQ CPU @ 2.30GHz
cast strings to intervals: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
string w/ interval 420 435 18 2.4 419.8 1.0X
string w/o interval 359 365 10 2.8 358.7 1.2X
1 units w/ interval 752 759 8 1.3 752.0 0.6X
1 units w/o interval 762 766 4 1.3 762.0 0.6X
2 units w/ interval 961 970 8 1.0 960.7 0.4X
2 units w/o interval 970 976 9 1.0 970.2 0.4X
3 units w/ interval 1130 1136 7 0.9 1130.4 0.4X
3 units w/o interval 1150 1158 9 0.9 1150.3 0.4X
4 units w/ interval 1333 1336 3 0.7 1333.5 0.3X
4 units w/o interval 1354 1359 4 0.7 1354.5 0.3X
5 units w/ interval 1523 1525 2 0.7 1523.3 0.3X
5 units w/o interval 1549 1551 3 0.6 1549.4 0.3X
6 units w/ interval 1661 1663 2 0.6 1660.8 0.3X
6 units w/o interval 1691 1704 13 0.6 1691.2 0.2X
7 units w/ interval 1811 1817 8 0.6 1810.6 0.2X
7 units w/o interval 1853 1854 1 0.5 1853.2 0.2X
8 units w/ interval 2029 2037 8 0.5 2028.7 0.2X
8 units w/o interval 2075 2075 1 0.5 2074.5 0.2X
9 units w/ interval 2170 2175 5 0.5 2170.0 0.2X
9 units w/o interval 2204 2212 8 0.5 2203.6 0.2X

102 changes: 102 additions & 0 deletions sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/IntervalBenchmark.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.spark.sql.execution.benchmark

import scala.collection.mutable.ListBuffer

import org.apache.spark.benchmark.Benchmark
import org.apache.spark.sql.Column
import org.apache.spark.sql.SaveMode.Overwrite
import org.apache.spark.sql.functions._
import org.apache.spark.sql.internal.SQLConf

/**
* Synthetic benchmark for interval functions.
* To run this benchmark:
* {{{
* 1. without sbt:
* bin/spark-submit --class <this class> --jars <spark core test jar> <sql core test jar>
* 2. build/sbt "sql/test:runMain <this class>"
* 3. generate result:
* SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt "sql/test:runMain <this class>"
* Results will be written to "benchmarks/IntervalBenchmark-results.txt".
* }}}
*/
object IntervalBenchmark extends SqlBasedBenchmark {
import spark.implicits._

private def doBenchmark(cardinality: Long, exprs: Column*): Unit = {
withSQLConf(SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> "true") {
spark
.range(0, cardinality, 1, 1)
.select(exprs: _*)
.write
.format("noop")
.mode(Overwrite)
.save()
}
}

private def addCase(
benchmark: Benchmark,
cardinality: Long,
name: String,
exprs: Column*): Unit = {
benchmark.addCase(name, numIters = 3) { _ =>
doBenchmark(cardinality, exprs: _*)
}
}

private def buildString(withPrefix: Boolean, units: Seq[String] = Seq.empty): Column = {
val init = lit(if (withPrefix) "interval" else "") ::
($"id" % 10000).cast("string") ::
lit("years") :: Nil

concat_ws(" ", (init ++ units.map(lit)): _*)
Copy link
Member

@dongjoon-hyun dongjoon-hyun Oct 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we avoid concat_ws cost in order to focus interval benchmark more? Due to id column, this seems to be not a foldable expression.

Copy link
Member

@dongjoon-hyun dongjoon-hyun Oct 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops. Sorry, this is already mentioned comment.

Copy link
Member

@dongjoon-hyun dongjoon-hyun Oct 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add more description about how the result should be interpreted?
It's not clear that the first two results should be subtracted from the other results.

}

private def addCase(benchmark: Benchmark, cardinality: Long, units: Seq[String]): Unit = {
Seq(true, false).foreach { withPrefix =>
val expr = buildString(withPrefix, units).cast("interval")
val note = if (withPrefix) "w/ interval" else "w/o interval"
benchmark.addCase(s"${units.length + 1} units $note", numIters = 3) { _ =>
doBenchmark(cardinality, expr)
}
}
}

override def runBenchmarkSuite(mainArgs: Array[String]): Unit = {
val N = 1000000
val timeUnits = Seq(
"13 months", "100 weeks", "9 days", "12 hours",
"5 minutes", "45 seconds", "123 milliseconds", "567 microseconds")
val intervalToTest = ListBuffer[String]()

val benchmark = new Benchmark("cast strings to intervals", N, output = output)
addCase(benchmark, N, "string w/ interval", buildString(true, timeUnits))
addCase(benchmark, N, "string w/o interval", buildString(false, timeUnits))
addCase(benchmark, N, intervalToTest) // Only years

for (unit <- timeUnits) {
intervalToTest.append(unit)
addCase(benchmark, N, intervalToTest)
}

benchmark.run()
}
}