Skip to content

[SPARK-27707][SQL] Prune unnecessary nested fields from Generate #24637

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
viirya wants to merge 17 commits into from
Closed

[SPARK-27707][SQL] Prune unnecessary nested fields from Generate #24637

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
6cab5ac
Prune unnecessary nested fields from Generate.
viirya May 18, 2019
f036649
Add one more test in ColumnPruningSuite.
viirya May 19, 2019
beb8993
Remove the regression.
viirya May 20, 2019
caea246
Update benchmark results.
viirya May 20, 2019
7a790ff
Use DSL.
viirya May 22, 2019
ef97ffc
Merge remote-tracking branch 'upstream/master' into SPARK-27707
viirya Jun 14, 2019
93b508f
Only apply the pruning when NESTED_SCHEMA_PRUNING_ENABLED is enabled.
viirya Jul 7, 2019
86e795f
Create separate config. Push down nested fields below Generate to help
viirya Jul 8, 2019
454aa9f
Enable SQLConf.NESTED_PRUNING_ON_EXPRESSIONS for new benchmark.
viirya Jul 11, 2019
456e6d2
Fix style.
viirya Jul 11, 2019
f49b282
new result
dongjoon-hyun Jul 11, 2019
a453d8c
Merge pull request #7 from dongjoon-hyun/PR-24637-2
viirya Jul 11, 2019
ff3b472
Merge remote-tracking branch 'upstream/master' into SPARK-27707
viirya Jul 17, 2019
dde0dbd
Add more generator.
viirya Jul 17, 2019
9c225f3
Merge branch 'SPARK-27707' of github.com:viirya/spark-1 into SPARK-27707
viirya Jul 17, 2019
5511445
More generators.
viirya Jul 18, 2019
0821444
Add white-list.
viirya Jul 19, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions ...atalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ object NestedColumnAliasing {
/**
* Return a replaced project list.
*/
private def getNewProjectList(
def getNewProjectList(
projectList: Seq[NamedExpression],
nestedFieldToAlias: Map[GetStructField, Alias]): Seq[NamedExpression] = {
projectList.map(_.transform {
Expand All @@ -77,7 +77,7 @@ object NestedColumnAliasing {
/**
* Returns true for those operators that project can be pushed through.
*/
private def canProjectPushThrough(plan: LogicalPlan) = plan match {
def canProjectPushThrough(plan: LogicalPlan): Boolean = plan match {
case _: GlobalLimit => true
case _: LocalLimit => true
case _: Repartition => true
Expand All @@ -100,7 +100,7 @@ object NestedColumnAliasing {
* 1. GetStructField -> Alias: A new alias is created for each nested field.
* 2. ExprId -> Seq[Alias]: A reference attribute has multiple aliases pointing it.
*/
private def getAliasSubMap(projectList: Seq[NamedExpression])
def getAliasSubMap(projectList: Seq[NamedExpression])
: Option[(Map[GetStructField, Alias], Map[ExprId, Seq[Alias]])] = {
val (nestedFieldReferences, otherRootReferences) =
projectList.flatMap(collectRootReferenceAndGetStructField).partition {
Expand Down
10 changes: 10 additions & 0 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -588,6 +588,16 @@ object ColumnPruning extends Rule[LogicalPlan] {
.map(_._2)
p.copy(child = g.copy(child = newChild, unrequiredChildIndex = unrequiredIndices))

// prune unrequired nested fields
case p @ Project(projectList, g: Generate) =>

@cloud-fan cloud-fan May 21, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why we need to special case Generate? I see there is a general case for nested column pruning below.

@viirya viirya May 21, 2019

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is more general case you are looking for:

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala

Lines 623 to 624 in 163a6e2

case p @ NestedColumnAliasing(nestedFieldToAlias, attrToAliases) =>
NestedColumnAliasing.replaceToAliases(p, nestedFieldToAlias, attrToAliases)

In the general case, it's doing pruning only when the flag (nestedSchemaPruningEnabled) is enabled. The case considers some operators that nested project can be pushed through. Generate isn't one of them. So the general case doesn't work on this case.

@viirya viirya Jun 19, 2019

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generate is special for another reason. We can't prune an output from its child even just a nested field of the output is used in the top project list. The generator could use it.

@dongjoon-hyun dongjoon-hyun Jun 20, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we add if SQLConf.get.nestedSchemaPruningEnabled?

@viirya viirya Jun 20, 2019

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nestedSchemaPruningEnabled is for pruning nested fields from logical relation. But this fix isn't due to the same cause. For the data sources can't be pruned nested fields, it is also useful to apply this fix.

@HyukjinKwon HyukjinKwon Jul 2, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, why don't we need another configuration then if spark.sql.optimizer.nestedSchemaPruning.enabled isn't part of this optimization, or fixing the doc of spark.sql.optimizer.nestedSchemaPruning.enabled?

@viirya viirya Jul 3, 2019

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks a general fix no matter the setting of nestedSchemaPruning is. Do we need a config to disable this?

@HyukjinKwon HyukjinKwon Jul 4, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My impression was that we need a configuration but I think you or @dongjoon-hyun have more context then me about nested pruning stuff. @cloud-fan, @dongjoon-hyun, @gatorsmile, can you make a call here if we need a config or not?

@viirya viirya Jul 6, 2019

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nestedSchemaPruning is vague as it can point to nested pruning in scan, or general nested pruning in other operators. Currently the config affects scan nested pruning. I feel it is not tightly related to the fix here, because the fix isn't for scan.

If we are considering a config here, I think a different config or fix the doc of nestedSchemaPruning to explicitly indicate it also helps general nested pruning.

@viirya viirya Jul 7, 2019

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me fix the doc of nestedSchemaPruning and apply this pruning when nestedSchemaPruning is enabled.

NestedColumnAliasing.getAliasSubMap(projectList).map {
case (nestedFieldToAlias, attrToAliases) =>
val newChild = g.withNewChildren(g.children.map { child =>
Project(child.output ++ attrToAliases.values.flatten, child)

@dongjoon-hyun dongjoon-hyun Jul 7, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks addition instead of a replacing. So, this may optimize explode, but we cannot avoid the full scan of underlying parquet files?

@viirya viirya Jul 8, 2019

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is good point. I was planning to deal this as separate issue. As you mentioned and we control it with a pruning on expressions config now, I think it is good to deal with it here. I will update.

})
Project(NestedColumnAliasing.getNewProjectList(projectList, nestedFieldToAlias), newChild)
}.getOrElse(p)

// Eliminate unneeded attributes from right side of a Left Existence Join.
case j @ Join(_, right, LeftExistence(_), _, _) =>
j.copy(right = prunedChild(right, j.references))
Expand Down
28 changes: 27 additions & 1 deletion sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ColumnPruningSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.plans.{Inner, PlanTest}
import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.catalyst.rules.RuleExecutor
import org.apache.spark.sql.types.StringType
import org.apache.spark.sql.types.{StringType, StructType}

class ColumnPruningSuite extends PlanTest {

Expand Down Expand Up @@ -101,6 +101,32 @@ class ColumnPruningSuite extends PlanTest {
comparePlans(optimized, correctAnswer)
}

test("Nested column pruning for Generate") {
val strcutType = StructType.fromDDL("d string, e array<string>")
val input = LocalRelation('a.int, 'b.int, 'c.struct(strcutType))

val query =
input
.generate(Explode(GetStructField('c, 1, Some("e"))), outputNames = "explode" :: Nil)
.select('a, GetStructField('c, 0, Some("d")), 'explode)
.analyze

val optimized = Optimize.execute(query)

val aliases = NestedColumnAliasingSuite.collectGeneratedAliases(optimized)

val correctAnswer =
input
.select('a, 'c, GetStructField('c, 0, Some("d")).as(aliases(0)))
.generate(Explode(GetStructField('c, 1, Some("e"))),
unrequiredChildIndex = Seq(1),
outputNames = "explode" :: Nil)
.select('a, $"${aliases(0)}".as("c.d"), 'explode)
.analyze

comparePlans(optimized, correctAnswer)
}

test("Column pruning for Project on Sort") {
val input = LocalRelation('a.int, 'b.string, 'c.double)

Expand Down
6 changes: 5 additions & 1 deletion ...st/src/test/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasingSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ import org.apache.spark.sql.catalyst.rules.RuleExecutor
import org.apache.spark.sql.types.{StringType, StructType}

class NestedColumnAliasingSuite extends SchemaPruningTest {
import NestedColumnAliasingSuite._

object Optimize extends RuleExecutor[LogicalPlan] {
val batches = Batch("Nested column pruning", FixedPoint(100),
Expand Down Expand Up @@ -220,8 +221,11 @@ class NestedColumnAliasingSuite extends SchemaPruningTest {

comparePlans(optimized, expected)
}
}


private def collectGeneratedAliases(query: LogicalPlan): ArrayBuffer[String] = {
object NestedColumnAliasingSuite {
Comment thread
dongjoon-hyun marked this conversation as resolved.
def collectGeneratedAliases(query: LogicalPlan): ArrayBuffer[String] = {
val aliases = ArrayBuffer[String]()
query.transformAllExpressions {
case a @ Alias(_, name) if name.startsWith("_gen_alias_") =>
Expand Down
129 changes: 68 additions & 61 deletions sql/core/benchmarks/MiscBenchmark-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,119 +2,126 @@
filter & aggregate without group
================================================================================================

OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
OpenJDK 64-Bit Server VM 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03 on Linux 4.15.0-1021-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
range/filter/sum: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
range/filter/sum wholestage off 47752 / 48952 43.9 22.8 1.0X
range/filter/sum wholestage on 3123 / 3558 671.5 1.5 15.3X
range/filter/sum: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
range/filter/sum wholestage off 46703 47444 1048 44.9 22.3 1.0X
range/filter/sum wholestage on 3109 3506 222 674.5 1.5 15.0X


================================================================================================
range/limit/sum
================================================================================================

OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
OpenJDK 64-Bit Server VM 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03 on Linux 4.15.0-1021-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
range/limit/sum: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
range/limit/sum wholestage off 229 / 236 2288.9 0.4 1.0X
range/limit/sum wholestage on 257 / 267 2041.0 0.5 0.9X
range/limit/sum: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
range/limit/sum wholestage off 191 205 19 2738.4 0.4 1.0X
range/limit/sum wholestage on 112 124 13 4699.4 0.2 1.7X

@dongjoon-hyun dongjoon-hyun Jun 20, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ur, this is irrelevant, but the ratio looks weird.

@dongjoon-hyun dongjoon-hyun Jul 11, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to be improved by some reason. This is consistently better in @viirya and my tests.

[info] OpenJDK 64-Bit Server VM 1.8.0_212-b04 on Linux 3.10.0-862.3.2.el7.x86_64
[info] Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
[info] range/limit/sum:                          Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
[info] ------------------------------------------------------------------------------------------------------------------------
[info] range/limit/sum wholestage off                      222            226           6       2359.5           0.4       1.0X
[info] range/limit/sum wholestage on                       114            121           8       4608.4           0.2       2.0X



================================================================================================
sample
================================================================================================

OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
OpenJDK 64-Bit Server VM 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03 on Linux 4.15.0-1021-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
sample with replacement: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
sample with replacement wholestage off 12908 / 13076 10.2 98.5 1.0X
sample with replacement wholestage on 7334 / 7346 17.9 56.0 1.8X
sample with replacement: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
sample with replacement wholestage off 12545 12789 344 10.4 95.7 1.0X
sample with replacement wholestage on 7666 7687 12 17.1 58.5 1.6X

OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
OpenJDK 64-Bit Server VM 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03 on Linux 4.15.0-1021-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
sample without replacement: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
sample without replacement wholestage off 3082 / 3095 42.5 23.5 1.0X
sample without replacement wholestage on 1125 / 1211 116.5 8.6 2.7X
sample without replacement: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
sample without replacement wholestage off 2972 2976 6 44.1 22.7 1.0X
sample without replacement wholestage on 1091 1098 6 120.1 8.3 2.7X


================================================================================================
collect
================================================================================================

OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
OpenJDK 64-Bit Server VM 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03 on Linux 4.15.0-1021-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
collect: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
collect 1 million 291 / 311 3.6 277.3 1.0X
collect 2 millions 552 / 564 1.9 526.6 0.5X
collect 4 millions 1104 / 1108 0.9 1053.0 0.3X
collect: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
collect 1 million 329 362 41 3.2 314.0 1.0X
collect 2 millions 545 566 19 1.9 519.8 0.6X
collect 4 millions 1199 2117 1299 0.9 1143.3 0.3X


================================================================================================
collect limit
================================================================================================

OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
OpenJDK 64-Bit Server VM 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03 on Linux 4.15.0-1021-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
collect limit: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
collect limit 1 million 311 / 340 3.4 296.2 1.0X
collect limit 2 millions 581 / 614 1.8 554.4 0.5X
collect limit: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
collect limit 1 million 341 348 5 3.1 325.6 1.0X
collect limit 2 millions 654 665 9 1.6 623.8 0.5X


================================================================================================
generate explode
================================================================================================

OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
OpenJDK 64-Bit Server VM 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03 on Linux 4.15.0-1021-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
generate explode array: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
generate explode array wholestage off 15211 / 15368 1.1 906.6 1.0X
generate explode array wholestage on 10761 / 10776 1.6 641.4 1.4X
generate explode array: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
generate explode array wholestage off 15383 15663 395 1.1 916.9 1.0X
generate explode array wholestage on 10593 10638 40 1.6 631.4 1.5X

OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
OpenJDK 64-Bit Server VM 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03 on Linux 4.15.0-1021-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
generate explode map: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
generate explode map wholestage off 22128 / 22578 0.8 1318.9 1.0X
generate explode map wholestage on 16421 / 16520 1.0 978.8 1.3X
generate explode map: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
generate explode map wholestage off 50364 50656 413 0.3 3001.9 1.0X
generate explode map wholestage on 43484 43833 313 0.4 2591.9 1.2X
Comment thread
dongjoon-hyun marked this conversation as resolved.
Outdated

OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
OpenJDK 64-Bit Server VM 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03 on Linux 4.15.0-1021-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
generate posexplode array: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
generate posexplode array wholestage off 17108 / 18019 1.0 1019.7 1.0X
generate posexplode array wholestage on 11715 / 11804 1.4 698.3 1.5X
generate posexplode array: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
generate posexplode array wholestage off 17162 18868 2412 1.0 1023.0 1.0X
generate posexplode array wholestage on 11506 11536 27 1.5 685.8 1.5X

OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
OpenJDK 64-Bit Server VM 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03 on Linux 4.15.0-1021-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
generate inline array: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
generate inline array wholestage off 16358 / 16418 1.0 975.0 1.0X
generate inline array wholestage on 11152 / 11472 1.5 664.7 1.5X
generate inline array: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
generate inline array wholestage off 15148 15181 48 1.1 902.9 1.0X
generate inline array wholestage on 9713 9761 28 1.7 578.9 1.6X

OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
OpenJDK 64-Bit Server VM 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03 on Linux 4.15.0-1021-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
generate big struct array: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
generate big struct array wholestage off 708 / 776 0.1 11803.5 1.0X
generate big struct array wholestage on 535 / 589 0.1 8913.9 1.3X
generate big struct array: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
generate big struct array wholestage off 663 705 59 0.1 11054.4 1.0X
generate big struct array wholestage on 576 612 34 0.1 9602.9 1.2X

OpenJDK 64-Bit Server VM 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03 on Linux 4.15.0-1021-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
generate big nested struct array: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
Comment thread
dongjoon-hyun marked this conversation as resolved.
------------------------------------------------------------------------------------------------------------------------
generate big nested struct array wholestage off 635 683 68 0.1 10582.3 1.0X
generate big nested struct array wholestage on 647 679 38 0.1 10785.4 1.0X

@dongjoon-hyun dongjoon-hyun Jul 10, 2019 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ur, I got a different result on this. I rebased this PR to the master.

-generate big nested struct array wholestage off            635            683          68          0.1       10582.3       1.0X
-generate big nested struct array wholestage on            647            679          38          0.1       10785.4       1.0X
+generate big nested struct array wholestage off          57843          62617        2949          0.0      964042.1       1.0X
+generate big nested struct array wholestage on          49793          53627         NaN          0.0      829888.7       1.2X

@dongjoon-hyun dongjoon-hyun Jul 10, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll make a PR to you for comparison.



================================================================================================
generate regular generator
================================================================================================

OpenJDK 64-Bit Server VM 1.8.0_181-b13 on Linux 3.10.0-862.3.2.el7.x86_64
OpenJDK 64-Bit Server VM 1.8.0_212-8u212-b03-0ubuntu1.18.04.1-b03 on Linux 4.15.0-1021-aws
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
generate stack: Best/Avg Time(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------
generate stack wholestage off 29082 / 29393 0.6 1733.4 1.0X
generate stack wholestage on 21066 / 21128 0.8 1255.6 1.4X
generate stack: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
------------------------------------------------------------------------------------------------------------------------
generate stack wholestage off 27872 27987 164 0.6 1661.3 1.0X
generate stack wholestage on 18665 18710 35 0.9 1112.5 1.5X


Loading