[SPARK-16526][SQL] Benchmarking Performance for Fast HashMap Implementations #14266
Conversation
ooq
changed the title
|
Test build #62539 has finished for PR 14266 at commit
|
|
Test build #62541 has finished for PR 14266 at commit
|
ooq
force-pushed
the
rowbasedfastaggmap-pr3
branch
from
c984914 to
c2b276f
Compare
ooq
changed the title
|
Test build #62945 has finished for PR 14266 at commit
|
|
Test build #62947 has finished for PR 14266 at commit
|
| sparkSession.conf.set("spark.sql.codegen.aggregate.map.columns.max", "30") | ||
|
|
||
| // scalastyle:off | ||
| println(Benchmark.getJVMOSInfo()) |
There was a problem hiding this comment.
nit: to minimize duplication, maybe create a small utility function that can then be reused in all test cases.
|
Just a minor nit. LGTM. |
| */ | ||
| } | ||
|
|
||
| ignore("varying key fields, varying value field, 16 linear distinct keys") { |
There was a problem hiding this comment.
The performance difference between column-based and row-based are cache locality, could you increase the number of distinct keys to make sure that not all the keys/values are fit in L1 cache? for example, 4k. We could also increase that to 64k in first two cases (single key, single value).
|
@davies Added some test results with larger number of distinct keys. |
|
Test build #63296 has finished for PR 14266 at commit
|
|
Test build #63299 has finished for PR 14266 at commit
|
| */ | ||
| } | ||
|
|
||
| ignore("single key field, single value field, varying linear distinct keys") { |
There was a problem hiding this comment.
Should we access them in random way?
|
(gentle ping @ooq) |
## What changes were proposed in this pull request? This PR proposes to close stale PRs. What I mean by "stale" here includes that there are some review comments by reviewers but the author looks inactive without any answer to them more than a month. I left some comments roughly a week ago to ping and the author looks still inactive in these PR below These below includes some PR suggested to be closed and a PR against another branch which seems obviously inappropriate. Given the comments in the last three PRs below, they are probably worth being taken over by anyone who is interested in it. Closes apache#7963 Closes apache#8374 Closes apache#11192 Closes apache#11374 Closes apache#11692 Closes apache#12243 Closes apache#12583 Closes apache#12620 Closes apache#12675 Closes apache#12697 Closes apache#12800 Closes apache#13715 Closes apache#14266 Closes apache#15053 Closes apache#15159 Closes apache#15209 Closes apache#15264 Closes apache#15267 Closes apache#15871 Closes apache#15861 Closes apache#16319 Closes apache#16324 Closes apache#16890 Closes apache#12398 Closes apache#12933 Closes apache#14517 ## How was this patch tested? N/A Author: hyukjinkwon <[email protected]> Closes apache#16937 from HyukjinKwon/stale-prs-close.
5 participants
What changes were proposed in this pull request?
The 3rd PR in its series to resolve SPARK-16523.
This patch adds benchmark tests for vectorized hashmap vs. row-based hashmap (along with results in the comments). Those tests are ignored by default as they take long to run.
How was this patch tested?
This patch are mostly tests itself.