Skip to content

[SPARK-35345][SQL] Add Parquet tests to BloomFilterBenchmark #34594

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
huaxingao wants to merge 3 commits into from
Closed

[SPARK-35345][SQL] Add Parquet tests to BloomFilterBenchmark #34594

huaxingao wants to merge 3 commits into from

Conversation

@huaxingao
Copy link
Contributor

@huaxingao huaxingao commented Nov 14, 2021

What changes were proposed in this pull request?

Add BloomFilter Benchmark test for Parquet

Why are the changes needed?

Currently, we only have BloomFilter Benchmark test for ORC. Will add one for Parquet too.

Does this PR introduce any user-facing change?

no

How was this patch tested?

new test

@github-actions github-actions bot added the SQL label Nov 14, 2021
@huaxingao huaxingao mentioned this pull request Nov 14, 2021
Closed
@HyukjinKwon HyukjinKwon marked this pull request as draft November 15, 2021 00:32
@SparkQA
Copy link

SparkQA commented Nov 15, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49682/

@SparkQA
Copy link

SparkQA commented Nov 15, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49682/

@SparkQA
Copy link

SparkQA commented Nov 15, 2021

Test build #145213 has finished for PR 34594 at commit 3688e50.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 17, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49765/

@HyukjinKwon HyukjinKwon marked this pull request as ready for review November 17, 2021 01:21
HyukjinKwon
HyukjinKwon reviewed Nov 17, 2021
View reviewed changes
sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/BloomFilterBenchmark.scala Outdated
.option("parquet.block.size", blocksize)
.parquet(path + "/withBF")

runBenchmark(s"Parquet Read") {
Copy link
Member

@HyukjinKwon HyukjinKwon Nov 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
runBenchmark(s"Parquet Read") {
runBenchmark("Parquet Read") {

sql/core/src/test/scala/org/apache/spark/sql/execution/benchmark/BloomFilterBenchmark.scala Outdated
withTempPath { dir =>
val path = dir.getCanonicalPath

runBenchmark(s"Parquet Write") {
Copy link
Member

@HyukjinKwon HyukjinKwon Nov 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
runBenchmark(s"Parquet Write") {
runBenchmark("Parquet Write") {

@SparkQA
Copy link

SparkQA commented Nov 17, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49765/

@HyukjinKwon
Copy link
Member

HyukjinKwon commented Nov 17, 2021

Merged to master.

@huaxingao
Copy link
Contributor Author

huaxingao commented Nov 17, 2021

Thank you Hyukjin!

@huaxingao huaxingao deleted the bloomfilter_bm branch November 17, 2021 03:38
@SparkQA
Copy link

SparkQA commented Nov 17, 2021

Test build #145295 has finished for PR 34594 at commit dd9cbb4.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 17, 2021

Test build #145306 has finished for PR 34594 at commit bac368c.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@HyukjinKwon HyukjinKwon HyukjinKwon approved these changes

Assignees

No one assigned

Labels

SQL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@huaxingao @SparkQA @HyukjinKwon