[SPARK-17556] [CORE] [SQL] Executor side broadcast for broadcast joins #15240

scwf · 2016-09-26T11:44:10Z

What changes were proposed in this pull request?

JIRA: https://issues.apache.org/jira/browse/SPARK-17556
Design doc :
https://issues.apache.org/jira/secure/attachment/12830668/executor%20broadcast.pdf

Added two api for RDD to perform executor side broadcast and apply it on sql's broadcast join.

[1]. def broadcast[U: ClassTag](f: Iterator[T] => U): Broadcast[U]

User only need pass a function to describe how to translate all the element of the rdd to the value they want to broadcast

[2]. def broadcast[U: ClassTag](transFunc: TransFunc[T, U]): Broadcast[U]

This is only used in spark sql(spark internal), TransFunc is a interface to describe how to translate all the element of the rdd to a single value.
TransFunc is inherited by BroadcastMode in spark sql.

When construct broadcast, firstly it write blocks to block manager from executor and then create Broadcast from driver(not write blocks)

How was this patch tested?

added unit test and manual test.

SparkQA · 2016-09-26T11:49:04Z

Test build #65907 has finished for PR 15240 at commit 89336e7.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-09-26T12:18:01Z

Test build #65909 has finished for PR 15240 at commit f0a26c3.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-09-26T14:10:49Z

Test build #65913 has finished for PR 15240 at commit 9a334df.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-09-26T14:28:30Z

Test build #65916 has finished for PR 15240 at commit 90217c0.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-09-26T17:02:35Z

Test build #65914 has finished for PR 15240 at commit 5e2ee76.

This patch fails from timeout after a configured wait of 250m.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-09-26T17:03:03Z

Test build #65918 has finished for PR 15240 at commit 5db6a3c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-09-26T19:07:38Z

Test build #65925 has finished for PR 15240 at commit 4714cae.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-09-28T10:38:58Z

Test build #66030 has finished for PR 15240 at commit 0b80837.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-09-29T08:40:24Z

Test build #66088 has finished for PR 15240 at commit 3e77821.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-09-29T11:59:02Z

Test build #66095 has finished for PR 15240 at commit 147b053.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

scwf · 2016-10-08T00:57:41Z

/cc @rxin can you help review this?

SparkQA · 2016-12-30T11:15:49Z

Test build #70745 has finished for PR 15240 at commit cdab885.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-01-03T07:17:43Z

Test build #70804 has started for PR 15240 at commit cdab885.

SparkQA · 2017-01-03T10:15:48Z

Test build #70807 has finished for PR 15240 at commit 26b03b2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-01-03T14:44:50Z

Test build #70818 has finished for PR 15240 at commit 26b03b2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

scwf · 2017-01-03T14:49:23Z

retest this please

This reverts commit 76dfc20.

…adcast

SparkQA · 2017-01-04T12:20:41Z

Test build #70867 has finished for PR 15240 at commit 4a547a6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

jiangxb1987 · 2017-06-13T02:57:56Z

Are you still working on this? @scwf

viirya · 2017-06-13T04:41:21Z

@jiangxb1987 there's another solution to this JIRA at #15178. We take different approaches. You can also take a look.

## What changes were proposed in this pull request? This PR proposes to close stale PRs, mostly the same instances with apache#18017 Closes apache#14085 - [SPARK-16408][SQL] SparkSQL Added file get Exception: is a directory … Closes apache#14239 - [SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism to accelerate shuffle stage. Closes apache#14567 - [SPARK-16992][PYSPARK] Python Pep8 formatting and import reorganisation Closes apache#14579 - [SPARK-16921][PYSPARK] RDD/DataFrame persist()/cache() should return Python context managers Closes apache#14601 - [SPARK-13979][Core] Killed executor is re spawned without AWS key… Closes apache#14830 - [SPARK-16992][PYSPARK][DOCS] import sort and autopep8 on Pyspark examples Closes apache#14963 - [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in lint-python Closes apache#15227 - [SPARK-17655][SQL]Remove unused variables declarations and definations in a WholeStageCodeGened stage Closes apache#15240 - [SPARK-17556] [CORE] [SQL] Executor side broadcast for broadcast joins Closes apache#15405 - [SPARK-15917][CORE] Added support for number of executors in Standalone [WIP] Closes apache#16099 - [SPARK-18665][SQL] set statement state to "ERROR" after user cancel job Closes apache#16445 - [SPARK-19043][SQL]Make SparkSQLSessionManager more configurable Closes apache#16618 - [SPARK-14409][ML][WIP] Add RankingEvaluator Closes apache#16766 - [SPARK-19426][SQL] Custom coalesce for Dataset Closes apache#16832 - [SPARK-19490][SQL] ignore case sensitivity when filtering hive partition columns Closes apache#17052 - [SPARK-19690][SS] Join a streaming DataFrame with a batch DataFrame which has an aggregation may not work Closes apache#17267 - [SPARK-19926][PYSPARK] Make pyspark exception more user-friendly Closes apache#17371 - [SPARK-19903][PYSPARK][SS] window operator miss the `watermark` metadata of time column Closes apache#17401 - [SPARK-18364][YARN] Expose metrics for YarnShuffleService Closes apache#17519 - [SPARK-15352][Doc] follow-up: add configuration docs for topology-aware block replication Closes apache#17530 - [SPARK-5158] Access kerberized HDFS from Spark standalone Closes apache#17854 - [SPARK-20564][Deploy] Reduce massive executor failures when executor count is large (>2000) Closes apache#17979 - [SPARK-19320][MESOS][WIP]allow specifying a hard limit on number of gpus required in each spark executor when running on mesos Closes apache#18127 - [SPARK-6628][SQL][Branch-2.1] Fix ClassCastException when executing sql statement 'insert into' on hbase table Closes apache#18236 - [SPARK-21015] Check field name is not null and empty in GenericRowWit… Closes apache#18269 - [SPARK-21056][SQL] Use at most one spark job to list files in InMemoryFileIndex Closes apache#18328 - [SPARK-21121][SQL] Support changing storage level via the spark.sql.inMemoryColumnarStorage.level variable Closes apache#18354 - [SPARK-18016][SQL][CATALYST][BRANCH-2.1] Code Generation: Constant Pool Limit - Class Splitting Closes apache#18383 - [SPARK-21167][SS] Set kafka clientId while fetch messages Closes apache#18414 - [SPARK-21169] [core] Make sure to update application status to RUNNING if executors are accepted and RUNNING after recovery Closes apache#18432 - resolve com.esotericsoftware.kryo.KryoException Closes apache#18490 - [SPARK-21269][Core][WIP] Fix FetchFailedException when enable maxReqSizeShuffleToMem and KryoSerializer Closes apache#18585 - SPARK-21359 Closes apache#18609 - Spark SQL merge small files to big files Update InsertIntoHiveTable.scala Added: Closes apache#18308 - [SPARK-21099][Spark Core] INFO Log Message Using Incorrect Executor I… Closes apache#18599 - [SPARK-21372] spark writes one log file even I set the number of spark_rotate_log to 0 Closes apache#18619 - [SPARK-21397][BUILD]Maven shade plugin adding dependency-reduced-pom.xml to … Closes apache#18667 - Fix the simpleString used in error messages Closes apache#18782 - Branch 2.1 Added: Closes apache#17694 - [SPARK-12717][PYSPARK] Resolving race condition with pyspark broadcasts when using multiple threads Added: Closes apache#16456 - [SPARK-18994] clean up the local directories for application in future by annother thread Closes apache#18683 - [SPARK-21474][CORE] Make number of parallel fetches from a reducer configurable Closes apache#18690 - [SPARK-21334][CORE] Add metrics reporting service to External Shuffle Server Added: Closes apache#18827 - Merge pull request 1 from apache/master ## How was this patch tested? N/A Author: hyukjinkwon <[email protected]> Closes apache#18780 from HyukjinKwon/close-prs.

scwf force-pushed the SPARK-17556 branch from 4714cae to 0b80837 Compare September 28, 2016 08:12

scwf changed the title ~~[SPARK-17556] Executor side broadcast for broadcast joins~~ [SPARK-17556] [CORE] [SQL] Executor side broadcast for broadcast joins Sep 29, 2016

scwf and others added 9 commits December 30, 2016 15:14

executor-side broadcast

0b0a252

renaming better

31eaefb

add finally

9dbdd8c

improve the rdd broadcast api

55c2b8e

rename improvement

8347ad6

var => val

a8c7a54

add shutdown hook to delete dir

41110f4

fix dagschedulersuite

1bb8a44

improve and fix compile error

cdab885

scwf force-pushed the SPARK-17556 branch from 147b053 to cdab885 Compare December 30, 2016 09:09

checkSum leads to NPE

26b03b2

clean check sum

76dfc20

scwf added 2 commits January 4, 2017 16:38

Revert "clean check sum"

482c0c8

This reverts commit 76dfc20.

added transient for obj, option leads to obj as a field of TorrentBro…

4a547a6

…adcast

HyukjinKwon mentioned this pull request Jul 31, 2017

[INFRA] Close stale PRs #18780

Closed

asfgit closed this in 3a45c7f Aug 5, 2017

[SPARK-17556] [CORE] [SQL] Executor side broadcast for broadcast joins #15240

[SPARK-17556] [CORE] [SQL] Executor side broadcast for broadcast joins #15240

Uh oh!

Conversation

scwf commented Sep 26, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Sep 26, 2016

Uh oh!

SparkQA commented Sep 26, 2016

Uh oh!

SparkQA commented Sep 26, 2016

Uh oh!

SparkQA commented Sep 26, 2016

Uh oh!

SparkQA commented Sep 26, 2016

Uh oh!

SparkQA commented Sep 26, 2016

Uh oh!

SparkQA commented Sep 26, 2016

Uh oh!

SparkQA commented Sep 28, 2016

Uh oh!

SparkQA commented Sep 29, 2016

Uh oh!

SparkQA commented Sep 29, 2016

Uh oh!

scwf commented Oct 8, 2016

Uh oh!

SparkQA commented Dec 30, 2016

Uh oh!

SparkQA commented Jan 3, 2017

Uh oh!

SparkQA commented Jan 3, 2017

Uh oh!

SparkQA commented Jan 3, 2017

Uh oh!

scwf commented Jan 3, 2017

Uh oh!

SparkQA commented Jan 4, 2017

Uh oh!

jiangxb1987 commented Jun 13, 2017

Uh oh!

viirya commented Jun 13, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

scwf commented Sep 26, 2016 •

edited

Loading

viirya commented Jun 13, 2017 •

edited

Loading