Skip to content

Update #28

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
YanTangZhai merged 46 commits into from
Jan 24, 2015
Merged

Update #28

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
851b6a9
SPARK-5217 Spark UI should report pending stages during job execution…
ScrapCodes Jan 19, 2015
3453d57
[SPARK-3288] All fields in TaskMetrics should be private and use gett…
Jan 19, 2015
4a4f9cc
[SPARK-5088] Use spark-class for running executors directly
jongyoul Jan 19, 2015
1ac1c1d
MAINTENANCE: Automated closing of pull requests.
pwendell Jan 19, 2015
4432568
[SPARK-5282][mllib]: RowMatrix easily gets int overflow in the memory…
hhbyyh Jan 19, 2015
cd5da42
[SPARK-5284][SQL] Insert into Hive throws NPE when a inner complex ty…
yhuai Jan 19, 2015
2604bc3
[SPARK-5286][SQL] Fail to drop an invalid table when using the data s…
yhuai Jan 19, 2015
74de94e
[SPARK-4504][Examples] fix run-example failure if multiple assembly j…
gvramana Jan 19, 2015
e69fb8c
[SPARK-5214][Core] Add EventLoop and change DAGScheduler to an EventLoop
zsxwing Jan 20, 2015
306ff18
SPARK-5270 [CORE] Provide isEmpty() function in RDD API
srowen Jan 20, 2015
debc031
[SQL][minor] Add a log4j file for catalyst test.
rxin Jan 20, 2015
4afad9c
[SPARK-4803] [streaming] Remove duplicate RegisterReceiver message
ilayaperumalg Jan 20, 2015
9d9294a
[SPARK-5333][Mesos] MesosTaskLaunchData occurs BufferUnderflowException
jongyoul Jan 20, 2015
8140802
[SQL][Minor] Refactors deeply nested FP style code in BooleanSimplifi…
liancheng Jan 20, 2015
c93a57f
SPARK-4660: Use correct class loader in JavaSerializer (copy of PR #3…
jacek-lewandowski Jan 20, 2015
769aced
[SPARK-5329][WebUI] UIWorkloadGenerator should stop SparkContext.
sarutak Jan 20, 2015
23e2554
SPARK-5019 [MLlib] - GaussianMixtureModel exposes instances of Multiv…
tgaloppo Jan 20, 2015
bc20a52
[SPARK-5287][SQL] Add defaultSizeOf to every data type.
yhuai Jan 20, 2015
d181c2a
[SPARK-5323][SQL] Remove Row's Seq inheritance.
rxin Jan 20, 2015
2f82c84
[SPARK-5186] [MLLIB] Vector.equals and Vector.hashCode are very inef…
hhbyyh Jan 20, 2015
9a151ce
[SPARK-5294][WebUI] Hide tables in AllStagePages for "Active Stages, …
sarutak Jan 21, 2015
bad6c57
[SPARK-5275] [Streaming] include python source code
Jan 21, 2015
ec5b0f2
[HOTFIX] Update pom.xml to pull MapR's Hadoop version 2.4.1.
rkannan82 Jan 21, 2015
424d8c6
[SPARK-5297][Streaming] Fix Java file stream type erasure problem
jerryshao Jan 21, 2015
8c06a5f
[SPARK-5336][YARN]spark.executor.cores must not be less than spark.ta…
WangTaoTheTonic Jan 21, 2015
2eeada3
SPARK-1714. Take advantage of AMRMClient APIs to simplify logic in Ya…
sryza Jan 21, 2015
aa1e22b
[MLlib] [SPARK-5301] Missing conversions and operations on IndexedRow…
Jan 21, 2015
7450a99
[SPARK-4749] [mllib]: Allow initializing KMeans clusters using a seed
str-janus Jan 21, 2015
3ee3ab5
[SPARK-5064][GraphX] Add numEdges upperbound validation for R-MAT gra…
Jan 21, 2015
812d367
[SPARK-5244] [SQL] add coalesce() in sql parser
adrian-wang Jan 21, 2015
8361078
[SPARK-5009] [SQL] Long keyword support in SQL Parsers
chenghao-intel Jan 21, 2015
b328ac6
Revert "[SPARK-5244] [SQL] add coalesce() in sql parser"
JoshRosen Jan 21, 2015
ba19689
[SQL] [Minor] Remove deprecated parquet tests
liancheng Jan 21, 2015
3be2a88
[SPARK-4984][CORE][WEBUI] Adding a pop-up containing the full job des…
scwf Jan 21, 2015
9bad062
[SPARK-5355] make SparkConf thread-safe
Jan 22, 2015
27bccc5
[SPARK-5202] [SQL] Add hql variable substitution support
chenghao-intel Jan 22, 2015
ca7910d
[SPARK-3424][MLLIB] cache point distances during k-means|| init
mengxr Jan 22, 2015
fcb3e18
[SPARK-5317]Set BoostingStrategy.defaultParams With Enumeration Algo.…
Peishen-Jia Jan 22, 2015
3027f06
[SPARK-5147][Streaming] Delete the received data WAL log periodically
tdas Jan 22, 2015
246111d
[SPARK-5365][MLlib] Refactor KMeans to reduce redundant data
viirya Jan 22, 2015
820ce03
SPARK-5370. [YARN] Remove some unnecessary synchronization in YarnAll…
sryza Jan 22, 2015
3c3fa63
[SPARK-5233][Streaming] Fix error replaying of WAL introduced bug
jerryshao Jan 23, 2015
e0f7fb7
[SPARK-5315][Streaming] Fix reduceByWindow Java API not work bug
jerryshao Jan 23, 2015
ea74365
[SPARK-3541][MLLIB] New ALS implementation with improved storage
mengxr Jan 23, 2015
cef1f09
[SPARK-5063] More helpful error messages for several invalid operations
JoshRosen Jan 24, 2015
e224dbb
[SPARK-5351][GraphX] Do not use Partitioner.defaultPartitioner as a p…
maropu Jan 24, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 15 additions & 12 deletions bin/compute-classpath.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,22 +72,25 @@ else
assembly_folder="$ASSEMBLY_DIR"
fi

num_jars="$(ls "$assembly_folder" | grep "spark-assembly.*hadoop.*\.jar$" | wc -l)"
if [ "$num_jars" -eq "0" ]; then
echo "Failed to find Spark assembly in $assembly_folder"
echo "You need to build Spark before running this program."
exit 1
fi
num_jars=0

for f in ${assembly_folder}/spark-assembly*hadoop*.jar; do
if [[ ! -e "$f" ]]; then
echo "Failed to find Spark assembly in $assembly_folder" 1>&2
echo "You need to build Spark before running this program." 1>&2
exit 1
fi
ASSEMBLY_JAR="$f"
num_jars=$((num_jars+1))
done

if [ "$num_jars" -gt "1" ]; then
jars_list=$(ls "$assembly_folder" | grep "spark-assembly.*hadoop.*.jar$")
echo "Found multiple Spark assembly jars in $assembly_folder:"
echo "$jars_list"
echo "Please remove all but one jar."
echo "Found multiple Spark assembly jars in $assembly_folder:" 1>&2
ls ${assembly_folder}/spark-assembly*hadoop*.jar 1>&2
echo "Please remove all but one jar." 1>&2
exit 1
fi

ASSEMBLY_JAR="$(ls "$assembly_folder"/spark-assembly*hadoop*.jar 2>/dev/null)"

# Verify that versions of java used to build the jars and run Spark are compatible
jar_error_check=$("$JAR_CMD" -tf "$ASSEMBLY_JAR" nonexistent/class/path 2>&1)
if [[ "$jar_error_check" =~ "invalid CEN header" ]]; then
Expand Down
27 changes: 21 additions & 6 deletions bin/run-example
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,17 +35,32 @@ else
fi

if [ -f "$FWDIR/RELEASE" ]; then
export SPARK_EXAMPLES_JAR="`ls "$FWDIR"/lib/spark-examples-*hadoop*.jar`"
elif [ -e "$EXAMPLES_DIR"/target/scala-$SPARK_SCALA_VERSION/spark-examples-*hadoop*.jar ]; then
export SPARK_EXAMPLES_JAR="`ls "$EXAMPLES_DIR"/target/scala-$SPARK_SCALA_VERSION/spark-examples-*hadoop*.jar`"
JAR_PATH="${FWDIR}/lib"
else
JAR_PATH="${EXAMPLES_DIR}/target/scala-${SPARK_SCALA_VERSION}"
fi

if [[ -z "$SPARK_EXAMPLES_JAR" ]]; then
echo "Failed to find Spark examples assembly in $FWDIR/lib or $FWDIR/examples/target" 1>&2
echo "You need to build Spark before running this program" 1>&2
JAR_COUNT=0

for f in ${JAR_PATH}/spark-examples-*hadoop*.jar; do
if [[ ! -e "$f" ]]; then
echo "Failed to find Spark examples assembly in $FWDIR/lib or $FWDIR/examples/target" 1>&2
echo "You need to build Spark before running this program" 1>&2
exit 1
fi
SPARK_EXAMPLES_JAR="$f"
JAR_COUNT=$((JAR_COUNT+1))
done

if [ "$JAR_COUNT" -gt "1" ]; then
echo "Found multiple Spark examples assembly jars in ${JAR_PATH}" 1>&2
ls ${JAR_PATH}/spark-examples-*hadoop*.jar 1>&2
echo "Please remove all but one jar." 1>&2
exit 1
fi

export SPARK_EXAMPLES_JAR

EXAMPLE_MASTER=${MASTER:-"local[*]"}

if [[ ! $EXAMPLE_CLASS == org.apache.spark.examples* ]]; then
Expand Down
2 changes: 2 additions & 0 deletions bin/spark-class
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,8 @@ case "$1" in
'org.apache.spark.executor.MesosExecutorBackend')
OUR_JAVA_OPTS="$SPARK_JAVA_OPTS $SPARK_EXECUTOR_OPTS"
OUR_JAVA_MEM=${SPARK_EXECUTOR_MEMORY:-$DEFAULT_MEM}
export PYTHONPATH="$FWDIR/python:$PYTHONPATH"
export PYTHONPATH="$FWDIR/python/lib/py4j-0.8.2.1-src.zip:$PYTHONPATH"
;;

# Spark submit uses SPARK_JAVA_OPTS + SPARK_SUBMIT_OPTS +
Expand Down
1 change: 1 addition & 0 deletions core/src/main/resources/org/apache/spark/log4j-defaults.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,4 @@ log4j.logger.org.eclipse.jetty=WARN
log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.hadoop.yarn.util.RackResolver=WARN
8 changes: 8 additions & 0 deletions core/src/main/resources/org/apache/spark/ui/static/webui.css
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,14 @@ pre {
border: none;
}

.description-input {
overflow: hidden;
text-overflow: ellipsis;
width: 100%;
white-space: nowrap;
display: block;
}

.stacktrace-details {
max-height: 300px;
overflow-y: auto;
Expand Down
8 changes: 4 additions & 4 deletions core/src/main/scala/org/apache/spark/Aggregator.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,8 +61,8 @@ case class Aggregator[K, V, C] (
// Update task metrics if context is not null
// TODO: Make context non optional in a future release
Option(context).foreach { c =>
c.taskMetrics.memoryBytesSpilled += combiners.memoryBytesSpilled
c.taskMetrics.diskBytesSpilled += combiners.diskBytesSpilled
c.taskMetrics.incMemoryBytesSpilled(combiners.memoryBytesSpilled)
c.taskMetrics.incDiskBytesSpilled(combiners.diskBytesSpilled)
}
combiners.iterator
}
Expand Down Expand Up @@ -95,8 +95,8 @@ case class Aggregator[K, V, C] (
// Update task metrics if context is not null
// TODO: Make context non-optional in a future release
Option(context).foreach { c =>
c.taskMetrics.memoryBytesSpilled += combiners.memoryBytesSpilled
c.taskMetrics.diskBytesSpilled += combiners.diskBytesSpilled
c.taskMetrics.incMemoryBytesSpilled(combiners.memoryBytesSpilled)
c.taskMetrics.incDiskBytesSpilled(combiners.diskBytesSpilled)
}
combiners.iterator
}
Expand Down
2 changes: 1 addition & 1 deletion core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,7 @@ private[spark] class ExecutorAllocationManager(
"shuffle service. You may enable this through spark.shuffle.service.enabled.")
}
if (tasksPerExecutor == 0) {
throw new SparkException("spark.executor.cores must not be less than spark.task.cpus.cores")
throw new SparkException("spark.executor.cores must not be less than spark.task.cpus.")
}
}

Expand Down
5 changes: 3 additions & 2 deletions core/src/main/scala/org/apache/spark/SparkConf.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
package org.apache.spark

import scala.collection.JavaConverters._
import scala.collection.concurrent.TrieMap
import scala.collection.mutable.{HashMap, LinkedHashSet}
import org.apache.spark.serializer.KryoSerializer

Expand Down Expand Up @@ -46,7 +47,7 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable with Logging {
/** Create a SparkConf that loads defaults from system properties and the classpath */
def this() = this(true)

private[spark] val settings = new HashMap[String, String]()
private[spark] val settings = new TrieMap[String, String]()

if (loadDefaults) {
// Load any spark.* system properties
Expand Down Expand Up @@ -177,7 +178,7 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable with Logging {
}

/** Get all parameters as a list of pairs */
def getAll: Array[(String, String)] = settings.clone().toArray
def getAll: Array[(String, String)] = settings.toArray

/** Get a parameter as an integer, falling back to a default if not set */
def getInt(key: String, defaultValue: Int): Int = {
Expand Down
Loading