Branch 1.6#12407

Closed

thinkborm wants to merge 831 commits intomasterfrom

This pull request is big! We're only showing the most recent 250 commits

Commits on Dec 16, 2015

[SPARK-12364][ML][SPARKR] Add ML example for SparkR

yanboliang
authored and
jkbradley
committed
[SPARK-12380] [PYSPARK] use SQLContext.getOrCreate in mllib

Davies Liu
authored and
davies
committed

Commits on Dec 17, 2015

[MINOR] Add missing interpolation in NettyRPCEnv

Andrew Or
authored and
zsxwing
committed
[SPARK-10248][CORE] track exceptions in dagscheduler event loop in tests

squito
authored and
Andrew Or
committed
[SPARK-12365][CORE] Use ShutdownHookManager where Runtime.getRuntime.addShutdownHook() is called

tedyu
authored and
Andrew Or
committed
[SPARK-12186][WEB UI] Send the complete request URI including the query string when redirecting.

Rohit Agarwal
authored and
Andrew Or
committed
[SPARK-12386][CORE] Fix NPE when spark.executor.port is set.

Marcelo Vanzin
authored and
Andrew Or
committed
[SPARK-12057][SQL] Prevent failure on corrupt JSON records

yhuai
authored and
rxin
committed
Once driver register successfully, stop it to connect to master.

echoTomei
authored and
davies
committed
Revert "Once driver register successfully, stop it to connect to master."
davies
committed
[SPARK-12395] [SQL] fix resulting columns of outer join

Davies Liu
authored and
davies
committed
[SQL] Update SQLContext.read.text doc

yanboliang
authored and
rxin
committed
[SPARK-12220][CORE] Make Utils.fetchFile support files that contain special characters
zsxwing
committed
[SPARK-12345][MESOS] Properly filter out SPARK_HOME in the Mesos REST server

dragos
authored and
sarutak
committed
[SPARK-12390] Clean up unused serializer parameter in BlockManager
Andrew Or
committed
[SPARK-12410][STREAMING] Fix places that use '.' and '|' directly in split
zsxwing
committed
[SPARK-12397][SQL] Improve error messages for data sources when they are not found

rxin
authored and
marmbrus
committed
[SPARK-12376][TESTS] Spark Streaming Java8APISuite fails in assertOrderInvariantEquals method

Evan Chen
authored and
zsxwing
committed

Commits on Dec 18, 2015

[SPARK-11749][STREAMING] Duplicate creating the RDD in file stream when recovering from checkpoint data

jhu-chang
authored and
zsxwing
committed
[SPARK-12413] Fix Mesos ZK persistence

Michael Gummelt
authored and
sarutak
committed
[SPARK-12218][SQL] Invalid splitting of nested AND expressions in Data Source filter API
yhuai
committed
Revert "[SPARK-12365][CORE] Use ShutdownHookManager where Runtime.getRuntime.addShutdownHook() is called"
Andrew Or
committed
[SPARK-12404][SQL] Ensure objects passed to StaticInvoke is Serializable

sarutak
authored and
marmbrus
committed
[SPARK-11985][STREAMING][KINESIS][DOCS] Update Kinesis docs

brkyvz
authored and
zsxwing
committed

Commits on Dec 19, 2015

[SQL] Fix mistake doc of join type for dataframe.join

yanboliang
authored and
rxin
committed

Commits on Dec 21, 2015

Doc typo: ltrim = trim from left end, not right

pshearer
authored and
Andrew Or
committed
[SPARK-12466] Fix harmless NPE in tests
Andrew Or
committed

Commits on Dec 23, 2015

[SPARK-12429][STREAMING][DOC] Add Accumulator and Broadcast example for Streaming

zsxwing
authored and
tdas
committed
[SPARK-12477][SQL] - Tungsten projection fails for null values in array fields

pierre-borckmans
authored and
rxin
committed

Commits on Dec 24, 2015

[SPARK-12499][BUILD] don't force MAVEN_OPTS

abridgett
authored and
JoshRosen
committed
[SPARK-12411][CORE] Decrease executor heartbeat timeout to match heartbeat interval

nongli
authored and
Andrew Or
committed
[SPARK-12502][BUILD][PYTHON] Script /dev/run-tests fails when IBM Java is used

kiszk
authored and
sarutak
committed
[SPARK-12010][SQL] Spark JDBC requires support for column-name-free INSERT syntax

CK50
authored and
srowen
committed

Commits on Dec 29, 2015

[SPARK-11394][SQL] Throw IllegalArgumentException for unsupported types in postgresql

maropu
authored and
yhuai
committed
[SPARK-12526][SPARKR] ifelse`, `when`, `otherwise` unable to take Column as value

saurfang
authored and
shivaram
committed

Commits on Dec 30, 2015

[SPARK-12300] [SQL] [PYSPARK] fix schema inferance on local collections

holdenk
authored and
davies
committed
[SPARK-12399] Display correct error message when accessing REST API with an unknown app Id

carsonwang
authored and
Marcelo Vanzin
committed

Commits on Jan 3, 2016

[SPARK-12327][SPARKR] fix code for lintr warning for commented code

felixcheung
authored and
shivaram
committed

Commits on Jan 4, 2016

[SPARK-12562][SQL] DataFrame.write.format(text) requires the column name to be called value

xguo27
authored and
rxin
committed
[SPARK-12486] Worker should kill the executors more forcefully if possible.

nongli
authored and
Andrew Or
committed
[SPARK-12470] [SQL] Fix size reduction calculation

robbinspg
authored and
davies
committed
[SPARK-12579][SQL] Force user-specified JDBC driver to take precedence

JoshRosen
authored and
yhuai
committed
[DOC] Adjust coverage for partitionBy()

tedyu
authored and
marmbrus
committed
[SPARK-12589][SQL] Fix UnsafeRowParquetRecordReader to properly set the row length.

nongli
authored and
yhuai
committed

Commits on Jan 5, 2016

[SPARKR][DOC] minor doc update for version in migration guide

felixcheung
authored and
shivaram
committed
[SPARK-12568][SQL] Add BINARY to Encoders
marmbrus
committed
[SPARK-12647][SQL] Fix o.a.s.sqlexecution.ExchangeCoordinatorSuite.determining the number of reducers: aggregate operator

robbinspg
authored and
yhuai
committed
[SPARK-12617] [PYSPARK] Clean up the leak sockets of Py4J

zsxwing
authored and
davies
committed
[SPARK-12511] [PYSPARK] [STREAMING] Make sure PythonDStream.registerSerializer is called only once

zsxwing
authored and
davies
committed
[SPARK-12450][MLLIB] Un-persist broadcasted variables in KMeans

rnowling
authored and
jkbradley
committed
[SPARK-12453][STREAMING] Remove explicit dependency on aws-java-sdk

BrianLondon
authored and
srowen
committed

Commits on Jan 6, 2016

Commits on Jan 7, 2016

[SPARK-12016] [MLLIB] [PYSPARK] Wrap Word2VecModel when loading it in pyspark

viirya
authored and
jkbradley
committed
[SPARK-12673][UI] Add missing uri prepending for job description

jerryshao
authored and
zsxwing
committed
[SPARK-12678][CORE] MapPartitionsRDD clearDependencies

gpoulin
authored and
rxin
committed
Revert "[SPARK-12006][ML][PYTHON] Fix GMM failure if initialModel is not None"
yhuai
committed
[DOC] fix 'spark.memory.offHeap.enabled' default value to false

zzcclp
authored and
rxin
committed
[SPARK-12006][ML][PYTHON] Fix GMM failure if initialModel is not None

zero323
authored and
jkbradley
committed
[SPARK-12662][SQL] Fix DataFrame.randomSplit to avoid creating overlapping splits

sameeragarwal
authored and
rxin
committed
[SPARK-12598][CORE] bug in setMinPartitions

datafarmer
authored and
srowen
committed

Commits on Jan 8, 2016

[SPARK-12507][STREAMING][DOCUMENT] Expose closeFileAfterWrite and allowBatching configurations for Streaming

zsxwing
authored and
tdas
committed
[SPARK-12591][STREAMING] Register OpenHashMapBasedStateMap for Kryo (branch 1.6)

zsxwing
authored and
tdas
committed
[DOCUMENTATION] doc fix of job scheduling

zjffdu
authored and
Marcelo Vanzin
committed
fixed numVertices in transitive closure example

udoklein
authored and
srowen
committed
[SPARK-12654] sc.wholeTextFiles with spark.hadoop.cloneConf=true fail…

tgravescs
authored and
Tom Graves
committed
[SPARK-12696] Backport Dataset Bug fixes to 1.6
marmbrus
committed

Commits on Jan 9, 2016

[SPARK-12645][SPARKR] SparkR support hash function

yanboliang
authored and
shivaram
committed

Commits on Jan 10, 2016

[SPARK-10359][PROJECT-INFRA] Backport dev/test-dependencies script to branch-1.6
JoshRosen
committed

Commits on Jan 12, 2016

Commits on Jan 13, 2016

[HOT-FIX] bypass hive test when parse logical plan to json

cloud-fan
authored and
marmbrus
committed
[SPARK-12558][SQL] AnalysisException when multiple functions applied in GROUP BY clause

dilipbiswal
authored and
yhuai
committed
[SPARK-12805][MESOS] Fixes documentation on Mesos run modes

Luc Bourlier
authored and
rxin
committed
[SPARK-12685][MLLIB][BACKPORT TO 1.4] word2vec trainWordsCount gets overflow

hhbyyh
authored and
jkbradley
committed
[SPARK-12268][PYSPARK] Make pyspark shell pythonstartup work under python3

erikselin
authored and
JoshRosen
committed
[SPARK-12690][CORE] Fix NPE in UnsafeInMemorySorter.free()

carsonwang
authored and
JoshRosen
committed

Commits on Jan 14, 2016

Commits on Jan 15, 2016

[SPARK-12708][UI] Sorting task error in Stages Page when yarn mode.

yoshidakuy
authored and
sarutak
committed
[SPARK-11031][SPARKR] Method str() on a DataFrame

Oscar D. Lara Yejas
authored and
shivaram
committed
[SPARK-12701][CORE] FileAppender should use join to ensure writing thread completion

BryanCutler
authored and
srowen
committed

Commits on Jan 16, 2016

[SPARK-12722][DOCS] Fixed typo in Pipeline example

Jeff Lam
authored and
srowen
committed

Commits on Jan 18, 2016

[SPARK-12558][FOLLOW-UP] AnalysisException when multiple functions applied in GROUP BY clause

dilipbiswal
authored and
yhuai
committed
[SPARK-12346][ML] Missing attribute names in GLM for vector-type features

ericl
authored and
mengxr
committed
[SPARK-12814][DOCUMENT] Add deploy instructions for Python in flume integration doc

zsxwing
authored and
tdas
committed

Commits on Jan 19, 2016

Commits on Jan 21, 2016

[SPARK-12921] Use SparkHadoopUtil reflection in SpecificParquetRecordReaderBase
JoshRosen
committed

Commits on Jan 22, 2016

[SPARK-12747][SQL] Use correct type name for Postgres JDBC's real array

viirya
authored and
rxin
committed

Commits on Jan 23, 2016

Commits on Jan 24, 2016

[SPARK-12120][PYSPARK] Improve exception message when failing to init…

zjffdu
authored and
JoshRosen
committed

Commits on Jan 25, 2016

[SPARK-12624][PYSPARK] Checks row length when converting Java arrays to Python rows

liancheng
authored and
yhuai
committed
[SPARK-12932][JAVA API] improved error message for java type inference failure

andygrove
authored and
srowen
committed
[SPARK-12755][CORE] Stop the event logger before the DAG scheduler

Michael Allman
authored and
srowen
committed

Commits on Jan 26, 2016

Commits on Jan 27, 2016

Commits on Jan 29, 2016

[SPARK-13082][PYSPARK] Backport the fix of 'read.json(rdd)' in #10559 to branch-1.6
zsxwing
committed

Commits on Jan 30, 2016

[SPARK-13088] Fix DAG viz in latest version of chrome
Andrew Or
committed

Commits on Feb 1, 2016

Commits on Feb 2, 2016

[SPARK-13087][SQL] Fix group by function for sort based aggregation

marmbrus
authored and
yhuai
committed
[SPARK-13094][SQL] Add encoders for seq/array of primitives
marmbrus
committed
[SPARK-12780][ML][PYTHON][BACKPORT] Inconsistency returning value of ML python models' properties

yinxusen
authored and
jkbradley
committed
[SPARK-12629][SPARKR] Fixes for DataFrame saveAsTable method

NarineK
authored and
shivaram
committed
[SPARK-13121][STREAMING] java mapWithState mishandles scala Option

sparkyengine
authored and
zsxwing
committed
[SPARK-12711][ML] ML StopWordsRemover does not protect itself from column name duplication

grzegorz-chilkiewicz
authored and
jkbradley
committed
[SPARK-13056][SQL] map column would throw NPE if value is null

adrian-wang
authored and
marmbrus
committed
[DOCS] Update StructType.scala

swkimme
authored and
marmbrus
committed

Commits on Feb 3, 2016

[SPARK-13122] Fix race condition in MemoryStore.unrollSafely()

Adam Budde
authored and
Andrew Or
committed
[SPARK-12739][STREAMING] Details of batch in Streaming tab uses two Duration columns

mariobriggs
authored and
zsxwing
committed

Commits on Feb 4, 2016

Commits on Feb 5, 2016

[SPARK-13214][DOCS] update dynamicAllocation documentation

Bill Chambers
authored and
Andrew Or
committed

Commits on Feb 8, 2016

[SPARK-13210][SQL] catch OOM when allocate memory and expand array

Davies Liu
authored and
JoshRosen
committed

Commits on Feb 9, 2016

[SPARK-12807][YARN] Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3

steveloughran
authored and
Marcelo Vanzin
committed

Commits on Feb 10, 2016

Commits on Feb 11, 2016

[SPARK-13274] Fix Aggregator Links on GroupedDataset Scala API

raelawang
authored and
rxin
committed
[SPARK-13265][ML] Refactoring of basic ML import/export for other file system besides HDFS

yu-iskw
authored and
mengxr
committed

Commits on Feb 12, 2016

[SPARK-13047][PYSPARK][ML] Pyspark Params.hasParam should not throw an error

sethah
authored and
mengxr
committed
[SPARK-13153][PYSPARK] ML persistence failed when handle no default value parameter

Wenpei
authored and
mengxr
committed

Commits on Feb 13, 2016

[SPARK-13142][WEB UI] Problem accessing Web UI /logPage/ on Microsoft Windows

markpavey
authored and
srowen
committed
[SPARK-12363][MLLIB] Remove setRun and fix PowerIterationClustering failed test

viirya
authored and
mengxr
committed

Commits on Feb 14, 2016

[SPARK-13300][DOCUMENTATION] Added pygments.rb dependancy

amitdev
authored and
srowen
committed

Commits on Feb 15, 2016

[SPARK-13312][MLLIB] Update java train-validation-split example in ml-guide

JeremyNixon
authored and
srowen
committed

Commits on Feb 16, 2016

Correct SparseVector.parse documentation

mgyucht
authored and
srowen
committed

Commits on Feb 17, 2016

[SPARK-13279] Remove O(n^2) operation from scheduler.

Sital Kedia
authored and
kayousterhout
committed
[SPARK-13350][DOCS] Config doc updated to state that PYSPARK_PYTHON's default is "python2.7"

chrisaycock
authored and
JoshRosen
committed

Commits on Feb 18, 2016

[SPARK-13371][CORE][STRING] TaskSetManager.dequeueSpeculativeTask compares Option and String directly.

srowen
authored and
Andrew Or
committed

Commits on Feb 22, 2016

[SPARK-12546][SQL] Change default number of open parquet files
marmbrus
committed

Commits on Feb 23, 2016

Commits on Feb 24, 2016

Commits on Feb 25, 2016

[SPARK-13482][MINOR][CONFIGURATION] Make consistency of the configuraiton named in TransportConf.

SaintBacchus
authored and
rxin
committed
[SPARK-13473][SQL] Don't push predicate through project with nondeterministic field(s)

liancheng
authored and
cloud-fan
committed
[SPARK-13444][MLLIB] QuantileDiscretizer chooses bad splits on large DataFrames

Oliver Pierson
authored and
srowen
committed
[SPARK-13441][YARN] Fix NPE in yarn Client.createConfArchive method

chtyim
authored and
srowen
committed
[SPARK-13439][MESOS] Document that spark.mesos.uris is comma-separated

Michael Gummelt
authored and
srowen
committed
[SPARK-12316] Wait a minutes to avoid cycle calling.

SaintBacchus
authored and
Tom Graves
committed
Revert "[SPARK-13444][MLLIB] QuantileDiscretizer chooses bad splits on large DataFrames"
mengxr
committed
[SPARK-12874][ML] ML StringIndexer does not protect itself from column name duplication

yu-iskw
authored and
mengxr
committed

Commits on Feb 26, 2016

[SPARK-13454][SQL] Allow users to drop a table with a name starting with an underscore.
yhuai
committed

Commits on Feb 29, 2016

[SPARK-12941][SQL][MASTER] Spark-SQL JDBC Oracle dialect fails to map string datatypes to Oracle VARCHAR datatype

thomastechs
authored and
yhuai
committed

Commits on Mar 3, 2016

[SPARK-13465] Add a task failure listener to TaskContext

Davies Liu
authored and
davies
committed
[SPARK-13601] call failure callbacks before writer.close()
davies
committed

Commits on Mar 4, 2016

[SPARK-13601] [TESTS] use 1 partition in tests to avoid race conditions

Davies Liu
authored and
davies
committed
[SPARK-13652][CORE] Copy ByteBuffer in sendRpcSync as it will be recycled
zsxwing
committed
[SPARK-11515][ML] QuantileDiscretizer should take random seed

yu-iskw
authored and
srowen
committed
[SPARK-12941][SQL][MASTER] Spark-SQL JDBC Oracle dialect fails to map string datatypes to Oracle VARCHAR datatype mapping

thomastechs
authored and
yhuai
committed
[SPARK-13444][MLLIB] QuantileDiscretizer chooses bad splits on large DataFrames

Oliver Pierson
authored and
mengxr
committed
[HOTFIX] fix the conflict when cherry-pick
davies
committed

Commits on Mar 6, 2016

[SPARK-13697] [PYSPARK] Fix the missing module name of TransformFunctionSerializer.loads

zsxwing
authored and
davies
committed

Commits on Mar 7, 2016

[SPARK-13705][DOCS] UpdateStateByKey Operation documentation incorrectly refers to StatefulNetworkWordCount

rmishra
authored and
srowen
committed
[SPARK-13599][BUILD] remove transitive groovy dependencies from spark-hive and spark-hiveserver (branch 1.6)

steveloughran
authored and
srowen
committed
[MINOR][DOC] improve the doc for "spark.memory.offHeap.size"

CodingCat
authored and
zsxwing
committed
[SPARK-13648] Add Hive Cli to classes for isolated classloader

preecet
authored and
marmbrus
committed

Commits on Mar 8, 2016

[SPARK-13711][CORE] Don't call SparkUncaughtExceptionHandler in AppClient as it's in driver
zsxwing
committed

Commits on Mar 9, 2016

[SPARK-13755] Escape quotes in SQL plan visualization node labels
JoshRosen
committed
[SPARK-13631][CORE] Thread-safe getLocationsWithLargestOutputs

Andy Sloane
authored and
srowen
committed
[SPARK-13242] [SQL] codegen fallback in case-when if there many branches

Davies Liu
authored and
davies
committed

Commits on Mar 10, 2016

Commits on Mar 11, 2016

[MINOR][DOC] Fix supported hive version in doc

dongjoon-hyun
authored and
rxin
committed
[SPARK-13327][SPARKR] Added parameter validations for colnames<-

Oscar D. Lara Yejas
authored and
shivaram
committed

Commits on Mar 13, 2016

[SPARK-13810][CORE] Add Port Configuration Suggestions on Bind Exceptions

bjornjon
authored and
srowen
committed

Commits on Mar 14, 2016

[SQL] fix typo in DataSourceRegister

Jacky Li
authored and
rxin
committed
[MINOR][DOCS] Added Missing back slashes

danielsan
authored and
Andrew Or
committed
[MINOR][COMMON] Fix copy-paste oversight in variable naming

bjornjon
authored and
Andrew Or
committed

Commits on Mar 15, 2016

[SPARK-13207][SQL][BRANCH-1.6] Make partitioning discovery ignore _SUCCESS files.

yhuai
authored and
rxin
committed
[SPARK-13803] restore the changes in SPARK-3411

CodingCat
authored and
srowen
committed

Commits on Mar 17, 2016

[SPARK-13901][CORE] correct the logDebug information when jump to the next locality level

trueyao
authored and
srowen
committed

Commits on Mar 18, 2016

[SPARK-13958] Executor OOM due to unbounded growth of pointer array in…

Sital Kedia
authored and
davies
committed

Commits on Mar 22, 2016

[SPARK-14058][PYTHON] Incorrect docstring in Window.order

zero323
authored and
rxin
committed
[SPARK-13772][SQL] Fix data type mismatch for decimal

cenyuhai
authored and
liancheng
committed
[SPARK-14006][SPARKR] Fix SparkR lint-r test errors in branch-1.6.

Sun Rui
authored and
mengxr
committed
[SPARK-13806] [SQL] fix rounding mode of negative float/double

Davies Liu
authored and
davies
committed

Commits on Mar 23, 2016

[SPARK-13642][YARN][1.6-BACKPORT] Properly handle signal kill in ApplicationMaster

jerryshao
authored and
Tom Graves
committed
[SPARK-14074][SPARKR] Specify commit sha1 ID when using install_github to install intr package.

Sun Rui
authored and
mengxr
committed

Commits on Mar 25, 2016

Commits on Mar 28, 2016

[SPARK-14187][MLLIB] Fix incorrect use of binarySearch in SparseMatrix

Chenliang Xu
authored and
mengxr
committed
[SPARK-13622][YARN] Issue creating level db for YARN shuffle service

ashangit
authored and
Marcelo Vanzin
committed

Commits on Mar 29, 2016

[SPARK-14219][GRAPHX] Fix `pickRandomVertex` not to fall into infinit…

dongjoon-hyun
authored and
rxin
committed
[SPARK-13845][CORE][BACKPORT-1.6] Using onBlockUpdated to replace onTaskEnd avioding driver OOM

jeanlyn
authored and
Andrew Or
committed
[SPARK-14232][WEBUI] Fix event timeline display issue when an executor is removed with a multiple line reason.

carsonwang
authored and
Andrew Or
committed

Commits on Mar 30, 2016

[SPARK-11507][MLLIB] add compact in Matrices fromBreeze

hhbyyh
authored and
jkbradley
committed

Commits on Mar 31, 2016

[SPARK-14138] [SQL] Fix generated SpecificColumnarIterator code can exceed JVM size limit for cached DataFrames

kiszk
authored and
davies
committed

Commits on Apr 4, 2016

[SPARK-11327][MESOS] Backport dispatcher does not respect all args f…

jayv
authored and
Andrew Or
committed

Commits on Apr 5, 2016

[SPARK-14368][PYSPARK] Support python.spark.worker.memory with upper-case unit.

yongtang
authored and
sarutak
committed
[SPARK-14243][CORE][BACKPORT-1.6] update task metrics when removing blocks

jeanlyn
authored and
Andrew Or
committed

Commits on Apr 6, 2016

[SPARK-14322][MLLIB] Use treeAggregate instead of reduce in OnlineLDAOptimizer

hhbyyh
authored and
jkbradley
committed

Commits on Apr 8, 2016

[DOCS][MINOR] Remove sentence about Mesos not supporting cluster mode.

Michael Gummelt
authored and
Andrew Or
committed
[SPARK-14468] Always enable OutputCommitCoordinator
Andrew Or
committed

Commits on Apr 10, 2016

[SPARK-14357][CORE] Properly handle the root cause being a commit denied exception

jasonmoore2k
authored and
Andrew Or
committed

Commits on Apr 12, 2016

[SPARK-14242][CORE][NETWORK] avoid copy in compositeBuffer for frame decoder

liyezhang556520
authored and
davies
committed
[SPARK-14563][ML] use a random table name instead of __THIS__ in SQLTransformer
mengxr
committed
[SPARK-14544] [SQL] improve performance of SQL UI tab

Davies Liu
authored and
davies
committed
[SPARK-14363] Fix executor OOM due to memory leak in the Sorter

Sital Kedia
authored and
davies
committed

Commits on Apr 14, 2016

[SPARK-14618][ML][DOC] Updated RegressionEvaluator.metricName param doc
jkbradley
committed