SPARK - August 24, 2017#5

Merged

pgandhi999 merged 152 commits intopgandhi999:masterfrom

Aug 24, 2017

Commits on Aug 1, 2017

[SPARK-21522][CORE] Fix flakiness in LauncherServerSuite.
Marcelo Vanzin
committed
[SPARK-20079][YARN] Fix client AM not allocating executors after restart.
Marcelo Vanzin
committed
[SPARK-21592][BUILD] Skip maven-compiler-plugin main and test compilations in Maven build

gslowikowski
authored and
srowen
committed
[SPARK-21593][DOCS] Fix 2 rendering errors on configuration page
srowen
committed
[SPARK-21339][CORE] spark-shell --packages option does not add jars to classpath on windows

Devaraj K
authored and
Marcelo Vanzin
committed
[SPARK-12717][PYTHON] Adding thread-safe broadcast pickle registry

BryanCutler
authored and
HyukjinKwon
committed

Commits on Aug 2, 2017

Commits on Aug 3, 2017

[SPARK-21615][ML][MLLIB][DOCS] Fix broken redirect in collaborative filtering docs to databricks training repo

Ayush Singh
authored and
srowen
committed
[SPARK-21604][SQL] if the object extends Logging, i suggest to remove the var LOG which is useless.

zuotingbing
authored and
srowen
committed
[SPARK-21611][SQL] Error class name for log in several classes.

zuotingbing
authored and
srowen
committed
[SPARK-21605][BUILD] Let IntelliJ IDEA correctly detect Language level and Target byte code version

baibaichen
authored and
srowen
committed
[SPARK-21602][R] Add map_keys and map_values functions to R
HyukjinKwon
committed
[SPARK-21599][SQL] Collecting column statistics for datasource tables may fail with java.util.NoSuchElementException

dilipbiswal
authored and
gatorsmile
committed
[SPARK-20713][SPARK CORE] Convert CommitDenied to TaskKilled.

louis lyu
authored and
Tom Graves
committed
Fix Java SimpleApp spark application

christiam
authored and
srowen
committed

Commits on Aug 4, 2017

Commits on Aug 5, 2017

[SPARK-21374][CORE] Fix reading globbed paths from S3 into DF with disabled FS cache

zsxwing
authored and
gatorsmile
committed
[SPARK-21580][SQL] Integers in aggregation expressions are wrongly taken as group-by ordinal

10110346
authored and
gatorsmile
committed
[INFRA] Close stale PRs
HyukjinKwon
committed
[SPARK-21485][FOLLOWUP][SQL][DOCS] Describes examples and arguments separately, and note/since in SQL built-in function documentation

HyukjinKwon
authored and
gatorsmile
committed
[SPARK-21640] Add errorifexists as a valid string for ErrorIfExists save mode

arodriguez
authored and
gatorsmile
committed

Commits on Aug 6, 2017

[SPARK-21637][SPARK-21451][SQL] get `spark.hadoop.*` properties from sysProps to hiveconf

yaooqinn
authored and
gatorsmile
committed
[SPARK-20963][SQL] Support column aliases for join relations in FROM clause

maropu
authored and
gatorsmile
committed
[SPARK-21588][SQL] SQLContext.getConf(key, null) should return null

vinodkc
authored and
gatorsmile
committed
[SPARKR][BUILD] AppVeyor change to latest R version

felixcheung
authored and
HyukjinKwon
committed
[SPARK-21574][SQL] Point out user to set hive config before SparkSession is initialized

wangyum
authored and
gatorsmile
committed
[SPARK-20963][SQL][FOLLOW-UP] Use UnresolvedSubqueryColumnAliases for visitTableName

maropu
authored and
gatorsmile
committed
[SPARK-21622][ML][SPARKR] Support offset in SparkR GLM

actuaryzhang
authored and
Felix Cheung
committed
Add "full_outer" name to join types

BartekH
authored and
gatorsmile
committed
[MINOR][BUILD] Remove duplicate test-jar:test spark-sql dependency from Hive module

srowen
authored and
gatorsmile
committed

Commits on Aug 7, 2017

[SPARK-21621][CORE] Reset numRecordsWritten after DiskBlockObjectWriter.commitAndGet called

ConeyLiu
authored and
cloud-fan
committed
[SPARK-13041][MESOS] Adds sandbox uri to spark dispatcher ui

skonto
authored and
srowen
committed
[SPARK-21623][ML] fix RF doc

Peng Meng
authored and
srowen
committed
[SPARK-21544][DEPLOY][TEST-MAVEN] Tests jar of some module should not upload twice

caneGuy
authored and
srowen
committed
[SPARK-21647][SQL] Fix SortMergeJoin when using CROSS

gatorsmile
authored and
cloud-fan
committed
[SPARK][DOCS] Added note on meaning of position to substring function

maclockard
authored and
srowen
committed
[SPARK-21565][SS] Propagate metadata in attribute replacement.

Jose Torres
authored and
zsxwing
committed
[SPARK-21648][SQL] Fix confusing assert failure in JDBC source when parallel fetching parameters are not properly provided.
gatorsmile
committed

Commits on Aug 8, 2017

[SPARK-21542][ML][PYTHON] Python persistence helper functions

ajaysaini725
authored and
jkbradley
committed
[SPARK-19270][FOLLOW-UP][ML] PySpark GLR model.summary should return a printable representation.
yanboliang
committed
[SPARK-21640][FOLLOW-UP][SQL] added errorifexists on IllegalArgumentException message

Marcos P. Sanchez
authored and
gatorsmile
committed
[SPARK-21567][SQL] Dataset should work with type alias

viirya
authored and
cloud-fan
committed
[MINOR][R][BUILD] More reliable detection of R version for Windows in AppVeyor
HyukjinKwon
committed
[SPARK-20655][CORE] In-memory KVStore implementation.
Marcelo Vanzin
committed
[SPARK-21671][CORE] Move kvstore to "util" sub-package, add private annotation.
Marcelo Vanzin
committed

Commits on Aug 9, 2017

[SPARK-20433][BUILD] Bump jackson from 2.6.5 to 2.6.7.1

srowen
authored and
Marcelo Vanzin
committed
[SPARK-21596][SS] Ensure places calling HDFSMetadataLog.get check the return value

zsxwing
authored and
tdas
committed
[SPARK-21608][SPARK-9221][SQL] Window rangeBetween() API should allow literal boundary

jiangxb1987
authored and
cloud-fan
committed
[SPARK-21503][UI] Spark UI shows incorrect task status for a killed Executor Process

pgandhi
authored and
cloud-fan
committed
[SPARK-21176][WEB UI] Use a single ProxyServlet to proxy all workers and applications

aosagie
authored and
cloud-fan
committed
[SPARK-21523][ML] update breeze to 0.13.2 for an emergency bugfix in strong wolfe line search

WeichenXu123
authored and
yanboliang
committed
[SPARK-21663][TESTS] test("remote fetch below max RPC message size") should call masterTracker.stop() in MapOutputTrackerSuite

wangjiaochun
authored and
cloud-fan
committed
[SPARK-21665][CORE] Need to close resources after use

vinodkc
authored and
srowen
committed
[SPARK-21276][CORE] Update lz4-java to the latest (v1.4.0)

maropu
authored and
srowen
committed
[SPARK-21504][SQL] Add spark version info into table metadata
gatorsmile
committed
[SPARK-21587][SS] Added filter pushdown through watermarks.

Jose Torres
authored and
tdas
committed
[SPARK-21551][PYTHON] Increase timeout for PythonRDD.serveIterator

peay
authored and
rxin
committed

Commits on Aug 10, 2017

[SPARK-14932][SQL] Allow DataFrame.replace() to replace values with None

jiayue-zhang
authored and
gatorsmile
committed
[SPARK-21669] Internal API for collecting metrics/stats during FileFormatWriter jobs

adrian-ionescu
authored and
rxin
committed
[SPARK-21638][ML] Fix RF/GBT Warning message error

Peng Meng
authored and
srowen
committed

Commits on Aug 11, 2017

[SPARK-21699][SQL] Remove unused getTableOption in ExternalCatalog
rxin
committed
[SPARK-21675][WEBUI] Add a navigation bar at the bottom of the Details for Stage Page

yaooqinn
authored and
srowen
committed
[SPARK-21519][SQL] Add an option to the JDBC data source to initialize the target DB environment

LucaCanali
authored and
gatorsmile
committed
[SPARK-21595] Separate thresholds for buffering and spilling in ExternalAppendOnlyUnsafeRowArray

tejasapatil
authored and
hvanhovell
committed
[SPARK-19122][SQL] Unnecessary shuffle+sort added if join predicates ordering differ from bucketing and sorting order

tejasapatil
authored and
gatorsmile
committed
[SPARK-12559][SPARK SUBMIT] fix --packages for stand-alone cluster mode

skonto
authored and
Marcelo Vanzin
committed

Commits on Aug 12, 2017

[MINOR][BUILD] Download RAT and R version info over HTTPS; use RAT 0.12

srowen
authored and
HyukjinKwon
committed
[SPARK-17025][ML][PYTHON] Persistence for Pipelines with Python-only Stages

ajaysaini725
authored and
jkbradley
committed
[SPARK-21709][BUILD] sbt 0.13.16 and some plugin updates

pj.fanning
authored and
srowen
committed

Commits on Aug 14, 2017

[MINOR][SQL] Additional test case for CheckCartesianProducts rule

aokolnychyi
authored and
gatorsmile
committed
[SPARK-21176][WEB UI] Format worker page links to work with proxy

aosagie
authored and
srowen
committed
[SPARK-21563][CORE] Fix race condition when serializing TaskDescriptions and adding jars

ash211
authored and
cloud-fan
committed
[SPARK-21658][SQL][PYSPARK] Add default None for value in na.replace in PySpark

chihhanyu
authored and
HyukjinKwon
committed
[MINOR][SQL][TEST] no uncache table in joinsuite test

heary-cao
authored and
gatorsmile
committed
[SPARK-19471][SQL] AggregationIterator does not initialize the generated result projection before using it

DonnyZone
authored and
gatorsmile
committed
[SPARK-21696][SS] Fix a potential issue that may generate partial snapshot files

zsxwing
authored and
tdas
committed

Commits on Aug 15, 2017

[SPARK-21721][SQL] Clear FileSystem deleteOnExit cache when paths are successfully removed

viirya
authored and
gatorsmile
committed
[SPARK-21724][SQL][DOC] Adds since information in the documentation of date functions

HyukjinKwon
authored and
gatorsmile
committed
[SPARK-21732][SQL] Lazily init hive metastore client

zsxwing
authored and
gatorsmile
committed
[SPARK-19471][SQL] AggregationIterator does not initialize the generated result projection before using it

DonnyZone
authored and
gatorsmile
committed
[SPARK-18464][SQL][FOLLOWUP] support old table which doesn't store schema in table properties

cloud-fan
authored and
gatorsmile
committed
[SPARK-17742][CORE] Handle child process exit in SparkLauncher.
Marcelo Vanzin
committed
[SPARK-21731][BUILD] Upgrade scalastyle to 0.9.
Marcelo Vanzin
committed
[MINOR] Fix a typo in the method name `UserDefinedFunction.asNonNullabe`

jiangxb1987
authored and
gatorsmile
committed

Commits on Aug 16, 2017

[SPARK-21712][PYSPARK] Clarify type error for Column.substr()

nchammas
authored and
HyukjinKwon
committed
[SPARK-19634][ML] Multivariate summarizer - dataframes API

WeichenXu123
authored and
yanboliang
committed
[SPARK-21422][BUILD] Depend on Apache ORC 1.4.0

dongjoon-hyun
authored and
gatorsmile
committed
[SPARK-21723][ML] Fix writing LibSVM (key not found: numFeatures)

Jan Vrsovsky
authored and
srowen
committed
[SPARK-13969][ML] Add FeatureHasher transformer
Nick Pentreath
committed
[SPARK-21656][CORE] spark dynamic allocation should not idle timeout executors when tasks still to run

John Lee
authored and
Tom Graves
committed
[SPARK-21603][SQL] The wholestage codegen will be much slower then that is closed when the function is too long

eatoncys
authored and
gatorsmile
committed
[SPARK-21738] Thriftserver doesn't cancel jobs when session is closed

mgaido91
authored and
gatorsmile
committed
[SPARK-21680][ML][MLLIB] optimize Vector compress

Peng Meng
authored and
srowen
committed

Commits on Aug 17, 2017

[SPARK-3151][BLOCK MANAGER] DiskStore.getBytes fails for files larger than 2GB

Eyal Farago
authored and
cloud-fan
committed
[SPARK-21743][SQL] top-most limit should not cause memory leak

cloud-fan
authored and
gatorsmile
committed
[SPARK-21642][CORE] Use FQDN for DRIVER_HOST_ADDRESS instead of ip address

akitanaka
authored and
cloud-fan
committed
[SPARK-21428] Turn IsolatedClientLoader off while using builtin Hive jars for reusing CliSessionState

yaooqinn
authored and
cloud-fan
committed
[SQL][MINOR][TEST] Set spark.unsafe.exceptionOnMemoryLeak to true
gatorsmile
committed
[SPARK-18394][SQL] Make an AttributeSet.toSeq output order consistent

maropu
authored and
hvanhovell
committed
[SPARK-16742] Mesos Kerberos Support

ArtRand
authored and
Marcelo Vanzin
committed
[SPARK-21677][SQL] json_tuple throws NullPointException when column is null as string type

jmchung
authored and
gatorsmile
committed
[SPARK-21767][TEST][SQL] Add Decimal Test For Avro in VersionSuite
gatorsmile
committed

Commits on Aug 18, 2017

[SPARK-21739][SQL] Cast expression should initialize timezoneId when it is called statically to convert something into TimestampType

DonnyZone
authored and
gatorsmile
committed
[SPARK-21778][SQL] Simpler Dataset.sample API in Scala / Java

rxin
authored and
HyukjinKwon
committed
[SPARK-21213][SQL] Support collecting partition-level statistics: rowCount and sizeInBytes

mbasmanova
authored and
gatorsmile
committed
[SPARK-21743][SQL][FOLLOW-UP] top-most limit should not cause memory leak

cloud-fan
authored and
gatorsmile
committed
[MINOR][TYPO] Fix typos: runnning and Excecutors

ash211
authored and
rxin
committed

Commits on Aug 19, 2017

[SPARK-21566][SQL][PYTHON] Python method for summary

aray
authored and
holdenk
committed
[SPARK-21790][TESTS] Fix Docker-based Integration Test errors.

wangyum
authored and
gatorsmile
committed

Commits on Aug 20, 2017

Commits on Aug 21, 2017

[SPARK-21782][CORE] Repartition creates skews when numPartitions is a power of 2

megaserg
authored and
srowen
committed
[SPARK-21718][SQL] Heavy log of type: "Skipping partition based on stats ..."

srowen
authored and
hvanhovell
committed
[SPARK-21468][PYSPARK][ML] Python API for FeatureHasher
Nick Pentreath
committed
[SPARK-21790][TESTS][FOLLOW-UP] Add filter pushdown verification back.

wangyum
authored and
gatorsmile
committed
[SPARK-21617][SQL] Store correct table metadata when altering schema in Hive metastore.

Marcelo Vanzin
authored and
gatorsmile
committed

Commits on Aug 22, 2017

[SPARK-19762][ML][FOLLOWUP] Add necessary comments to L2Regularization.
yanboliang
committed
[SPARK-21070][PYSPARK] Attempt to update cloudpickle again

rgbkrk
authored and
HyukjinKwon
committed
[SPARK-21584][SQL][SPARKR] Update R method for summary to call new implementation

aray
authored and
Felix Cheung
committed
[SPARK-21803][TEST] Remove the HiveDDLCommandSuite

gatorsmile
authored and
cloud-fan
committed
[SPARK-20641][CORE] Add missing kvstore module in Laucher and SparkSubmit code

jerryshao
authored and
Marcelo Vanzin
committed
[SPARK-21499][SQL] Support creating persistent function for Spark UDAF(UserDefinedAggregateFunction)
gatorsmile
committed
[SPARK-21769][SQL] Add a table-specific option for always respecting schemas inferred/controlled by Spark SQL

gatorsmile
authored and
sameeragarwal
committed
[SPARK-21681][ML] fix bug of MLOR do not work correctly when featureStd contains zero

WeichenXu123
authored and
jkbradley
committed

Commits on Aug 23, 2017

[SPARK-10931][ML][PYSPARK] PySpark Models Copy Param Values from Estimator

BryanCutler
authored and
holdenk
committed
[SPARK-21765] Set isStreaming on leaf nodes for streaming plans.

Jose Torres
authored and
tdas
committed
[ML][MINOR] Make sharedParams update.
yanboliang
committed
[SPARK-19326] Speculated task attempts do not get launched in few scenarios

janewangfb
authored and
cloud-fan
committed
[SPARK-12664][ML] Expose probability in mlp model

WeichenXu123
authored and
jkbradley
committed
[SPARK-21501] Change CacheLoader to limit entries based on memory footprint

Sanket Chintapalli
authored and
Tom Graves
committed
[SPARK-21603][SQL][FOLLOW-UP] Change the default value of maxLinesPerFunction into 4000

maropu
authored and
gatorsmile
committed

Commits on Aug 24, 2017

[SPARK-21807][SQL] Override ++ operation in ExpressionSet to reduce clone time

eatoncys
authored and
gatorsmile
committed
[SPARK-21805][SPARKR] Disable R vignettes code on Windows

felixcheung
authored and
Felix Cheung
committed
[SPARK-21694][MESOS] Support Mesos CNI network labels

susanxhuynh
authored and
srowen
committed
[MINOR][SQL] The comment of Class ExchangeCoordinator exist a typing and context error

figo77
authored and
srowen
committed
[SPARK-21804][SQL] json_tuple returns null values within repeated columns except the first one

jmchung
authored and
HyukjinKwon
committed
[SPARK-19165][PYTHON][SQL] PySpark APIs using columns as arguments should validate input types for column
HyukjinKwon
committed
[SPARK-21745][SQL] Refactor ColumnVector hierarchy to make ColumnVector read-only and to introduce WritableColumnVector.

ueshin
authored and
cloud-fan
committed
[SPARK-21759][SQL] In.checkInputDataTypes should not wrongly report unresolved plans for IN correlated subquery

viirya
authored and
cloud-fan
committed
[SPARK-21826][SQL] outer broadcast hash join should not throw NPE

cloud-fan
authored and
hvanhovell
committed
[SPARK-21788][SS] Handle more exceptions when stopping a streaming query

zsxwing
authored and
tdas
committed
[SPARK-21701][CORE] Enable RPC client to use ` SO_RCVBUF` and ` SO_SNDBUF` in SparkConf.

xu.zhang
authored and
zsxwing
committed