Add more types#6

Closed

yinxusen wants to merge 383 commits intoBryanCutler:wip-toPandas_with_arrow-SPARK-13534from

yinxusen:wip-toPandas_with_arrow-SPARK-13534

This pull request is big! We're only showing the most recent 250 commits

Commits on Nov 5, 2016

[SPARK-17849][SQL] Fix NPE problem when using grouping sets

wangyang
authored and
hvanhovell
committed
[SPARK-17964][SPARKR] Enable SparkR with Mesos client mode and cluster mode

susanxhuynh
authored and
srowen
committed

Commits on Nov 6, 2016

[MINOR][DOCUMENTATION] Fix some minor descriptions in functions consistently with expressions

HyukjinKwon
authored and
Felix Cheung
committed
[SPARK-18276][ML] ML models should copy the training summary and set parent

sethah
authored and
yanboliang
committed
[SPARK-17854][SQL] rand/randn allows null/long as input seed

HyukjinKwon
authored and
srowen
committed
[SPARK-18210][ML] Pipeline.copy does not create an instance with the same UID

wojtek-szymanski
authored and
yanboliang
committed

Commits on Nov 7, 2016

Commits on Nov 8, 2016

[SPARK-18236] Reduce duplicate objects in Spark UI and HistoryServer
JoshRosen
committed
[SPARK-18295][SQL] Make to_json function null safe (matching it to from_json)

HyukjinKwon
authored and
marmbrus
committed
[SPARK-18086] Add support for Hive session vars.

rdblue
authored and
rxin
committed
[SPARK-18261][STRUCTURED STREAMING] Add statistics to MemorySink for joining

lw-lin
authored and
rxin
committed
[SPARK-18217][SQL] Disallow creating permanent views based on temporary views or UDFs

gatorsmile
authored and
rxin
committed
[SPARK-16575][CORE] partition calculation mismatch with sc.binaryFiles

fidato13
authored and
rxin
committed
[SPARK-18207][SQL] Fix a compilation error due to HashExpression.doGenCode

kiszk
authored and
hvanhovell
committed
[SPARK-18137][SQL] Fix RewriteDistinctAggregates UnresolvedException when a UDAF has a foldable TypeCheck

root
authored and
hvanhovell
committed
[SPARK-13770][DOCUMENTATION][ML] Document the ML feature Interaction

chie8842
authored and
srowen
committed
[MINOR][DOC] Unify example marks

zhengruifeng
authored and
srowen
committed
[SPARK-17868][SQL] Do not use bitmasks during parsing and analysis of CUBE/ROLLUP/GROUPING SETS

jiangxb1987
authored and
hvanhovell
committed
[SPARK-18346][SQL] TRUNCATE TABLE should fail if no partition is matched for the given non-partial partition spec
cloud-fan
committed
[SPARK-18191][CORE] Port RDD API to use commit protocol

jiangxb1987
authored and
rxin
committed
[SPARK-18357] Fix yarn files/archive broken issue andd unit tests

kishorvpatil
authored and
Tom Graves
committed
[SPARK-17748][ML] Minor cleanups to one-pass linear regression with elastic net
jkbradley
committed
[SPARK-18280][CORE] Fix potential deadlock in `StandaloneSchedulerBackend.dead`
zsxwing
committed
[SPARK-18342] Make rename failures fatal in HDFSBackedStateStore

brkyvz
authored and
tdas
committed

Commits on Nov 9, 2016

[SPARK-18239][SPARKR] Gradient Boosted Tree for R

felixcheung
authored and
Felix Cheung
committed
[SPARK-18333][SQL] Revert hacks in parquet and orc reader to support case insensitive resolution

ericl
authored and
cloud-fan
committed
[SPARK-18368] Fix regexp_replace with task serialization.

rdblue
authored and
rxin
committed
[SPARK-17659][SQL] Partitioned View is Not Supported By SHOW CREATE TABLE
gatorsmile
committed
[SPARK-18292][SQL] LogicalPlanToSQLSuite should not use resource dependent path for golden file generation

dongjoon-hyun
authored and
srowen
committed
[SPARK-18338][SQL][TEST-MAVEN] Fix test case initialization order under Maven builds

liancheng
authored and
yhuai
committed
[SPARK-16808][CORE] History Server main page does not honor APPLICATION_WEB_PROXY_BASE

vijoshi
authored and
Marcelo Vanzin
committed
Revert "[SPARK-18368] Fix regexp_replace with task serialization."
yhuai
committed
[SPARK-18368][SQL] Fix regexp replace when serialized

rdblue
authored and
yhuai
committed
[SPARK-18370][SQL] Add table information to InsertIntoHadoopFsRelationCommand

hvanhovell
authored and
rxin
committed
[SPARK-18191][CORE][FOLLOWUP] Call `setConf` if `OutputFormat` is `Configurable`.

jiangxb1987
authored and
rxin
committed
[SPARK-17829][SQL] Stable format for offset log

Tyson Condie
authored and
zsxwing
committed

Commits on Nov 10, 2016

[SPARK-18147][SQL] do not fail for very complex aggregator result type
cloud-fan
committed
[MINOR][PYSPARK] Improve error message when running PySpark with different minor versions

viirya
authored and
srowen
committed
[SPARK-18268][ML][MLLIB] ALS fail with better message if ratings is empty rdd

techaddict
authored and
srowen
committed
[SPARK-14914][CORE] Fix Resource not closed after using, for unit tests and example

wangmiao1981
authored and
srowen
committed
[SPARK-18262][BUILD][SQL] JSON.org license is now CatX

srowen
authored and
rxin
committed
[SPARK-17993][SQL] Fix Parquet log output redirection

Michael Allman
authored and
rxin
committed
[SPARK-17990][SPARK-18302][SQL] correct several partition related behaviours of ExternalCatalog

cloud-fan
authored and
rxin
committed
[SPARK-18403][SQL] Temporarily disable flaky ObjectHashAggregateSuite

liancheng
authored and
rxin
committed

Commits on Nov 11, 2016

[SPARK-18185] Fix all forms of INSERT / OVERWRITE TABLE for Datasource tables

ericl
authored and
rxin
committed
[SPARK-18401][SPARKR][ML] SparkR random forest should support output original label.
yanboliang
committed
[SPARK-13331] AES support for over-the-wire encryption

Junjie Chen
authored and
Marcelo Vanzin
committed
[SPARK-17843][WEB UI] Indicate event logs pending for processing on history server UI

vijoshi
authored and
Tom Graves
committed
[SPARK-17982][SQL] SQLBuilder should wrap the generated SQL with parenthesis for LIMIT

dongjoon-hyun
authored and
gatorsmile
committed
[SPARK-18387][SQL] Add serialization to checkEvaluation.

rdblue
authored and
rxin
committed
[SPARK-18264][SPARKR] build vignettes with package, update vignettes for CRAN release build and add info on release

felixcheung
authored and
shivaram
committed

Commits on Nov 12, 2016

Commits on Nov 14, 2016

[SPARK-18426][STRUCTURED STREAMING] Python Documentation Fix for Structured Streaming Programming Guide

Denny Lee
authored and
rxin
committed
[SPARK-18412][SPARKR][ML] Fix exception for some SparkR ML algorithms training on libsvm data
yanboliang
committed
[SPARK-18382][WEBUI] "run at null:-1" in UI when no file/line info in call site info

srowen
authored and
sarutak
committed
[SPARK-18166][MLLIB] Fix Poisson GLM bug due to wrong requirement of response values

actuaryzhang
authored and
srowen
committed
[SPARK-18396][HISTORYSERVER] Duration" column makes search result confused, maybe we should make it unsearchable

WangTaoTheTonic
authored and
srowen
committed
[SPARK-18432][DOC] Changed HDFS default block size from 64MB to 128MB

moomindani
authored and
sarutak
committed
[SPARK-18416][STRUCTURED STREAMING] Fixed temp file leak in state store

tdas
authored and
zsxwing
committed
[SPARK-17510][STREAMING][KAFKA] config max rate on a per-partition basis

koeninger
authored and
zsxwing
committed
[SPARK-11496][GRAPHX][FOLLOWUP] Add param checking for runParallelPersonalizedPageRank

zhengruifeng
authored and
dbtsai
committed
[SPARK-17348][SQL] Incorrect results from subquery transformation

nsyca
authored and
hvanhovell
committed

Commits on Nov 15, 2016

[SPARK-18124] Observed delay based Event Time Watermarks

marmbrus
authored and
tdas
committed
[SPARK-18428][DOC] Update docs for GraphX

zhengruifeng
authored and
rxin
committed
[SPARK-18430][SQL] Fixed Exception Messages when Hitting an Invocation Exception of Function Lookup

gatorsmile
authored and
rxin
committed
[SPARK-18232][MESOS] Support CNI

Michael Gummelt
authored and
rxin
committed
[SPARK-18427][DOC] Update docs of mllib.KMeans

zhengruifeng
authored and
srowen
committed
[SPARK-18300][SQL] Do not apply foldable propagation with expand as a child.

hvanhovell
authored and
gatorsmile
committed
[SPARK-18379][SQL] Make the parallelism of parallelPartitionDiscovery configurable.

genmao.ygm
authored and
yhuai
committed
[SPARK-13027][STREAMING] Added batch time as a parameter to updateStateByKey

Aaditya Ramesh
authored and
zsxwing
committed
[SPARK-18337] Complete mode memory sinks should be able to recover from checkpoints

brkyvz
authored and
tdas
committed
[SPARK-18417][YARN] Define 'spark.yarn.am.port' in yarn config object

weiqingy
authored and
Marcelo Vanzin
committed
[SPARK-18440][STRUCTURED STREAMING] Pass correct query execution to FileFormatWriter

tdas
authored and
marmbrus
committed
[SPARK-18423][STREAMING] ReceiverTracker should close checkpoint dir when stopped even if it was not started

HyukjinKwon
authored and
zsxwing
committed
[SPARK-17732][SQL] ALTER TABLE DROP PARTITION should support comparators

dongjoon-hyun
authored and
hvanhovell
committed

Commits on Nov 16, 2016

[SPARK-18300][SQL] Fix scala 2.10 build for FoldablePropagation

hvanhovell
authored and
zsxwing
committed
[SPARK-18377][SQL] warehouse path should be a static conf

cloud-fan
authored and
rxin
committed
[SPARK-18438][SPARKR][ML] spark.mlp should support RFormula.
yanboliang
committed
[SPARK-18433][SQL] Improve DataSource option keys to be more case-insensitive

dongjoon-hyun
authored and
cloud-fan
committed
[DOC][MINOR] Kafka doc: breakup into lines

lw-lin
authored and
srowen
committed
[SPARK-18400][STREAMING] NPE when resharding Kinesis Stream
srowen
committed
[SPARK-18410][STREAMING] Add structured kafka example

uncleGen
authored and
srowen
committed
[MINOR][DOC] Fix typos in the 'configuration', 'monitoring' and 'sql-programming-guide' documentation

weiqingy
authored and
srowen
committed
[SPARK-18434][ML] Add missing ParamValidations for ML algos

zhengruifeng
authored and
yanboliang
committed
[SPARK-18446][ML][DOCS] Add links to API docs for ML algos

zhengruifeng
authored and
srowen
committed
[SPARK-18420][BUILD] Fix the errors caused by lint check in Java

Xianyang Liu
authored and
srowen
committed
[SPARK-18415][SQL] Weird Plan Output when CTE used in RunnableCommand

gatorsmile
authored and
hvanhovell
committed
[SPARK-18459][SPARK-18460][STRUCTUREDSTREAMING] Rename triggerId to batchId and add triggerDetails to json in StreamingQueryStatus

tdas
authored and
zsxwing
committed
[SPARK-18461][DOCS][STRUCTUREDSTREAMING] Added more information about monitoring streaming queries

tdas
authored and
marmbrus
committed
[SPARK-1267][SPARK-18129] Allow PySpark to be pip installed

holdenk
authored and
JoshRosen
committed
[SPARK-18186] Migrate HiveUDAFFunction to TypedImperativeAggregate for partial aggregation support

liancheng
authored and
yhuai
committed
[YARN][DOC] Increasing NodeManager's heap size with External Shuffle Service

Devian-ua
authored and
rxin
committed

Commits on Nov 17, 2016

[SPARK-18442][SQL] Fix nullability of WrapOption.

ueshin
authored and
cloud-fan
committed
[SPARK-18464][SQL] support old table which doesn't store schema in metastore

cloud-fan
authored and
rxin
committed
[YARN][DOC] Remove non-Yarn specific configurations from running-on-yarn.md

weiqingy
authored and
srowen
committed
[SPARK-18365][DOCS] Improve Sample Method Documentation

bllchmbrs
authored and
srowen
committed
[SPARK-17462][MLLIB]use VersionUtils to parse Spark version strings

VinceShieh
authored and
srowen
committed
[SPARK-18480][DOCS] Fix wrong links for ML guide docs

zhengruifeng
authored and
srowen
committed
[SPARK-18490][SQL] duplication nodename extrainfo for ShuffleExchange

root
authored and
srowen
committed

Commits on Nov 18, 2016

[SPARK-18360][SQL] default table path of tables in default database should depend on the location of default database

cloud-fan
authored and
yhuai
committed
[SPARK-18462] Fix ClassCastException in SparkListenerDriverAccumUpdates event

JoshRosen
authored and
rxin
committed
refine connect and read code
yinxusen
committed
[SPARK-18187][SQL] CompactibleFileStreamLog should not use "compactInterval" direcly with user setting.

Tyson Condie
authored and
zsxwing
committed
[SPARK-18457][SQL] ORC and other columnar formats using HiveShim read all columns when doing a simple count

aray
authored and
rxin
committed
refine test suite of arrow vectors
yinxusen
committed
merge with bryan
yinxusen
committed
[SPARK-18422][CORE] Fix wholeTextFiles test to pass on Windows in JavaAPISuite

HyukjinKwon
authored and
srowen
committed
update supported types
yinxusen
committed

Commits on Nov 19, 2016

Commits on Nov 20, 2016

[SPARK-17062][MESOS] add conf option to mesos dispatcher

skonto
authored and
Marcelo Vanzin
committed
[SPARK-18456][ML][FOLLOWUP] Use matrix abstraction for coefficients in LogisticRegression training

sethah
authored and
dbtsai
committed
[SPARK-18458][CORE] Fix signed integer overflow problem at an expression in RadixSort.java

kiszk
authored and
rxin
committed
[SPARK-18508][SQL] Fix documentation error for DateDiff
rxin
committed
[SQL] Fix documentation for Concat and ConcatWs
rxin
committed
[SPARK-15214][SQL] Code-generation for Generate

hvanhovell
authored and
rxin
committed
[SPARK-3359][BUILD][DOCS] Print examples and disable group and tparam tags in javadoc

HyukjinKwon
authored and
srowen
committed
Fix Mesos build break for Scala 2.10.
rxin
committed

Commits on Nov 21, 2016

[HOTFIX][SQL] Fix DDLSuite failure.
rxin
committed
[SPARK-18467][SQL] Extracts method for preparing arguments from StaticInvoke, Invoke and NewInstance and modify to short circuit if arguments have null when `needNullCheck == true`.

ueshin
authored and
cloud-fan
committed
[SPARK-18282][ML][PYSPARK] Add python clustering summaries for GMM and BKM

sethah
authored and
yanboliang
committed
[SPARK-18398][SQL] Fix nullabilities of MapObjects and ExternalMapToCatalyst.

ueshin
authored and
hvanhovell
committed
[SPARK-18413][SQL] Add `maxConnections` JDBCOption

dongjoon-hyun
authored and
srowen
committed
[SPARK-18361][PYSPARK] Expose RDD localCheckpoint in PySpark

Gabriel Huang
authored and
andrewor14
committed
[SPARK-18517][SQL] DROP TABLE IF EXISTS should not warn for non-existing tables

dongjoon-hyun
authored and
andrewor14
committed
[SPARK-17765][SQL] Support for writing out user-defined type in ORC datasource

HyukjinKwon
authored and
rxin
committed

Commits on Nov 22, 2016

Commits on Nov 23, 2016

Commits on Nov 24, 2016

[SPARK-18510][SQL] Follow up to address comments in #15951

zsxwing
authored and
tdas
committed
[SPARK-18520][ML] Add missing setXXXCol methods for BisectingKMeansModel and GaussianMixtureModel

zhengruifeng
authored and
yanboliang
committed
[SPARK-18578][SQL] Full outer join in correlated subquery returns incorrect results

nsyca
authored and
hvanhovell
committed

Commits on Nov 25, 2016

[SPARK-18575][WEB] Keep same style: adjust the position of driver log links

uncleGen
authored and
srowen
committed
[SPARK-18119][SPARK-CORE] Namenode safemode check is only performed on one namenode which can stuck the startup of SparkHistory server

n.fraison
authored and
srowen
committed
[SPARK-3359][BUILD][DOCS] More changes to resolve javadoc 8 errors that will help unidoc/genjavadoc compatibility

HyukjinKwon
authored and
srowen
committed
[SPARK-18559][SQL] Fix HLL++ with small relative error

wzhfy
authored and
hvanhovell
committed
[SPARK-18356][ML] Improve MLKmeans Performance

ZakariaHili
authored and
srowen
committed
[SPARK-18413][SQL][FOLLOW-UP] Use `numPartitions` instead of `maxConnections`

dongjoon-hyun
authored and
rxin
committed
[SPARK-18436][SQL] isin causing SQL syntax error with JDBC

jiangxb1987
authored and
hvanhovell
committed

Commits on Nov 26, 2016

Commits on Nov 28, 2016

Commits on Nov 29, 2016

[SPARK-18403][SQL] Fix unsafe data false sharing issue in ObjectHashAggregateExec

liancheng
authored and
cloud-fan
committed
[SPARK-18523][PYSPARK] Make SparkContext.stop more reliable

kxepal
authored and
rxin
committed
[SPARK-16282][SQL] Follow-up: remove "percentile" from temp function detection after implementing it natively

lins05
authored and
gatorsmile
committed
[SPARK-18588][SS][KAFKA] Ignore the flaky kafka test
zsxwing
committed
[SPARK-18547][CORE] Propagate I/O encryption key when executors register.

Marcelo Vanzin
authored and
zsxwing
committed
[SPARK-18058][SQL][TRIVIAL] Use dataType.sameResult(...) instead equality on asNullable datatypes

hvanhovell
authored and
rxin
committed
[SPARK-18544][SQL] Append with df.saveAsTable writes data to wrong location

ericl
authored and
rxin
committed
[SPARK-18339][SPARK-18513][SQL] Don't push down current_timestamp for filters in StructuredStreaming and persist batch and watermark timestamps to offset log.

Tyson Condie
authored and
tdas
committed
[SPARK-18188] add checksum for blocks of broadcast

Davies Liu
authored and
zsxwing
committed
[SPARK-3359][DOCS] Make javadoc8 working for unidoc/genjavadoc compatibility in Java API documentation

HyukjinKwon
authored and
srowen
committed
[MINOR][DOCS] Updates to the Accumulator example in the programming guide. Fixed typos, AccumulatorV2 in Java

aokolnychyi
authored and
srowen
committed
[SPARK-18615][DOCS] Switch to multi-line doc to avoid a genjavadoc bug for backticks

HyukjinKwon
authored and
srowen
committed
[SPARK-18592][ML] Move DT/RF/GBT Param setter methods to subclasses

yanboliang
authored and
jkbradley
committed
[SPARK-18498][SQL] Revise HDFSMetadataLog API for better testing

Tyson Condie
authored and
marmbrus
committed
[SPARK-18429][SQL] implement a new Aggregate for CountMinSketch

wangzhenhua
authored and
rxin
committed
[SPARK-18631][SQL] Changed ExchangeCoordinator re-partitioning to avoid more data skew

markhamstra
authored and
yhuai
committed
[SPARK-18614][SQL] Incorrect predicate pushdown from ExistenceJoin

nsyca
authored and
hvanhovell
committed

Commits on Nov 30, 2016

[SPARK-18553][CORE] Fix leak of TaskSetManager following executor loss
JoshRosen
committed
[SPARK-18516][SQL] Split state and progress in streaming

tdas
authored and
marmbrus
committed
[SPARK-18319][ML][QA2.1] 2.1 QA: API: Experimental, DeveloperApi, final, sealed audit

YY-OnCall
authored and
jkbradley
committed
[SPARK-18632][SQL] AggregateFunction should not implement ImplicitCastInputTypes

hvanhovell
authored and
rxin
committed
[SPARK-18145] Update documentation for hive partition management in 2.1

ericl
authored and
rxin
committed
[SPARK-15819][PYSPARK][ML] Add KMeanSummary in KMeans of PySpark

zjffdu
authored and
yanboliang
committed
[SPARK-18516][STRUCTURED STREAMING] Follow up PR to add StreamingQuery.status to Python
tdas
committed
[SPARK-17680][SQL][TEST] Added a Testcase for Verifying Unicode Character Support for Column Names and Comments

gatorsmile
authored and
cloud-fan
committed
[SPARK-18622][SQL] Fix the datatype of the Sum aggregate function

hvanhovell
authored and
cloud-fan
committed
[SPARK-18617][CORE][STREAMING] Close "kryo auto pick" feature for Spark Streaming

uncleGen
authored and
rxin
committed
[SPARK-18366][PYSPARK][ML] Add handleInvalid to Pyspark for QuantileDiscretizer and Bucketizer

techaddict
authored and
Nick Pentreath
committed
[SPARK-18612][MLLIB] Delete broadcasted variable in LBFGS CostFun

Anthony Truchet
authored and
srowen
committed
[SPARK-17897][SQL] Fixed IsNotNull Constraint Inference Rule

gatorsmile
authored and
cloud-fan
committed
[SPARK-17932][SQL] Support SHOW TABLES EXTENDED LIKE 'identifier_with_wildcards' statement

jiangxb1987
authored and
hvanhovell
committed
[SPARK-18220][SQL] read Hive orc table with varchar column should not fail

cloud-fan
authored and
rxin
committed
[SPARK][EXAMPLE] Added missing semicolon in quick-start-guide example

manishAtGit
authored and
andrewor14
committed
[SPARK-18640] Add synchronization to TaskScheduler.runningTasksByExecutors

JoshRosen
authored and
andrewor14
committed
[SPARK-18318][ML] ML, Graph 2.1 QA: API: New Scala APIs, docs

yanboliang
authored and
jkbradley
committed
[SPARK-18251][SQL] the type of Dataset can't be Option of non-flat type

cloud-fan
authored and
liancheng
committed
[SPARK-18546][CORE] Fix merging shuffle spills when using encryption.
Marcelo Vanzin
committed

Commits on Dec 1, 2016

Commits on Dec 2, 2016

add string support
yinxusen
committed

Commits on Dec 3, 2016

fix string type convert
yinxusen
committed