merge#3

Merged

ulysses-you merged 102 commits intoulysses-you:masterfrom

Aug 2, 2019

Commits on Jul 19, 2019

[SPARK-28287][SQL][PYTHON][TESTS] Convert and port 'udaf.sql' into UDF test base

vinodkc
authored and
HyukjinKwon
committed
[SPARK-28285][SQL][PYTHON][TESTS] Convert and port 'outer-join.sql' into UDF test base

huaxingao
authored and
HyukjinKwon
committed
[SPARK-28277][SQL][PYTHON][TESTS] Convert and port 'except.sql' into UDF test base

huaxingao
authored and
HyukjinKwon
committed
[SPARK-28389][SQL][FOLLOW-UP] Use one example in 'add_months' behavior change at migration guide
HyukjinKwon
committed
[SPARK-27707][SQL] Prune unnecessary nested fields from Generate

viirya
and
dongjoon-hyun
committed
[SPARK-28440][MLLIB][TEST] Use TestingUtils to compare floating point values

Ievgen Prokhorenko
authored and
dongjoon-hyun
committed
[SPARK-28284][SQL][PYTHON][TESTS] Convert and port 'join-empty-relation.sql' into UDF test base

imback82
authored and
HyukjinKwon
committed
[SPARK-28199][SS][FOLLOWUP] Remove unnecessary annotations for private API

HeartSaVioR
authored and
dongjoon-hyun
committed
[SPARK-28433][SQL][TEST] Remove hardware-dependent `0.0/0.0` and NaN comparison assertions

huangtianhua
authored and
dongjoon-hyun
committed

Commits on Jul 21, 2019

[SPARK-28189][SQL][FOLLOW-UP] Remove the unnecessary test in DataFrameSuite

maropu
authored and
dongjoon-hyun
committed
[SPARK-28464][DOC][SS] Document Kafka source minPartitions option

Arun Pandian
authored and
dongjoon-hyun
committed
[SPARK-28462][SQL][TEST] Add a prefix '*' to non-nullable attribute names in PlanTestBase.comparePlans failures

maropu
authored and
dongjoon-hyun
committed

Commits on Jul 22, 2019

[SPARK-25705][DOC][FOLLOWUP] Recover links to structured-streaming-kafka-integration

dongjoon-hyun
authored and
HyukjinKwon
committed
[SPARK-28369][SQL] Honor spark.sql.decimalOperations.nullOnOverflow in ScalaUDF result

mgaido91
authored and
cloud-fan
committed
[SPARK-27724][SQL] Implement REPLACE TABLE and REPLACE TABLE AS SELECT with V2

mccheah
authored and
cloud-fan
committed
[SPARK-28456][SQL] Add a public API `Encoder.makeCopy` to allow creating Encoder without touching Scala Reflection

zsxwing
authored and
cloud-fan
committed
[SPARK-28321][DOCS][FOLLOW-UP] Update migration guide by 0-args Java UDF's internal behaviour change

HyukjinKwon
authored and
cloud-fan
committed
[SPARK-28280][SQL][PYTHON][TESTS] Convert and port 'group-by.sql' into UDF test base

skonto
authored and
HyukjinKwon
committed
[SPARK-28455][CORE] Avoid overflow when calculating executor timeout time

Marcelo Vanzin
authored and
dongjoon-hyun
committed
[SPARK-28432][SQL] Add `make_date` function

MaxGekk
authored and
dongjoon-hyun
committed

Commits on Jul 23, 2019

Commits on Jul 24, 2019

[SPARK-28391][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_implicit.sql' into UDF test base

Udbhav30
authored and
HyukjinKwon
committed
[SPARK-27234][SS][PYTHON] Use InheritableThreadLocal for current epoch in EpochTracker (to support Python UDFs)
HyukjinKwon
committed
[SPARK-28421][ML] SparseVector.apply performance optimization

zhengruifeng
authored and
srowen
committed
[SPARK-28435][SQL] Support accepting the interval keyword in the schema string

wangyum
authored and
dongjoon-hyun
committed
[SPARK-28390][SQL][PYTHON][TESTS] Convert and port 'pgSQL/select_having.sql' into UDF test base

shivusondur
authored and
HyukjinKwon
committed
[SPARK-28293][SQL] Implement Spark's own GetTableTypesOperation

wangyum
authored and
gatorsmile
committed
[SPARK-28496][INFRA] Use branch name instead of tag during dry-run

dongjoon-hyun
authored and
Marcelo Vanzin
committed
[SPARK-28465][K8S] Fix integration tests which fail due to missing ceph-nano image

skonto
authored and
Marcelo Vanzin
committed

Commits on Jul 25, 2019

[SPARK-28497][SQL] Disallow upcasting complex data types to string type

gengliangwang
authored and
HyukjinKwon
committed
[SPARK-28288][SQL][PYTHON][TESTS] Convert and port 'window.sql' into UDF test base

younggyu chun
authored and
HyukjinKwon
committed
[SPARK-28511][INFRA] Get REV from RELEASE_VERSION instead of VERSION
dongjoon-hyun
committed
[SPARK-27845][SQL] DataSourceV2: InsertTable

authored and
brkyvz
committed

Commits on Jul 26, 2019

[MINOR][SQL] Fix log messages of DataWritingSparkTask

dongjoon-hyun
authored and
HyukjinKwon
committed
[SPARK-28289][SQL][PYTHON][TESTS] Convert and port 'union.sql' into UDF test base

yiheng
authored and
HyukjinKwon
committed
[SPARK-28365][ML] Fallback locale to en_US in StopWordsRemover if system default locale isn't in available locales in JVM

viirya
authored and
HyukjinKwon
committed
[SPARK-28237][SQL] Enforce Idempotence for Once batches in RuleExecutor

yeshengm
authored and
gatorsmile
committed
[SPARK-28489][SS] Fix a bug that KafkaOffsetRangeCalculator.getRanges may drop offsets

zsxwing
authored and
dongjoon-hyun
committed
[SPARK-28499][ML] Optimize MinMaxScaler

zhengruifeng
authored and
srowen
committed
[SPARK-28524][SQL] Fix ThriftServerTab lost error message

wangyum
authored and
gatorsmile
committed
[SPARK-28507][ML][PYSPARK] Remove deprecated API context(self, sqlContext) from pyspark/ml/util.py

huaxingao
authored and
srowen
committed
[SPARK-28463][SQL] Thriftserver throws BigDecimal incompatible with HiveDecimal

wangyum
authored and
gatorsmile
committed
[SPARK-25285][CORE] Add startedTasks and finishedTasks to the metrics system in the executor instance

LucaCanali
authored and
Marcelo Vanzin
committed
[SPARK-28518][SQL][TEST] Refer to ChecksumFileSystem#isChecksumFile to fix StatisticsCollectionTestBase#getDataSize

wangyum
authored and
dongjoon-hyun
committed

Commits on Jul 27, 2019

Commits on Jul 28, 2019

Commits on Jul 29, 2019

[SPARK-28549][BUILD][CORE][SQL] Use `text.StringEscapeUtils` instead `lang3.StringEscapeUtils`

dongjoon-hyun
authored and
HyukjinKwon
committed
[SPARK-28471][SQL] Replace `yyyy` by `uuuu` in date-timestamp patterns without era

MaxGekk
authored and
dongjoon-hyun
committed
[SPARK-28534][K8S][TEST] Update node affinity for DockerForDesktop backend in PVTestsSuite

dongjoon-hyun
authored and
HyukjinKwon
committed
[SPARK-28086][SQL] Add a function alias `random` for Rand

dongjoon-hyun
authored and
HyukjinKwon
committed
[MINOR] Trivial cleanups

dongjinleekr
authored and
HyukjinKwon
committed
[SPARK-28042][K8S] Support using volume mount as local storage

authored and
Marcelo Vanzin
committed
[SPARK-28550][K8S][TESTS] Unset SPARK_HOME environment variable in K8S integration preparation

HyukjinKwon
authored and
dongjoon-hyun
committed
[SPARK-28459][SQL] Add `make_timestamp` function

MaxGekk
authored and
dongjoon-hyun
committed

Commits on Jul 30, 2019

[SPARK-28556][SQL] QueryExecutionListener should also notify Error

zsxwing
authored and
HyukjinKwon
committed
[SPARK-28326][SQL][TEST] Port join.sql

wangyum
authored and
dongjoon-hyun
committed
[SPARK-28406][SQL][TEST] Port union.sql

wangyum
authored and
dongjoon-hyun
committed
[SPARK-28178][SQL] DataSourceV2: DataFrameWriter.insertInfo

jzhuge
authored and
cloud-fan
committed
[SPARK-28071][SQL][TEST] Port strings.sql

wangyum
authored and
maropu
committed
[SPARK-26755][SCHEDULER] : Optimize Spark Scheduler to dequeue speculative tasks…

authored and
squito
committed
[SPARK-28399][ML][PYTHON] implement RobustScaler

zhengruifeng
authored and
srowen
committed
[SPARK-28525][DEPLOY] Allow Launcher to be applied Java options

sarutak
authored and
Marcelo Vanzin
committed
[SPARK-28209][CORE][SHUFFLE] Proposed new shuffle writer API

mccheah
authored and
Marcelo Vanzin
committed
[SPARK-28375][SQL] Make pullupCorrelatedPredicate idempotent

dilipbiswal
authored and
gatorsmile
committed

Commits on Jul 31, 2019

[SPARK-26175][PYTHON] Redirect the standard input of the forked child to devnull in daemon

WeichenXu123
authored and
HyukjinKwon
committed
[SPARK-28038][SQL][TEST] Port text.sql

wangyum
authored and
HyukjinKwon
committed
[MINOR][CORE][DOCS] Fix inconsistent description of showConsoleProgress

beliefer
authored and
HyukjinKwon
committed
[SPARK-25382][SQL][PYSPARK] Remove ImageSchema.readImages in 3.0

WeichenXu123
authored and
HyukjinKwon
committed
[SPARK-27924][SQL] Support ANSI SQL Boolean-Predicate syntax

beliefer
authored and
dongjoon-hyun
committed
[SPARK-28153][PYTHON] Use AtomicReference at InputFileBlockHolder (to support input_file_name with Python UDF)

HyukjinKwon
authored and
cloud-fan
committed
[SPARK-28581][SQL] Replace _FUNC_ in UDF ExpressionInfo

wangyum
authored and
dongjoon-hyun
committed
[SPARK-28564][CORE] Access history application defaults to the last attempt id

cxzl25
authored and
Marcelo Vanzin
committed

Commits on Aug 1, 2019

[SPARK-24352][CORE][TESTS] De-flake StandaloneDynamicAllocationSuite blacklist test

Marcelo Vanzin
authored and
dongjoon-hyun
committed
[SPARK-26218][SQL] Overflow on arithmetic operations returns incorrect result

mgaido91
authored and
cloud-fan
committed
[SPARK-28366][CORE] Logging in driver when loading single large unsplittable file

WeichenXu123
authored and
cloud-fan
committed
[SPARK-26329][CORE] Faster polling of executor memory metrics.

wypoon
authored and
squito
committed
[SPARK-25584][ML][DOC] datasource for libsvm user guide

zhengruifeng
authored and
srowen
committed
[SPARK-28586][INFRA] Make merge-spark-pr script compatible with Python 3

HyukjinKwon
authored and
dongjoon-hyun
committed
[SPARK-28568][SHUFFLE][DOCS] Make Javadoc in org.apache.spark.shuffle.api visible

HyukjinKwon
authored and
dongjoon-hyun
committed
[SPARK-28584][CORE] Fix thread safety issue in blacklist timer, tests

Marcelo Vanzin
authored and
dongjoon-hyun
committed
[SPARK-28521][SQL] Fix error message for built-in functions

wangyum
authored and
srowen
committed