merge#4

Merged

ulysses-you merged 102 commits intoulysses-you:masterfrom

Aug 15, 2019

Commits on Aug 2, 2019

[SPARK-28475][CORE] Add regex MetricFilter to GraphiteSink

nkarpov
authored and
jerryshao
committed
[SPARK-28445][SQL][PYTHON] Fix error when PythonUDF is used in both group by and aggregate expression

viirya
authored and
HyukjinKwon
committed
[SPARK-28582][PYSPARK] Fix flaky test DaemonTests.do_termination_test which fail on Python 3.7

WeichenXu123
authored and
HyukjinKwon
committed
[SPARK-28579][ML] MaxAbsScaler avoids conversion to breeze.vector

zhengruifeng
authored and
srowen
committed
[SPARK-28510][SQL] Implement Spark's own GetFunctionsOperation

wangyum
authored and
gatorsmile
committed
[SPARK-23469][ML] HashingTF should use corrected MurmurHash3 implementation

huaxingao
authored and
srowen
committed
[SPARK-20604][ML] Allow imputer to handle numeric types

authored and
srowen
committed
[SPARK-28519][SQL] Use StrictMath log, pow functions for platform independence
srowen
committed
[MINOR][DOC][SS] Correct description of minPartitions in Kafka option

HeartSaVioR
authored and
dongjoon-hyun
committed
Revert "[SPARK-28582][PYSPARK] Fix flaky test DaemonTests.do_termination_test which fail on Python 3.7"
dongjoon-hyun
committed
[SPARK-28532][SPARK-28530][SQL][FOLLOWUP] Inline doc for FixedPoint(1) batches "Subquery" and "Join Reorder"

gatorsmile
and
yeshengm
committed
[SPARK-28574][CORE] Allow to config different sizes for event queues

yunzoud
authored and
zsxwing
committed
[SPARK-28606][INFRA] Update CRAN key to recover docker image generation

dongjoon-hyun
authored and
dbtsai
committed

Commits on Aug 3, 2019

[SPARK-28582][PYTHON] Fix flaky test DaemonTests.do_termination_test which fail on Python 3.7

WeichenXu123
authored and
HyukjinKwon
committed

Commits on Aug 4, 2019

Commits on Aug 5, 2019

Commits on Aug 6, 2019

Commits on Aug 7, 2019

[SPARK-28615][SQL][DOCS] Add a guide line for dataframe functions to say column signature function is by default

WeichenXu123
authored and
HyukjinKwon
committed
[SPARK-28238][SQL] Implement DESCRIBE TABLE for Data Source V2 Tables

mccheah
authored and
cloud-fan
committed
[SPARK-27924][SQL][TEST][FOLLOW-UP] Enable Boolean-Predicate syntax tests

beliefer
authored and
dongjoon-hyun
committed
[SPARK-28595][SQL] explain should not trigger partition listing
cloud-fan
committed
[SPARK-28472][SQL][TEST] Add test for thriftserver protocol versions

wangyum
authored and
gatorsmile
committed
[SPARK-28583][SQL] Subqueries should not call `onUpdatePlan` in Adaptive Query Execution

maryannxue
authored and
hvanhovell
committed
[SPARK-28470][SQL] Cast to decimal throws ArithmeticException on overflow

mgaido91
authored and
maropu
committed
[SPARK-28331][SQL] Catalogs.load() should be able to load built-in catalogs

gengliangwang
authored and
brkyvz
committed
[SPARK-28617][SQL][TEST] Fix misplacement when comment is at the end of the query

wangyum
authored and
dongjoon-hyun
committed
[SPARK-28649][INFRA] Add Python .eggs to .gitignore

rvesse
authored and
dongjoon-hyun
committed

Commits on Aug 8, 2019

[SPARK-28395][FOLLOW-UP][SQL] Make spark.sql.function.preferIntegralDivision internal

wangyum
authored and
HyukjinKwon
committed
[MINOR][DOC] Fix a typo 'lister' -> 'listener'

lys0716
authored and
HyukjinKwon
committed
[SPARK-28454][PYTHON] Validate LongType in `createDataFrame(verifySchema=True)`

simplylizz
authored and
HyukjinKwon
committed
[SPARK-28474][SQL] Hive 0.12 JDBC client can not handle binary type

wangyum
authored and
HyukjinKwon
committed
[SPARK-28644][SQL] Port HIVE-10646: ColumnValue does not handle NULL_TYPE

wangyum
authored and
HyukjinKwon
committed
[SPARK-28654][SQL] Move "Extract Python UDFs" to the last in optimizer

HyukjinKwon
authored and
cloud-fan
committed
[SPARK-28642][SQL] Hide credentials in SHOW CREATE TABLE

wangyum
authored and
dongjoon-hyun
committed
[SPARK-28660][SQL][TEST] Port AGGREGATES.sql [Part 4]

wangyum
authored and
dongjoon-hyun
committed

Commits on Aug 9, 2019

[SPARK-28572][SQL] Simple analyzer checks for v2 table creation code paths

brkyvz
authored and
cloud-fan
committed
[SPARK-28017][SQL] Support additional levels of truncations by DATE_TRUNC/TRUNC

MaxGekk
authored and
cloud-fan
committed
[SPARK-28565][SQL] DataFrameWriter saveAsTable support for V2 catalogs
brkyvz
committed
[SPARK-28163][SS] Use CaseInsensitiveMap for KafkaOffsetReader

gaborgsomogyi
authored and
cloud-fan
committed
[SPARK-28651][SS] Force the schema of Streaming file source to be nullable

zsxwing
authored and
HyukjinKwon
committed
[SPARK-27889][INFRA] Make development scripts under dev/ support Python 3

WeichenXu123
authored and
HyukjinKwon
committed
[SPARK-28077][SQL][TEST][FOLLOW-UP] Enable Overlay function tests

beliefer
authored and
maropu
committed
[SPARK-27371][CORE] Support GPU-aware resources scheduling in Standalone

Ngone51
authored and
tgravescs
committed
[SPARK-21481][ML][FOLLOWUP] HashingTF Cleanup

zhengruifeng
authored and
srowen
committed
[SPARK-28656][SQL] Support `millennium`, `century` and `decade` at `extract()`

MaxGekk
authored and
dongjoon-hyun
committed
[SPARK-28676][CORE] Avoid Excessive logging from ContextCleaner

ajithme
authored and
dongjoon-hyun
committed

Commits on Aug 10, 2019

[MINOR] Fix typos in comments and replace an explicit type with <>

younggyu chun
authored and
srowen
committed
[SPARK-28677][WEBUI] "Select All" checkbox in StagePage doesn't work properly

sarutak
authored and
srowen
committed
[SPARK-28675][SQL] Remove maskCredentials and use redactOptions

wangyum
authored and
dongjoon-hyun
committed

Commits on Aug 11, 2019

[SPARK-28685][SQL][TEST] Test HMS 2.0.0+ in VersionsSuite/HiveClientSuites on JDK 11

wangyum
authored and
dongjoon-hyun
committed
[SPARK-28639][CORE][DOC] Configuration doc for Barrier Execution Mode

sarutak
authored and
srowen
committed

Commits on Aug 12, 2019

[SPARK-28652][TESTS][K8S] Add python version check for executor

viirya
authored and
HyukjinKwon
committed
[SPARK-28686][SQL][TEST] Move udf_radians from HiveCompatibilitySuite to HiveQuerySuite

wangyum
authored and
dongjoon-hyun
committed
[SPARK-28688][SQL][TEST] Skip `VersionsSuite.read hive materialized view` test for HMS 3.0+ on JDK9+

wangyum
authored and
dongjoon-hyun
committed
[SPARK-28366][CORE][FOLLOW-UP] Refine logging in driver when loading single large unsplittable file

WeichenXu123
authored and
cloud-fan
committed
[SPARK-28538][UI] Document SQL page

zhengruifeng
authored and
srowen
committed
[SPARK-28017][SPARK-28656][SQL][FOLLOW-UP] Restore comments in date.sql

MaxGekk
authored and
dongjoon-hyun
committed
[SPARK-28638][WEBUI] Task summary should only contain successful tasks' metrics

gengliangwang
authored and
Marcelo Vanzin
committed
[SPARK-26969][SQL] Using ODBC client not able to see the query data when column datatype is decimal

sujith71955
authored and
dongjoon-hyun
committed

Commits on Aug 13, 2019

[SPARK-28647][WEBUI] Recover additional metric feature and remove additional-metrics.js

sarutak
authored and
dongjoon-hyun
committed
[SPARK-28280][PYTHON][SQL][TESTS][FOLLOW-UP] Add UDF cases into group by clause in 'udf-group-by.sql'

skonto
authored and
HyukjinKwon
committed
[SPARK-28703][SQL][TEST] Skip HiveExternalCatalogVersionsSuite and 3 tests in HiveSparkSubmitSuite at JDK9+

wangyum
authored and
dongjoon-hyun
committed
[SPARK-28383][SQL] SHOW CREATE TABLE is not supported on a temporary view

wangyum
authored and
dongjoon-hyun
committed
[SPARK-28561][WEBUI] DAG viz for barrier-execution mode

sarutak
authored and
jiangxb1987
committed
[SPARK-28706][SQL] Allow cast null type to any types

jiangxb1987
authored and
cloud-fan
committed
[SPARK-28698][SQL] Support user-specified output schema in `to_avro`

gengliangwang
authored and
cloud-fan
committed
[SPARK-28422][SQL][PYTHON] GROUPED_AGG pandas_udf should work without group by clause

viirya
authored and
HyukjinKwon
committed
[SPARK-28714][SQL][TEST] Add `hive.aux.jars.path` test for spark-sql shell

wangyum
authored and
dongjoon-hyun
committed
[SPARK-28713][BUILD] Bump checkstyle from 8.14 to 8.23

Fokko
authored and
dongjoon-hyun
committed
[SPARK-28708][SQL] IsolatedClientLoader will not load hive classes from application jars on JDK9+

wangyum
authored and
dongjoon-hyun
committed
[SPARK-28718][SQL] Support `field` synonyms at `extract`

MaxGekk
authored and
dongjoon-hyun
committed
[SPARK-28700][SQL] Use DECIMAL type for `sec` in `make_timestamp()`

MaxGekk
authored and
dongjoon-hyun
committed
[SPARK-28719][BUILD] Enable Github Actions for master
dbtsai
committed
[SPARK-28720][BUILD][R] Update AppVeyor R version to 3.6.1

dongjoon-hyun
authored and
dbtsai
committed
[SPARK-27768][SQL] Support Infinity/NaN-related float/double literals case-insensitively

dilipbiswal
authored and
dongjoon-hyun
committed

Commits on Aug 14, 2019

[SPARK-28487][K8S] More responsive dynamic allocation with K8S
Marcelo Vanzin
committed
[SPARK-28719][BUILD] [FOLLOWUP] Add JDK11 for Github Actions
dbtsai
committed
[SPARK-28265][SQL] Add renameTable to TableCatalog API

edgarRd
authored and
cloud-fan
committed
[SPARK-27739][SQL] df.persist should save stats from optimized plan

jzhuge
authored and
cloud-fan
committed
[SPARK-28351][SQL] Support DELETE in DataSource V2

xy_xin
authored and
cloud-fan
committed
[SPARK-28687][SQL] Support `epoch`, `isoyear`, `milliseconds` and `microseconds` at `extract()`

MaxGekk
authored and
dongjoon-hyun
committed

Commits on Aug 15, 2019

[SPARK-28203][CORE][PYTHON] PythonRDD should respect SparkContext's hadoop configuration

advancedxy
authored and
HyukjinKwon
committed
[SPARK-28666] Support saveAsTable for V2 tables through Session Catalog

brkyvz
authored and
cloud-fan
committed
[SPARK-28695][SS] Use CaseInsensitiveMap in KafkaSourceProvider to make source param handling more robust

gaborgsomogyi
authored and
cloud-fan
committed
[SPARK-27592][SQL] Set the bucketed data source table SerDe correctly

wangyum
authored and
cloud-fan
committed
[SPARK-28543][DOCS][WEBUI] Document Spark Jobs page

planga82
authored and
srowen
committed