[SPARK-24544][SQL] Print actual failure cause when look up function failed#21552

Closed

caneGuy wants to merge 194 commits intoapache:masterfrom

caneGuy:zhoukang/print-warning

Commits on Jun 14, 2018

[MINOR][CORE][TEST] Remove unnecessary sort in UnsafeInMemorySorterSuite

jiangxb1987
authored and
HyukjinKwon
committed
[SPARK-24495][SQL] EnsureRequirement returns wrong plan when reordering equal keys

mgaido91
authored and
gatorsmile
committed
[SPARK-24563][PYTHON] Catch TypeError when testing existence of HiveConf when creating pysp…

icexelloss
authored and
Marcelo Vanzin
committed
[SPARK-24543][SQL] Support any type as DDL string for from_json's schema

MaxGekk
authored and
cloud-fan
committed
[SPARK-24319][SPARK SUBMIT] Fix spark-submit execution where no main class is required.

gaborgsomogyi
authored and
Marcelo Vanzin
committed
[SPARK-24248][K8S] Use level triggering and state reconciliation in scheduling and lifecycle
mccheah
committed

Commits on Jun 15, 2018

[SPARK-24478][SQL] Move projection and filter push down to physical conversion

rdblue
authored and
cloud-fan
committed
[PYTHON] Fix typo in serializer exception

rberenguel
authored and
HyukjinKwon
committed
[SPARK-24490][WEBUI] Use WebUI.addStaticHandler in web UIs

jaceklaskowski
authored and
Marcelo Vanzin
committed
[SPARK-24396][SS][PYSPARK] Add Structured Streaming ForeachWriter for python
tdas
committed
[SPARK-24452][SQL][CORE] Avoid possible overflow in int add or multiple

kiszk
authored and
cloud-fan
committed
[SPARK-24525][SS] Provide an option to limit number of rows in a MemorySink

Mukul Murthy
authored and
brkyvz
committed

Commits on Jun 16, 2018

add one supported type missing from the javadoc

ispot-james
authored and
rxin
committed

Commits on Jun 18, 2018

Commits on Jun 19, 2018

Commits on Jun 20, 2018

[SPARK-23778][CORE] Avoid unneeded shuffle when union gets an empty RDD

mgaido91
authored and
cloud-fan
committed
[MINOR][SQL] Remove invalid comment from SparkStrategies

HeartSaVioR
authored and
hvanhovell
committed
[SPARK-24575][SQL] Prohibit window expressions inside WHERE and HAVING clauses

aokolnychyi
authored and
hvanhovell
committed
[SPARK-24578][CORE] Cap sub-region's size of returned nio buffer

WenboZhao
authored and
zsxwing
committed

Commits on Jun 21, 2018

[SPARK-24547][K8S] Allow for building spark on k8s docker images without cache and don't forget to push spark-py container.

Ray Burgemeestre
authored and
Anirudh Ramanathan
committed
[SPARK-23912][SQL] add array_distinct

huaxingao
authored and
ueshin
committed
[SPARK-24571][SQL] Support Char literals

MaxGekk
authored and
gatorsmile
committed
[SPARK-24574][SQL] array_contains, array_position, array_remove and element_at functions deal with Column type

chongguang
authored and
HyukjinKwon
committed
[SPARK-24614][PYSPARK] Fix for SyntaxWarning on tests.py

rekhajoshm
authored and
HyukjinKwon
committed
[SPARK-16630][YARN] Blacklist a node if executors won't launch on it

attilapiros
authored and
squito
committed
[SPARK-24589][CORE] Correctly identify tasks in output commit coordinator.

Marcelo Vanzin
authored and
tgravescs
committed
[SPARK-24613][SQL] Cache with UDF could not be matched with subsequent dependent caches

maryannxue
authored and
gatorsmile
committed
[SPARK-24588][SS] streaming join should require HashClusteredPartitioning from children

cloud-fan
authored and
gatorsmile
committed

Commits on Jun 22, 2018

[SPARK-23934][SQL] Adding map_from_entries function

mn-mikke
authored and
ueshin
committed
[SPARK-24519] Make the threshold for highly compressed map status configurable

Hieu Huynh
authored and
tgravescs
committed
[SPARK-24518][CORE] Using Hadoop credential provider API to store password

jerryshao
authored and
Marcelo Vanzin
committed
[SPARK-24372][BUILD] Add scripts to help with preparing releases.

Marcelo Vanzin
authored and
tgravescs
committed

Commits on Jun 24, 2018

[SPARK-24190][SQL] Allow saving of JSON files in UTF-16 and UTF-32

MaxGekk
authored and
gatorsmile
committed
[SPARK-24206][SQL] Improve FilterPushdownBenchmark benchmark code

maropu
authored and
gatorsmile
committed
[SPARK-24324][PYTHON] Pandas Grouped Map UDF should assign result columns by name

BryanCutler
authored and
HyukjinKwon
committed

Commits on Jun 25, 2018

[SPARK-24327][SQL] Verify and normalize a partition column name based on the JDBC resolved schema

maropu
authored and
gatorsmile
committed
[SPARK-23931][SQL][FOLLOW-UP] Make `arrays_zip` in function.scala `@scala.annotation.varargs`.

ueshin
authored and
gatorsmile
committed
Fix minor typo in docs/cloud-integration.md

Jim Kleckner
authored and
HyukjinKwon
committed
[SPARK-24596][SQL] Non-cascading Cache Invalidation

maryannxue
authored and
gatorsmile
committed
[SPARK-24633][SQL] Fix codegen when split is required for arrays_zip

mgaido91
authored and
cloud-fan
committed
[SPARK-24648][SQL] SqlMetrics should be threadsafe

dbkerkela
authored and
hvanhovell
committed
[INFRA] Close stale PR.
Marcelo Vanzin
committed
[SPARK-24552][CORE][SQL] Use task ID instead of attempt number for writes.
Marcelo Vanzin
committed

Commits on Jun 26, 2018

[SPARK-24324][PYTHON][FOLLOWUP] Grouped Map positional conf should have deprecation note
BryanCutler
committed
[SPARK-23776][DOC] Update instructions for running PySpark after building with SBT

bersprockets
authored and
HyukjinKwon
committed
[SPARK-24418][BUILD] Upgrade Scala to 2.11.12 and 2.12.6

dbtsai
authored and
jerryshao
committed
[SPARK-24636][SQL] Type coercion of arrays for array_join function

mn-mikke
authored and
HyukjinKwon
committed
[SPARK-24658][SQL] Remove workaround for ANTLR bug

wangyum
authored and
gatorsmile
committed
[SPARK-24423][SQL] Add a new option for JDBC sources

dilipbiswal
authored and
gatorsmile
committed
[SPARK-6237][NETWORK] Network-layer changes to allow stream upload.

squito
authored and
Marcelo Vanzin
committed

Commits on Jun 27, 2018

[SPARK-24659][SQL] GenericArrayData.equals should respect element type differences

rednaxelafx
authored and
cloud-fan
committed
[SPARK-24605][SQL] size(null) returns null instead of -1

MaxGekk
authored and
cloud-fan
committed
[SPARK-23927][SQL] Add "sequence" expression

wajda
authored and
ueshin
committed
[SPARK-23927][SQL][FOLLOW-UP] Fix a build failure.

ueshin
authored and
cloud-fan
committed
[SPARK-21335][DOC] doc changes for disallowed un-aliased subquery use case

cnZach
authored and
cloud-fan
committed
[SPARK-24215][PYSPARK][FOLLOW UP] Implement eager evaluation for DataFrame APIs in PySpark

xuanyuanking
authored and
gatorsmile
committed
[SPARK-24446][YARN] Properly quote library path for YARN.
Marcelo Vanzin
committed
[SPARK-21687][SQL] Spark SQL should set createTime for Hive partition

debugger87
authored and
gatorsmile
committed
[SPARK-24660][SHS] Show correct error pages when downloading logs

mgaido91
authored and
Marcelo Vanzin
committed
[SPARK-24533] Typesafe rebranded to lightbend. Changing the build downloads path

Sanket Chintapalli
authored and
Marcelo Vanzin
committed
[SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFileFormat

maropu
authored and
gatorsmile
committed
[SPARK-24553][WEB-UI] http 302 fixes for href redirect

SJKallman
authored and
gatorsmile
committed

Commits on Jun 28, 2018

[SPARK-24645][SQL] Skip parsing when csvColumnPruning enabled and partitions scanned only

maropu
authored and
HyukjinKwon
committed
[SPARK-24206][SQL][FOLLOW-UP] Update DataSourceReadBenchmark benchmark results

maropu
authored and
HyukjinKwon
committed
[SPARK-24603][SQL] Fix findTightestCommonType reference in comments

Fokko Driesprong
authored and
HyukjinKwon
committed
[SPARK-24564][TEST] Add test suite for RecordBinaryComparator

jiangxb1987
authored and
cloud-fan
committed
[SPARK-14712][ML] LogisticRegressionModel.toString should summarize model

jiayue-zhang
authored and
holdenk
committed
[SPARK-23120][PYSPARK][ML] Add basic PMML export support to PySpark
holdenk
committed
[SPARK-24408][SQL][DOC] Move abs function to math_funcs group

jaceklaskowski
authored and
holdenk
committed
[SPARK-24439][ML][PYTHON] Add distanceMeasure to BisectingKMeans in PySpark

huaxingao
authored and
BryanCutler
committed
[SPARK-24386][SS] coalesce(1) aggregates in continuous processing

jose-torres
authored and
tdas
committed

Commits on Jun 29, 2018

[SPARK-24566][CORE] Fix spark.storage.blockManagerSlaveTimeoutMs default config

xueyumusic
authored and
zsxwing
committed

Commits on Jun 30, 2018

[SPARK-24638][SQL] StringStartsWith support push down

wangyum
authored and
cloud-fan
committed
[SPARK-24696][SQL] ColumnPruning rule fails to remove extra Project

maryannxue
authored and
gatorsmile
committed
simplify rand in dsl/package.scala
gatorsmile
committed

Commits on Jul 1, 2018

[SPARK-24654][BUILD] Update, fix LICENSE and NOTICE, and specialize for source vs binary
srowen
committed

Commits on Jul 2, 2018

[SPARK-24665][PYSPARK] Use SQLConf in PySpark to manage all sql configs

xuanyuanking
authored and
HyukjinKwon
committed
[SPARK-24715][BUILD] Override jline version as 2.14.3 in SBT

viirya
authored and
HyukjinKwon
committed
[SPARK-24507][DOCUMENTATION] Update streaming guide

rekhajoshm
authored and
HyukjinKwon
committed
[SPARK-24683][K8S] Fix k8s no resource
mccheah
committed
[SPARK-24428][K8S] Fix unused code

Stavros Kontopoulos
authored and
Anirudh Ramanathan
committed

Commits on Jul 3, 2018

Commits on Jul 4, 2018

[SPARK-24709][SQL] schema_of_json() - schema inference from an example

MaxGekk
authored and
HyukjinKwon
committed
[SPARK-23698] Remove raw_input() from Python 2

cclauss
authored and
HyukjinKwon
committed
[BUILD] Close stale PRs

srowen
authored and
HyukjinKwon
committed
[SPARK-24732][SQL] Type coercion between MapTypes.

ueshin
authored and
HyukjinKwon
committed
[SPARK-24704][WEBUI] Fix the order of stages in the DAG graph

stanzhai
authored and
hvanhovell
committed
[SPARK-24727][SQL] Add a static config to control cache size for generated classes

maropu
authored and
cloud-fan
committed
[SPARK-24716][SQL] Refactor ParquetFilters

wangyum
authored and
cloud-fan
committed
[SPARK-24635][SQL] Remove Blocks class from JavaCode class hierarchy

viirya
authored and
cloud-fan
committed
[MINOR][ML] Minor correction in the powerIterationSuite

shahidki31
authored and
srowen
committed

Commits on Jul 5, 2018

[SPARK-22384][SQL][FOLLOWUP] Refine partition pruning when attribute is wrapped in Cast

cloud-fan
authored and
gatorsmile
committed
[SPARK-17213][SPARK-17213][FOLLOW-UP] Improve the test of

gatorsmile
authored and
HyukjinKwon
committed
[SPARK-24698][PYTHON] Fixed typo in pyspark.ml's Identifiable class.

mcteo
authored and
HyukjinKwon
committed
[SPARK-24673][SQL] scala sql function from_utc_timestamp second argument could be Column instead of String

agilelab-tmnd1991
authored and
HyukjinKwon
committed
[SPARK-24361][SQL] Polish code block manipulation API

viirya
authored and
cloud-fan
committed
[SPARK-23820][CORE] Enable use of long form of callsite in logs

michaelmior
authored and
srowen
committed
[SPARK-24711][K8S] Fix tags for integration tests

Stavros Kontopoulos
authored and
srowen
committed
[SPARK-24743][EXAMPLES] Update the JavaDirectKafkaWordCount example to support the new API of kafka

cluo512
authored and
koeninger
committed
[SPARK-24675][SQL] Rename table: validate existence of new location

gengliangwang
authored and
gatorsmile
committed
[SPARK-24694][K8S] Pass all app args to integration tests

Stavros Kontopoulos
authored and
srowen
committed

Commits on Jul 6, 2018

[SPARK-24737][SQL] Type coercion between StructTypes.

ueshin
authored and
HyukjinKwon
committed
[SPARK-24692][TESTS] Improvement FilterPushdownBenchmark

wangyum
authored and
HyukjinKwon
committed
[SPARK-24535][SPARKR] fix tests on java check error

felixcheung
authored and
Felix Cheung
committed
[SPARK-24673][SQL][PYTHON][FOLLOWUP] Support Column arguments in timezone of from_utc_timestamp/to_utc_timestamp

maropu
authored and
HyukjinKwon
committed

Commits on Jul 7, 2018

Commits on Jul 11, 2018

[SPARK-24730][SS] Add policy to choose max as global watermark when streaming query has multiple watermarks
tdas
committed
[SPARK-24530][PYTHON] Add a control to force Python version in Sphinx via environment variable, SPHINXPYTHON
HyukjinKwon
committed
[SPARK-24165][SQL] Fixing conditional expressions to handle nullability of nested types

mn-mikke
authored and
cloud-fan
committed
[SPARK-23529][K8S] Support mounting volumes

Andrew Korzhuev
authored and
Felix Cheung
committed
[SPARK-23461][R] vignettes should include model predictions for some ML models

huaxingao
authored and
Felix Cheung
committed
[SPARK-24562][TESTS] Support different configs for same test in SQLQueryTestSuite

mgaido91
authored and
cloud-fan
committed
[SPARK-24208][SQL] Fix attribute deduplication for FlatMapGroupsInPandas

mgaido91
authored and
gatorsmile
committed
[SPARK-24470][CORE] RestSubmissionClient to be robust against 404 & non json responses

rekhajoshm
authored and
srowen
committed
[SPARK-23254][ML] Add user guide entry and example for DataFrame multivariate summary

WeichenXu123
authored and
srowen
committed
[SPARK-24697][SS] Fix the reported start offsets in streaming query progress
tdas
committed

Commits on Jul 12, 2018

[SPARK-24782][SQL] Simplify conf retrieval in SQL expressions

mgaido91
authored and
gatorsmile
committed
[SPARK-24761][SQL] Adding of isModifiable() to RuntimeConfig

MaxGekk
authored and
gatorsmile
committed
[SPARK-24529][BUILD][TEST-MAVEN] Add spotbugs into maven build process

kiszk
authored and
HyukjinKwon
committed
[SPARK-23914][SQL] Add array_union function

kiszk
authored and
ueshin
committed
Refine comment
caneGuy
committed
Refine comment
caneGuy
committed
Update
caneGuy
committed
[SPARK-24691][SQL] Dispatch the type support check in FileFormat implementation

gengliangwang
authored and
cloud-fan
committed
[SPARK-20168][STREAMING KINESIS] Setting the timestamp directly would cause exception on …

yashs360
authored and
srowen
committed
[SPARK-24610] fix reading small files via wholeTextFiles

dhruve
authored and
tgravescs
committed
[SPARK-24768][SQL] Have a built-in AVRO data source implementation

gengliangwang
authored and
gatorsmile
committed
[SPARK-23007][SQL][TEST] Add read schema suite for file-based data sources

dongjoon-hyun
authored and
gatorsmile
committed
[SPARK-24208][SQL][FOLLOWUP] Move test cases to proper locations

mgaido91
authored and
gatorsmile
committed
[SPARK-24790][SQL] Allow complex aggregate expressions in Pivot

maryannxue
authored and
gatorsmile
committed

Commits on Jul 13, 2018

[SPARK-24537][R] Add array_remove / array_zip / map_from_arrays / array_distinct

huaxingao
authored and
HyukjinKwon
committed
[SPARK-23486] cache the function name from the external catalog for lookupFunctions

kevinyu98
authored and
gatorsmile
committed
[SPARK-24713] AppMatser of spark streaming kafka OOM if there are hund…

Yuanbo Liu
authored and
koeninger
committed
[SPARK-24781][SQL] Using a reference from Dataset in Filter/Sort might not work

viirya
authored and
gatorsmile
committed
[SPARK-24776][SQL] Avro unit test: use SQLTestUtils and replace deprecated methods

gengliangwang
authored and
gatorsmile
committed
Revert "[SPARK-24776][SQL] Avro unit test: use SQLTestUtils and replace deprecated methods"
gatorsmile
committed
[SPARK-23528][ML] Add numIter to ClusteringSummary

mgaido91
authored and
holdenk
committed
[SPARK-23831][SQL] Add org.apache.derby to IsolatedClientLoader

wangyum
authored and
gatorsmile
committed

Commits on Jul 14, 2018

Commits on Jul 15, 2018

Commits on Jul 16, 2018

[SPARK-24813][TESTS][HIVE][HOTFIX] HiveExternalCatalogVersionsSuite still flaky; fall back to Apache archive

srowen
authored and
HyukjinKwon
committed
[TRIVIAL][ML] GMM unpersist RDD after training

郑瑞峰
authored and
Felix Cheung
committed
[SPARK-24676][SQL] Project required data from CSV parsed data when column pruning disabled

maropu
authored and
gatorsmile
committed
[SPARK-24810][SQL] Fix paths to test files in AvroSuite

MaxGekk
authored and
gatorsmile
committed
[SPARK-24558][CORE] wrong Idle Timeout value is used in case of the cacheBlock.

sandeep-katta
authored and
cloud-fan
committed
[SPARK-24549][SQL] Support Decimal type push down to the parquet data sources

wangyum
authored and
cloud-fan
committed
[SPARK-18230][MLLIB] Throw a better exception, if the user or product doesn't exist

shahidki31
authored and
srowen
committed
[SPARK-24734][SQL] Fix type coercions and nullabilities of nested data types of some functions.

ueshin
authored and
cloud-fan
committed
[SPARK-23901][SQL] Removing masking functions

mn-mikke
authored and
gatorsmile
committed
[SPARK-24805][SQL] Do not ignore avro files without extensions by default

MaxGekk
authored and
gatorsmile
committed
[SPARK-24402][SQL] Optimize `In` expression when only one element in the collection or collection is empty

dbtsai
authored and
gatorsmile
committed

Commits on Jul 17, 2018