SPARK-22256 Updates#1

Open

dmcwhorter wants to merge 7069 commits intopmackles:paul-SPARK-22256from

dmcwhorter:dmcwhorter-SPARK-22256

This pull request is big! We're only showing the most recent 250 commits

Commits on Nov 17, 2020

[SPARK-33407][PYTHON] Simplify the exception message from Python UDFs (disabled by default)
HyukjinKwon
committed
[SPARK-33416][SQL] Avoid Hive metastore stack overflow when InSet predicate have many values

wangyum
authored and
cloud-fan
committed
[SPARK-33427][SQL] Add subexpression elimination for interpreted expression evaluation

viirya
authored and
cloud-fan
committed
[SPARK-32222][K8S][TESTS] Add K8s IT for conf propagation

ScrapCodes
authored and
dongjoon-hyun
committed
[SPARK-33471][K8S][BUILD] Upgrade kubernetes-client to 4.12.0

ramesh-muthusamy
authored and
dongjoon-hyun
committed

Commits on Nov 18, 2020

[MINOR][SQL][DOCS] Update schema_of_csv and schema_of_json doc

viirya
authored and
HyukjinKwon
committed
[SPARK-31069][CORE] Avoid repeat compute `chunksBeingTransferred` cause hight cpu cost in external shuffle service when `maxChunksBeingTransferred` use default value

authored and
Mridul Muralidharan
committed
[SPARK-24554][PYTHON][SQL] Add MapType support for PySpark with Arrow

BryanCutler
authored and
HyukjinKwon
committed
[SPARK-33475][BUILD] Bump ANTLR runtime version to 4.8-1

maropu
authored and
HyukjinKwon
committed
[SPARK-32852][SQL][DOC][FOLLOWUP] Revise the documentation of spark.sql.hive.metastore.jars
gengliangwang
committed
[SPARK-32907][ML][PYTHON] Adaptively blockify instances - AFT,LiR,LoR

zhengruifeng
authored and
WeichenXu123
committed
[SPARK-33476][CORE] Generalize ExecutorSource to expose user-given file system schemes
dongjoon-hyun
committed
[SPARK-27936][K8S] Support python deps

authored and
dongjoon-hyun
committed
[SPARK-32381][CORE][SQL][FOLLOWUP] More cleanup on HadoopFSUtils

sunchao
authored and
holdenk
committed
[SPARK-31255][SQL] Add SupportsMetadataColumns to DSv2

rdblue
authored and
brkyvz
committed
[SPARK-33464][INFRA] Add/remove (un)necessary cache and restructure GitHub Actions yaml

HyukjinKwon
authored and
dongjoon-hyun
committed

Commits on Nov 19, 2020

Commits on Nov 20, 2020

Commits on Nov 21, 2020

Commits on Nov 22, 2020

Commits on Nov 23, 2020

[SPARK-33512][BUILD] Upgrade test libraries
dongjoon-hyun
committed
[MINOR][INFRA] Suppress warning in check-license

williamhyun
authored and
HyukjinKwon
committed
[SPARK-33427][SQL][FOLLOWUP] Put key and value into IdentityHashMap sequantially

viirya
authored and
HyukjinKwon
committed
[SPARK-33143][PYTHON] Add configurable timeout to python server and client

gaborgsomogyi
and
HyukjinKwon
committed
[SPARK-33510][BUILD] Update SBT to 1.4.4

williamhyun
authored and
dongjoon-hyun
committed
Revert "[SPARK-32481][CORE][SQL] Support truncate table to move data to trash"

gatorsmile
authored and
HyukjinKwon
committed
[SPARK-33515][SQL] Improve exception messages while handling UnresolvedTable

imback82
authored and
cloud-fan
committed
[SPARK-33511][SQL] Respect case sensitivity while resolving V2 partition specs

MaxGekk
authored and
cloud-fan
committed
[SPARK-33278][SQL][FOLLOWUP] Improve OptimizeWindowFunctions to avoid transfer first to nth_value

beliefer
authored and
cloud-fan
committed
[SPARK-32918][SHUFFLE] RPC implementation to support control plane coordination for push-based shuffle

authored and
Mridul Muralidharan
committed

Commits on Nov 24, 2020

Commits on Nov 25, 2020

Commits on Nov 26, 2020

Commits on Nov 27, 2020

[SPARK-33563][PYTHON][R][SQL] Expose inverse hyperbolic trig functions in PySpark and SparkR

zero323
authored and
HyukjinKwon
committed
[SPARK-33566][CORE][SQL][SS][PYTHON] Make unescapedQuoteHandling option configurable when read CSV

LuciferYang
authored and
HyukjinKwon
committed
[SPARK-33575][SQL] Fix misleading exception for "ANALYZE TABLE ... FOR COLUMNS" on temporary views

imback82
authored and
cloud-fan
committed
[SPARK-33522][SQL] Improve exception messages while handling UnresolvedTableOrView

imback82
authored and
cloud-fan
committed
[SPARK-28645][SQL] ParseException is thrown when the window is redefined

beliefer
authored and
cloud-fan
committed
[SPARK-33498][SQL] Datetime parsing should fail if the input string can't be parsed, or the pattern string is invalid

leanken-zz
authored and
cloud-fan
committed
[SPARK-33141][SQL] Capture SQL configs when creating permanent views

luluorta
authored and
cloud-fan
committed
Spelling r common dev mlib external project streaming resource managers python

jsoref
authored and
srowen
committed

Commits on Nov 28, 2020

Commits on Nov 29, 2020

Commits on Nov 30, 2020

[SPARK-33517][SQL][DOCS] Fix the correct menu items and page links in PySpark Usage Guide for Pandas with Apache Arrow

liucht-inspur
authored and
HyukjinKwon
committed
[SPARK-33589][SQL] Close opened session if the initialization fails

wangyum
authored and
HyukjinKwon
committed
[SPARK-33582][SQL] Hive Metastore support filter by not-equals

wangyum
authored and
HyukjinKwon
committed
[SPARK-33567][SQL] DSv2: Use callback instead of passing Spark session and v2 relation for refreshing cache

sunchao
authored and
cloud-fan
committed
[MINOR] Spelling bin core docs external mllib repl

jsoref
authored and
maropu
committed
[SPARK-32976][SQL] Support column list in INSERT statement

yaooqinn
authored and
cloud-fan
committed
[SPARK-33448][SQL] Support CACHE/UNCACHE TABLE commands for v2 tables

imback82
authored and
cloud-fan
committed
[SPARK-33498][SQL][FOLLOW-UP] Deduplicate the unittest by using checkCastWithParseError

leanken-zz
authored and
HyukjinKwon
committed
[SPARK-28646][SQL] Fix bug of Count so as consistent with mainstream databases

beliefer
authored and
HyukjinKwon
committed
[SPARK-33480][SQL] Support char/varchar type
cloud-fan
committed
[SPARK-33579][UI] Fix executor blank page behind proxy

Pascal Gillet
authored and
sarutak
committed
[SPARK-33452][SQL] Support v2 SHOW PARTITIONS

MaxGekk
authored and
cloud-fan
committed
[SPARK-33569][SQL] Remove getting partitions by an identifier prefix

MaxGekk
authored and
cloud-fan
committed
[SPARK-33569][SPARK-33452][SQL][FOLLOWUP] Fix a build error in `ShowPartitionsExec`

MaxGekk
authored and
cloud-fan
committed
[SPARK-33185][YARN][FOLLOW-ON] Leverage RM's RPC API instead of REST to fetch driver log links in yarn.Client

xkrogen
authored and
Mridul Muralidharan
committed
[SPARK-33545][CORE] Support Fallback Storage during Worker decommission
dongjoon-hyun
committed
[SPARK-33440][CORE] Use current timestamp with warning log in HadoopFSDelegationTokenProvider when the issue date for token is not set up properly
HeartSaVioR
committed

Commits on Dec 1, 2020

Commits on Dec 2, 2020

Commits on Dec 3, 2020

[MINOR][INFRA] Use the latest image for GitHub Action jobs

dongjoon-hyun
authored and
HyukjinKwon
committed
[SPARK-31953][SS] Add Spark Structured Streaming History Server Support

authored and
zsxwing
committed
[SPARK-33610][ML] Imputer transform skip duplicate head() job
zhengruifeng
committed
[SPARK-32896][SS][FOLLOW-UP] Rename the API to `toTable`

xuanyuanking
authored and
zsxwing
committed
[SPARK-22798][PYTHON][ML][FOLLOWUP] Add labelsArray to PySpark StringIndexer

viirya
authored and
HyukjinKwon
committed
[SPARK-33636][PYTHON][ML][FOLLOWUP] Update since tag of labelsArray in StringIndexer

viirya
authored and
HyukjinKwon
committed
[SPARK-20044][SQL] Add new function DATE_FROM_UNIX_DATE and UNIX_DATE

gengliangwang
authored and
cloud-fan
committed
[SPARK-26218][SQL][FOLLOW UP] Fix the corner case of codegen when casting float to Integer

luluorta
authored and
cloud-fan
committed
[SPARK-30098][SQL] Add a configuration to use default datasource as provider for CREATE TABLE command
cloud-fan
committed
[SPARK-33629][PYTHON] Make spark.buffer.size configuration visible on driver side

gaborgsomogyi
authored and
HyukjinKwon
committed
[SPARK-33623][SQL] Add canDeleteWhere to SupportsDelete

aokolnychyi
authored and
dongjoon-hyun
committed
[SPARK-33634][SQL][TESTS] Use Analyzer in PlanResolutionSuite

cloud-fan
authored and
dongjoon-hyun
committed

Commits on Dec 4, 2020

Commits on Dec 5, 2020

[SPARK-33651][SQL] Allow CREATE EXTERNAL TABLE with LOCATION for data source tables

cloud-fan
authored and
dongjoon-hyun
committed

Commits on Dec 6, 2020

[MINOR] Fix string interpolation in CommandUtils.scala and KafkaDataConsumer.scala

imback82
authored and
HyukjinKwon
committed
[SPARK-33668][K8S][TEST] Fix flaky test "Verify logging configuration is picked from the provided

ScrapCodes
authored and
dongjoon-hyun
committed
[SPARK-33652][SQL] DSv2: DeleteFrom should refresh cache

sunchao
authored and
dongjoon-hyun
committed
[SPARK-33256][PYTHON][DOCS] Clarify PySpark follows NumPy documentation style

HyukjinKwon
authored and
dongjoon-hyun
committed
[SPARK-33667][SQL] Respect the `spark.sql.caseSensitive` config while resolving partition spec in v1 `SHOW PARTITIONS`

MaxGekk
authored and
dongjoon-hyun
committed
[SPARK-33674][TEST] Show Slowpoke notifications in SBT tests

gatorsmile
authored and
wangyum
committed

Commits on Dec 7, 2020

[SPARK-33663][SQL] Uncaching should not be called on non-existing temp views

imback82
authored and
HyukjinKwon
committed
[SPARK-33675][INFRA] Add GitHub Action job to publish snapshot

dongjoon-hyun
authored and
HyukjinKwon
committed
[SPARK-33670][SQL] Verify the partition provider is Hive in v1 SHOW TABLE EXTENDED

MaxGekk
authored and
HyukjinKwon
committed
[SPARK-33683][INFRA] Remove -Djava.version=11 from Scala 2.13 build in GitHub Actions

sarutak
authored and
dongjoon-hyun
committed
[SPARK-33680][SQL][TESTS] Fix PrunePartitionSuiteBase/BucketedReadWithHiveSupportSuite not to depend on the default conf
dongjoon-hyun
committed
[SPARK-33684][BUILD] Upgrade httpclient from 4.5.6 to 4.5.13

sarutak
authored and
dongjoon-hyun
committed
[SPARK-33671][SQL] Remove VIEW checks from V1 table commands

MaxGekk
authored and
dongjoon-hyun
committed
[SPARK-33676][SQL] Require exact matching of partition spec to the schema in V2 `ALTER TABLE .. ADD/DROP PARTITION`

MaxGekk
authored and
cloud-fan
committed
[SPARK-33617][SQL] Add default parallelism configuration for Spark SQL queries

wangyum
authored and
HyukjinKwon
committed
[SPARK-32680][SQL] Don't Preprocess V2 CTAS with Unresolved Query

linhongliu-db
authored and
cloud-fan
committed
[SPARK-33641][SQL] Invalidate new char/varchar types in public APIs that produce incorrect results

yaooqinn
authored and
cloud-fan
committed
[MINOR] Spelling sql not core

jsoref
authored and
srowen
committed
[SPARK-33693][SQL] deprecate spark.sql.hive.convertCTAS

cloud-fan
authored and
dongjoon-hyun
committed
[SPARK-33480][SQL][FOLLOWUP] do not expose user data in error message

cloud-fan
authored and
dongjoon-hyun
committed
[SPARK-33621][SQL] Add a way to inject data source rewrite rules

aokolnychyi
authored and
dongjoon-hyun
committed

Commits on Dec 8, 2020

Commits on Dec 9, 2020

[MINOR][ML] Increase Bounded MLOR (without regularization) test error tolerance
WeichenXu123
committed
[SPARK-33703][SQL] Migrate MSCK REPAIR TABLE to use UnresolvedTable to resolve the identifier

imback82
authored and
cloud-fan
committed
[SPARK-33641][SQL][DOC][FOLLOW-UP] Add migration guide for CHAR VARCHAR types

yaooqinn
authored and
cloud-fan
committed
[SPARK-33669] Wrong error message from YARN application state monitor when sc.stop in yarn client mode

suqilong
authored and
Mridul Muralidharan
committed
[SPARK-33655][SQL] Improve performance of processing FETCH_PRIOR

Dooyoung Hwang
authored and
HyukjinKwon
committed
[SPARK-33719][DOC] Add make_date/make_timestamp/make_interval into the doc of ANSI Compliance

gengliangwang
authored and
HyukjinKwon
committed
[SPARK-33071][SPARK-33536][SQL][FOLLOW-UP] Rename deniedMetadataKeys to nonInheritableMetadataKeys in Alias
HyukjinKwon
committed
[SPARK-33722][SQL] Handle DELETE in ReplaceNullWithFalseInPredicate

aokolnychyi
authored and
dongjoon-hyun
committed
[SPARK-33725][BUILD] Upgrade snappy-java to 1.1.8.2

viirya
authored and
dongjoon-hyun
committed

Commits on Dec 10, 2020

[SPARK-33727][K8S] Fall back from gnupg.net to openpgp.org

holdenk
authored and
HyukjinKwon
committed
[SPARK-33724][K8S] Add decom script as a configuration param

holdenk
authored and
dongjoon-hyun
committed
[SPARK-33558][SQL][TESTS] Unify v1 and v2 ALTER TABLE .. ADD PARTITION tests

MaxGekk
authored and
cloud-fan
committed
[SPARK-33714][SQL] Migrate ALTER VIEW ... SET/UNSET TBLPROPERTIES commands to use UnresolvedView to resolve the identifier

imback82
authored and
cloud-fan
committed
[SPARK-33732][K8S][TESTS] Kubernetes integration tests doesn't work with Minikube 1.9+

sarutak
authored and
dongjoon-hyun
committed
[SPARK-32670][SQL][FOLLOWUP] Group exception messages in Catalyst Analyzer in one file

beliefer
authored and
cloud-fan
committed
[SPARK-33692][SQL] View should use captured catalog and namespace to lookup function

linhongliu-db
authored and
cloud-fan
committed

Commits on Dec 11, 2020

Commits on Dec 15, 2020

[SPARK-22256] Update comments for driverContainerMemory to accruately describe behavior, add ticket number to test cases
dmcwhorter
committed