Branch 2.3#20185

Closed

jimmy144 wants to merge 142 commits intomasterfrom

Commits on Jan 2, 2018

[SPARK-22897][CORE] Expose stageAttemptId in TaskContext

advancedxy
authored and
cloud-fan
committed

Commits on Jan 3, 2018

[SPARK-22938] Assert that SQLConf.get is accessed only on the driver.

juliuszsompolski
authored and
cloud-fan
committed
[SPARK-22934][SQL] Make optional clauses order insensitive for CREATE TABLE SQL statement
gatorsmile
committed
[SPARK-20236][SQL] dynamic partition overwrite

cloud-fan
authored and
gatorsmile
committed
[SPARK-22932][SQL] Refactor AnalysisContext
gatorsmile
committed
[SPARK-22896] Improvement in String interpolation

chetkhatri
authored and
srowen
committed
[SPARK-20960][SQL] make ColumnVector public

cloud-fan
authored and
gatorsmile
committed

Commits on Jan 4, 2018

[SPARK-22944][SQL] improve FoldablePropagation

cloud-fan
authored and
gatorsmile
committed
[SPARK-22933][SPARKR] R Structured Streaming API for withWatermark, trigger, partitionBy

felixcheung
authored and
Felix Cheung
committed
[SPARK-22950][SQL] Handle ChildFirstURLClassLoader's parent

yaooqinn
authored and
cloud-fan
committed
[SPARK-22945][SQL] add java UDF APIs in the functions object
cloud-fan
committed
[SPARK-22939][PYSPARK] Support Spark UDF in registerFunction
gatorsmile
committed
[SPARK-22771][SQL] Add a missing return statement in Concat.checkInputDataTypes

maropu
authored and
gatorsmile
committed
[SPARK-21475][CORE][2ND ATTEMPT] Change to use NIO's Files API for external shuffle service

jerryshao
authored and
zsxwing
committed
[SPARK-22850][CORE] Ensure queued events are delivered to all event queues.

Marcelo Vanzin
authored and
squito
committed
[SPARK-22948][K8S] Move SparkPodInitContainer to correct package.
Marcelo Vanzin
committed
[SPARK-22953][K8S] Avoids adding duplicated secret volumes when init-container is used

liyinan926
authored and
Marcelo Vanzin
committed

Commits on Jan 5, 2018

[SPARK-22960][K8S] Make build-push-docker-images.sh more dev-friendly.
Marcelo Vanzin
committed
[SPARK-22957] ApproxQuantile breaks if the number of rows exceeds MaxInt

juliuszsompolski
authored and
cloud-fan
committed
[SPARK-22825][SQL] Fix incorrect results of Casting Array to String

maropu
authored and
cloud-fan
committed
[SPARK-22949][ML] Apply CrossValidator approach to Driver/Distributed memory tradeoff for TrainValidationSplit

MrBago
authored and
jkbradley
committed
[SPARK-22757][K8S] Enable spark.jars and spark.files in KUBERNETES mode

liyinan926
authored and
Felix Cheung
committed
[SPARK-22961][REGRESSION] Constant columns should generate QueryPlanConstraints

adrian-ionescu
authored and
gatorsmile
committed
[SPARK-22940][SQL] HiveExternalCatalogVersionsSuite should succeed on platforms that don't have wget

bersprockets
authored and
Marcelo Vanzin
committed
[SPARK-13030][ML] Follow-up cleanups for OneHotEncoderEstimator
jkbradley
committed

Commits on Jan 6, 2018

[SPARK-22914][DEPLOY] Register history.ui.port

gerashegalov
authored and
Marcelo Vanzin
committed
[SPARK-22937][SQL] SQL elt output binary for binary inputs

maropu
authored and
gatorsmile
committed
[SPARK-22960][K8S] Revert use of ARG base_image in images

liyinan926
authored and
Marcelo Vanzin
committed
[SPARK-22930][PYTHON][SQL] Improve the description of Vectorized UDFs for non-deterministic cases

icexelloss
authored and
gatorsmile
committed
[SPARK-22793][SQL] Memory leak in Spark Thrift Server

zuotingbing
authored and
gatorsmile
committed
[SPARK-21786][SQL] When acquiring 'compressionCodecClassName' in 'ParquetOptions', `parquet.compression` needs to be considered.

fjh100456
authored and
gatorsmile
committed
[SPARK-22901][PYTHON][FOLLOWUP] Adds the doc for asNondeterministic for wrapped UDF function

HyukjinKwon
authored and
gatorsmile
committed
[HOTFIX] Fix style checking failure
gatorsmile
committed

Commits on Jan 7, 2018

[SPARK-22973][SQL] Fix incorrect results of Casting Map to String

maropu
authored and
cloud-fan
committed

Commits on Jan 8, 2018

[SPARK-22985] Fix argument escaping bug in from_utc_timestamp / to_utc_timestamp codegen

JoshRosen
authored and
gatorsmile
committed
[SPARK-22566][PYTHON] Better error message for `_merge_type` in Pandas to Spark DF conversion

gberger-palantir
authored and
ueshin
committed
[SPARK-22979][PYTHON][SQL] Avoid per-record type dispatch in Python data conversion (EvaluatePython.fromJava)

HyukjinKwon
authored and
cloud-fan
committed
[SPARK-22983] Don't push filters beneath aggregates with empty grouping expressions

JoshRosen
authored and
gatorsmile
committed
[SPARK-21865][SQL] simplify the distribution semantic of Spark SQL
cloud-fan
committed
[SPARK-22952][CORE] Deprecate stageAttemptId in favour of stageAttemptNumber

advancedxy
authored and
cloud-fan
committed
[SPARK-22992][K8S] Remove assumption of the DNS domain

foxish
authored and
Marcelo Vanzin
committed

Commits on Jan 9, 2018

[SPARK-22972] Couldn't find corresponding Hive SerDe for data source provider org.apache.spark.sql.hive.orc

xubo245
authored and
gatorsmile
committed
[SPARK-22990][CORE] Fix method isFairScheduler in JobsTab and StagesTab

gengliangwang
authored and
cloud-fan
committed
[SPARK-22984] Fix incorrect bitmap copying and offset adjustment in GenerateUnsafeRowJoiner

JoshRosen
authored and
cloud-fan
committed
[SPARK-21292][DOCS] refreshtable example

felixcheung
authored and
Felix Cheung
committed
[SPARK-21293][SPARKR][DOCS] structured streaming doc update

felixcheung
authored and
Felix Cheung
committed
[SPARK-23000] Fix Flaky test suite DataSourceWithHiveMetastoreCatalogSuite in Spark 2.3

gatorsmile
authored and
cloud-fan
committed
[SPARK-22998][K8S] Set missing value for SPARK_MOUNTED_CLASSPATH in the executors

liyinan926
authored and
Felix Cheung
committed
[SPARK-16060][SQL] Support Vectorized ORC Reader

dongjoon-hyun
authored and
cloud-fan
committed
[SPARK-22981][SQL] Fix incorrect results of Casting Struct to String

maropu
authored and
cloud-fan
committed
[SPARK-22912] v2 data source support in MicroBatchExecution

jose-torres
authored and
tdas
committed

Commits on Jan 10, 2018

[SPARK-23005][CORE] Improve RDD.take on small number of partitions

gengliangwang
authored and
cloud-fan
committed
[MINOR] fix a typo in BroadcastJoinSuite
cloud-fan
committed
[SPARK-23018][PYTHON] Fix createDataFrame from Pandas timestamp series assignment

BryanCutler
authored and
ueshin
committed
[SPARK-22982] Remove unsafe asynchronous close() call from FileDownloadChannel

JoshRosen
authored and
cloud-fan
committed
[SPARK-16060][SQL][FOLLOW-UP] add a wrapper solution for vectorized orc reader
cloud-fan
committed
[SPARK-22993][ML] Clarify HasCheckpointInterval param doc

sethah
authored and
Felix Cheung
committed
[SPARK-22997] Add additional defenses against use of freed MemoryBlocks
JoshRosen
committed
[SPARK-23019][CORE] Wait until SparkContext.stop() finished in SparkLauncherSuite

gengliangwang
authored and
Marcelo Vanzin
committed
[SPARK-22951][SQL] fix aggregation after dropDuplicates on empty data frames

Feng Liu
authored and
liancheng
committed

Commits on Jan 11, 2018

[SPARK-23009][PYTHON] Fix for non-str col names to createDataFrame from Pandas

BryanCutler
authored and
HyukjinKwon
committed
[SPARK-22587] Spark job fails if fs.defaultFS and application jar are different url

Mingjie Tang
authored and
jerryshao
committed
[SPARK-23001][SQL] Fix NullPointerException when DESC a database with NULL description

gatorsmile
authored and
cloud-fan
committed
[SPARK-20657][CORE] Speed up rendering of the stages page.

Marcelo Vanzin
authored and
cloud-fan
committed
[SPARK-22967][TESTS] Fix VersionSuite's unit tests by change Windows path into URI path

Ngone51
authored and
HyukjinKwon
committed
[SPARK-23000][TEST-HADOOP2.6] Fix Flaky test suite DataSourceWithHiveMetastoreCatalogSuite
gatorsmile
committed
[SPARK-19732][FOLLOW-UP] Document behavior changes made in na.fill and fillna

gatorsmile
authored and
HyukjinKwon
committed
[SPARK-22994][K8S] Use a single image for all Spark containers.
Marcelo Vanzin
committed
[SPARK-22908] Add kafka source and sink for continuous processing.

jose-torres
authored and
tdas
committed
[SPARK-23046][ML][SPARKR] Have RFormula include VectorSizeHint in pipeline

MrBago
authored and
jkbradley
committed
Preparing Spark release v2.3.0-rc1
sameeragarwal
committed
Preparing development version 2.3.1-SNAPSHOT
sameeragarwal
committed

Commits on Jan 12, 2018

[SPARK-23008][ML] OnehotEncoderEstimator python API

WeichenXu123
authored and
jkbradley
committed
[SPARK-22986][CORE] Use a cache to avoid instantiating multiple instances of broadcast variable values

ho3rexqj
authored and
cloud-fan
committed
[SPARK-23008][ML][FOLLOW-UP] mark OneHotEncoder python API deprecated

WeichenXu123
authored and
Nick Pentreath
committed
[SPARK-23025][SQL] Support Null type in scala reflection

mgaido91
authored and
gatorsmile
committed
[MINOR][BUILD] Fix Java linter errors

dongjoon-hyun
authored and
sameeragarwal
committed
[SPARK-22975][SS] MetricsReporter should not throw exception when there was no progress reported

mgaido91
authored and
zsxwing
committed
Revert "[SPARK-22908] Add kafka source and sink for continuous processing."
sameeragarwal
committed

Commits on Jan 13, 2018

Commits on Jan 14, 2018

[SPARK-23063][K8S] K8s changes for publishing scripts (and a couple of other misses)

foxish
authored and
Felix Cheung
committed
[SPARK-23038][TEST] Update docker/spark-test (JDK/OS)

dongjoon-hyun
authored and
Felix Cheung
committed
[SPARK-23069][DOCS][SPARKR] fix R doc for describe missing text

felixcheung
authored and
HyukjinKwon
committed
[SPARK-23021][SQL] AnalysisBarrier should override innerChildren to print correct explain output

maropu
authored and
gatorsmile
committed
[SPARK-23051][CORE] Fix for broken job description in Spark UI

smurakozi
authored and
srowen
committed
[SPARK-22999][SQL] show databases like command' can remove the like keyword

guoxiaolong
authored and
gatorsmile
committed

Commits on Jan 15, 2018

[SPARK-23054][SQL] Fix incorrect results of casting UserDefinedType to String

maropu
authored and
cloud-fan
committed
[SPARK-23049][SQL] `spark.sql.files.ignoreCorruptFiles` should work for ORC files

dongjoon-hyun
authored and
cloud-fan
committed
[SPARK-23023][SQL] Cast field data to strings in showString

maropu
authored and
cloud-fan
committed
[SPARK-19550][BUILD][FOLLOW-UP] Remove MaxPermSize for sql module

wangyum
authored and
srowen
committed
[SPARK-23070] Bump previousSparkVersion in MimaBuild.scala to be 2.2.0
gatorsmile
committed
[SPARK-23035][SQL] Fix improper information of TempTableAlreadyExistsException

xubo245
authored and
gatorsmile
committed

Commits on Jan 16, 2018

[SPARK-23080][SQL] Improve error message for built-in functions

mgaido91
authored and
HyukjinKwon
committed
[SPARK-23000] Use fully qualified table names in HiveMetastoreCatalogSuite

sameeragarwal
authored and
gatorsmile
committed
[SPARK-22956][SS] Bug fix for 2 streams union failover scenario

xuanyuanking
authored and
zsxwing
committed
[SPARK-23020][CORE] Fix races in launcher code, test.

Marcelo Vanzin
authored and
sameeragarwal
committed
[SPARK-22978][PYSPARK] Register Vectorized UDFs for SQL Statement

gatorsmile
authored and
HyukjinKwon
committed
[SPARK-22392][SQL] data source v2 columnar batch reader
cloud-fan
committed
[SPARK-23045][ML][SPARKR] Update RFormula to use OneHotEncoderEstimator.

MrBago
authored and
jkbradley
committed

Commits on Jan 17, 2018

[SPARK-23095][SQL] Decorrelation of scalar subquery fails with java.util.NoSuchElementException

dilipbiswal
authored and
gatorsmile
committed
[SPARK-22361][SQL][TEST] Add unit test for Window Frames

gaborgsomogyi
authored and
gatorsmile
committed
[SPARK-22908][SS] Roll forward continuous processing Kafka support with fix to continuous Kafka data reader

jose-torres
authored and
tdas
committed
Revert "[SPARK-23020][CORE] Fix races in launcher code, test."
sameeragarwal
committed
Fix merge between 07ae39d0ec and 1667057851

jose-torres
authored and
zsxwing
committed
[SPARK-23072][SQL][TEST] Add a Unicode schema test for file-based data sources

dongjoon-hyun
authored and
cloud-fan
committed
[SPARK-23062][SQL] Improve EXCEPT documentation

Henry Robinson
authored and
gatorsmile
committed
[SPARK-21783][SQL] Turn on ORC filter push-down by default

dongjoon-hyun
authored and
cloud-fan
committed
[SPARK-23079][SQL] Fix query constraints propagation with aliases

gengliangwang
authored and
cloud-fan
committed
[SPARK-23020] Ignore Flaky Test: SparkLauncherSuite.testInProcessLauncher
sameeragarwal
committed
[SPARK-23033][SS] Don't use task level retry for continuous processing

jose-torres
authored and
tdas
committed
[SPARK-23093][SS] Don't change run id when reconfiguring a continuous processing query.

jose-torres
authored and
zsxwing
committed
[SPARK-23047][PYTHON][SQL] Change MapVector to NullableMapVector in ArrowColumnVector

icexelloss
authored and
HyukjinKwon
committed
[SPARK-23132][PYTHON][ML] Run doctests in ml.image when testing
HyukjinKwon
committed

Commits on Jan 18, 2018

[SPARK-23119][SS] Minor fixes to V2 streaming APIs

tdas
authored and
zsxwing
committed
[SPARK-23064][DOCS][SS] Added documentation for stream-stream joins

tdas
authored and
zsxwing
committed
[SPARK-21996][SQL] read files with space in name for streaming

xysun
authored and
zsxwing
committed
[SPARK-23122][PYTHON][SQL] Deprecate register* for UDFs in SQLContext and Catalog in PySpark

HyukjinKwon
authored and
ueshin
committed
[SPARK-23052][SS] Migrate ConsoleSink to data source V2 api.

jose-torres
authored and
tdas
committed
[SPARK-23140][SQL] Add DataSourceV2Strategy to Hive Session state's planner

jerryshao
authored and
cloud-fan
committed
[SPARK-22036][SQL] Decimal multiplication with high precision/scale often returns NULL

mgaido91
authored and
cloud-fan
committed
[SPARK-23141][SQL][PYSPARK] Support data type string as a returnType for registerJavaFunction.

ueshin
authored and
HyukjinKwon
committed
[SPARK-23147][UI] Fix task page table IndexOutOfBound Exception

jerryshao
authored and
Marcelo Vanzin
committed
[SPARK-23029][DOCS] Specifying default units of configuration entries

ferdonline
authored and
srowen
committed
[SPARK-23143][SS][PYTHON] Added python API for setting continuous trigger
tdas
committed
[SPARK-23144][SS] Added console sink for continuous processing
tdas
committed
[SPARK-23133][K8S] Fix passing java options to Executor

andrusha
authored and
Marcelo Vanzin
committed
[SPARK-23094] Fix invalid character handling in JsonDataSource

brkyvz
authored and
HyukjinKwon
committed
[SPARK-22962][K8S] Fail fast if submission client local files are used

liyinan926
authored and
Marcelo Vanzin
committed

Commits on Jan 19, 2018

[SPARK-23142][SS][DOCS] Added docs for continuous processing
tdas
committed
[DOCS] change to dataset for java code in structured-streaming-kafka-integration document

brandonJY
authored and
srowen
committed
[SPARK-23054][SQL][PYSPARK][FOLLOWUP] Use sqlType casting when casting PythonUserDefinedType to String.

ueshin
authored and
cloud-fan
committed
[BUILD][MINOR] Fix java style check issues
sameeragarwal
committed
[SPARK-23127][DOC] Update FeatureHasher guide for categoricalCols parameter
Nick Pentreath
committed
[SPARK-23048][ML] Add OneHotEncoderEstimator document and examples

viirya
authored and
Nick Pentreath
committed
[SPARK-23089][STS] Recreate session log directory if it doesn't exist

mgaido91
authored and
cloud-fan
committed
[SPARK-23000][TEST] Keep Derby DB Location Unchanged After Session Cloning

gatorsmile
authored and
cloud-fan
committed
[SPARK-23149][SQL] polish ColumnarBatch

cloud-fan
authored and
gatorsmile
committed
[SPARK-23104][K8S][DOCS] Changes to Kubernetes scheduler documentation

foxish
authored and
Marcelo Vanzin
committed