[HOPSWORKS-1081] Upgrade Spark to 2.4.3#11

Merged

tkakantousis merged 263 commits intologicalclocks:branch-2.4from

kai-chi:HOPSWORKS-1081

Aug 6, 2019

This pull request is big! We're only showing the most recent 250 commits

Commits on Aug 1, 2019

[SPARK-25899][TESTS] Fix flaky CoarseGrainedSchedulerBackendSuite

zsxwing
authored and
kai-chi
committed
[SPARK-25837][CORE] Fix potential slowdown in AppStatusListener when cleaning up stages

patrickbrownsync
authored and
kai-chi
committed
[SPARK-25918][SQL] LOAD DATA LOCAL INPATH should handle a relative path

dongjoon-hyun
authored and
kai-chi
committed
[SPARK-25023] Clarify Spark security documentation

Thomas Graves
authored and
kai-chi
committed
[SPARK-25909] fix documentation on cluster managers

jameslamb
authored and
kai-chi
committed
[SPARK-25827][CORE] Avoid converting incoming encrypted blocks to byte buffers

squito
authored and
kai-chi
committed
[SPARK-25933][DOCUMENTATION] Fix pstats.Stats() reference in configuration.md

Alex Hagerman
authored and
kai-chi
committed
[SPARK-25930][K8S] Fix scala string detection in k8s tests

Stavros Kontopoulos
authored and
kai-chi
committed
[MINOR] Fix typos and misspellings

srowen
authored and
kai-chi
committed
[SPARK-25906][SHELL] Documents '-I' option (from Scala REPL) in spark-shell

HyukjinKwon
authored and
kai-chi
committed
[SPARK-22148][SPARK-15815][SCHEDULER] Acquire new executors to avoid hang because of blacklisting

authored and
kai-chi
committed
[SPARK-25866][ML] Update KMeans formatVersion

mgaido91
authored and
kai-chi
committed
[MINOR] update known_translations

cloud-fan
authored and
kai-chi
committed
Revert "[SPARK-23831][SQL] Add org.apache.derby to IsolatedClientLoader"

HyukjinKwon
authored and
kai-chi
committed
[SPARK-25904][CORE] Allocate arrays smaller than Int.MaxValue

squito
authored and
kai-chi
committed
hot fix: add missing import

cloud-fan
authored and
kai-chi
committed
[MINOR] update HiveExternalCatalogVersionsSuite to test 2.4.0

cloud-fan
authored and
kai-chi
committed
[SPARK-25988][SQL] Keep names unchanged when deduplicating the column names in Analyzer

gatorsmile
authored and
kai-chi
committed
[SPARK-25979][SQL] Window function: allow parentheses around window reference

gengliangwang
authored and
kai-chi
committed
[SPARK-26029][BUILD][2.4] Bump previousSparkVersion in MimaBuild.scala to be 2.3.0

cloud-fan
authored and
kai-chi
committed
[SPARK-26010][R] fix vignette eval with Java 11

felixcheung
authored and
kai-chi
committed
[SPARK-25921][PYSPARK] Fix barrier task run without BarrierTaskContext while python worker reuse

xuanyuanking
authored and
kai-chi
committed
[SPARK-26042][SS][TESTS] Fix a potential hang in KafkaContinuousSourceTopicDeletionSuite

zsxwing
authored and
kai-chi
committed
[SPARK-25778] WriteAheadLogBackedBlockRDD in YARN Cluster Mode Fails …

gss2002
authored and
kai-chi
committed
[SPARK-26057][SQL] Transform also analyzed plans when dedup references

mgaido91
authored and
kai-chi
committed
[SPARK-26011][SPARK-SUBMIT] Yarn mode pyspark app without python main resource does not honor "spark.jars.packages"

shanyu
authored and
kai-chi
committed
[SPARK-25883][BACKPORT][SQL][MINOR] Override method `prettyName` in `from_avro`/`to_avro`

gengliangwang
authored and
kai-chi
committed
[SPARK-23207][SQL][FOLLOW-UP] Use `SQLConf.get.enableRadixSort` instead of `SparkEnv.get.conf.get(SQLConf.RADIX_SORT_ENABLED)`.

ueshin
authored and
kai-chi
committed
[SPARK-25023] More detailed security guidance for K8S

rvesse
authored and
kai-chi
committed
[SPARK-25934][MESOS] Don't propagate SPARK_CONF_DIR from spark submit

mpmolek
authored and
kai-chi
committed
[SPARK-26069][TESTS] Fix flaky test: RpcIntegrationSuite.sendRpcWithStreamFailures

zsxwing
authored and
kai-chi
committed
[SPARK-26092][SS] Use CheckpointFileManager to write the streaming metadata file

zsxwing
authored and
kai-chi
committed
[SPARK-26079][SQL] Ensure listener event delivery in StreamingQueryListenersConfSuite.

Marcelo Vanzin
authored and
kai-chi
committed
[MINOR][SQL] Fix typo in CTAS plan database string

dongjoon-hyun
authored and
kai-chi
committed
[SPARK-26118][WEB UI] Introducing spark.ui.requestHeaderSize for setting HTTP requestHeaderSize

attilapiros
authored and
kai-chi
committed
[SPARK-26084][SQL] Fixes unresolved AggregateExpression.references exception

ssimeonov
authored and
kai-chi
committed
[SPARK-26120][TESTS][SS][SPARKR] Fix a streaming query leak in Structured Streaming R tests

zsxwing
authored and
kai-chi
committed
[SPARK-26109][WEBUI] Duration in the task summary metrics table and the task table are different

shahidki31
authored and
kai-chi
committed
[SPARK-26021][SQL] replace minus zero with zero in Platform.putDouble/Float

Alon Doron
authored and
kai-chi
committed
[SPARK-24553][UI][FOLLOWUP][2.4 BACKPORT] Fix unnecessary UI redirect

jerryshao
authored and
kai-chi
committed
[SPARK-26069][TESTS][FOLLOWUP] Add another possible error message

zsxwing
authored and
kai-chi
committed
[SPARK-25786][CORE] If the ByteBuffer.hasArray is false , it will throw UnsupportedOperationException for Kryo

10110346
authored and
kai-chi
committed
[MINOR][K8S] Invalid property "spark.driver.pod.name" is referenced in docs.

Leemoonsoo
authored and
kai-chi
committed
[SPARK-25451][SPARK-26100][CORE] Aggregated metrics table doesn't show the right number of the total tasks

shahidki31
authored and
kai-chi
committed
[SPARK-26114][CORE] ExternalSorter's readingIterator field leak

szhem
authored and
kai-chi
committed
[SPARK-26147][SQL] only pull out unevaluable python udf from join condition

cloud-fan
authored and
kai-chi
committed
[SPARK-26137][CORE] Use Java system property "file.separator" inste…

Mark Pavey
authored and
kai-chi
committed
[SPARK-26211][SQL] Fix InSet for binary, and struct and array with null.

ueshin
authored and
kai-chi
committed
[SPARK-26186][SPARK-26184][CORE] Last updated time is not getting updated for the Inprogress application

shahidki31
authored and
kai-chi
committed
[SPARK-26188][SQL] FileIndex: don't infer data types of partition columns if user specifies schema

gengliangwang
authored and
kai-chi
committed
[SPARK-26201] Fix python broadcast with encryption

schintap
authored and
kai-chi
committed
[MINOR][DOC] Correct some document description errors

10110346
authored and
kai-chi
committed
[SPARK-26080][PYTHON] Skips Python resource limit on Windows in Python worker

HyukjinKwon
authored and
kai-chi
committed
[SPARK-26198][SQL] Fix Metadata serialize null values throw NPE

wangyum
authored and
kai-chi
committed
[SPARK-26181][SQL] the `hasMinMaxStats` method of `ColumnStatsMap` is not correct

adrian-wang
authored and
kai-chi
committed
[SPARK-26256][K8S] Fix labels for pod deletion

Stavros Kontopoulos
authored and
kai-chi
committed
[SPARK-26219][CORE][BRANCH-2.4] Executor summary should get updated for failure jobs in the history server UI

shahidki31
authored and
kai-chi
committed
[SPARK-24423][FOLLOW-UP][SQL] Fix error example

wangyum
authored and
kai-chi
committed
[SPARK-26119][CORE][WEBUI] Task summary table should contain only successful tasks' metrics

shahidki31
authored and
kai-chi
committed
[SPARK-26133][ML][FOLLOWUP] Fix doc for OneHotEncoder

viirya
authored and
kai-chi
committed
Revert "[SPARK-26133][ML][FOLLOWUP] Fix doc for OneHotEncoder"

HyukjinKwon
authored and
kai-chi
committed
[SPARK-26233][SQL][BACKPORT-2.4] CheckOverflow when encoding a decimal value

mgaido91
authored and
kai-chi
committed
[SPARK-26266][BUILD] Update to Scala 2.12.8 (branch-2.4)

srowen
authored and
kai-chi
committed
[SPARK-25132][SQL][FOLLOWUP][DOC] Add migration doc for case-insensitive field resolution when reading from Parquet

seancxmao
authored and
kai-chi
committed
[SPARK-26021][2.4][SQL][FOLLOWUP] only deal with NaN and -0.0 in UnsafeWriter

cloud-fan
authored and
kai-chi
committed
[SPARK-26307][SQL] Fix CTAS when INSERT a partitioned table using Hive serde

gatorsmile
authored and
kai-chi
committed
[SPARK-26327][SQL][BACKPORT-2.4] Bug fix for `FileSourceScanExec` metrics update

xuanyuanking
authored and
kai-chi
committed
[SPARK-26265][CORE][BRANCH-2.4] Fix deadlock in BytesToBytesMap.MapIterator when locking both BytesToBytesMap.MapIterator and TaskMemoryManager

viirya
authored and
kai-chi
committed
This is a dummy commit to trigger AFS gitbox sync

dongjoon-hyun
authored and
kai-chi
committed
[SPARK-26355][PYSPARK] Add a workaround for PyArrow 0.11.

ueshin
authored and
kai-chi
committed
[SPARK-26370][SQL] Fix resolution of higher-order function for the same identifier.

ueshin
authored and
kai-chi
committed
[SPARK-26265][CORE][FOLLOWUP] Put freePage into a finally block

viirya
authored and
kai-chi
committed
[SPARK-26315][PYSPARK] auto cast threshold from Integer to Float in approxSimilarityJoin of BucketedRandomProjectionLSHModel

Jing Chen He
authored and
kai-chi
committed
[SPARK-26316][SPARK-21052][BRANCH-2.4] Revert hash join metrics in that causes performance degradation

JkSelf
authored and
kai-chi
committed
[SPARK-26352][SQL] join reorder should not change the order of output attributes

rednaxelafx
authored and
kai-chi
committed
[SPARK-26352][SQL][FOLLOWUP-2.4] Fix missing sameOutput in branch-2.4

rednaxelafx
authored and
kai-chi
committed
[SPARK-26382][CORE] prefix comparator should handle -0.0

cloud-fan
authored and
kai-chi
committed
[SPARK-26394][CORE] Fix annotation error for Utils.timeStringAsMs

Jackey Lee
authored and
kai-chi
committed
[SPARK-26366][SQL] ReplaceExceptWithFilter should consider NULL as False

mgaido91
authored and
kai-chi
committed
[SPARK-24687][CORE] Avoid job hanging when generate task binary causes fatal error

caneGuy
authored and
kai-chi
committed
[SPARK-26392][YARN] Cancel pending allocate requests by taking locality preference into account

Ngone51
authored and
kai-chi
committed
[SPARK-26409][SQL][TESTS] SQLConf should be serializable in test sessions

gengliangwang
authored and
kai-chi
committed
[SPARK-26422][R] Support to disable Hive support in SparkR even for Hadoop versions unsupported by Hive fork

HyukjinKwon
authored and
kai-chi
committed
[SPARK-26402][SQL] Accessing nested fields with different cases in case insensitive mode

dbtsai
authored and
kai-chi
committed
[SPARK-26426][SQL] fix ExpresionInfo assert error in windows operation system.

wangyanlin01
authored and
kai-chi
committed
Revert [SPARK-26021][SQL] replace minus zero with zero in Platform.putDouble/Float

cloud-fan
authored and
kai-chi
committed
[SPARK-26444][WEBUI] Stage color doesn't change with it's status

seancxmao
authored and
kai-chi
committed
[SPARK-26496][SS][TEST] Avoid to use Random.nextString in StreamingInnerJoinSuite

HyukjinKwon
authored and
kai-chi
committed
[SPARK-26019][PYSPARK] Allow insecure py4j gateways

squito
authored and
kai-chi
committed
[SPARK-26501][CORE][TEST] Fix unexpected overriden of exitFn in SparkSubmitSuite

Liupengcheng
authored and
kai-chi
committed
[MINOR][NETWORK][TEST] Fix TransportFrameDecoderSuite to use ByteBuf instead of ByteBuffer

dongjoon-hyun
authored and
kai-chi
committed
[SPARK-26078][SQL][BACKPORT-2.4] Dedup self-join attributes on IN subqueries

mgaido91
authored and
kai-chi
committed
[SPARK-26537][BUILD] change git-wip-us to gitbox

shaneknapp
authored and
kai-chi
committed
[SPARK-26545] Fix typo in EqualNullSafe's truth table comment

rednaxelafx
authored and
kai-chi
committed
[SPARK-26554][BUILD] Update `release-util.sh` to avoid GitBox fake 200 headers

dongjoon-hyun
authored and
kai-chi
committed
[MINOR][BUILD] Fix script name in `release-tag.sh` usage message

dongjoon-hyun
authored and
kai-chi
committed
[SPARK-26559][ML][PYSPARK] ML image can't work with numpy versions prior to 1.9

viirya
authored and
kai-chi
committed
[SPARK-26269][YARN][BRANCH-2.4] Yarnallocator should have same blacklist behaviour with yarn to maxmize use of cluster resource

Ngone51
authored and
kai-chi
committed
[SPARK-26267][SS] Retry when detecting incorrect offsets from Kafka (2.4)

zsxwing
authored and
kai-chi
committed
[SPARK-26554][BUILD][FOLLOWUP] Use GitHub instead of GitBox to check HEADER

dongjoon-hyun
authored and
kai-chi
committed
[SPARK-26571][SQL] Update Hive Serde mapping with canonical name of Parquet and Orc FileFormat

gengliangwang
authored and
kai-chi
committed
[SPARK-22128][CORE][BUILD] Add `paranamer` dependency to `core` module

dongjoon-hyun
authored and
kai-chi
committed
[SPARK-26576][SQL] Broadcast hint not applied to partitioned table

jzhuge
authored and
kai-chi
committed
[SPARK-26551][SQL] Fix schema pruning error when selecting one complex field and having is not null predicate on another one

viirya
authored and
kai-chi
committed
[SPARK-26586][SS] Fix race condition that causes streams to run with unexpected confs

mukulmurthy
authored and
kai-chi
committed
[SPARK-26607][SQL][TEST] Remove Spark 2.2.x testing from HiveExternalCatalogVersionsSuite

dongjoon-hyun
authored and
kai-chi
committed
[SPARK-26538][SQL] Set default precision and scale for elements of postgres numeric array

authored and
kai-chi
committed
[MINOR][BUILD] Remove binary license/notice files in a source release for branch-2.4+ only

maropu
authored and
kai-chi
committed
[SPARK-26615][CORE] Fixing transport server/client resource leaks in the core unittests

attilapiros
authored and
kai-chi
committed
[MINOR][PYTHON] Fix SQLContext to SparkSession in Python API main page

HyukjinKwon
authored and
kai-chi
committed
[SPARK-25992][PYTHON] Document SparkContext cannot be shared for multiprocessing

HyukjinKwon
authored and
kai-chi
committed
[SPARK-26629][SS] Fixed error with multiple file stream in a query + restart on a batch that has no data for one file stream

tdas
authored and
kai-chi
committed
[SPARK-26633][REPL] Add ExecutorClassLoader.getResourceAsStream

rednaxelafx
authored and
kai-chi
committed
[SPARK-26638][PYSPARK][ML] Pyspark vector classes always return error for unary negation

srowen
authored and
kai-chi
committed
[SPARK-26351][MLLIB] Update doc and minor correction in the mllib evaluation metrics

shahidki31
authored and
kai-chi
committed
[SPARK-26665][CORE] Fix a bug that BlockTransferService.fetchBlockSync may hang forever

zsxwing
authored and
kai-chi
committed
[SPARK-26605][YARN] Update AM's credentials when creating tokens.

Marcelo Vanzin
authored and
kai-chi
committed
[SPARK-26228][MLLIB] OOM issue encountered when computing Gramian matrix

srowen
authored and
kai-chi
committed
[SPARK-26706][SQL] Fix `illegalNumericPrecedence` for ByteType

aokolnychyi
authored and
kai-chi
committed
[SPARK-26682][SQL] Use taskAttemptID instead of attemptNumber for Hadoop.

rdblue
authored and
kai-chi
committed
[SPARK-26680][SQL] Eagerly create inputVars while conditions are appropriate

bersprockets
authored and
kai-chi
committed
[SPARK-26709][SQL] OptimizeMetadataOnlyQuery does not handle empty records correctly

authored and
kai-chi
committed
[SPARK-26427][BUILD][BACKPORT-2.4] Upgrade Apache ORC to 1.5.4

dongjoon-hyun
authored and
kai-chi
committed
[SPARK-26379][SS] Fix issue on adding current_timestamp/current_date to streaming query

HeartSaVioR
authored and
kai-chi
committed
[SPARK-26379][SS][FOLLOWUP] Use dummy TimeZoneId to avoid UnresolvedException in CurrentBatchTimestamp

dongjoon-hyun
authored and
kai-chi
committed
[SPARK-26708][SQL][BRANCH-2.4] Incorrect result caused by inconsistency between a SQL cache's cached RDD and its physical plan

maryannxue
authored and
kai-chi
committed
[SPARK-26718][SS][BRANCH-2.4] Fixed integer overflow in SS kafka rateLimit calculation

ryne.yang
authored and
kai-chi
committed
[SPARK-26732][CORE][TEST] Wait for listener bus to process events in SparkContextInfoSuite.

Marcelo Vanzin
authored and
kai-chi
committed
[SPARK-26726] Synchronize the amount of memory used by the broadcast variable to the UI display

httfighter
authored and
kai-chi
committed
[SPARK-26757][GRAPHX] Return 0 for `count` on empty Edge/Vertex RDDs

huonw
authored and
kai-chi
committed
[SPARK-26745][SPARK-24959][SQL][BRANCH-2.4] Revert count optimization in JSON datasource by

HyukjinKwon
authored and
kai-chi
committed
[SPARK-26806][SS] EventTimeStats.merge should handle zeros correctly

zsxwing
authored and
kai-chi
committed
[SPARK-26677][BUILD] Update Parquet to 1.10.1 with notEq pushdown fix.

authored and
kai-chi
committed
[SPARK-26751][SQL] Fix memory leak when statement run in background and throw exception which is not HiveSQLException

caneGuy
authored and
kai-chi
committed
[SPARK-26758][CORE] Idle Executors are not getting killed after spark.dynamiAllocation.executorIdleTimeout value

sandeep-katta
authored and
kai-chi
committed
[SPARK-26677][FOLLOWUP][BRANCH-2.4] Update Parquet manifest with Hadoop-2.6

dongjoon-hyun
authored and
kai-chi
committed
[SPARK-26734][STREAMING] Fix StackOverflowError with large block queue

rlodge
authored and
kai-chi
committed
[SPARK-26082][MESOS] Fix mesos fetch cache config name

mwlon
authored and
kai-chi
committed
[SPARK-26082][MESOS][FOLLOWUP] Add UT on fetcher cache option on MesosClusterScheduler

HeartSaVioR
authored and
kai-chi
committed
Revert "[SPARK-26082][MESOS][FOLLOWUP] Add UT on fetcher cache option on MesosClusterScheduler"

dongjoon-hyun
authored and
kai-chi
committed
[SPARK-26082][MESOS][FOLLOWUP][BRANCH-2.4] Add UT on fetcher cache option on MesosClusterScheduler

HeartSaVioR
authored and
kai-chi
committed
Preparing Spark release v2.4.1-rc1

dbtsai
authored and
kai-chi
committed
Preparing development version 2.4.2-SNAPSHOT

dbtsai
authored and
kai-chi
committed
[MINOR][DOCS] Fix for contradiction in condition formula of keeping intermediate state of window in structured streaming docs

Viktor Tarasenko
authored and
kai-chi
committed
[SPARK-26572][SQL] fix aggregate codegen result evaluation

peter-toth
authored and
kai-chi
committed
[SPARK-26873][SQL] Use a consistent timestamp to build Hadoop Job IDs.

rdblue
authored and
kai-chi
committed
[SPARK-25922][K8S] Spark Driver/Executor "spark-app-selector" label mismatch (branch-2.4)

suxingfate
authored and
kai-chi
committed
[SPARK-26864][SQL][BACKPORT-2.4] Query may return incorrect result when python udf is used as a join condition and the udf uses attributes from both legs of left semi join

dilipbiswal
authored and
kai-chi
committed
[SPARK-26897][SQL][TEST] Update Spark 2.3.x testing from HiveExternalCatalogVersionsSuite

maropu
authored and
kai-chi
committed
[SPARK-26897][SQL][TEST][FOLLOW-UP] Remove workaround for 2.2.0 and 2.1.x in HiveExternalCatalogVersionsSuite

maropu
authored and
kai-chi
committed
[SPARK-26740][SQL][BRANCH-2.4] Read timestamp/date column stats written by Spark 3.0

MaxGekk
authored and
kai-chi
committed
[MINOR][DOCS] Fix the update rule in StreamingKMeansModel documentation

joelgenter
authored and
kai-chi
committed
Preparing Spark release v2.4.1-rc2

dbtsai
authored and
kai-chi
committed
Preparing development version 2.4.2-SNAPSHOT

dbtsai
authored and
kai-chi
committed
[SPARK-26859][SQL] Fix field writer index bug in non-vectorized ORC deserializer

authored and
kai-chi
committed
Preparing Spark release v2.4.1-rc3

dbtsai
authored and
kai-chi
committed
Preparing development version 2.4.2-SNAPSHOT

dbtsai
authored and
kai-chi
committed
[R][BACKPORT-2.4] update package description

felixcheung
authored and
kai-chi
committed
Preparing Spark release v2.4.1-rc4

dbtsai
authored and
kai-chi
committed
Preparing development version 2.4.2-SNAPSHOT

dbtsai
authored and
kai-chi
committed
[R][BACKPORT-2.3] update package description

felixcheung
authored and
kai-chi
committed
Revert "[R][BACKPORT-2.3] update package description"

HyukjinKwon
authored and
kai-chi
committed
[SPARK-26950][SQL][TEST] Make RandomDataGenerator use Float.NaN or Double.NaN for all NaN values

dongjoon-hyun
authored and
kai-chi
committed
Preparing Spark release v2.4.1-rc5

dbtsai
authored and
kai-chi
committed
Preparing development version 2.4.2-SNAPSHOT

dbtsai
authored and
kai-chi
committed
[MINOR][BUILD] Update all checkstyle dtd to use "https://checkstyle.org"

HeartSaVioR
authored and
kai-chi
committed
[SPARK-26990][SQL][BACKPORT-2.4] FileIndex: use user specified field names if possible

gengliangwang
authored and
kai-chi
committed
[SPARK-27046][DSTREAMS] Remove SPARK-19185 related references from documentation

gaborgsomogyi
authored and
kai-chi
committed
[MINOR][DOCS] Clarify that Spark apps should mark Spark as a 'provided' dependency, not package it

srowen
authored and
kai-chi
committed
[SPARK-26932][DOC] Add a warning for Hive 2.1.1 ORC reader issue

haiboself
authored and
kai-chi
committed
[SPARK-24669][SQL] Invalidate tables in case of DROP DATABASE CASCADE

Udbhav30
authored and
kai-chi
committed
[SPARK-23433][SPARK-25250][CORE] Later created TaskSet should learn about the finished partitions

Ngone51
authored and
kai-chi
committed
[SPARK-27065][CORE] avoid more than one active task set managers for a stage

cloud-fan
authored and
kai-chi
committed
[SPARK-27019][SQL][WEBUI] onJobStart happens after onExecutionEnd shouldn't overwrite kvstore

shahidki31
authored and
kai-chi
committed
[SPARK-27078][SQL] Fix NoSuchFieldError when read Hive materialized views

wangyum
authored and
kai-chi
committed
[SPARK-25863][SPARK-21871][SQL] Check if code size statistics is empty or not in updateAndGetCompilationStats

maropu
authored and
kai-chi
committed
[SPARK-26604][CORE][BACKPORT-2.4] Clean up channel registration for StreamManager

authored and
kai-chi
committed
Preparing Spark release v2.4.1-rc7

dbtsai
authored and
kai-chi
committed
Preparing development version 2.4.2-SNAPSHOT

dbtsai
authored and
kai-chi
committed
[SPARK-27080][SQL] bug fix: mergeWithMetastoreSchema with uniform lower case comparison

CodeGod
authored and
kai-chi
committed
[SPARK-27111][SS] Fix a race that a continuous query may fail with InterruptedException

zsxwing
authored and
kai-chi
committed
[SPARK-27097][CHERRY-PICK 2.4] Avoid embedding platform-dependent offsets literally in whole-stage generated code

authored and
kai-chi
committed
Preparing Spark release v2.4.1-rc8

dbtsai
authored and
kai-chi
committed
Preparing development version 2.4.2-SNAPSHOT

dbtsai
authored and
kai-chi
committed
[SPARK-26927][CORE] Ensure executor is active when processing events in dynamic allocation manager.

Liupengcheng
authored and
kai-chi
committed
[MINOR][CORE] Use https for bintray spark-packages repository

HeartSaVioR
authored and
kai-chi
committed
[SPARK-26742][K8S][BRANCH-2.4] Update k8s client version to 4.1.2

Stavros Kontopoulos
authored and
kai-chi
committed
[SPARK-27165][SPARK-27107][BRANCH-2.4][BUILD][SQL] Upgrade Apache ORC to 1.5.5

dongjoon-hyun
authored and
kai-chi
committed
[SPARK-27134][SQL] array_distinct function does not work correctly with columns containing array of array

dilipbiswal
authored and
kai-chi
committed
[SPARK-27178][K8S][BRANCH-2.4] adding nss package to fix tests

shaneknapp
authored and
kai-chi
committed
[SPARK-27112][CORE] : Create a resource ordering between threads to r…

pgandhi
authored and
kai-chi
committed
[SPARK-26606][CORE] Handle driver options properly when submitting to standalone cluster mode via legacy Client

HeartSaVioR
authored and
kai-chi
committed
[SPARK-27160][SQL] Fix DecimalType when building orc filters

da-liii
authored and
kai-chi
committed
Revert "[SPARK-26606][CORE] Handle driver options properly when submitting to standalone cluster mode via legacy Client"

Marcelo Vanzin
authored and
kai-chi
committed
[SPARK-24935][SQL] fix Hive UDAF with two aggregation buffers

authored and
kai-chi
committed
Revert "Revert "[SPARK-26606][CORE] Handle driver options properly when submitting to standalone cluster mode via legacy Client""

HeartSaVioR
authored and
kai-chi
committed
[SPARK-27094][YARN][BRANCH-2.4] Work around RackResolver swallowing thread interrupt.

Marcelo Vanzin
authored and
kai-chi
committed
[SPARK-27198][CORE] Heartbeat interval mismatch in driver and executor

ajithme
authored and
kai-chi
committed
[SPARK-27274][DOCS] Fix references to scala 2.11 in 2.4.1+ docs; Note 2.11 support is deprecated in 2.4.1+

srowen
authored and
kai-chi
committed
[SPARK-26961][CORE] Enable parallel classloading capability

ajithme
authored and
kai-chi
committed
Preparing Spark release v2.4.1-rc9

dbtsai
authored and
kai-chi
committed
Preparing development version 2.4.2-SNAPSHOT

dbtsai
authored and
kai-chi
committed
[SPARK-27275][CORE] Fix potential corruption in EncryptedMessage.transferTo (2.4)

zsxwing
authored and
kai-chi
committed
[SPARK-27244][CORE] Redact Passwords While Using Option logConf=true

Ninad Ingole
authored and
kai-chi
committed
[SPARK-27301][DSTREAM] Shorten the FileSystem cached life cycle to the cleanup method inner scope

yaooqinn
authored and
kai-chi
committed
[SPARK-27267][CORE] Update snappy to avoid error when decompressing empty serialized data

srowen
authored and
kai-chi
committed
[SPARK-27267][FOLLOWUP][BRANCH-2.4] Update hadoop-2.6 dependency manifest

dongjoon-hyun
authored and
kai-chi
committed
[MINOR][R] fix R project description

felixcheung
authored and
kai-chi
committed
[SPARK-27244][CORE][TEST][FOLLOWUP] toDebugString redacts sensitive information

gatorsmile
authored and
kai-chi
committed
[SPARK-26998][CORE] Remove SSL configuration from executors

gaborgsomogyi
authored and
kai-chi
committed
[SPARK-27346][SQL] Loosen the newline assert condition on 'examples' field in ExpressionInfo

HyukjinKwon
authored and
kai-chi
committed
[MINOR][DOC][SQL] Remove out-of-date doc about ORC in DataFrameReader and Writer

viirya
authored and
kai-chi
committed
[SPARK-27338][CORE] Fix deadlock in UnsafeExternalSorter.SpillableIterator when locking both UnsafeExternalSorter.SpillableIterator and TaskMemoryManager

Venkata krishnan Sowrirajan
authored and
kai-chi
committed
[SPARK-27338][CORE][FOLLOWUP] remove trailing space

cloud-fan
authored and
kai-chi
committed
[SPARK-27382][SQL][TEST] Update Spark 2.4.x testing in HiveExternalCatalogVersionsSuite

dongjoon-hyun
authored and
kai-chi
committed
[SPARK-27216][CORE][BACKPORT-2.4] Upgrade RoaringBitmap to 0.7.45 to fix Kryo unsafe ser/dser issue

LantaoJin
authored and
kai-chi
committed
[MINOR][DOC] Fix html tag broken in configuration.md

HeartSaVioR
authored and
kai-chi
committed
[SPARK-27358][UI] Update jquery to 1.12.x to pick up security fixes

srowen
authored and
kai-chi
committed
[SPARK-27390][CORE][SQL][TEST] Fix package name mismatch

dongjoon-hyun
authored and
kai-chi
committed
[SPARK-27391][SS] Don't initialize a lazy val in ContinuousExecution job.

jose-torres
authored and
kai-chi
committed
[SPARK-27419][CORE] Avoid casting heartbeat interval to seconds (2.4)

zsxwing
authored and
kai-chi
committed
[SPARK-27406][SQL] UnsafeArrayData serialization breaks when two machi…

mingbo_pb
authored and
kai-chi
committed
[SPARK-27394][WEBUI] Flush LiveEntity if necessary when receiving SparkListenerExecutorMetricsUpdate (backport 2.4)

zsxwing
authored and
kai-chi
committed
Revert "[SPARK-23433][SPARK-25250][CORE] Later created TaskSet should learn about the finished partitions"

cloud-fan
authored and
kai-chi
committed
[SPARK-27351][SQL] Wrong outputRows estimation after AggregateEstimation wit…

authored and
kai-chi
committed
[SPARK-27453] Pass partitionBy as options in DataFrameWriter

liwensun
authored and
kai-chi
committed
[SPARK-27479][BUILD] Hide API docs for org.apache.spark.util.kvstore

gatorsmile
authored and
kai-chi
committed
[SPARK-27403][SQL] Fix `updateTableStats` to update table stats always with new stats or None

sujith71955
authored and
kai-chi
committed
Preparing Spark release v2.4.2-rc1

cloud-fan
authored and
kai-chi
committed
Preparing development version 2.4.3-SNAPSHOT

cloud-fan
authored and
kai-chi
committed
[MINOR][TEST] Expand spark-submit test to allow python2/3 executable

srowen
authored and
kai-chi
committed
[SPARK-25079][PYTHON][BRANCH-2.4] update python3 executable to 3.6.x

shaneknapp
authored and
kai-chi
committed
[SPARK-24601][SPARK-27051][BACKPORT][CORE] Update to Jackson 2.9.8

srowen
authored and
kai-chi
committed
[SPARK-27496][CORE] Fatal errors should also be sent back to the sender

zsxwing
authored and
kai-chi
committed
[SPARK-27419][FOLLOWUP][DOCS] Add note about spark.executor.heartbeatInterval change to migration guide

srowen
authored and
kai-chi
committed
[SPARK-27469][BUILD][BRANCH-2.4] Unify commons-beanutils deps to latest 1.9.3

srowen
authored and
kai-chi
committed
[SPARK-27539][SQL] Fix inaccurate aggregate outputRows estimation with column containing null values

pengbo
authored and
kai-chi
committed
[SPARK-27544][PYTHON][TEST][BRANCH-2.4] Fix Python test script to work on Scala-2.12 build

dongjoon-hyun
authored and
kai-chi
committed
[SPARK-27550][TEST][BRANCH-2.4] Fix `test-dependencies.sh` not to use `kafka-0-8`profile for Scala-2.12

dongjoon-hyun
authored and
kai-chi
committed
[MINOR][TEST] switch from 2.4.1 to 2.4.2 in HiveExternalCatalogVersionsSuite

cloud-fan
authored and
kai-chi
committed
[SPARK-27494][SS] Null values don't work in Kafka source v2

uncleGen
authored and
kai-chi
committed
add missing import and fix compilation

cloud-fan
authored and
kai-chi
committed
[SPARK-27563][SQL][TEST] automatically get the latest Spark versions in HiveExternalCatalogVersionsSuite

cloud-fan
authored and
kai-chi
committed
[SPARK-26891][BACKPORT-2.4][YARN] Fixing flaky test in YarnSchedulerBackendSuite

attilapiros
authored and
kai-chi
committed
[SPARK-25535][CORE][BRANCH-2.4] Work around bad error handling in commons-crypto.

Marcelo Vanzin
authored and
kai-chi
committed
[SPARK-26745][SQL][TESTS] JsonSuite test case: empty line -> 0 record count

Branden Smith
authored and
kai-chi
committed
[SPARK-24935][SQL][FOLLOWUP] support INIT -> UPDATE -> MERGE -> FINISH in Hive UDAF adapter

cloud-fan
authored and
kai-chi
committed
Revert "[SPARK-24601][SPARK-27051][BACKPORT][CORE] Update to Jackson 2.9.8

gatorsmile
authored and
kai-chi
committed
Preparing Spark release v2.4.3-rc1

Xiao Li
authored and
kai-chi
committed
[HOPSWORKS-1081] Upgrade Spark to 2.4.3
kai-chi
committed