[WIP][SPARK-20628][CORE][K8S] Keep track of nodes (/ spot instances) which are going to be shutdown#19045

Closed

holdenk wants to merge 213 commits intoapache:masterfrom

holdenk:SPARK-20628-keep-track-of-nodes-which-are-going-to-be-shutdown-r2

Commits on Jun 21, 2017

Commits on Jun 22, 2017

Remove sketchy println debugging
holdenk
committed

Commits on Jul 6, 2017

Commits on Aug 16, 2017

Merge branch 'master' into SPARK-20628-keep-track-of-nodes-which-are-going-to-be-shutdown-r2
holdenk
committed

Commits on Aug 17, 2017

Merge branch 'master' into SPARK-20628-keep-track-of-nodes-which-are-going-to-be-shutdown-r2
holdenk
committed

Commits on Aug 13, 2018

Merge in master
holdenk
committed

Commits on Aug 27, 2018

Commits on Sep 8, 2018

Commits on Sep 19, 2018

Commits on Sep 21, 2018

Add a WIP Decom suite work
holdenk
committed

Commits on Oct 12, 2018

Commits on Oct 16, 2018

Add missing sys import
holdenk
committed

Commits on Oct 25, 2018

Merge branch 'master' into SPARK-20628-keep-track-of-nodes-which-are-going-to-be-shutdown-r2
holdenk
committed

Commits on Oct 26, 2018

Add back appArgs since despite Ilan's comments during the stream they aren't optional for this code path :p
holdenk
committed

Commits on Nov 8, 2018

Remove import
holdenk
committed

Commits on Dec 28, 2018

Commits on Jan 5, 2019

[SPARK-26193][SQL][FOLLOW UP] Read metrics rename and display text changes

xuanyuanking
authored and
holdenk
committed
[SPARK-19827][R][FOLLOWUP] spark.ml R API for PIC

srowen
authored and
holdenk
committed
[SPARK-24102][ML][MLLIB] ML Evaluators should use weight column - added weight column for regression evaluator

imatiach-msft
authored and
holdenk
committed
[SPARK-25877][K8S] Move all feature logic to feature classes.

Marcelo Vanzin
authored and
holdenk
committed
[SPARK-25277][YARN] YARN applicationMaster metrics should not register static metrics

LucaCanali
authored and
holdenk
committed
[SPARK-26322][SS] Add spark.kafka.sasl.token.mechanism to ease delegation token configuration.

gaborgsomogyi
authored and
holdenk
committed
[SPARK-26297][SQL] improve the doc of Distribution/Partitioning

cloud-fan
authored and
holdenk
committed
[SPARK-26348][SQL][TEST] make sure expression is resolved during test

cloud-fan
authored and
holdenk
committed
[SPARK-26355][PYSPARK] Add a workaround for PyArrow 0.11.

ueshin
authored and
holdenk
committed
[MINOR][R] Fix indents of sparkR welcome message to be consistent with pyspark and spark-shell

AzureQ
authored and
holdenk
committed
[MINOR][DOC] Fix comments of ConvertToLocalRelation rule

seancxmao
authored and
holdenk
committed
[MINOR][DOC] update the condition description of BypassMergeSortShuffle

lcqzte10192193
authored and
holdenk
committed
[SPARK-26340][CORE] Ensure cores per executor is greater than cpu per task

n.fraison
authored and
holdenk
committed
[SPARK-26313][SQL] move `newScanBuilder` from Table to read related mix-in traits

cloud-fan
authored and
holdenk
committed
[SPARK-26098][WEBUI] Show associated SQL query in Job page

gengliangwang
authored and
holdenk
committed
[SPARK-23886][SS] Update query status for ContinuousExecution

gaborgsomogyi
authored and
holdenk
committed
[SPARK-26364][PYTHON][TESTING] Clean up imports in test_pandas_udf*

icexelloss
authored and
holdenk
committed
[SPARK-26360] remove redundant validateQuery call

JasonWayne
authored and
holdenk
committed
[SPARK-26337][SQL][TEST] Add benchmark for LongToUnsafeRowMap

viirya
authored and
holdenk
committed
[SPARK-26368][SQL] Make it clear that getOrInferFileFormatSchema doesn't create InMemoryFileIndex

rxin
authored and
holdenk
committed
[SPARK-26370][SQL] Fix resolution of higher-order function for the same identifier.

ueshin
authored and
holdenk
committed
[MINOR][SQL] Some errors in the notes.

CarolinPeng
authored and
holdenk
committed
[SPARK-26265][CORE][FOLLOWUP] Put freePage into a finally block

viirya
authored and
holdenk
committed
[SPARK-26362][CORE] Remove 'spark.driver.allowMultipleContexts' to disallow multiple creation of SparkContexts

HyukjinKwon
authored and
holdenk
committed
[SPARK-26315][PYSPARK] auto cast threshold from Integer to Float in approxSimilarityJoin of BucketedRandomProjectionLSHModel

Jing Chen He
authored and
holdenk
committed
[SPARK-26243][SQL] Use java.time API for parsing timestamps and dates from JSON

MaxGekk
authored and
holdenk
committed
[SPARK-26078][SQL] Dedup self-join attributes on IN subqueries

mgaido91
authored and
holdenk
committed
[SPARK-26372][SQL] Don't reuse value from previous row when parsing bad CSV input field

bersprockets
authored and
holdenk
committed
[SPARK-26248][SQL] Infer date type from CSV

MaxGekk
authored and
holdenk
committed
[MINOR][DOCS] Fix the "not found: value Row" error on the "programmatic_schema" example

kjmrknsn
authored and
holdenk
committed
Revert "[SPARK-26248][SQL] Infer date type from CSV"

HyukjinKwon
authored and
holdenk
committed
[SPARK-26352][SQL] join reorder should not change the order of output attributes

rednaxelafx
authored and
holdenk
committed
[SPARK-26327][SQL][FOLLOW-UP] Refactor the code and restore the metrics name

gatorsmile
authored and
holdenk
committed
[SPARK-20636] Add the rule TransposeWindow to the optimization batch

gatorsmile
authored and
holdenk
committed
[SPARK-26243][SQL][FOLLOWUP] fix code style issues in TimestampFormatter.scala

cloud-fan
authored and
holdenk
committed
[SPARK-20351][ML] Add trait hasTrainingSummary to replace the duplicate code

YY-OnCall
authored and
holdenk
committed
[SPARK-26255][YARN] Apply user provided UI filters to SQL tab in yarn mode

chakravarthiT
authored and
holdenk
committed
[SPARK-26371][SS] Increase kafka ConfigUpdater test coverage.

gaborgsomogyi
authored and
holdenk
committed
[SPARK-24933][SS] Report numOutputRows in SinkProgress

authored and
holdenk
committed
[SPARK-25922][K8] Spark Driver/Executor "spark-app-selector" label mismatch

authored and
holdenk
committed
[SPARK-24561][SQL][PYTHON] User-defined window aggregation functions with Pandas UDF (bounded window)

icexelloss
authored and
holdenk
committed
[SPARK-26246][SQL] Inferring TimestampType from JSON

MaxGekk
authored and
holdenk
committed
[SPARK-26081][SQL][FOLLOW-UP] Use foreach instead of misuse of map (for Unit)

HyukjinKwon
authored and
holdenk
committed
[SPARK-24680][DEPLOY] Support spark.executorEnv.JAVA_HOME in Standalone mode

stanzhai
authored and
holdenk
committed
[SPARK-26384][SQL] Propagate SQL configs for CSV schema inferring

MaxGekk
authored and
holdenk
committed
[SPARK-26382][CORE] prefix comparator should handle -0.0

cloud-fan
authored and
holdenk
committed
[SPARK-26394][CORE] Fix annotation error for Utils.timeStringAsMs

Jackey Lee
authored and
holdenk
committed
[SPARK-25815][K8S] Support kerberos in client mode, keytab-based token renewal.

Marcelo Vanzin
authored and
holdenk
committed
[SPARK-26366][SQL] ReplaceExceptWithFilter should consider NULL as False

mgaido91
authored and
holdenk
committed
[SPARK-26390][SQL] ColumnPruning rule should only do column pruning

cloud-fan
authored and
holdenk
committed
[SPARK-26262][SQL] Runs SQLQueryTestSuite on mixed config sets: WHOLESTAGE_CODEGEN_ENABLED and CODEGEN_FACTORY_MODE

maropu
authored and
holdenk
committed
[SPARK-25271][SQL] Hive ctas commands should use data source if it is convertible

viirya
authored and
holdenk
committed
[SPARK-26318][SQL] Deprecate Row.merge

KyleLi1985
authored and
holdenk
committed
[SPARK-26308][SQL] Avoid cast of decimals for ScalaUDF

mgaido91
authored and
holdenk
committed
[SPARK-24687][CORE] Avoid job hanging when generate task binary causes fatal error

caneGuy
authored and
holdenk
committed
[SPARK-26324][DOCS] Add Spark docs for Running in Mesos with SSL

authored and
holdenk
committed
[SPARK-26409][SQL][TESTS] SQLConf should be serializable in test sessions

gengliangwang
authored and
holdenk
committed
[SPARK-26392][YARN] Cancel pending allocate requests by taking locality preference into account

Ngone51
authored and
holdenk
committed
[SPARK-25970][ML] Add Instrumentation to PrefixSpan

zhengruifeng
authored and
holdenk
committed
[MINOR][SQL] Locality does not need to be implemented

10110346
authored and
holdenk
committed
[SPARK-26422][R] Support to disable Hive support in SparkR even for Hadoop versions unsupported by Hive fork

HyukjinKwon
authored and
holdenk
committed
[SPARK-26267][SS] Retry when detecting incorrect offsets from Kafka

zsxwing
authored and
holdenk
committed
[SPARK-26269][YARN] Yarnallocator should have same blacklist behaviour with yarn to maxmize use of cluster resource

Ngone51
authored and
holdenk
committed
[SPARK-25642][YARN] Adding two new metrics to record the number of registered connections as well as the number of active connections to YARN Shuffle Service

pgandhi
authored and
holdenk
committed
[SPARK-26216][SQL][FOLLOWUP] use abstract class instead of trait for UserDefinedFunction

cloud-fan
authored and
holdenk
committed
[SPARK-26427][BUILD] Upgrade Apache ORC to 1.5.4

dongjoon-hyun
authored and
holdenk
committed
[SPARK-26428][SS][TEST] Minimize deprecated `ProcessingTime` usage

dongjoon-hyun
authored and
holdenk
committed
[SPARK-26430][BUILD][TEST-MAVEN] Upgrade Surefire plugin to 3.0.0-M2

dongjoon-hyun
authored and
holdenk
committed
[SPARK-26285][CORE] accumulator metrics sources for LongAccumulator and Doub…

authored and
holdenk
committed
[SPARK-25245][DOCS][SS] Explain regarding limiting modification on "spark.sql.shuffle.partitions" for structured streaming

HeartSaVioR
authored and
holdenk
committed
[SPARK-26402][SQL] Accessing nested fields with different cases in case insensitive mode

dbtsai
authored and
holdenk
committed
[SPARK-26178][SPARK-26243][SQL][FOLLOWUP] Replacing SimpleDateFormat by DateTimeFormatter in comments

MaxGekk
authored and
holdenk
committed
[SPARK-14023][CORE][SQL] Don't reference 'field' in StructField errors for clarity in exceptions

srowen
authored and
holdenk
committed
[SPARK-26426][SQL] fix ExpresionInfo assert error in windows operation system.

wangyanlin01
authored and
holdenk
committed
[SPARK-26424][SQL] Use java.time API in date/timestamp expressions

MaxGekk
authored and
holdenk
committed
[SPARK-26435][SQL] Support creating partitioned table using Hive CTAS by specifying partition column names

viirya
authored and
holdenk
committed
[SPARK-26191][SQL] Control truncation of Spark plans via maxFields parameter

MaxGekk
authored and
holdenk
committed
[SPARK-25892][SQL] Change AttributeReference.withMetadata's return type to AttributeReference

kevinyu98
authored and
holdenk
committed
[SPARK-26451][SQL] Change lead/lag argument name from count to offset

deepyaman
authored and
holdenk
committed
[SPARK-26446][CORE] Add cachedExecutorIdleTimeout docs at ExecutorAllocationManager

wuqingxin
authored and
holdenk
committed
[SPARK-26444][WEBUI] Stage color doesn't change with it's status

seancxmao
authored and
holdenk
committed
[SPARK-26424][SQL][FOLLOWUP] Fix DateFormatClass/UnixTime codegen

dongjoon-hyun
authored and
holdenk
committed
Maybe we have a race condition with the watcher? idk why we aren't getting any pods.
holdenk
committed
wtf is going on with this eventually block
holdenk
committed
Rewrite the eventually's to use should be which I had accidently removed.
holdenk
committed
Change how we handle decom tests to actually decom workers and check that workers are decomed eventually.
holdenk
committed
Revert "Fix using SparkPI for decom test."
holdenk
committed
Revert "Python tests aren't registering the executors, lets avoid that noise and just do SparkPI since we don't really need any special logic in the driver."
holdenk
committed
Match the wait logic (TODO refactor)
holdenk
committed
Debug pods not becoming ready.
holdenk
committed

Commits on Jan 6, 2019

Commits on Jan 8, 2019

Use POD_RUNNING_TIMEOUT
holdenk
committed

Commits on Jan 10, 2019

Merge branch 'master' into SPARK-20628-keep-track-of-nodes-which-are-going-to-be-shutdown-r2
holdenk
committed

Commits on Feb 11, 2019

Commits on Feb 15, 2019

Commits on Feb 16, 2019

Fix pod check
holdenk
committed

Commits on Mar 7, 2019

Commits on Mar 13, 2019

Commits on Mar 14, 2019