Add auto-tuning of max executor and hash partition

[vermapratyush]

Test Plan

Added unit test to the classes modified.
Ran presto server on local and submitted queries on "customers" table to confirm correct hash_partition_count is used.

Summary

Added 2 configuration options (spark_executor_allocation_strategy_enabled, spark_hash_partition_count_allocation_strategy_enabled) to auto-tune, spark.dynamicAllocation.maxExecutors and hash_partition_count based on the input table stats, respectively.

The auto tune configuration option take precedence over explicitly provided values for spark.dynamicAllocation.maxExecutors and hash_partition_count.

If spark_resource_allocation_strategy_enabled is enabled, both executor and hash partition are considered to be auto-tune enabled. This configuration option could be removed in subsequent revision, as it is redundant.

== RELEASE NOTES ==

General Changes
* Add property ``spark_executor_allocation_strategy_enabled`` to auto-tune spark max executor count (``spark.dynamicAllocation.maxExecutors``)  based on input data. Only required if ``spark_resource_allocation_strategy_enabled`` is not already enabled.
* Add property ``spark_hash_partition_count_allocation_strategy_enabled`` to auto-tune hash partition count (``hash_partition_count``) based on input data. Only required if ``spark_resource_allocation_strategy_enabled`` is not already enabled.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: vermapratyush / name: Pratyush Verma (1ca1e3296362bb6a3b977061034fd96af0843237)

[v-jizhang]

@bot kick off tests

[vermapratyush]

@bot kick off tests

[ajaygeorge]

@vermapratyush Can you please sign the CLA

[ajaygeorge]

@vermapratyush Please squash the commits as well.

[vermapratyush]

@vermapratyush Can you please sign the CLA

@ajaygeorge Already signed the CLA, but somehow it still keeps failing.

[ajaygeorge]

@vermapratyush Can you please sign the CLA

@ajaygeorge Already signed the CLA, but somehow it still keeps failing.

Can you try signing the individual contributor CLA as well if not already.

[ajaygeorge]

Added some comments

[vermapratyush]

Unrelated failure in module presto-cache.
Re-ran the test locally and was successful.

[pgupta2]

We can do the check for the main property here: isSparkResourceAllocationStrategyEnabled(session) and if its disabled, we return back. This is basically a single switch to disable allocation strategy completely.

Basically, my thinking is that spark_resource_allocation_strategy_enabled control this whole logic. If this is disabled, we skip this logic completely. If spark_resource_allocation_strategy_enabled is true, then only we check for individual strategy properties and trigger the one that is enabled.

I know this is bit weird. Ideally, we should just remove spark_resource_allocation_strategy_enabled session property to make this logic cleaner

@highker : What is the right approach of deprecating/removing a property?

[highker]

What is the right approach of deprecating/removing a property?

• We use @DefunctConfig annotation in a config class. Usually, we also rename the property to deprecated.XXX. Check FeatureConfig for examples. We don't usually delete the property directly.
• We have release note to tell users what config has the deprecated one migrated to

[pgupta2]

Check for isSparkResourceAllocationStrategyEnabled(session) can be removed here.

[vermapratyush]

Will deprecate this config in subsequent PR.

[pgupta2]

Should we make it an AND condition? That will give us the ability to control individual optimizations separately. Else if isSparkResourceAllocationStrategyEnabled(session) is true, both optimizations will run regardless.

[pgupta2]

Same as above comment.

[vermapratyush]

Will deprecate this config in subsequent PR.

[ajaygeorge]

LGTM % a few nits.

[pgupta2]

Just 1 final comment.

[pgupta2]

Should we make it an AND condition? That will give us the ability to control individual optimizations separately. Else if isSparkResourceAllocationStrategyEnabled(session) is true, both optimizations will run regardless.

[vermapratyush]

@pgupta2 I don't want to couple isSparkResourceAllocationStrategyEnabled and isSparkHashPartitionCountAllocationStrategyEnabled together since isSparkResourceAllocationStrategyEnabled will be going away in subsequent PR anyway.