Add adaptive execution policy

[tdcmeehan]

Add a simple adaptive execution policy that delegates to all-at-once scheduling for low stage queries, and phased scheduling for high stage queries. The aim is to keep lower stage queries at a reasonable latency, while making high stage queries more reliable at the potential cost of higher latency--reducing the maximum concurrent running task count can improve reliability in heavily loaded clusters that run very high-stage queries, as may be generated by BI tools, ETL jobs, or complex analytical queries.

Test plan - Ran this configuration with a workload that skewed toward high stage queries and observed improvement in running task count. While this test may not always reflect the real world, in reality reliability issues are typically seen when a burst of high stage queries are executed in tandem in a cluster. In other words, this change should keep typical task counts constant, but improve (lower) the running task count in the worst case where reliability issues are encountered.

== RELEASE NOTES ==

General Changes


* Add an adaptive stage scheduling policy that switches to phased execution mode once a query's stage count exceeds a configurable upper bound. This can be enabled by setting the session property ``execution_policy`` to ``phased`` and the stage count limit can be configured by the session property ``max_stage_count_for_eager_scheduling``.

[mayankgarg1990]

This looks good to me . While query latency is an acceptable impact to protect the cluster, do we expect some sort of memory impact as well ? I am not completely familiar with how phased execution is implemented, so I maybe wrong here - but since some stages might not be scheduled, we might have data staying in memory for longer and that can increase the memory-second value of a query?

[mayankgarg1990]

Add an adaptive stage scheduling policy that switches to phased execution mode once a query's stage count exceeds a configurable upper bound. This can be enabled by setting the session property ``execution_policy`` to ``phased`` and the stage count limit can be configured by the session property ``max_stage_count_for_eager_scheduling``.

[swapsmagic]

It make more sense to call this config Adaptive scheduling over eager scheduling. wdyt?

[tdcmeehan]

Hmm, I'm putting "max for eager scheduling", meaning the maximum level before which we use eager scheduling, and after which we use phased scheduling.

[swapsmagic]

Can we add some unit test to verify this works as expected?

[tdcmeehan]

This looks good to me . While query latency is an acceptable impact to protect the cluster, do we expect some sort of memory impact as well ? I am not completely familiar with how phased execution is implemented, so I maybe wrong here - but since some stages might not be scheduled, we might have data staying in memory for longer and that can increase the memory-second value of a query?

I actually think this will likely improve memory, since we won't create tasks for the entire plan for large stage queries, and task-related overhead dominates memory usage in heavily loaded clusters.