Support stage level schedule for final write stage

[ulysses-you]

Why are the changes needed?

This pr adds a new rule FinalStageResourceManager to inject custom reousrce profile for the final write stage.

It nowprovide two features:

• Kill redundant executors
  We first get the final stage partition which is the actually required cores, then kill the redundant executors. The priority of kill executors follow:

  1. kill executor who is younger than other (The older the JIT works better)
  2. kill executor who produces less shuffle data first

• Custom resource profile
  We can specify the custom executor resource for final write stage. It now supports:

  executor core
  executor memory
  executor memory overhead
  executor off heap memory
  

The reason why add this feature is that, if the previous stage contains lots executors but final stage has less, then the tasks of final stage would be scheduled randomly in all exists executors which may cause resource waste. e.g., each executor only run 1 or 2 tasks but holds 4 or 5 cores.

How was this patch tested?

test manually

• test for the kill executor

• test for custom resource profile

[codecov-commenter]

Codecov Report

Merging #4574 (5ce2c5d) into master (465e23a) will increase coverage by 0.03%.
The diff coverage is 96.87%.

❗ Current head 5ce2c5d differs from pull request most recent head 54afa36. Consider uploading reports for the commit 54afa36 to get more accurate results

@@             Coverage Diff              @@
##             master    #4574      +/-   ##
============================================
+ Coverage     53.32%   53.36%   +0.03%     
  Complexity       13       13              
============================================
  Files           573      577       +4     
  Lines         31501    31589      +88     
  Branches       4239     4245       +6     
============================================
+ Hits          16797    16856      +59     
- Misses        13128    13147      +19     
- Partials       1576     1586      +10     

Impacted Files	Coverage Δ
...in/scala/org/apache/kyuubi/sql/KyuubiSQLConf.scala	98.61% <96.87%> (-0.50%)	⬇️

... and 22 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[ulysses-you]

It is really hard to add some tests, so I test all of this manully. cc @yaooqinn @pan3793 @bowenliang123 @cfmcgrady if you have some other idea about this.

[bowenliang123]

Is there any description available for the manual test results other than the screenshots? I don't quite understand the screenshots showing consequence related to the purposed features.

[ulysses-you]

@bowenliang123 I have added some tags, ideally, the screenshots show that:

1. kill executor
2. inject custom resource profile

[bowenliang123]

LGTM. Thx for the supplement.

[bowenliang123]

LGTM. Thx for the supplement.

[bowenliang123]

LGTM. Thx for the supplement.

[yaooqinn]

please separate this PR to small ones

[ulysses-you]

I seperate thie pr into two parts: 1. kill executor, 2. inject custom resource profile.

here is the pr #4592 for kill executors