Add NativeExecutionShuffleManager for presto spark native execution

[miaoever]

PrestoSparkNativeExecutionShuffleManager is the shuffle manager implementing the Spark shuffle manager interface specifically for native execution. The reasons we have this new shuffle manager are:

1. To bypass calling into Spark java shuffle writer/reader since the actual shuffle read/write will happen in C++ side. In PrestoSparkNativeExecutionShuffleManager, we registered a pair of no-op shuffle reader/writer to hook-up with regular Spark shuffle workflow.
2. To capture the shuffle metadata (eg. {@link ShuffleHandle}) for later use. These metadata are only available during shuffle writer creation internally which is beyond the whole Presto-Spark native execution flow. By using the {@link PrestoSparkNativeExecutionShuffleManager}, we capture and store these metadata inside the shuffle manager and provide the APIs to allow native execution runtime access.

[tdcmeehan]

@bot kick off test

[tdcmeehan]

@prestodbbot kick off test

[tdcmeehan]

@presto-release-bot kick off test

[tanjialiang]

@presto-release-bot kick off test

What is this arcane trick about?

[highker]

nit comments only; will leave to others to review first

[highker]

unused clazz

[highker]

int shuffleId

[highker]

new EmptyShuffleWriter<>

[highker]

same, no "K, C"

[highker]

you don't need this

[highker]

remove this line

[highker]

Can use specialize the type for BypassMergeSortShuffleHandle: BypassMergeSortShuffleHandle<.., ..>

[v-jizhang]

We are deploying a bot that can restart failed tests. It's not ready yet.

[chenyangfb]

Could you explain a bit more about why we need
resetSparkContext(Map<String, String> additionalSparkConfigs)
get(Map<String, String> additionalSparkConfigs)
My understanding is you want add additional configs to existing sparkContext? And this is only needed for testing
Is this correct or not?

[chenyangfb]

looks good overall, have one question about changes in PrestoSparkQueryRunner

[miaoever]

The reason we need reset: the Spark shuffle manager will be only created/initialized once at Spark context/Env creation time, and it'll be used throughout the life time of that Spark context. On other side, In PrestoSpark test suite, all the test cases in one suite will be sharing one Spark context (held by the PrestoSparkQueryRunner) by default. With these two constraints, we have to 'reset' the Spark Context to create a new one to register our new shuffle manager for our testing purpose.

[highker]

This is a private method. Also, move it to the end of the class

[miaoever]

Done.

[v-jizhang]

@bot kick off tests