[SPARK-10984] Simplify *MemoryManager class structure

[JoshRosen]

This patch refactors the MemoryManager class structure. After #9000, Spark had the following classes:

• MemoryManager
• StaticMemoryManager
• ExecutorMemoryManager
• TaskMemoryManager
• ShuffleMemoryManager

This is fairly confusing. To simplify things, this patch consolidates several of these classes:

• ShuffleMemoryManager and ExecutorMemoryManager were merged into MemoryManager.
• TaskMemoryManager is moved into Spark Core.

Key changes and tasks:

• Merge ExecutorMemoryManager into MemoryManager.
  • Move pooling logic into Allocator.
• Move TaskMemoryManager from spark-unsafe to spark-core.
• Refactor the existing Tungsten TaskMemoryManager interactions so Tungsten code use only this and not both this and ShuffleMemoryManager.
• Refactor non-Tungsten code to use the TaskMemoryManager instead of ShuffleMemoryManager.
• Merge ShuffleMemoryManager into MemoryManager.
  • Move code
  • Simplify 1/n calculation. Will defer to followup, since this needs more work.
• Port ShuffleMemoryManagerSuite tests.
• Move classes from unsafe package to memory package.
• Figure out how to handle the hacky use of the memory managers in HashedRelation's broadcast variable construction.
• Test porting and cleanup: several tests relied on mock functionality (such as TestShuffleMemoryManager.markAsOutOfMemory) which has been changed or broken during the memory manager consolidation
  • AbstractBytesToBytesMapSuite
  • UnsafeExternalSorterSuite
  • UnsafeFixedWidthAggregationMapSuite
  • UnsafeKVExternalSorterSuite

Compatiblity notes:

• This patch introduces breaking changes in ExternalAppendOnlyMap, which is marked as @DevloperAPI (likely for legacy reasons): this class now cannot be used outside of a task.

[SparkQA]

Test build #43753 has finished for PR 9127 at commit 25ba4b5.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

Jenkins, retest this please.

[SparkQA]

Test build #43789 has finished for PR 9127 at commit 25ba4b5.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #43858 has finished for PR 9127 at commit 8f93e94.

• This patch fails MiMa tests.
• This patch does not merge cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #43862 has finished for PR 9127 at commit 3bbc54d.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #43866 has finished for PR 9127 at commit ec48ff9.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #43875 has finished for PR 9127 at commit 7d6a37f.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

Alright, the first part of this change should be ready for review. At this point, there are a number of remaining small tasks for updating mocks which are used in unit tests, but the basic "client" APIs for interacting with the memory managers have been sketched out. Please take a look at those and provide feedback on whether the new, combined interfaces make sense. Once we agree on those interfaces, we can loop back and do a second review pass looking at the test refactoring code.

[SparkQA]

Test build #44170 has finished for PR 9127 at commit f21b767.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

This use of the memory managers in HashedRelation was messy before and is slightly messier now; I'd like to find a clean way to deal with this before merging this refactoring.

[SparkQA]

Test build #44253 has finished for PR 9127 at commit 04ec429.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #44255 has finished for PR 9127 at commit e56d039.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #44261 has finished for PR 9127 at commit aa14113.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #44269 has finished for PR 9127 at commit 5af0b17.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #44274 has finished for PR 9127 at commit a264703.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

This PR is logically blocked on #9260.

[SparkQA]

Test build #44290 has finished for PR 9127 at commit 0b5c72f.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #44326 has finished for PR 9127 at commit f68fdb1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[JoshRosen]

Woohoo, this passes tests!

There are still a few minor follow-up tasks that I'd like to do for this, but I'm going to defer them to separate patches: this patch is fairly large and has conflicts with several other memory-management-related patches that are in-flight or which will be opened soon. @andrewor14, if you have any post-hoc review comments, I'll handle them in a followup. @davies, this should unblock your open patch.

Merging to master.

[JoshRosen]

I filed https://issues.apache.org/jira/browse/SPARK-11309 as one followup.

[davies]

@JoshRosen Great, thanks!