[SPARK-7889] [Core] WiP History Caching: xu tingjun's patch against master

[steveloughran]

This is @XuTingjun's patch (#6545) reapplied to trunk with a bit of cleanup (the probe in the history provider is now isComplete(String appId, Option[String] attemptId; the conditions around its use and is internals slightly tweaked.

Looking at this code now, I do agree the history provider should have a say in the state of the attempts, I'm not sure if this is the right approach. Having 3x different maps to track app state (including two that never get cleaned up) isn't ideal, and if it were to be so, then the two new ones would have to be thread safe, so as to handle the scenario of "multiple requests coming in at the same time"

I'd rather retain the history server's information alongside the cached app, (as my patch does), though that still leaves the question of how to probe for a updated version and then reload it, especially in a way that is thread safe.

Oh, and testing, obviously.

(once this PR has been tested I'm going to close it; it's here as reference)

[SparkQA]

Test build #46543 has finished for PR 9913 at commit 865808e.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.