[SPARK-29637][CORE] Add description to Job SHS web API

[gaborgsomogyi]

Why are the changes needed?

Starting from Spark 2.3, the SHS REST API endpoint /applications/<app_id>/jobs/ is not including description in the JobData returned. This is not the case until Spark 2.2.

In this PR I've added the mentioned field.

Does this PR introduce any user-facing change?

Yes.

Old API response:

[ {
  "jobId" : 0,
  "name" : "foreach at <console>:26",
  "submissionTime" : "2019-10-28T12:41:54.301GMT",
  "completionTime" : "2019-10-28T12:41:54.731GMT",
  "stageIds" : [ 0 ],
  "jobGroup" : "test",
  "status" : "SUCCEEDED",
  "numTasks" : 1,
  "numActiveTasks" : 0,
  "numCompletedTasks" : 1,
  "numSkippedTasks" : 0,
  "numFailedTasks" : 0,
  "numKilledTasks" : 0,
  "numCompletedIndices" : 1,
  "numActiveStages" : 0,
  "numCompletedStages" : 1,
  "numSkippedStages" : 0,
  "numFailedStages" : 0,
  "killedTasksSummary" : { }
} ]

New API response:

[ {
  "jobId" : 0,
  "name" : "foreach at <console>:26",
  "description" : "job",                            <= This is the addition here
  "submissionTime" : "2019-10-28T13:37:24.107GMT",
  "completionTime" : "2019-10-28T13:37:24.613GMT",
  "stageIds" : [ 0 ],
  "jobGroup" : "test",
  "status" : "SUCCEEDED",
  "numTasks" : 1,
  "numActiveTasks" : 0,
  "numCompletedTasks" : 1,
  "numSkippedTasks" : 0,
  "numFailedTasks" : 0,
  "numKilledTasks" : 0,
  "numCompletedIndices" : 1,
  "numActiveStages" : 0,
  "numCompletedStages" : 1,
  "numSkippedStages" : 0,
  "numFailedStages" : 0,
  "killedTasksSummary" : { }
} ]

How was this patch tested?

Extended + existing unit tests.

Manually:

• Open spark-shell

scala> sc.setJobGroup("test", "job", false); 
scala> val foo = sc.textFile("/user/foo.txt");
foo: org.apache.spark.rdd.RDD[String] = /user/foo.txt MapPartitionsRDD[1] at textFile at <console>:24
scala> foo.foreach(println);

• Access REST API http://SHS-host:port/api/v1/applications/<app-id>/jobs/

[gaborgsomogyi]

I'm fully aware that in case of bugs we normally create a new test containing the jira ID. In this case I've found a test which can be extended and thought a new test would be an overkill. If one thinks it's still worth to add a separate test just bring it up and we can make the adjustment.

[MaxGekk]

In the Does this PR introduce any user-facing change? section, I don't see the difference between New API response and Old API response or maybe I missed something?

[gaborgsomogyi]

@MaxGekk "description" : "job",

[gaborgsomogyi]

I've added a marker in the JSON to highlight it.

[SparkQA]

Test build #112850 has finished for PR 26295 at commit 005c3e5.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[vanzin]

Merging to master / 2.4.