[SPARK-10885][Streaming]Display the failed output op in Streaming UI

[zsxwing]

This PR implements the following features for both master and branch-1.5.

1. Display the failed output op count in the batch list
2. Display the failure reason of output op in the batch detail page

Screenshots:

There are still two remaining problems in the UI.

1. If an output operation doesn't run any spark job, we cannot get the its duration since now it's the sum of all jobs' durations.
2. If an output operation doesn't run any spark job, we cannot get the description since it's the latest job's call site.

We need to add new StreamingListenerEvent about output operations to fix them. So I'd like to fix them only for master in another PR.

[zsxwing]

/cc @tdas

[SparkQA]

Test build #43133 has finished for PR 8950 at commit f940ccf.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[tdas]

If we are showing "Succeeded" for successful output ops, then we should also show "Failed" for failed ones as well. In fact, there should be "Failed" and "Failed due to Spark job error". In the first case, the output op error will be below the "Failed" .

[zsxwing]

In fact, there should be "Failed" and "Failed due to Spark job error". In the first case, the output op error will be below the "Failed" .

I think now we cannot distinguish these two errors. They are both caught in org.apache.spark.streaming.scheduler.Job.run.

[tdas]

Cant we distinguish based on the whether there is a Spark job error in any of the jobs associated with the output op

[zsxwing]

I don't think so. E.g., the use may write the following codes:

stream.foreachRDD { rdd =>
  try {
    rdd.foreach(...)
  } catch {
     ...
  }
}

In this case, if a Spark job (rdd.foreach) fails, we cannot say the output op fails.

[tdas]

Here is a possible logic.

If no failure reason, then "Succeeded"
Else if failure reason contains "SparkException", then "Failed due to Spark job error"
Else "Failed"

This should work fine for most cases, where the user is not doing fancy things like catching exceptions themselves and ignoring/rethrowing them. Isnt it?

Consider your example. If the user catches Spark job exception (most probably SparkException) and rethrows it, the above logic should identify it as Spark job error and say "Failed due to Spark job error". On the other hand, if the user catches and ignore exception, then failure reason will be empty and the output will be marked as "Succeeded" (even though Spark job error will not be empty, which is okay).

[zsxwing]

Sounds great. I will update the logic.

[zsxwing]

New screenshot:

[SparkQA]

Test build #43270 has finished for PR 8950 at commit ca68ac8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[tdas]

This looks cool! Also I like that you used "details" for that.

1. In the case of "Failed", unless the details is opened, there is not indication of failure. So it might be better to show "Failed due to error: $exceptionMessage", and the full stacktrace in the detail.
2. In case of "Failed due to Spark error", does it make sense to have the error show up in both places?

[zsxwing]

In the case of "Failed", unless the details is opened, there is not indication of failure. So it might be better to show "Failed due to error: $exceptionMessage", and the full stacktrace in the detail.

Added it.

In case of "Failed due to Spark error", does it make sense to have the error show up in both places?

This seems unnecessary. The job error column has already shown it.

[SparkQA]

Test build #43278 has finished for PR 8950 at commit d7c42c9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[tdas]

LGTM. Merging this to master and 1.5. Thanks @zsxwing