SPARK-4111 [MLlib] add regression metrics

[yanboliang]

Add RegressionMetrics.scala as regression metrics used for evaluation and corresponding test case RegressionMetricsSuite.scala.

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

Typo in name

[srowen]

Update the title with SPARK-XXXX [MLLIB]

[srowen]

I don't see that stats for pred are used?

[yanboliang]

Yes, it's not used and I have remove it in a new commit.

[yanboliang]

Rename re_score() and remove unused column.

[srowen]

I think this might be worth a comment to explain what sums of squares you are trying to compute in the numerator and denominator. A link to the definition might be good, here and for explained variance, since they are related.

[srowen]

This is picky now, but you might write out "meanAverageError" instead of saying "mae". Is "r2_score" style-wise correct vs "r2Score"? (Sorry should have thought of that.) Finally consider using @return tags in your scaladoc to describe what's being returned instead of leaving it blank but writing docs in the body.

[srowen]

Also picky but you can avoid math.pow and avoid computing value - pred 3 times here with a local var. Might be cleaner. This LGTM for what it's worth.

[mengxr]

The third and the forth columns are not necessary. You can use normL1 and normL2 on the second column:

https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/stat/MultivariateOnlineSummarizer.scala#L219

[yanboliang]

Rename parameter and function names to be consistent with spark naming rules.
Delete unused columns and set prediction as the first column.
Add explanation and reference to r2Score and explained variance.
Other code style keeping.

[mengxr]

ok to test

[mengxr]

test this please

[SparkQA]

Test build #22457 has started for PR 2978 at commit a8ad3e3.

• This patch merges cleanly.

[SparkQA]

Test build #22457 timed out for PR 2978 at commit a8ad3e3 after a configured wait of 120m.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22457/
Test FAILed.

[mengxr]

space after ,

[mengxr]

space after ,

[SparkQA]

Test build #22527 has started for PR 2978 at commit 3d0bec1.

• This patch merges cleanly.

[SparkQA]

Test build #22528 has started for PR 2978 at commit 730d0a9.

• This patch merges cleanly.

[SparkQA]

Test build #22527 has finished for PR 2978 at commit 3d0bec1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class RegressionMetrics(predictionAndObservations: RDD[(Double, Double)]) extends Logging

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22527/
Test PASSed.

[SparkQA]

Test build #22528 has finished for PR 2978 at commit 730d0a9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class RegressionMetrics(predictionAndObservations: RDD[(Double, Double)]) extends Logging

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22528/
Test PASSed.

[mengxr]

LGTM. Merged into master. Thanks!