[SPARK-17001] [ML] Enable standardScaler to standardize sparse vectors when withMean=True

[srowen]

What changes were proposed in this pull request?

Allow centering / mean scaling of sparse vectors in StandardScaler, if requested. This is for compatibility with VectorAssembler in common usages.

How was this patch tested?

Jenkins tests, including new caes to reflect the new behavior.

[SparkQA]

Test build #63840 has finished for PR 14663 at commit 496a8df.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Another one where I'd welcome comments from ... @holdenk @MLnick @davies et al

[srowen]

going once, going twice. This would simply let an operation proceed where it errored before, at the cost of giving a user a little more rope to hang him/herself. I think it unblocks a legitimate and common set of use cases, so I think it's worth changing.

[MLnick]

As mentioned on the JIRA discussion, I'm neutral on this, though I tend to lean towards allowing the user to do what they want even if it might be "dangerous". I guess +0?

Though perhaps we may want to explicitly log a WARN message if sparse input is encountered and withMean=true, just so there is a very clear root cause message in the logs if things blow up with OOM (it's in the user guide and doc string, but still may be missed by many users).

[srowen]

Warning seems reasonable. I think you'd have to put in a flag to remember if the user has been warned in order to avoid spewing millions of them. Worth it, you think?

[MLnick]

Ah right, good point. Actually I realised that the doc in ml.feature.StandardScaler needs updating for withMean:

  /**
   * Whether to center the data with mean before scaling.
   * It will build a dense output, so this does not work on sparse input
   * and will raise an exception.
   * Default: false
   * @group param
   */
  val withMean: BooleanParam = new BooleanParam(this, "withMean",
    "Whether to center data with mean")

... as good a place as any to add a warning about using with sparse input (this can then show up in explainParam doc as well as ScalaDoc. Python side will probably need it too.

[srowen]

If I understood you correctly @MLnick you favored just adding warnings in the doc? I added to three more places that needed it.

[SparkQA]

Test build #64288 has finished for PR 14663 at commit e263084.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

I'll go for this tomorrow if there are no other comments.

[srowen]

Merged to master