Skip to content

[SPARK-15643][DOC][ML] Add breaking changes to ML migration guide #13924

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MLnick wants to merge 8 commits into from
Closed

[SPARK-15643][DOC][ML] Add breaking changes to ML migration guide #13924

Changes from 3 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
80872b3
Add breaking changes to ML migration guide
Jun 27, 2016
177a9e0
initial work
Jun 28, 2016
6ef09a3
Update guide for vector/matrix conversion and clean up
Jun 28, 2016
ac49f31
review comment and add single instance conversion details
Jun 29, 2016
c48b894
Add Python asML example code
Jun 30, 2016
91fd24c
fix comment
Jun 30, 2016
c2ce7cd
Fix java ;
Jun 30, 2016
919bfe9
Fix Java method invoc
Jun 30, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 83 additions & 3 deletions docs/mllib-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,9 +104,87 @@ and the migration guide below will explain all changes between releases.

## From 1.6 to 2.0

The deprecations and changes of behavior in the `spark.mllib` or `spark.ml` packages include:
### Breaking changes

Deprecations:
There were several breaking changes in Spark 2.0, which are outlined below.

**Linear algebra classes for DataFrame-based APIs**

Spark's linear algebra dependencies were moved to a new project, `spark-mllib-local`

@jkbradley jkbradley Jun 29, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be "mllib-local" (no "spark-")

(see [SPARK-13944](https://issues.apache.org/jira/browse/SPARK-13944)).
As part of this change, the linear algebra classes were moved to a new package, `spark.ml.linalg`.

@jkbradley jkbradley Jun 29, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"moved" --> "copied"

The DataFrame-based APIs in `spark.ml` now depend on the `spark.ml.linalg` classes,
leading to a few breaking changes, predominantly in various model classes
(see [SPARK-14810](https://issues.apache.org/jira/browse/SPARK-14810) for a full list).

**Note:** the RDD-based APIs in `spark.mllib` continue to depend on the previous package `spark.mllib.linalg`.

_Converting vectors and matrices_

@jkbradley jkbradley Jun 29, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you also please note the public conversion methods asML, fromML within spark.mllib.linalg Vector and Matrix types and subtypes for converting single instances?

@MLnick MLnick Jun 29, 2016 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, will do


While most pipeline components support backward compatibility for loading,
some existing `DataFrames` and pipelines in Spark versions prior to 2.0, that contain vector or matrix
columns, may need to be migrated to the new `spark.ml` vector and matrix types.
Utilities for converting `DataFrame` columns from `spark.mllib.linalg` to `spark.ml.linalg` types
(and vice versa) can be found in `spark.mllib.util.MLUtils`.

<div class="codetabs">
<div data-lang="scala" markdown="1">

{% highlight scala %}
import org.apache.spark.mllib.util.MLUtils

val convertedVecDF = MLUtils.convertVectorColumnsToML(vecDF);
val convertedMatrixDF = MLUtils.convertMatrixColumnsToML(matrixDF);
{% endhighlight %}

Refer to the [`MLUtils` Scala docs](api/scala/index.html#org.apache.spark.mllib.util.MLUtils$) for further detail.
</div>

<div data-lang="java" markdown="1">

{% highlight java %}
import org.apache.spark.mllib.util.MLUtils;
import org.apache.spark.sql.Dataset;

Dataset<Row> convertedVecDF = MLUtils.convertVectorColumnsToML(vecDF);
Dataset<Row> convertedMatrixDF = MLUtils.convertMatrixColumnsToML(matrixDF);
{% endhighlight %}

Refer to the [`MLUtils` Java docs](api/java/org/apache/spark/mllib/util/MLUtils.html) for further detail.
</div>

<div data-lang="python" markdown="1">

{% highlight python %}
from pyspark.mllib.util import MLUtils

convertedVecDF = MLUtils.convertVectorColumnsToML(vecDF)
convertedMatrxDF = MLUtils.convertMatrixColumnsToML(matrixDF)

@MLnick MLnick Jun 29, 2016 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note, it looks like we don't have single instance conversion methods asML / fromML in Python linalg classes (I commented on SPARK-15944.

Not sure if this is intended or we just missed them. One can do newVec = Vectors.dense(oldVec) (or vice versa for sparse) in Python directly, so if that is the expected way to do things I can add that here.

@jkbradley jkbradley Jun 29, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That may have just been overlooked, but that's a good point that there is already a decent way to do the conversion. Could you please just note that way here?

@MLnick MLnick Jun 30, 2016

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jkbradley Ah sorry - I mispoke. It happens to work for dense vectors because it effectively calls np.array(DenseVector), but not for sparse. Workaround is fairly ugly: mlSV = NewVectors.sparse(mllibSV.size, zip(mllibSV.indices, mllibSV.values)), or something similar.

I'd say we should have some convenience methods like in Scala/Java?

@MLnick MLnick Jun 30, 2016 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Created SPARK-16328 and #13997.

{% endhighlight %}

Refer to the [`MLUtils` Python docs](api/python/pyspark.mllib.html#pyspark.mllib.util.MLUtils) for further detail.
</div>
</div>

**Deprecated methods removed**

Several deprecated methods were removed in the `spark.mllib` and `spark.ml` packages:

* `setScoreCol` in `ml.evaluation.BinaryClassificationEvaluator`
* `weights` in `LinearRegression` and `LogisticRegression` in `spark.ml`
* `setMaxNumIterations` in `mllib.optimization.LBFGS` (marked as `DeveloperApi`)
* `treeReduce` and `treeAggregate` in `mllib.rdd.RDDFunctions` (these functions are available on `RDD`s directly, and were marked as `DeveloperApi`)
* `defaultStategy` in `mllib.tree.configuration.Strategy`
* `build` in `mllib.tree.Node`
* libsvm loaders for multiclass and load/save labeledData methods in `mllib.util.MLUtils`

A full list of breaking changes can be found at [SPARK-14810](https://issues.apache.org/jira/browse/SPARK-14810).

### Deprecations and changes of behavior

**Deprecations**

Deprecations in the `spark.mllib` and `spark.ml` packages include:

* [SPARK-14984](https://issues.apache.org/jira/browse/SPARK-14984):
In `spark.ml.regression.LinearRegressionSummary`, the `model` field has been deprecated.
Expand All @@ -125,7 +203,9 @@ Deprecations:
In `spark.ml.util.MLReader` and `spark.ml.util.MLWriter`, the `context` method has been deprecated in favor of `session`.
* In `spark.ml.feature.ChiSqSelectorModel`, the `setLabelCol` method has been deprecated since it was not used by `ChiSqSelectorModel`.

Changes of behavior:
**Changes of behavior**

Changes of behavior in the `spark.mllib` and `spark.ml` packages include:

* [SPARK-7780](https://issues.apache.org/jira/browse/SPARK-7780):
`spark.mllib.classification.LogisticRegressionWithLBFGS` directly calls `spark.ml.classification.LogisticRegresson` for binary classification now.
Expand Down