[MLLIB] SPARK-1547: Add Gradient Boosting to MLlib #2607
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,93 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one or more | ||
| * contributor license agreements. See the NOTICE file distributed with | ||
| * this work for additional information regarding copyright ownership. | ||
| * The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| * (the "License"); you may not use this file except in compliance with | ||
| * the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| package org.apache.spark.mllib.tree.configuration | ||
|
|
||
| import org.apache.spark.annotation.Experimental | ||
| import org.apache.spark.mllib.tree.configuration.Algo._ | ||
| import org.apache.spark.mllib.tree.configuration.QuantileStrategy._ | ||
| import org.apache.spark.mllib.tree.impurity.{Variance, Impurity} | ||
| import org.apache.spark.mllib.tree.loss.{SquaredError, Loss} | ||
|
|
||
| /** | ||
| * :: Experimental :: | ||
| * Stores all the configuration options for the boosting algorithms | ||
| * @param algo Learning goal. Supported: | ||
| * [[org.apache.spark.mllib.tree.configuration.Algo.Classification]], | ||
| * [[org.apache.spark.mllib.tree.configuration.Algo.Regression]] | ||
| * @param numEstimators Number of estimators used in boosting stages. In other words, | ||
| * number of boosting iterations performed. | ||
| * @param loss Loss function used for minimization during gradient boosting. | ||
| * @param maxDepth Maximum depth of the tree. | ||
| * E.g., depth 0 means 1 leaf node; depth 1 means 1 internal node + 2 leaf nodes. | ||
| * @param learningRate Learning rate for shrinking the contribution of each estimator. The | ||
| * learning rate should be between in the interval (0, 1] | ||
| * @param subsample Fraction of the training data used for learning the decision tree. | ||
| * @param numClassesForClassification Number of classes for classification. | ||
| * (Ignored for regression.) | ||
| * Default value is 2 (binary classification). | ||
| * @param maxBins Maximum number of bins used for discretizing continuous features and | ||
| * for choosing how to split on features at each node. | ||
| * More bins give higher granularity. | ||
| * @param quantileCalculationStrategy Algorithm for calculating quantiles. Supported: | ||
| * [[org.apache.spark.mllib.tree.configuration.QuantileStrategy.Sort]] | ||
| * @param categoricalFeaturesInfo A map storing information about the categorical variables and the | ||
| * number of discrete values they take. For example, an entry (n -> | ||
| * k) implies the feature n is categorical with k categories 0, | ||
| * 1, 2, ... , k-1. It's important to note that features are | ||
| * zero-indexed. | ||
| * @param minInstancesPerNode Minimum number of instances each child must have after split. | ||
| * Default value is 1. If a split cause left or right child | ||
| * to have less than minInstancesPerNode, | ||
| * this split will not be considered as a valid split. | ||
| * @param minInfoGain Minimum information gain a split must get. Default value is 0.0. | ||
| * If a split has less information gain than minInfoGain, | ||
| * this split will not be considered as a valid split. | ||
| * @param maxMemoryInMB Maximum memory in MB allocated to histogram aggregation. Default value is | ||
| * 256 MB. | ||
| */ | ||
| @Experimental | ||
| case class BoostingStrategy( | ||
| // Required boosting parameters | ||
| algo: Algo, | ||
| numEstimators: Int, | ||
| loss: Loss, | ||
| // Required tree parameters | ||
| maxDepth: Int, | ||
| // Optional boosting parameters | ||
| learningRate: Double = 0.1, | ||
| subsample: Double = 1, | ||
| numClassesForClassification: Int = 2, | ||
| // Optional tree parametes | ||
| maxBins: Int = 32, | ||
| quantileCalculationStrategy: QuantileStrategy = Sort, | ||
| categoricalFeaturesInfo: Map[Int, Int] = Map[Int, Int](), | ||
| minInstancesPerNode: Int = 1, | ||
| minInfoGain: Double = 0.0, | ||
| maxMemoryInMB: Int = 256) extends Serializable { | ||
|
|
||
| // Note: Regression tree used even for classification for GBT. | ||
| val strategy = new Strategy(Regression, Variance, maxDepth, numClassesForClassification, maxBins, | ||
| quantileCalculationStrategy, categoricalFeaturesInfo, minInstancesPerNode, minInfoGain, | ||
| maxMemoryInMB, subsample) | ||
|
|
||
| require(learningRate <= 1, "Learning rate should be <= 1. Provided learning rate is " + | ||
| s"$learningRate.") | ||
| require(learningRate > 0, "Learning rate should be > 0. Provided learning rate is " + | ||
| s"$learningRate.") | ||
|
|
||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one or more | ||
| * contributor license agreements. See the NOTICE file distributed with | ||
| * this work for additional information regarding copyright ownership. | ||
| * The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| * (the "License"); you may not use this file except in compliance with | ||
| * the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| package org.apache.spark.mllib.tree.configuration | ||
|
|
||
| import org.apache.spark.annotation.DeveloperApi | ||
|
|
||
| /** | ||
| * :: Experimental :: | ||
| * Enum to select ensemble combining strategy for base learners | ||
| */ | ||
| @DeveloperApi | ||
| object EnsembleCombiningStrategy extends Enumeration { | ||
|
Member
There was a problem hiding this comment. I think this strategy option would be useful at some point, but not yet.
Contributor
Author
There was a problem hiding this comment. Why are sum and average the same? In RF you average predictions and in GBT you add predictions.
Member
There was a problem hiding this comment. You're right that they are a bit different; I was thinking in terms of thresholding for classification, but it would be important to sum, not average, for regression. I also revoke what I said about supporting things like median.
Contributor
Author
There was a problem hiding this comment. I think we use average or majority for the random forest ensemble calculations. I moved the RF code to also return a WeightedEnsembleModel. I am okay with removing EnsembleCombiningStrategy class but we still need to support both sum and averaging operations for combining ensemble predictions.
Member
There was a problem hiding this comment. OK, I agree; keeping it sounds good. |
||
| type EnsembleCombiningStrategy = Value | ||
| val Sum, Average = Value | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I personally think that Boosted Model can be a separate one from RandomForestModel. E.g., it's not inconceivable to have boosted models to use RandomForestModel as its base learners.
And if this were a truly generic weighted ensemble model, then it could probably live outside of tree.model namespace, since boosting at least in theory doesn't care whether base learners are trees or not.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These generalizations will rely on the new ML API (for which there will be a PR any day now); it makes sense to keep it in the tree namespace since there is not generic Estimator concept currently. But once we can, I agree it will be important to generalize meta-algorithms.
With respect to the models, I don't see how the model concepts are different. The learning algorithms are different, but that will not prevent a meta-algorithm to use another meta-algorithm as a weak learner (once the new API is available). (I think it's good to separate the concepts of Estimator (learning algorithm) and Transformer (learned model) here.) What do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yea, I guess from the design perspective, it's tempting to unify these under the same umbrella.
IMO, RandomForest is mostly a specific instance of a generic ensemble model, so this makes sense.
However, I think that boosted models have some specific things about them due to their sequential nature (as opposed to parallel nature of RandomForest). E.g., if you have 1000 models, you can potentially predict based on the first 100 models whereas with RandomForest you can pick any 100. You also have to do overfitting/underfitting analyses on boosted models sequentially, etc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@codedeft I started with a separate model for boosting but @jkbradley (quite correctly IMO) convinced me otherwise. :-)
I agree methods like boosting require support such as early stopping, sequential selection of models, etc. but may be we can handle it as a part of the model configuration. AdaBoost and RF in some ways are more similar than AdaBoost and GBT in their combining operation. It might be better to capture all these nuances in one place. Of course, we can always split them later if we end up writing a lot of custom logic for each algorithm. Thoughts?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@manishamde Sounds good.
Just a side note. Because RF models tend to be much bigger than boosted ensembles, we've encountered situations where the model was too big to fit in a single machine memory. RandomForest model is in a way a good model for embarassingly parallel predictions so a model could potentially reside in a distributed fashion.
But we haven't yet decided whether we really want to do this (i.e. are humongous models really useful in practice and do we really expect crazy scenarios of gigantic models surpassing dozens of GBs?)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@codedeft Agree about the distributed storage though I never bothered to check the size of deep trees in memory! :-) In fact, such a storage might be a good option for Partial Forest implementation.