Skip to content
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -39,9 +39,13 @@ private[ml] trait DifferentiableRegularization[T] extends DiffFunction[T] {
*
* @param regParam The magnitude of the regularization.
* @param shouldApply A function (Int => Boolean) indicating whether a given index should have
* regularization applied to it.
* regularization applied to it. Usually we don't apply regularization to
* the intercept.
* @param applyFeaturesStd Option for a function which maps coefficient index (column major) to the
* feature standard deviation. If `None`, no standardization is applied.
* feature standard deviation. Since we always standardize the data during
* training, if `standardization` is false, we have to reverse
* standardization by penalizing each component differently by this param.
* If `standardization` is true, this should be `None`.
*/
private[ml] class L2Regularization(
override val regParam: Double,
Expand All @@ -57,6 +61,11 @@ private[ml] class L2Regularization(
val coef = coefficients(j)
applyFeaturesStd match {
case Some(getStd) =>
// If `standardization` is false, we still standardize the data
// to improve the rate of convergence; as a result, we have to
// perform this reverse standardization by penalizing each component
// differently to get effectively the same objective function when
// the training dataset is not standardized.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes this is an important comment we need keep.

val std = getStd(j)
if (std != 0.0) {
val temp = coef / (std * std)
Expand All @@ -66,6 +75,7 @@ private[ml] class L2Regularization(
0.0
}
case None =>
// If `standardization` is true, compute L2 regularization normally.
sum += coef * coef
gradient(j) = coef * regParam
}
Expand Down