ortools-9.12.4544 breaking some binnings

[bmreiniger]

I get this error from certain optbinning fittings:

TypeError                                 Traceback (most recent call last)
[<ipython-input-2-f32626932f5e>](https://localhost:8080/#) in <cell line: 0>()
     10 # for col in X.columns:
     11 #     bp.fit_transform(X[col], y)
---> 12 bp.fit_transform(X['mean area'], y)
     13 
     14 # bp = OptimalBinning()

[/usr/local/lib/python3.11/dist-packages/optbinning/binning/binning.py](https://localhost:8080/#) in fit_transform(self, x, y, sample_weight, metric, metric_special, metric_missing, show_digits, check_input)
    606             Transformed array.
    607         """
--> 608         return self.fit(x, y, sample_weight, check_input).transform(
    609             x, metric, metric_special, metric_missing, show_digits,
    610             check_input)

[/usr/local/lib/python3.11/dist-packages/optbinning/binning/binning.py](https://localhost:8080/#) in fit(self, x, y, sample_weight, check_input)
    556             Fitted optimal binning.
    557         """
--> 558         return self._fit(x, y, sample_weight, check_input)
    559 
    560     def fit_transform(self, x, y, sample_weight=None, metric="woe",

[/usr/local/lib/python3.11/dist-packages/optbinning/binning/binning.py](https://localhost:8080/#) in _fit(self, x, y, sample_weight, check_input)
    828 
    829         # Optimization
--> 830         self._fit_optimizer(splits, n_nonevent, n_event)
    831 
    832         # Post-processing

[/usr/local/lib/python3.11/dist-packages/optbinning/binning/binning.py](https://localhost:8080/#) in _fit_optimizer(self, splits, n_nonevent, n_event)
   1018             logger.info("Optimizer: build model...")
   1019 
-> 1020         optimizer.build_model(self.divergence, n_nonevent, n_event,
   1021                               trend_change)
   1022 

[/usr/local/lib/python3.11/dist-packages/optbinning/binning/cp.py](https://localhost:8080/#) in build_model(self, divergence, n_nonevent, n_event, trend_change)
     78                            regularization * (pmax - pmin))
     79         else:
---> 80             model.Maximize(sum([(V[i][i] * x[i, i]) +
     81                            sum([(V[i][j] - V[i][j+1]) * x[i, j]
     82                                 for j in range(i)]) for i in range(n)]))

[/usr/local/lib/python3.11/dist-packages/optbinning/binning/cp.py](https://localhost:8080/#) in <listcomp>(.0)
     78                            regularization * (pmax - pmin))
     79         else:
---> 80             model.Maximize(sum([(V[i][i] * x[i, i]) +
     81                            sum([(V[i][j] - V[i][j+1]) * x[i, j]
     82                                 for j in range(i)]) for i in range(n)]))

TypeError: __radd__(): incompatible function arguments. The following argument types are supported:
    1. (self: object, other: int) -> ortools.sat.python.cp_model_helper.LinearExpr
    2. (self: object, arg0: float) -> ortools.sat.python.cp_model_helper.LinearExpr

Invoked with: SumArray(IntAffine(expr=x[5, 0](0..1), coeff=562277, offset=0), IntAffine(expr=x[5, 1](0..1), coeff=236900, offset=0), IntAffine(expr=x[5, 2](0..1), coeff=199303, offset=0), IntAffine(expr=x[5, 3](0..1), coeff=69286, offset=0), IntAffine(expr=x[5, 4](0..1), coeff=23302, offset=0)), IntConstant(0)

That sounds like a change from ortools, and I can confirm that things seem to work if I downgrade from ortools 9.12.4544 to 9.11.4210. But even with 9.12 I sometimes get no error, and I don't immediately see why. The above was generated from the following code, but replacing mean area by mean radius and it runs fine. (This is on Colab, which currently has python 3.11.11, optbinning version 0.20.0.)

from optbinning import OptimalBinning
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True, as_frame=True)

bp = OptimalBinning()
bp.fit_transform(X['mean area'], y)

[guillermo-navas-palencia]

Thanks for reporting this @bmreiniger. This seems an ortools change, indeed, see: #346. I didn't have time to investigate it.

Edit: quick search
https://github.com/google/or-tools/releases/tag/v9.12

[guillermo-navas-palencia]

Could you please try to change the solver to "mip"?

[bmreiniger]

Yes, changing the solver works.

For the BinningProcess class, can we pass solver through to the underlying binners?

[guillermo-navas-palencia]

Yes, via the binning_fit_params.

[guillermo-navas-palencia]

Hi @bmreiniger. After some investigation I think I found the source of the error with ortools 9.12.

It seems that the sum below is no longer supported since it does not consider V[i][i] * x[i, i]) an affine expression. The sum of a non-affine expression and the sum of a linear expression is not supported (thus, the __radd__ incompatibility)

model.Maximize(sum([(V[i][i] * x[i, i]) +
                    sum([(V[i][j] - V[i][j+1]) * x[i, j]
                         for j in range(i)]) for i in range(n)]))

I found two ways to fix the problem:
Option 1: Separate sum of diagonal and off-diagonal elements:

sum_diagonal = sum([V[i][i] * x[i, i] for i in range(n)])
model.Maximize(sum_diagonal +
               sum([
                   sum([(V[i][j] - V[i][j+1]) * x[i, j]
                        for j in range(i)]) for i in range(n)]))

Option 2: Cast the diagonal element to be an affine expression:

model.Maximize(
    sum([cp_model.LinearExpr.affine(x[i, i], V[i][i], 0) +
         sum([(V[i][j] - V[i][j+1]) * x[i, j]
              for j in range(i)]) for i in range(n)]))

The first approach will work for all previous versions of ortools. I have to test if the second is compatible with older versions.

[guillermo-navas-palencia]

I just released version 0.20.1. A quick fix setting the maximum ortools version to <9.12. I will work on the changes to avoid this maximum in the coming weeks.