[Bridges] add final_touch

[blegat]

Closes #1665
Closes #1097

What

This PR adds the ability for bridges to implement Bridges.final_touch. It also implements or improves three bridges as a proof-of-concept to check for correctness of the idea.

Why

Some bridges require knowledge about the global state of the model, for example, variable bounds. A good example of this type of bridge is the new CountDistinctToMILPBridge.

final_touch acts as a callback that is run immediately prior to optimize!, which lets bridges update their reformulation of check that the assumptions are satisfied (e.g., validating other variable bounds).

Considerations

There are a few considerations:

• Bridges must report accurate statistics such as ListOfConstraintIndices and NumberOfVariables at all times. This safely obscures the difference between the internal state of the bridge and the wider world. Importantly, it means that we don't care about the current internal state of the reformulation at a particular point in time, because this is just an implementation decision.
• Bridges implementing final_touch should probably call delete at the start of the function in order to wipe any previous or partially constructed reformulations. Of course, they could be cleverer and only update the reformulation if the variable bounds have changed etc, but they must not do something like add two reformulations if optimize! is called twice in succession.

blegat's earlier summary

There are two options:

Keep track on exactly what a bridge depend on and warn them whenever a change is made. This has two issues:

1. It might update the model many times if many changes are made. Think of a Big-M constraint with thousands of variables. If the user change the bound for all of them and the re-solve, we will change the the constrains thousands of time instead of once before the re-solve.
2. If the solver need to error, it might need to wait for a bound to be set so that it can work and not error. If no bound is set, it should throw its error but how can we know that this bridge wanted to error if no bound was set? So we need a mechanism to keep track of bridges that want to error as discussed in Add variable constraint watcher bridges #1097. But when do we throw this error ? We need to throw it at final_touch or optimize! so now we need final_touch or optimize! to be called to be sure a bridged model is valid. Users that do b.model might have a model missing the constraints that want to error.

The second option is for bridges to declare that they need final_touch to be called. Two issues:

1. The first issue is that we need final_touch or optimize! to be called for the inner model to be valid but we have the same issue in the first approach as discussed above. We could expose a inner_model or bridged_model function so that the user has a better way that final_touch(b); b.model
2. The second issue is that if the user does not change much, we are still going to call final_touch on all bridges that need to even if nothing changed. That's only going to apply to bridges for which final_touch returns true though and that's done in a type-stable way so that might not be such a big issue.

This PR implements the second option and #1097 implements the first one.

[odow]

I think this final_touch approach makes more sense than #1907. You're really saying "I have this transformation, but I need a little more information before I can complete it." It also fits in with the rest of MOI without introducing new concepts.

The one potential drawback is that there's no priority for final touch. So what if you have two bridges that each needed information about the other? Although I guess that's hidden.

I think before we merge this, we need some evidence of actual bridges.

• SemiToBinary bound check: have bounds been set? If so, error.
• A QuadToSOC bridge where t >= 0
• AllDifferent to MILP

[odow]

Are there any situations in which this would be used by a variable or objective bridge? I guess I can't think of any, because it's not like a bridge can support add_constrained_variables(model, AllDifferent(3)).

And the objective bridges already catch modifying ConstraintSet.

[odow]

So I've been playing around with this locally. I think we need to think about it a bit more.

We really need a mix of both PRs:

• A way to get notified immediately if a new constraint is added
• A way to get called at the end a la final_touch.

You need the initial notification to throw the right error, but you need the final touch to clean up and fix bounds etc that you don't want to be doing repeatedly.

[odow]

This seems very promising:

julia> using JuMP, HiGHS
[ Info: Precompiling JuMP [4076af6c-e467-56ae-b986-b466b2749572]
[ Info: Precompiling HiGHS [87dc4568-4c63-4d18-b0c0-bb2238e4078b]

julia> model = Model(HiGHS.Optimizer)
A JuMP Model
Feasibility problem with:
Variables: 0
Model mode: AUTOMATIC
CachingOptimizer state: EMPTY_OPTIMIZER
Solver name: HiGHS

julia> set_silent(model)

julia> @variable(model, n, Int)
n

julia> @variable(model, 1 <= x <= 2, Int)
x

julia> @variable(model, 2 <= y <= 3, Int)
y

julia> @constraint(model, [n, x, y] in MOI.CountDistinct(3))
[n, x, y] ∈ MathOptInterface.CountDistinct(3)

julia> @objective(model, Min, x + y)
x + y

julia> optimize!(model)

julia> @show value.([n, x, y])
value.([n, x, y]) = [2.0, 1.0, 2.0]
3-element Vector{Float64}:
 2.0
 1.0
 2.0

julia> @objective(model, Max, x + y)
x + y

julia> optimize!(model)

julia> @show value.([n, x, y])
value.([n, x, y]) = [2.0, 2.0, 3.0]
3-element Vector{Float64}:
 2.0
 2.0
 3.0

julia> @objective(model, Max, x - y)
x - y

julia> optimize!(model)

julia> @show value.([n, x, y])
value.([n, x, y]) = [1.0, 2.0, 2.0]
3-element Vector{Float64}:
 1.0
 2.0
 2.0

[odow]

More things working well: a bridge from AllDifferent to CountDistinct, which is a bridge that bridges to a constraint that uses final_touch.

julia> using JuMP, HiGHS

julia> model = Model(HiGHS.Optimizer)
A JuMP Model
Feasibility problem with:
Variables: 0
Model mode: AUTOMATIC
CachingOptimizer state: EMPTY_OPTIMIZER
Solver name: HiGHS

julia> set_silent(model)

julia> @variable(model, 1 <= x <= 2, Int)
x

julia> @variable(model, 2 <= y <= 3, Int)
y

julia> @constraint(model, [x, y] in MOI.AllDifferent(2))
[x, y] ∈ MathOptInterface.AllDifferent(2)

julia> @objective(model, Min, x + y)
x + y

julia> optimize!(model)

julia> value.([x, y])
2-element Vector{Float64}:
 1.0
 2.0

julia> @objective(model, Max, x + y)
x + y

julia> optimize!(model)

julia> value.([x, y])
2-element Vector{Float64}:
 2.0
 3.0

julia> @objective(model, Min, x - y)
x - y

julia> optimize!(model)

julia> value.([x, y])
2-element Vector{Float64}:
 1.0
 3.0

[odow]

I'm pretty happy with this. I think it actually solves a nice pain-point in an elegant way. I've done three example bridges without trouble.

• SemiToBinary uses final_touch to check if an error needs to be thrown
• CountDistinctToMILP builds the reformulation in final_touch
• AllDifferentToCountDistinct is a non-final_touch bridge that bridges to CountDistinctToMILP.

All three work and were pretty trivial.

There's a weird thing that the inner-most model of the bridge may not correspond to the true model until final_touch is called, but that's okay. Public-facing, it presents the true model as if everything was fine, and we always call final_touch on optimize!. This would only cause problems if people try and navigate through the bridging layer (e.g., in JuMP someone partially builds the problem, then calls unsafe_backend), but people shouldn't really do that.

We might encounter some edge cases once this starts getting used in anger, and the performance impact of rebuilding the reformulation every time might be large. But something is better than nothing.

[odow]

Here's one edge-case:

julia> using JuMP, HiGHS

julia> let
           model = Model(HiGHS.Optimizer)
           set_silent(model)
           @variable(model, 1 <= x <= 2, Int)
           @variable(model, 2 <= y <= 3, Int)
           @constraint(model, [x, y] in MOI.AllDifferent(2))
           @objective(model, Min, x + y)
           optimize!(model)
           value.([x, y])
       end
2-element Vector{Float64}:
 1.0
 2.0

julia> let
           model = Model(HiGHS.Optimizer)
           set_silent(model)
           @variable(model, 1 <= x <= 2, Int)
           @variable(model, 2 <= y <= 3, Int)
           @constraint(model, [x + 1, y] in MOI.AllDifferent(2))
           @objective(model, Min, x + y)
           optimize!(model)
           value.([x, y])
       end
ERROR: Unable to use CountDistinctToMILPBridge because variable MathOptInterface.VariableIndex(3) has a non-finite domain.
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:33
  [2] final_touch(bridge::MathOptInterface.Bridges.Constraint.CountDistinctToMILPBridge{Float64}, model::MathOptInterface.Bridges.LazyBridgeOptimizer{HiGHS.Optimizer})
    @ MathOptInterface.Bridges.Constraint ~/.julia/dev/MathOptInterface/src/Bridges/Constraint/bridges/count_distinct.jl:244

Because the bridge doesn't support VectorAffineFunction, it uses a SlackBridge, which introduces a new variable z = x + 1 with [z, y] in AllDifferent(2). But now z doesn't have finite variable bounds.

I guess the fix for this is for the bridge to support VectorAffineFunction.

Edit: The other issue of course is that it is thrown from CountDistinctToMILPBridge...

[odow]

This is fixed:

julia> using JuMP, HiGHS

julia> let
           model = Model(HiGHS.Optimizer)
           set_silent(model)
           @variable(model, 1 <= x <= 2, Int)
           @variable(model, 2 <= y <= 3, Int)
           @constraint(model, [x, y] in MOI.AllDifferent(2))
           @objective(model, Min, x + y)
           optimize!(model)
           value.([x, y])
       end
2-element Vector{Float64}:
 1.0
 2.0

julia> let
           model = Model(HiGHS.Optimizer)
           set_silent(model)
           @variable(model, 1 <= x <= 2, Int)
           @variable(model, 2 <= y <= 3, Int)
           @constraint(model, [x + 1, y] in MOI.AllDifferent(2))
           @objective(model, Min, x + y)
           optimize!(model)
           value.([x, y])
       end
2-element Vector{Float64}:
 2.0
 2.0

[odow]

@blegat, I changed this to be Bridges.final_touch instead of Utilities.final_touch. It lets us document a bit better, and there's no need to use the same function.

[odow]

Removed the AllDifferentToCountDistinctBridge into #1923

[odow]

I just tagged v1.5.0, so my plan is to merge all of these bridges, but let them sit for a bit to test with MILP solvers before tagging v1.6.0.

[blegat]

@blegat, I changed this to be Bridges.final_touch instead of Utilities.final_touch. It lets us document a bit better, and there's no need to use the same function.

Bridges kind of use the MOI API acting like ConstraintIndex so that we can reuse MOI.get, MOI.set, MOI.delete.
Here there is no MOI.Utilities.final_touch with a ConstraintIndex so there is no consistency argument to reuse the same function so I'm fine with having another one.

[blegat]

If I understand correctly, this is because Semicontinuous sets bounds so it should be incompatible with existing bounds. In fact, if these bounds are removed before optimize! we shouldn't throw so we should really use final_touch here too. I guess it's another argument for removing these errors as suggested in #1879 (comment)

[blegat]

Maybe put that in another PR if it's not needed to pass the tests so that we can merge the rest

[blegat]

It's going to be a nightmare now that bridges mess up with bounds if we keep this MOI.LowerBoundAlreadySet errors. The fact that bridges can modify these bounds due to the fact that the constraint indices can be inferred from the variable indices is another arguments that these can be muted from everywhere.

[blegat]

If we do this in final_touch we shouldn't do it in bridge_constraint

[blegat]

Ok to merge if semi_to_binary is moved to a separate PR as I don't have any comment except for this one.

[odow]

Okay, I'll remove into a separate PR.

[odow]

Okay, I'll merge once CI is green.