Test with unsupported constraints

[blegat]

We should have a test like that but apparently it is not enough, see jump-dev/CSDP.jl#80
Implementing copy_to ignoring the unsupported constraints is too common so we should have a tests for that in runtests

[odow]

There is a test:

MathOptInterface.jl/src/Test/test_model.jl

Lines 567 to 593 in e9044c5

	"""
	test_model_copy_to_UnsupportedConstraint(model::MOI.ModelLike, ::Config)
	Test an error is thrown when a constraint is unsupported.
	!!! note
	This function uses the two-argument `MOI.optimize!`. This ensures that we
	test three cases:
	* models implementing the two-argument method
	* models using the generic fallback and erroring on `copy_to`
	* models using the generic fallback and erroring on `optimize!`.
	"""
	function test_model_copy_to_UnsupportedConstraint(
	model::MOI.ModelLike,
	::Config{T},
	) where {T}
	@test !MOI.supports_constraint(
	model,
	MOI.VariableIndex,
	UnknownScalarSet{T},
	)
	@test_throws(
	MOI.UnsupportedConstraint,
	MOI.optimize!(model, BadConstraintModel()),
	)
	return
	end

CSDP was silently skipping the test because it was throwing a weird error for some reason:

julia> MOI.Test.test_model_copy_to_UnsupportedConstraint(model, config)
Test Failed at /Users/oscar/.julia/packages/MathOptInterface/Ohzb2/src/Test/test_model.jl:588
  Expression: MOI.optimize!(model, BadConstraintModel())
    Expected: MathOptInterface.UnsupportedConstraint
      Thrown: MathOptInterface.GetAttributeNotAllowed{MathOptInterface.ConstraintFunction}
ERROR: There was an error during testing

[blegat]

That's a bit problematic that we have such silent failures, it also caused jump-dev/MosekTools.jl#110 to go unnoticed

[odow]

We had two choices with the tests:

1. Throw every error, and force packages to explicitly opt-out of tests that use behavior they don't support
2. Silently skip tests that the solver claims it doesn't support

1 is a problem because the overhead is too high, and introducing a new test to MOI would break solver tests. 2 has a problem that if the solver incorrectly claims not to support something, then the test would skip instead of error.

It's a trade-off, but I think we've gone with the right choice. Both CSDP and MosekTools skip a looooot of tests, which is the underlying cause.

[odow]

That's a bit problematic that we have such silent failures

There's

MathOptInterface.jl/src/Test/Test.jl

Lines 167 to 173 in 1b7bec4

	* If `warn_unsupported` is `false`, `runtests` will silently skip tests that
	fail with `UnsupportedConstraint` or `UnsupportedAttribute`. When
	`warn_unsupported` is `true`, a warning will be printed. For most cases the
	default behavior (`false`) is what you want, since these tests likely test
	functionality that is not supported by `model`. However, it can be useful to
	run `warn_unsupported = true` to check you are not skipping tests due to a
	missing `supports_constraint` method or equivalent.

We could turn it on by default?

[blegat]

Unsupported... is fine but Mosek was throwing ...NotAllowed since supports_constraint was true but add_constraint was not implemented

[odow]

I guess I should update the docstring. It skips anything that is "unsupported"

MathOptInterface.jl/src/Test/Test.jl

Lines 272 to 278 in 1b7bec4

	function _error_handler(
	err::Union{MOI.NotAllowedError,MOI.UnsupportedError,RequirementUnmet},
	name::String,
	warn_unsupported::Bool,
	)
	if warn_unsupported
	@warn("Skipping $(name): $(err)")

[blegat]

We should probably have an option to skip the UnsupportedError and RequirementUnmet error but not the NotAllowedError. Not sure if that will do it though, maybe that will show many expected errors too.

[odow]

The problem is that a lot of the tests don't have explicit @requires blocks, so they just fallback to NotAllowed or Unsupported.

MOSEK not calling add_constraint seems like an edge-case. It could have been caught by codecov, or by warn_unsupported = true, or by manual testing. I don't know if that's worth spamming the tests of every solver with warmings, especially if it means that people stop looking at the output and ignoring all warnings that are printed.

[odow]

I updated the docstring (#2037), so I don't know what else there is to do here.

[blegat]

I don't see anything else actionable so let's close