None in CategoricalHyperparamter can cause invalid configuration

[JarnoRFB]

Let's say I categorical hyperparameter can either be a string or None since it is optional. It is not possible to do

cs = CS.ConfigurationSpace()
cs.add_hyperparameter(
    CSH.CategoricalHyperparameter('expected_numbers_method', choices=[None, 'numeric', 'glm']),  
)

but, switching the order of the arguments in the list works

cs = CS.ConfigurationSpace()
cs.add_hyperparameter(
    CSH.CategoricalHyperparameter('expected_numbers_method', choices=['numeric', 'glm', None]),  
)

The exception is raised when checking the default configuration of a new hyperparameter here.
The problem seems to be that the first entry in the choices list is treated as a default value. However, setting None as a value, does not set None as the value of the hyperparameter, but rather excludes this hyperparameter from the configuration. This is a bit surprising and intuitive from a users perspective.
This issue seems related to #73 .

[mfeurer]

Thanks for reporting this. It seems that you trapped into an undocumented edge case of the configuration space. None is used to mark disabled hyperparameters and should therefore not be used as a hyperparameter value. I suggest that you use a string "None" for this.

Also, do you maybe want to contribute a fix to the categorical (and ordinal) hyperparameter which forbids the use of None as a value?

[JarnoRFB]

Sure would be happy to get more involved! What exactly would be the desired behavior? Should both of the above examples work, or none of them? IMHO it is a bit unfortunate that None is used to mark disabled parameters, as it is very often used for model parameters, say in sklearn. Usually it is very convenient to be able to do something like Model(**cs.sample_configuration()) to instantiate a fully configured model. The upside is that that None is often a default value, so if the respective hyperparameter is missing from the configuration it will essentially have the same outcome as setting it to None. However, it would be great, if one could use None as value explicitly. I see two options:

• changing the representation of disabled parameters, to something else. This is probably undesired, as it would need a lot of internal changes
• using some special representation for None as a parameter value, like CSH.Constant(None). This might be easier but feels a bit clumsy from the API perspective.

Let's discuss this a bit...

[mfeurer]

Should both of the above examples work, or none of them?

I think that either both or None should work. As a first step, it would be great to have some code to make sure that None of the examples work, and to then extend the configuration space to make it work.

However, it would be great, if one could use None as value explicitly.

I agree. Same holds for True and False.

changing the representation of disabled parameters, to something else. This is probably undesired, as it would need a lot of internal changes

I'd be fine with that, however, I will not have the time to implement that myself and you would have to do all the work. I think that having an additional boolean array would be a good representation of disabled hyperparameters.

using some special representation for None as a parameter value, like CSH.Constant(None). This might be easier but feels a bit clumsy from the API perspective.

I don't think I understand exactly how that would work. Do you envision that the representation behaves like a None when used with scikit-learn, but not as None when used as a comparison inside the configuration space?

[mfeurer]

I just added an error in case anyone tries to add None to a categorical in 140bc3c until we find a conclusive solution.

[eddiebergman]

Handled in #359 which depends on #346. Will allow None and arbitrary objects! :)