Invalid configurations generated with constraints

[Kerilk]

Hello,

Thanks for this great library. I ran into a small issue while double checking some results. The (not so) small sample here seems to be generating invalid configurations:

import ConfigSpace as CS
import ConfigSpace.hyperparameters as CSH


cs = CS.ConfigurationSpace(seed=1234)
p1 = CSH.CategoricalHyperparameter(name='p1', choices=['None', '#pragma omp #p3', '#pragma omp target #p3', '#pragma omp target #p5', '#pragma omp #p4'])
p3 = CSH.CategoricalHyperparameter(name='p3', choices=['None', '#parallel for #p4', '#parallel for #p6', '#parallel for #p7'])
p4 = CSH.CategoricalHyperparameter(name='p4', choices=['None', 'simd'])
p5 = CSH.CategoricalHyperparameter(name='p5', choices=['None', '#dist_schedule static', '#dist_schedule #p11'])
p6 = CSH.CategoricalHyperparameter(name='p6', choices=['None', '#schedule #p10', '#schedule #p11'])
p7 = CSH.CategoricalHyperparameter(name='p7', choices=['None', '#numthreads #p12'])
p10 = CSH.CategoricalHyperparameter(name='p10', choices=['static', 'dynamic'])
p11 = CSH.OrdinalHyperparameter(name='p11', sequence=['1', '8', '16'])
p12 = CSH.OrdinalHyperparameter(name='p12', sequence=['1', '8', '16'])

cs.add_hyperparameters([p1, p3, p4, p5, p6, p7, p10, p11, p12])

cond0 = CS.EqualsCondition(p3, p1, '#pragma omp #p3')
cond1 = CS.EqualsCondition(p3, p1, '#pragma omp target #p3')
cond2 = CS.EqualsCondition(p5, p1, '#pragma omp target #p5')
cond3 = CS.EqualsCondition(p4, p1, '#pragma omp #p4')


cond4 = CS.EqualsCondition(p4, p3, '#parallel for #p4')
cond5 = CS.EqualsCondition(p6, p3, '#parallel for #p6')
cond6 = CS.EqualsCondition(p7, p3, '#parallel for #p7')

cond7 = CS.EqualsCondition(p11, p5, '#dist_schedule #p11')

cond8 = CS.EqualsCondition(p10, p6, '#schedule #p10')
cond9 = CS.EqualsCondition(p11, p6, '#schedule #p11')

cond10 = CS.EqualsCondition(p12, p7, '#numthreads #p12')

cs.add_condition(CS.OrConjunction(cond0,cond1))
cs.add_condition(cond2) 
cs.add_condition(CS.OrConjunction(cond3,cond4))
cs.add_condition(cond5) 
cs.add_condition(cond6) 
cs.add_condition(cond8) 
cs.add_condition(CS.OrConjunction(cond7,cond9))
cs.add_condition(cond10)

for x in cs.sample_configuration(1000):
    d = dict(x)
    if 'p11' in d:
        if 'p6' not in d and 'p5' not in d:
            print("Invalid configuration Found:")
            print(d)
        elif 'p6' in d and 'p5' not in d:
            if d['p6'] != '#schedule #p11':
                print("Invalid configuration Found:")
                print(d)
        elif 'p5' in d and 'p6' not in d:
            if d['p5'] != '#dist_schedule #p11':
                print("Invalid configuration Found:")
                print(d)
        elif d['p6'] != '#schedule #p11' and d['p5'] != '#dist_schedule #p11':
            print("Invalid configuration Found:")
            print(d)

If my understanding of the constraints is correct (and if the constraints have been expressed correctly), p11 should only be active if p5 is active and equals #dist_schedule #p11 or p6 is active and equals #schedule #p11.

In practice, we find several invalid configurations being returned by ConfigSpace. Here is one example:

{'p1': '#pragma omp target #p5', 'p5': '#dist_schedule static', 'p11': '8'}

Is this a bug or did I miss something?

Thanks,
Brice

[mfeurer]

Thank you very much for reporting this. Indeed, this appears to be an issue in the ConfigSpace. I'm not aware that anyone is using the OrCondition, so I'm afraid the bug is related to that.

Could you please check whether this issue is caught by cs._check_configuration_rigorous(x)?

[Kerilk]

Well you raise an interesting point: cs._check_configuration_rigorous(x) triggers for valid configurations:

  for x in cs.sample_configuration(1000):
      d = dict(x)
      print(x)
      cs._check_configuration_rigorous(x)

outputs:

Configuration:
  p1, Value: '#pragma omp target #p5'
  p5, Value: 'None'

Configuration:
  p1, Value: '#pragma omp #p4'
  p4, Value: 'simd'

Traceback (most recent call last):
  File "/home/videau/temp/ConfigSpace/problem.py", line 47, in <module>
    cs._check_configuration_rigorous(x)
  File "ConfigSpace/configuration_space.pyx", line 1166, in ConfigSpace.configuration_space.ConfigurationSpace._check_configuration_rigorous
ValueError: Inactive hyperparameter 'p4' must not be specified, but has the vector value: '1.0'.

p4 is also conditioned by an OrCondition.

[thijssnelleman]

This issue is quite old; Is error still the case? Possibly related to #415