GraphComponentException if hidden_layers_size used with only one (text) overwritten key

[JulianGerhard21]

Rasa Open Source version

3.0.0

Rasa SDK version

3.0.0

Rasa X version

No response

Python version

3.8

What operating system are you using?

Windows

What happened?

I wanted to train a migrated AI with the following config for DIETClassifier:

- name: DIETClassifier
  random_seed: 42
  loss_type: cross_entropy
  model_confidence: softmax
  constrain_similarities: True
  intent_classification: True
  entity_recognition: False
  use_masked_language_model: True
  epochs: 80
  number_of_transformer_layers: 4
  transformer_size: 256
  drop_rate: 0.2
  batch_size: [64, 256]
  embedding_dimension: 50
  hidden_layers_sizes:
    text: [512, 128]

The result was the following error:

  File "...rasa\nlu\classifiers\diet_classifier.py", line 1277, in __init__
    self._prepare_layers()
  File "...rasa\nlu\classifiers\diet_classifier.py", line 1437, in _prepare_layers
    self.config[HIDDEN_LAYERS_SIZES][self.label_name],
  File "...wrapt\wrappers.py", line 394, in __getitem__
    return self.__wrapped__[key]
KeyError: 'label'

followed by

rasa.engine.exceptions.GraphComponentException: Error running graph component for node train_DIETClassifier5.

After digging into it, it seems to me, that even if the documentation sais:

hidden_layers_sizes: This parameter allows you to define the number of feed forward layers and their output dimensions for user messages and intents (default: text: [], label: []). 

it is not enough to specify only one of the keys. After changing the config to:

  hidden_layers_sizes:
    text: [512, 128]
    label: []

everything worked fine. I am not absolutely sure but this worked in Rasa 2.8.x.

Command / Request

No response

Relevant log output

2021-11-25 14:36:48 INFO     rasa.engine.training.hooks  - Restored component 'LexicalSyntacticFeaturizer' from cache.
2021-11-25 14:36:48 INFO     rasa.engine.training.hooks  - Restored component 'CountVectorsFeaturizer' from cache.
2021-11-25 14:36:48 INFO     rasa.engine.training.hooks  - Restored component 'CountVectorsFeaturizer' from cache.
2021-11-25 14:36:48 INFO     rasa.engine.training.hooks  - Restored component 'MemoizationPolicy' from cache.
2021-11-25 14:36:48 INFO     rasa.engine.training.hooks  - Restored component 'RulePolicy' from cache.
2021-11-25 14:36:49 INFO     rasa.engine.training.hooks  - Starting to train component 'DIETClassifier'.
...
  File "...rasa\nlu\classifiers\diet_classifier.py", line 1277, in __init__
    self._prepare_layers()
  File "...rasa\nlu\classifiers\diet_classifier.py", line 1437, in _prepare_layers
    self.config[HIDDEN_LAYERS_SIZES][self.label_name],
  File "...wrapt\wrappers.py", line 394, in __getitem__
    return self.__wrapped__[key]
KeyError: 'label'

Definition of Done:

• Discovered the problem
• Fixed the problem
• Made sure the same problem does not exist for other parameters of the component and also other ML components like TEDPolicy, ResponseSelector and UnexpecTEDIntentPolicy
• Model regression tests run on the PR
• PR is merged to 3.0.x
• New micro version released after the bugfix is merged.

[sara-tagger]

Thanks for raising this issue, @jupyterjazz will get back to you about it soon✨

Please also check out the docs and the forum in case your issue was raised there too 🤗

[TyDunn]

@dakshvar22 any thoughts about what might have happened here?

[koernerfelicia]

Made sure the same problem does not exist for other parameters of the component and also other ML components like TEDPolicy, ResponseSelector and UnexpecTEDIntentPolicy

@dakshvar22 why'd you restrict this to just ML components? Am I missing sth about the nature of how these params are specified?

[dakshvar22]

@koernerfelicia Mainly because those are the only components where a configuration parameter can have a dictionary of values with keys as some text and values being a list / scalar. The above reported issue suggests a problem with those kind of configuration parameters.

[kedz]

@koernerfelicia is reviewer

[mleimeister]

The config of hidden_layers_sizes with only setting the text key does not lead to the same issue in the other components. However, using the config

dense_dimension:
  text: 128

(i.e. not specifying the other available keys, such as label for DIET, action_name for TED, etc.) triggers a similar error for DIETClassifier as well as TEDPolicy and UnexpecTEDIntentPolicy:

rasa.engine.exceptions.GraphComponentException: Error running graph component for node train_UnexpecTEDIntentPolicy2.

File "/Users/matthias/Workspace/rasa3/rasa/utils/tensorflow/rasa_layers.py", line 291, in <listcomp>
    config[DENSE_DIMENSION][attribute]
KeyError: 'action_name'

I was not able to reproduce the same behaviour for ResponseSelector using the nested dictionary parameters available there.

This suggests that the issue comes from the general setting of parameters if they contain nested dictionaries and not all keys are specified. The errors that surface then could be related to whether the component actually tries to use those missing entries and needs further investigation.

[koernerfelicia]

Thanks for the update!

This suggests that the issue comes from the general setting of parameters if they contain nested dictionaries and not all keys are specified

I think that this means we shouldn't constrain the investigation to ML components. CountVectorsFeaturizer for example allows for nested dictionaries:

- name: CountVectorsFeaturizer
  additional_vocabulary_size:
    text: 1000
    response: 1000
    action_text: 1000

Haven't looked into the PRs you linked, so it's possible the solution is general enough to cover non-ML components as well.

[mleimeister]

Hi, thanks for the feedback! In that case I'll also check the CountVectorsFeaturizer config. I added some documentation on the potential cause I think I found and steps I implemented for a possible fix in the second PR's comment. Since it would touch a core part of the components, it would be great though to discuss if this is actually the correct root cause, and if so a viable fix.

[mleimeister]

Scheduled for next upcoming micro release with Enable squat.

[koernerfelicia]

@mleimeister you can close the issue when the PR is merged, you don't have to wait for it to be released! I believe the "Fixes" keyword only closes the associated issue if the PR targets main, that's why this didn't happen automatically.