(3.0) Error initializing ResponseSelector in `rasa test nlu` with `use_text_as_label: true` #10519

indam23 · 2021-12-13T16:06:05Z

Rasa Open Source version

3.0.2

Python version

3.8

What operating system are you using?

OSX

What happened?

Trained a model with the following config without error:

config.yml

language: en

pipeline:
  - name: WhitespaceTokenizer
  - name: RegexFeaturizer
  - name: LexicalSyntacticFeaturizer
  - name: CountVectorsFeaturizer
  - name: ResponseSelector
    epochs: 10
    use_text_as_label: true
    number_of_transformer_layers: 1

But running rasa test nlu fails with the error below. I tried a few variations on transformer configuration with the same result e.g.

  - name: ResponseSelector
    epochs: 10
    use_text_as_label: true
    number_of_transformer_layers: 1
    transformer_size: 256
    hidden_layers_sizes:
      text: []
      label: []

Command / Request

rasa train nlu
rasa test nlu

Relevant log output

2021-12-13 16:57:30 INFO     rasa.core.processor  - Loading model models/nlu-20211213-165606-muffled-procedure.tar.gz...
Traceback (most recent call last):
  File "/Users/melinda/.pyenv/versions/3.8.12/lib/python3.8/site-packages/rasa/engine/graph.py", line 393, in _load_component
    self._component: GraphComponent = constructor(  # type: ignore[no-redef]
  File "/Users/melinda/.pyenv/versions/3.8.12/lib/python3.8/site-packages/rasa/nlu/selectors/response_selector.py", line 669, in load
    model: ResponseSelector = super().load(
  File "/Users/melinda/.pyenv/versions/3.8.12/lib/python3.8/site-packages/rasa/nlu/classifiers/diet_classifier.py", line 1076, in load
    return cls._load(
  File "/Users/melinda/.pyenv/versions/3.8.12/lib/python3.8/site-packages/rasa/nlu/classifiers/diet_classifier.py", line 1107, in _load
    model = cls._load_model(
  File "/Users/melinda/.pyenv/versions/3.8.12/lib/python3.8/site-packages/rasa/nlu/classifiers/diet_classifier.py", line 1200, in _load_model
    model = cls._load_model_class(
  File "/Users/melinda/.pyenv/versions/3.8.12/lib/python3.8/site-packages/rasa/nlu/selectors/response_selector.py", line 640, in _load_model_class
    return cls.model_class(config[USE_TEXT_AS_LABEL]).load(
  File "/Users/melinda/.pyenv/versions/3.8.12/lib/python3.8/site-packages/rasa/utils/tensorflow/models.py", line 425, in load
    model = cls(*args, **kwargs)
  File "/Users/melinda/.pyenv/versions/3.8.12/lib/python3.8/site-packages/rasa/nlu/classifiers/diet_classifier.py", line 1277, in __init__
    self._prepare_layers()
  File "/Users/melinda/.pyenv/versions/3.8.12/lib/python3.8/site-packages/rasa/nlu/selectors/response_selector.py", line 803, in _prepare_layers
    ] = rasa_layers.RasaSequenceLayer(
  File "/Users/melinda/.pyenv/versions/3.8.12/lib/python3.8/site-packages/rasa/utils/tensorflow/rasa_layers.py", line 784, in __init__
    transformer_layers, transformer_units = self._prepare_transformer(
  File "/Users/melinda/.pyenv/versions/3.8.12/lib/python3.8/site-packages/rasa/utils/tensorflow/rasa_layers.py", line 818, in _prepare_transformer
    self._tf_layers[self.TRANSFORMER] = prepare_transformer_layer(
  File "/Users/melinda/.pyenv/versions/3.8.12/lib/python3.8/site-packages/rasa/utils/tensorflow/rasa_layers.py", line 1077, in prepare_transformer_layer
    units * 4,
TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/melinda/.pyenv/versions/3.8.12/bin/rasa", line 8, in <module>
    sys.exit(main())
  File "/Users/melinda/.pyenv/versions/3.8.12/lib/python3.8/site-packages/rasa/__main__.py", line 121, in main
    cmdline_arguments.func(cmdline_arguments)
  File "/Users/melinda/.pyenv/versions/3.8.12/lib/python3.8/site-packages/rasa/cli/test.py", line 246, in run_nlu_test
    asyncio.run(
  File "/Users/melinda/.pyenv/versions/3.8.12/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
  File "/Users/melinda/.pyenv/versions/3.8.12/lib/python3.8/site-packages/rasa/cli/test.py", line 237, in run_nlu_test_async
    await test_nlu(model_path, data_path, output, all_args)
  File "/Users/melinda/.pyenv/versions/3.8.12/lib/python3.8/site-packages/rasa/model_testing.py", line 214, in test_nlu
    _agent = Agent.load(model_path=model)
  File "/Users/melinda/.pyenv/versions/3.8.12/lib/python3.8/site-packages/rasa/core/agent.py", line 356, in load
    agent.load_model(model_path=model_path, fingerprint=fingerprint)
  File "/Users/melinda/.pyenv/versions/3.8.12/lib/python3.8/site-packages/rasa/core/agent.py", line 363, in load_model
    self.processor = MessageProcessor(
  File "/Users/melinda/.pyenv/versions/3.8.12/lib/python3.8/site-packages/rasa/core/processor.py", line 96, in __init__
    self.model_filename, self.model_metadata, self.graph_runner = self._load_model(
  File "/Users/melinda/.pyenv/versions/3.8.12/lib/python3.8/site-packages/rasa/core/processor.py", line 121, in _load_model
    metadata, runner = loader.load_predict_graph_runner(
  File "/Users/melinda/.pyenv/versions/3.8.12/lib/python3.8/site-packages/rasa/engine/loader.py", line 29, in load_predict_graph_runner
    runner = graph_runner_class.create(
  File "/Users/melinda/.pyenv/versions/3.8.12/lib/python3.8/site-packages/rasa/engine/runner/dask.py", line 56, in create
    return cls(graph_schema, model_storage, execution_context, hooks)
  File "/Users/melinda/.pyenv/versions/3.8.12/lib/python3.8/site-packages/rasa/engine/runner/dask.py", line 42, in __init__
    self._instantiated_nodes: Dict[Text, GraphNode] = self._instantiate_nodes(
  File "/Users/melinda/.pyenv/versions/3.8.12/lib/python3.8/site-packages/rasa/engine/runner/dask.py", line 65, in _instantiate_nodes
    return {
  File "/Users/melinda/.pyenv/versions/3.8.12/lib/python3.8/site-packages/rasa/engine/runner/dask.py", line 66, in <dictcomp>
    node_name: GraphNode.from_schema_node(
  File "/Users/melinda/.pyenv/versions/3.8.12/lib/python3.8/site-packages/rasa/engine/graph.py", line 531, in from_schema_node
    return cls(
  File "/Users/melinda/.pyenv/versions/3.8.12/lib/python3.8/site-packages/rasa/engine/graph.py", line 382, in __init__
    self._load_component()
  File "/Users/melinda/.pyenv/versions/3.8.12/lib/python3.8/site-packages/rasa/engine/graph.py", line 405, in _load_component
    raise GraphComponentException(
rasa.engine.exceptions.GraphComponentException: Error initializing graph component for node 'run_ResponseSelector5'.

Definition of done

Fix bug

The text was updated successfully, but these errors were encountered:

TyDunn · 2021-12-16T12:15:05Z

@mleimeister Given your work with #10391, any ideas of what might be going on here?

mleimeister · 2021-12-22T14:02:29Z

Hi @TyDunn @melindaloubser1, sorry I saw this notifcation just now. I ran the above config on the moodbot with chitchat responses example. With this setup, the error does not seem to be related to the use_text_as_label setting, since for me both

- name: ResponseSelector
  epochs: 10
  use_text_as_label: true
  number_of_transformer_layers: 1

and

- name: ResponseSelector
  epochs: 10
  use_text_as_label: false
  number_of_transformer_layers: 1

lead to the same issue. However, the error does not happen if additionally setting transfomer_size to a value >0.

I was not able to reproduce the issue using the second config

- name: ResponseSelector
    epochs: 10
    use_text_as_label: true
    number_of_transformer_layers: 1
    transformer_size: 256
    hidden_layers_sizes:
      text: []
      label: []

@melindaloubser1 Not sure why this isn't happening for me with the second config. Could this be dataset dependent? If so, the below analysis might be missing something. Stepping through the code, the root of the error message in the logs seems to come from the following:

ResponseSelector has as default number_of_transformer_layers: 0 and transformer_size: None. If the first is specified as >0, during training a parameter correction function sets the transformer size to a default value (see here and here). This is triggered in the DIETClassifier initializer, which is the parent class of ResponseSelector. Therefore during training, everything works fine.

During model loading, which happens when running rasa test nlu, before running the constructor at the end of DIET's _load function, the model weights are loaded here. This ends up in the transformer code, where the function prepare_transformer_layer is triggered. However, because this is before the constructor of ResponseSelector is run, the parameter correction step has not happened yet and the units input parameter, which comes from the transformer_size in the config, is None. This leads to the error as in the logs above

File "/Users/matthias/Workspace/rasa3/rasa/utils/tensorflow/rasa_layers.py", line 1077, in prepare_transformer_layer
    units * 4,
TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'

A potential solution would be to also add a parameter correction step in the load function, before the model loading is happening. Alternatively, clarify in the docs that both those parameters have to be set?

jupyterjazz · 2022-01-10T11:48:12Z

Following @mleimeister's insights I created a draft solution on this branch.

@melindaloubser1
I was also unable to reproduce the error with the following configuration:

- name: ResponseSelector
    epochs: 10
    use_text_as_label: true
    number_of_transformer_layers: 1
    transformer_size: 256
    hidden_layers_sizes:
      text: []
      label: []

Can you please try reproducing the error with above-mentioned config file using this branch? In case it still exists, can you help me reproduce it?

indam23 added type:bug 🐛 Inconsistencies or issues which will cause an issue or problem for users or implementors. area:rasa-oss 🎡 Anything related to the open source Rasa framework labels Dec 13, 2021

TyDunn added the effort:atom-squad/2 Label which is used by the Rasa Atom squad to do internal estimation of task sizes. label Dec 17, 2021

jupyterjazz self-assigned this Dec 31, 2021

This was referenced Jan 7, 2022

Correct transformer_size parameter if needed for ResponseSelector during loading #10646

Closed

Correct transformer_size parameter if needed for ResponseSelector during loading #10651

Merged

m-vdb closed this as completed Jan 13, 2022

jupyterjazz mentioned this issue Jan 20, 2022

remove _warn_and_correct_transformer_size as it's not needed anymore #10712

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

(3.0) Error initializing ResponseSelector in `rasa test nlu` with `use_text_as_label: true` #10519

(3.0) Error initializing ResponseSelector in `rasa test nlu` with `use_text_as_label: true` #10519

indam23 commented Dec 13, 2021 •

edited

Loading

TyDunn commented Dec 16, 2021

mleimeister commented Dec 22, 2021 •

edited

Loading

jupyterjazz commented Jan 10, 2022

(3.0) Error initializing ResponseSelector in rasa test nlu with use_text_as_label: true #10519

(3.0) Error initializing ResponseSelector in rasa test nlu with use_text_as_label: true #10519

Comments

indam23 commented Dec 13, 2021 • edited Loading

Rasa Open Source version

Python version

What operating system are you using?

What happened?

Command / Request

Relevant log output

TyDunn commented Dec 16, 2021

mleimeister commented Dec 22, 2021 • edited Loading

jupyterjazz commented Jan 10, 2022

(3.0) Error initializing ResponseSelector in `rasa test nlu` with `use_text_as_label: true` #10519

(3.0) Error initializing ResponseSelector in `rasa test nlu` with `use_text_as_label: true` #10519

indam23 commented Dec 13, 2021 •

edited

Loading

mleimeister commented Dec 22, 2021 •

edited

Loading