rasa validate KeyError during story conflict testing

[gdeluc1]

Rasa version: 2.2.1

Rasa SDK version: 2.2.0

Rasa X version: 0.34.0

Python version: 3.7.9

Operating system: Windows-7-6.1.7601-SP1

Issue: rasa data validate throws error when using retrieval intents - also ResponsesSeclector responses need to be out of domain.ym unlike indicated by the online documentation on chitchat and faq can be reproduced with RasaHQ responselectorbot

Error (including full traceback):

(ihi2) D:\GitLab\IHI\rasa-bot>rasa data validate
2020-12-18 14:58:36.368767: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll


2020-12-18 14:58:40.702200: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library nvcuda.dll
2020-12-18 14:58:40.946225: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:02:00.0 name: GeForce GTX 1080 computeCapability: 6.1
coreClock: 1.7335GHz coreCount: 20 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 298.32GiB/s
2020-12-18 14:58:40.957226: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 1 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 980 computeCapability: 5.2
coreClock: 1.2155GHz coreCount: 16 deviceMemorySize: 4.00GiB deviceMemoryBandwidth: 208.91GiB/s
2020-12-18 14:58:40.963226: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-12-18 14:58:40.973227: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2020-12-18 14:58:40.981228: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2020-12-18 14:58:40.986229: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2020-12-18 14:58:40.998230: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2020-12-18 14:58:41.004231: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2020-12-18 14:58:41.014232: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2020-12-18 14:58:41.042234: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0, 1
2020-12-18 14:58:44 INFO     rasa.validator  - Validating intents...
2020-12-18 14:58:44 INFO     rasa.validator  - Validating uniqueness of intents and stories...
2020-12-18 14:58:44 INFO     rasa.validator  - Validating utterances...
2020-12-18 14:58:44 INFO     rasa.validator  - Story structure validation...
Processed story blocks: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 999.83it/s, # trackers=1]
2020-12-18 14:58:44 INFO     rasa.core.training.story_conflict  - Considering all preceding turns for conflict analysis.
2020-12-18 14:58:44 INFO     rasa.nlu.utils.spacy_utils  - Trying to load spacy model with name 'en_core_web_lg'
2020-12-18 14:59:00 INFO     rasa.nlu.components  - Added 'SpacyNLP' to component cache. Key 'SpacyNLP-en_core_web_lg'.
Traceback (most recent call last):
  File "c:\programdata\anaconda3\envs\ihi2\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "c:\programdata\anaconda3\envs\ihi2\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\ProgramData\Anaconda3\envs\ihi2\Scripts\rasa.exe\__main__.py", line 7, in <module>
  File "c:\programdata\anaconda3\envs\ihi2\lib\site-packages\rasa\__main__.py", line 116, in main
    cmdline_arguments.func(cmdline_arguments)
  File "c:\programdata\anaconda3\envs\ihi2\lib\site-packages\rasa\cli\data.py", line 262, in validate_files
    and _validate_story_structure(validator, args)
  File "c:\programdata\anaconda3\envs\ihi2\lib\site-packages\rasa\cli\data.py", line 297, in _validate_story_structure
    not args.fail_on_warnings, max_history=args.max_history
  File "c:\programdata\anaconda3\envs\ihi2\lib\site-packages\rasa\validator.py", line 249, in verify_story_structure
    trackers, self.domain, max_history, self.nlu_config
  File "c:\programdata\anaconda3\envs\ihi2\lib\site-packages\rasa\core\training\story_conflict.py", line 185, in find_story_conflicts
    trackers, domain, max_history, tokenizer
  File "c:\programdata\anaconda3\envs\ihi2\lib\site-packages\rasa\core\training\story_conflict.py", line 245, in _find_conflicting_states
    for element in _sliced_states_iterator(trackers, domain, max_history, tokenizer):
  File "c:\programdata\anaconda3\envs\ihi2\lib\site-packages\rasa\core\training\story_conflict.py", line 333, in _sliced_states_iterator
    _apply_tokenizer_to_states(tokenizer, sliced_states)
  File "c:\programdata\anaconda3\envs\ihi2\lib\site-packages\rasa\core\training\story_conflict.py", line 351, in _apply_tokenizer_to_states
    Message({TEXT: state[USER][TEXT]}), TEXT
KeyError: 'text'

Command or request that led to error:

rasa data validate

Content of configuration file (config.yml) (if relevant):

language: en

pipeline:
  - name: "WhitespaceTokenizer"
  - name: "CountVectorsFeaturizer"
  - name: "DIETClassifier"
    entity_recognition: False
    epochs: 50
  - name: ResponseSelector
    epochs: 50

policies:
  - name: TEDPolicy
    max_history: 5
    epochs: 100
  - name: MemoizationPolicy
  - name: RulePolicy

Content of domain file (domain.yml) (if relevant):

version: "2.0"

intents:
  - greet
  - goodbye
  - affirm
  - deny
  - mood_great
  - mood_unhappy
  - bot_challenge
  - chitchat

responses:
  utter_greet:
  - text: "Hey! How are you?"
    buttons:
    - title: "great"
      payload: "/mood_great"
    - title: "super sad"
      payload: "/mood_unhappy"

  utter_cheer_up:
  - text: "Here is something to cheer you up:"
    image: "https://i.imgur.com/nGF1K8f.jpg"

  utter_did_that_help:
  - text: "Did that help you?"

  utter_happy:
  - text: "Great, carry on!"

  utter_goodbye:
  - text: "Bye"

  utter_iamabot:
  - text: "I am a bot, powered by Rasa."
  
  # utter_chitchat:
  # - text: "this is to make retrieval intent work with rasa 2.2.1"
  
  
  # utter_chitchat/ask_name:
    # - image: "https://i.imgur.com/zTvA58i.jpeg"
      # text: hello, my name is retrieval bot.
    # - text: Oh yeah, I am called the retrieval bot.

  # utter_chitchat/ask_weather:
    # - text: Oh, it does look sunny right now in Berlin.
      # image: "https://i.imgur.com/vwv7aHN.png"
    # - text: I am not sure of the whole week but I can see the sun is out today.  
  
  

session_config:
  session_expiration_time: 60  # value in minutes
  carry_over_slots_to_new_session: true

Content of nlu file (nlu.yml) (if relevant):

version: "2.0"

nlu:
- intent: greet
  examples: |
    - hey
    - hello
    - hi
    - hello there
    - good morning
    - good evening
    - moin
    - hey there
    - let's go
    - hey dude
    - goodmorning
    - goodevening
#    - good afternoon

- intent: goodbye
  examples: |
    - good afternoon
    - cu
    - good by
    - cee you later
    - good night
    - bye
    - goodbye
    - have a nice day
    - see you around
    - bye bye
    - see you later

- intent: affirm
  examples: |
    - yes
    - y
    - indeed
    - of course
    - that sounds good
    - correct

- intent: deny
  examples: |
    - no
    - n
    - never
    - I don't think so
    - don't like that
    - no way

- intent: mood_great
  examples: |
    - perfect
    - great
    - amazing
    - feeling like a king
    - wonderful
    - I am feeling very good
    - I am great
    - I am amazing
    - I am going to save the world
    - super stoked
    - extremely good
    - so so perfect
    - so good
    - so perfect

- intent: mood_unhappy
  examples: |
    - my day was horrible
    - I am sad
    - I don't feel very well
    - I am disappointed
    - super sad
    - I'm so sad
    - sad
    - very sad
    - unhappy
    - not good
    - not very good
    - extremly sad
    - so saad
    - so sad

- intent: bot_challenge
  examples: |
    - are you a bot?
    - are you a human?
    - am I talking to a bot?
    - am I talking to a human?

- intent: chitchat/ask_name
  examples: |
    - What is your name?
    - May I know your name?
    - What do people call you?
    - Do you have a name for yourself?

- intent: chitchat/ask_weather
  examples: |
    - What's the weather like today?
    - Does it look sunny outside today?
    - Oh, do you mind checking the weather for me please?
    - I like sunny days in Berlin.

responses:
  utter_chitchat/ask_name:
    - image: "https://i.imgur.com/zTvA58i.jpeg"
      text: hello, my name is retrieval bot.
    - text: Oh yeah, I am called the retrieval bot.

  utter_chitchat/ask_weather:
    - text: Oh, it does look sunny right now in Berlin.
      image: "https://i.imgur.com/vwv7aHN.png"
    - text: I am not sure of the whole week but I can see the sun is out today.

Content of stories file (stories.yml) (if relevant):

version: "2.0"

stories:

- story: happy path
  steps:
  - intent: greet
  - action: utter_greet
  - intent: mood_great 
  - action: utter_happy

- story: sad path 1
  steps:
  - intent: greet
  - action: utter_greet
  - intent: mood_unhappy
  - action: utter_cheer_up
  - action: utter_did_that_help
  - intent: affirm
  - action: utter_happy

- story: sad path 2
  steps:
  - intent: greet
  - action: utter_greet
  - intent: mood_unhappy
  - action: utter_cheer_up
  - action: utter_did_that_help
  - intent: deny
  - action: utter_goodbye

Content of rules file (rules.yml) (if relevant):

version: "2.0"

rules:
  
- rule: Say goodbye anytime the user says goodbye
  steps:
  - intent: goodbye
  - action: utter_goodbye
  
- rule: Say 'I am a bot' anytime the user challenges
  steps:
  - intent: bot_challenge
  - action: utter_iamabot

- rule: Response with a chitchat utterance whenever user indulges in some chitchat
  steps:
  - intent: chitchat
  - action: utter_chitchat

[sara-tagger]

Thanks for raising this issue, @koaning will get back to you about it soon✨

Please also check out the docs and the forum in case your issue was raised there too 🤗

[madanmeena]

I am also facing similar issue both on 2.2.1 and 2.2.2

(venv) madan@MadanM-L:~/code/TSI-Bot-Spanish$ rasa data validate --debug
2020-12-22 12:16:42.420683: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2020-12-22 12:16:42.420708: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2020-12-22 12:16:43.529856: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2020-12-22 12:16:43.529883: W tensorflow/stream_executor/cuda/cuda_driver.cc:312] failed call to cuInit: UNKNOWN ERROR (303)
2020-12-22 12:16:43.529896: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (MadanM-L): /proc/driver/nvidia/version does not exist
2020-12-22 12:16:43 DEBUG    rasa.shared.nlu.training_data.loading  - Training data format of 'data/nlu.yml' is 'rasa_yml'.
2020-12-22 12:16:43 DEBUG    rasa.shared.nlu.training_data.loading  - Training data format of 'data/responses.yml' is 'rasa_yml'.
2020-12-22 12:16:43 DEBUG    rasa.shared.nlu.training_data.loading  - Training data format of 'data/rules.yml' is 'unk'.
2020-12-22 12:16:43 DEBUG    rasa.shared.nlu.training_data.loading  - Training data format of 'data/stories.yml' is 'unk'.
2020-12-22 12:16:46 DEBUG    rasa.shared.nlu.training_data.loading  - Training data format of 'data/nlu.yml' is 'rasa_yml'.
2020-12-22 12:16:46 DEBUG    rasa.shared.nlu.training_data.loading  - Training data format of 'data/responses.yml' is 'rasa_yml'.
2020-12-22 12:16:46 INFO     rasa.validator  - Validating intents...
2020-12-22 12:16:46 DEBUG    rasa.validator  - The intent 'back' is listed in the domain file, but is not found in the NLU training data.
2020-12-22 12:16:46 DEBUG    rasa.validator  - The intent 'nlu_fallback' is listed in the domain file, but is not found in the NLU training data.
2020-12-22 12:16:46 DEBUG    rasa.validator  - The intent 'out_of_scope' is listed in the domain file, but is not found in the NLU training data.
2020-12-22 12:16:46 DEBUG    rasa.validator  - The intent 'restart' is listed in the domain file, but is not found in the NLU training data.
2020-12-22 12:16:46 DEBUG    rasa.validator  - The intent 'session_start' is listed in the domain file, but is not found in the NLU training data.
2020-12-22 12:16:46 DEBUG    rasa.validator  - The intent 'back' is not used in any story.
2020-12-22 12:16:46 DEBUG    rasa.validator  - The intent 'out_of_scope' is not used in any story.
2020-12-22 12:16:46 DEBUG    rasa.validator  - The intent 'restart' is not used in any story.
2020-12-22 12:16:46 DEBUG    rasa.validator  - The intent 'session_start' is not used in any story.
2020-12-22 12:16:46 INFO     rasa.validator  - Validating uniqueness of intents and stories...
2020-12-22 12:16:46 INFO     rasa.validator  - Validating utterances...
2020-12-22 12:16:46 DEBUG    rasa.validator  - The utterance 'utter_default' is not used in any story.
2020-12-22 12:16:46 INFO     rasa.validator  - Story structure validation...
2020-12-22 12:16:46 DEBUG    rasa.shared.core.generator  - Number of augmentation rounds is 0
2020-12-22 12:16:46 DEBUG    rasa.shared.core.generator  - Starting data generation round 0 ... (with 1 trackers)
Processed story blocks: 100%|███████████████████████████████████████████████████████████| 127/127 [00:00<00:00, 1064.75it/s, # trackers=1]
2020-12-22 12:16:46 DEBUG    rasa.shared.core.generator  - Finished phase (118 training samples found).
2020-12-22 12:16:46 DEBUG    rasa.shared.core.generator  - Data generation rounds finished.
2020-12-22 12:16:46 DEBUG    rasa.shared.core.generator  - Found 0 unused checkpoints
2020-12-22 12:16:46 DEBUG    rasa.shared.core.generator  - Found 118 training trackers.
2020-12-22 12:16:46 INFO     rasa.core.training.story_conflict  - Considering all preceding turns for conflict analysis.
Traceback (most recent call last):
  File "/home/madan/code/TSI-Bot-Spanish/venv/bin/rasa", line 8, in <module>
    sys.exit(main())
  File "/home/madan/code/TSI-Bot-Spanish/venv/lib/python3.8/site-packages/rasa/__main__.py", line 116, in main
2020-12-22 12:16:46 DEBUG    urllib3.connectionpool  - Starting new HTTPS connection (1): o251570.ingest.sentry.io:443
    cmdline_arguments.func(cmdline_arguments)
  File "/home/madan/code/TSI-Bot-Spanish/venv/lib/python3.8/site-packages/rasa/cli/data.py", line 262, in validate_files
    and _validate_story_structure(validator, args)
  File "/home/madan/code/TSI-Bot-Spanish/venv/lib/python3.8/site-packages/rasa/cli/data.py", line 296, in _validate_story_structure
    return validator.verify_story_structure(
  File "/home/madan/code/TSI-Bot-Spanish/venv/lib/python3.8/site-packages/rasa/validator.py", line 248, in verify_story_structure
    conflicts = rasa.core.training.story_conflict.find_story_conflicts(
  File "/home/madan/code/TSI-Bot-Spanish/venv/lib/python3.8/site-packages/rasa/core/training/story_conflict.py", line 184, in find_story_conflicts
    conflicting_state_action_mapping = _find_conflicting_states(
  File "/home/madan/code/TSI-Bot-Spanish/venv/lib/python3.8/site-packages/rasa/core/training/story_conflict.py", line 245, in _find_conflicting_states
    for element in _sliced_states_iterator(trackers, domain, max_history, tokenizer):
  File "/home/madan/code/TSI-Bot-Spanish/venv/lib/python3.8/site-packages/rasa/core/training/story_conflict.py", line 333, in _sliced_states_iterator
    _apply_tokenizer_to_states(tokenizer, sliced_states)
  File "/home/madan/code/TSI-Bot-Spanish/venv/lib/python3.8/site-packages/rasa/core/training/story_conflict.py", line 351, in _apply_tokenizer_to_states
    Message({TEXT: state[USER][TEXT]}), TEXT
KeyError: 'text'
2020-12-22 12:16:47 DEBUG    urllib3.connectionpool  - https://o251570.ingest.sentry.io:443 "POST /api/2801673/store/ HTTP/1.1" 200 41
(venv) madan@MadanM-L:~/code/TSI-Bot-Spanish$ 

[Nriver]

Downgrade rasa to 2.1.3 and retrain the model solves the problem for me.

pip3 install rasa==2.1.3

[gdeluc1]

Thanks Nriver. 2.1.3 is indeed a (temporary) solution but there are still issues, see this post from Rasa blog.
Would be good if the problem of syntax could be clarified.

[santteegt]

This bug was introduced in this commit 954e473#diff-b9a6da0b97982d86712af2c20891570d9fa8d662a4d4fbf0e018b085eaaaf8e9 See rasa/core/training/story_conflict.py file. Seems that there are some compatibility issues with the experimental end-to-end training feature while running the rasa data validate

[rgstephens]

Renamed issue since this issue is not specific to retrieval intents. Duplicated this issue with story:

- story: time_from
  steps:
  - intent: time_from
  - action: action_time_range

states:

{"prev_action": {"action_name": "action_listen"}, "user": {"intent": "time_from"}}

state[USER]:

{"intent": "time_from"}