Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Core re-asks for slot upon form interruption #7751

Closed
humcasma opened this issue Jan 19, 2021 · 4 comments
Closed

Core re-asks for slot upon form interruption #7751

humcasma opened this issue Jan 19, 2021 · 4 comments
Assignees
@ArjaanBuijk
Labels
area:rasa-oss 🎡 Anything related to the open source Rasa framework area:rasa-oss/forms Issues focused around rasa forms type:bug 🐛 Inconsistencies or issues which will cause an issue or problem for users or implementors.

Comments

@humcasma
Copy link

Rasa version: 2.2.5

Rasa SDK version (if used & relevant): 2.2.0

Python version: 3.6.9

Operating system (windows, osx, ...): Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic

Issue:
When a form is (temporarily) interrupted as part of handling an unhappy path, rasa core prints the utter_ask_<form_name>_<slot_name> message just before executing the actions specified to handle the deviation from the happy path.

An example can be seen executing the financial demo. Using rasa version 2.0.8, the expected dialogue is generated when trying to switch forms:

User: I want to pay my credit
Bot: How much do you want to pay?
User: I want to transfer money
Bot: We haven't completed the credit card payment yet. Are you sure you want to switch to money transfer?

However, using version 2.2.5, this is the generated dialogue:

User: I want to pay my credit
Bot: How much do you want to pay?
User: I want to transfer money
Bot: How much do you want to pay?  <---- REPEATED!!!
Bot: We haven't completed the credit card payment yet. Are you sure you want to switch to money transfer?

Error (including full traceback):
With Rasa version 2.0.8:

How much do you want to pay?
Your input ->  I want to transfer money
2021-01-19 10:46:32 DEBUG    rasa.core.processor  - Received user message 'I want to transfer money' with intent '{'id': -971893940168523848, 'name': 'transfer_money', 'confidence': 0.9983439445495605}' and entities '[]'
2021-01-19 10:46:32 DEBUG    rasa.core.processor  - Logged UserUtterance - tracker now has 18 events.
2021-01-19 10:46:32 DEBUG    rasa.core.policies.memoization  - Current tracker state [{}, {'user': {'intent': 'pay_cc'}, 'prev_action': {'action_name': 'action_listen'}}, {'user': {'intent': 'pay_cc'}, 'prev_action': {'action_name': 'cc_payment_form'}, 'active_loop': {'name': 'cc_payment_form'}}, {'user': {'intent': 'transfer_money'}, 'prev_action': {'action_name': 'action_listen'}, 'active_loop': {'name': 'cc_payment_form'}}]
2021-01-19 10:46:32 DEBUG    rasa.core.policies.memoization  - There is a memorised next action 'action_switch_forms_ask'
2021-01-19 10:46:32 DEBUG    rasa.core.policies.rule_policy  - Predicted loop 'cc_payment_form'.
2021-01-19 10:46:32 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_2_RulePolicy
2021-01-19 10:46:32 DEBUG    rasa.core.processor  - Predicted next action 'cc_payment_form' with confidence 1.00.
2021-01-19 10:46:32 DEBUG    rasa.core.actions.forms  - Validating user input 'UserUttered(text: I want to transfer money, intent: {'id': -971893940168523848, 'name': 'transfer_money', 'confidence': 0.9983439445495605}, entities: [])'.
2021-01-19 10:46:32 DEBUG    rasa.core.actions.forms  - Trying to extract requested slot 'amount-of-money' ...
2021-01-19 10:46:32 DEBUG    rasa.core.actions.forms  - Got mapping '{'entity': 'amount-of-money', 'not_intent': ['check_balance', 'check_earnings'], 'type': 'from_entity'}'
2021-01-19 10:46:32 DEBUG    rasa.core.actions.forms  - Got mapping '{'entity': 'number', 'not_intent': ['check_balance', 'check_earnings'], 'type': 'from_entity'}'
2021-01-19 10:46:32 DEBUG    rasa.core.actions.forms  - Got mapping '{'intent': ['inform', 'cc_payment_form'], 'type': 'from_text'}'
2021-01-19 10:46:32 DEBUG    rasa.core.actions.forms  - Failed to extract requested slot 'amount-of-money'
2021-01-19 10:46:32 DEBUG    rasa.core.policies.memoization  - Current tracker state [{}, {'user': {'intent': 'pay_cc'}, 'prev_action': {'action_name': 'action_listen'}}, {'user': {'intent': 'pay_cc'}, 'prev_action': {'action_name': 'cc_payment_form'}, 'active_loop': {'name': 'cc_payment_form'}}, {'user': {'intent': 'transfer_money'}, 'prev_action': {'action_name': 'action_listen'}, 'active_loop': {'name': 'cc_payment_form'}}]
2021-01-19 10:46:32 DEBUG    rasa.core.policies.memoization  - There is a memorised next action 'action_switch_forms_ask'
2021-01-19 10:46:32 DEBUG    rasa.core.policies.rule_policy  - Current tracker state: [{}, {'user': {'intent': 'pay_cc'}, 'prev_action': {'action_name': 'action_listen'}}, {'user': {'intent': 'pay_cc'}, 'prev_action': {'action_name': 'cc_payment_form'}, 'active_loop': {'name': 'cc_payment_form'}}, {'user': {'intent': 'transfer_money'}, 'prev_action': {'action_name': 'action_listen'}, 'active_loop': {'name': 'cc_payment_form'}}]
2021-01-19 10:46:32 DEBUG    rasa.core.policies.rule_policy  - There is no applicable rule.
2021-01-19 10:46:32 DEBUG    rasa.core.policies.ensemble  - Execution of 'cc_payment_form' was rejected. Setting its confidence to 0.0 in all predictions.
2021-01-19 10:46:32 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_0_AugmentedMemoizationPolicy
2021-01-19 10:46:32 DEBUG    rasa.core.processor  - Predicted next action 'action_switch_forms_ask' with confidence 1.00.
2021-01-19 10:46:32 DEBUG    rasa.core.actions.action  - Calling action endpoint to run action 'action_switch_forms_ask'.
2021-01-19 10:46:32 DEBUG    rasa.core.processor  - Action 'action_switch_forms_ask' ended with events '[BotUttered('We haven't completed the credit card payment yet. Are you sure you want to switch to money transfer?', {"elements": null, "quick_replies": null, "buttons": [{"payload": "/affirm", "title": "Yes"}, {"payload": "/deny", "title": "No"}], "attachment": null, "image": null, "custom": null}, {}, 1611053192.9989927), <rasa.shared.core.events.SlotSet object at 0x7f569c78f898>]'.

With Rasa version 2.2.5:

How much do you want to pay?
Your input ->  I want to transfer money
2021-01-19 12:19:53 DEBUG    rasa.core.lock_store  - Issuing ticket for conversation '7c980d75cdfd45e493add7dac02f35a0'.
2021-01-19 12:19:53 DEBUG    rasa.core.lock_store  - Acquiring lock for conversation '7c980d75cdfd45e493add7dac02f35a0'.
2021-01-19 12:19:53 DEBUG    rasa.core.lock_store  - Acquired lock for conversation '7c980d75cdfd45e493add7dac02f35a0'.
2021-01-19 12:19:53 DEBUG    rasa.core.tracker_store  - Recreating tracker for id '7c980d75cdfd45e493add7dac02f35a0'
2021-01-19 12:19:53 DEBUG    urllib3.connectionpool  - Starting new HTTP connection (1): localhost:8000
2021-01-19 12:19:53 ERROR    rasa.nlu.extractors.duckling_entity_extractor  - Failed to connect to duckling http server. Make sure the duckling server is running/healthy/not stale and the proper host and port are set in the configuration. More information on how to run the server can be found on github: https://github.com/facebook/duckling#quickstart Error: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /parse (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fbf5dac0e48>: Failed to establish a new connection: [Errno 111] Connection refused',))
2021-01-19 12:19:53 DEBUG    rasa.core.processor  - Received user message 'I want to transfer money' with intent '{'id': -8914028284748327624, 'name': 'transfer_money', 'confidence': 0.9995288252830505}' and entities '[]'
2021-01-19 12:19:53 DEBUG    rasa.core.processor  - Logged UserUtterance - tracker now has 19 events.
2021-01-19 12:19:53 DEBUG    rasa.core.policies.memoization  - Current tracker state:
[state 1] user intent: pay_cc | previous action name: action_listen
[state 2] user intent: pay_cc | previous action name: cc_payment_form | active loop: {'name': 'cc_payment_form'}
[state 3] user intent: transfer_money | previous action name: action_listen | active loop: {'name': 'cc_payment_form'}
2021-01-19 12:19:53 DEBUG    rasa.core.policies.memoization  - There is a memorised next action 'action_switch_forms_ask'
2021-01-19 12:19:53 DEBUG    rasa.core.policies.rule_policy  - Current tracker state:
[state 1] user intent: pay_cc | previous action name: action_listen
[state 2] user intent: pay_cc | previous action name: cc_payment_form | active loop: {'name': 'cc_payment_form'}
[state 3] user text: I want to transfer money | previous action name: action_listen | active loop: {'name': 'cc_payment_form'}
2021-01-19 12:19:53 DEBUG    rasa.core.policies.rule_policy  - There is no applicable rule.
2021-01-19 12:19:53 DEBUG    rasa.core.policies.rule_policy  - Predicted loop 'cc_payment_form'.
2021-01-19 12:19:53 DEBUG    rasa.core.policies.ensemble  - Made prediction using user intent.
2021-01-19 12:19:53 DEBUG    rasa.core.policies.ensemble  - Added `DefinePrevUserUtteredFeaturization(False)` event.
2021-01-19 12:19:53 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_2_RulePolicy.
2021-01-19 12:19:53 DEBUG    rasa.core.processor  - Predicted next action 'cc_payment_form' with confidence 1.00.
2021-01-19 12:19:53 DEBUG    rasa.core.actions.forms  - Validating user input 'UserUttered(text: I want to transfer money, intent: transfer_money, use_text_for_featurization: False)'.
2021-01-19 12:19:53 DEBUG    rasa.core.actions.forms  - Trying to extract requested slot 'amount-of-money' ...
2021-01-19 12:19:53 DEBUG    rasa.core.actions.forms  - Got mapping '{'entity': 'amount-of-money', 'not_intent': ['check_balance', 'check_earnings'], 'type': 'from_entity'}'
2021-01-19 12:19:53 DEBUG    rasa.core.actions.forms  - Got mapping '{'entity': 'number', 'not_intent': ['check_balance', 'check_earnings'], 'type': 'from_entity'}'
2021-01-19 12:19:53 DEBUG    rasa.core.actions.forms  - Got mapping '{'intent': ['inform', 'cc_payment_form'], 'type': 'from_text'}'
2021-01-19 12:19:53 DEBUG    rasa.core.actions.forms  - Failed to extract requested slot 'amount-of-money'
2021-01-19 12:19:53 DEBUG    rasa.core.actions.forms  - Validating extracted slots: {}
2021-01-19 12:19:53 DEBUG    rasa.core.actions.action  - Calling action endpoint to run action 'validate_cc_payment_form'.
2021-01-19 12:19:53 DEBUG    rasa.core.actions.forms  - Request next slot 'amount-of-money'
2021-01-19 12:19:53 DEBUG    rasa.core.processor  - Action 'cc_payment_form' ended with events '[<rasa.shared.core.events.LoopInterrupted object at 0x7fbf5dac0dd8>, <rasa.shared.core.events.ActionExecutionRejected object at 0x7fbf5dac0ef0>, <rasa.shared.core.events.SlotSet object at 0x7fbf5db0bbe0>, BotUttered('How much do you want to pay?', {"elements": null, "quick_replies": null, "buttons": null, "attachment": null, "image": null, "custom": null}, {"template_name": "utter_ask_cc_payment_form_amount-of-money"}, 1611058793.7025585)]'.
2021-01-19 12:19:53 DEBUG    rasa.core.processor  - Current slot values:
	requested_slot: amount-of-money
        ...
2021-01-19 12:19:53 DEBUG    rasa.core.policies.memoization  - Current tracker state:
[state 1] user intent: pay_cc | previous action name: action_listen
[state 2] user intent: pay_cc | previous action name: cc_payment_form | active loop: {'name': 'cc_payment_form'}
[state 3] user intent: transfer_money | previous action name: action_listen | active loop: {'name': 'cc_payment_form'}
2021-01-19 12:19:53 DEBUG    rasa.core.policies.memoization  - There is a memorised next action 'action_switch_forms_ask'
2021-01-19 12:19:53 DEBUG    rasa.core.policies.rule_policy  - Current tracker state:
[state 1] user intent: pay_cc | previous action name: action_listen
[state 2] user intent: pay_cc | previous action name: cc_payment_form | active loop: {'name': 'cc_payment_form'}
[state 3] previous action name: action_listen | active loop: {'name': 'cc_payment_form'}
2021-01-19 12:19:53 DEBUG    rasa.core.policies.rule_policy  - There is no applicable rule.
2021-01-19 12:19:53 DEBUG    rasa.core.policies.rule_policy  - Current tracker state:
[state 1] user intent: pay_cc | previous action name: action_listen
[state 2] user intent: pay_cc | previous action name: cc_payment_form | active loop: {'name': 'cc_payment_form'}
[state 3] user intent: transfer_money | previous action name: action_listen | active loop: {'name': 'cc_payment_form'}
2021-01-19 12:19:53 DEBUG    rasa.core.policies.rule_policy  - There is no applicable rule.
2021-01-19 12:19:53 DEBUG    rasa.core.policies.ensemble  - Execution of 'cc_payment_form' was rejected. Setting its confidence to 0.0 in all predictions.
2021-01-19 12:19:53 DEBUG    rasa.core.policies.ensemble  - Made prediction using user intent.
2021-01-19 12:19:53 DEBUG    rasa.core.policies.ensemble  - Added `DefinePrevUserUtteredFeaturization(False)` event.
2021-01-19 12:19:53 DEBUG    rasa.core.policies.ensemble  - Predicted next action using policy_0_AugmentedMemoizationPolicy.
2021-01-19 12:19:53 DEBUG    rasa.core.processor  - Predicted next action 'action_switch_forms_ask' with confidence 1.00.
2021-01-19 12:19:53 DEBUG    rasa.core.actions.action  - Calling action endpoint to run action 'action_switch_forms_ask'.
2021-01-19 12:19:53 DEBUG    rasa.core.processor  - Policy prediction ended with events '[<rasa.shared.core.events.DefinePrevUserUtteredFeaturization object at 0x7fbf5dc0f278>]'.
2021-01-19 12:19:53 DEBUG    rasa.core.processor  - Action 'action_switch_forms_ask' ended with events '[BotUttered('We haven't completed the credit card payment yet. Are you sure you want to switch to money transfer?', {"elements": null, "quick_replies": null, "buttons": [{"payload": "/affirm", "title": "Yes"}, {"payload": "/deny", "title": "No"}], "attachment": null, "image": null, "custom": null}, {}, 1611058793.7394376), <rasa.shared.core.events.SlotSet object at 0x7fbf5da445c0>]'.

Content of configuration file (config.yml) (if relevant):
See the financial demo

Content of domain file (domain.yml) (if relevant):
See the financial demo
@humcasma humcasma added area:rasa-oss 🎡 Anything related to the open source Rasa framework type:bug 🐛 Inconsistencies or issues which will cause an issue or problem for users or implementors. labels Jan 19, 2021
@sara-tagger
Copy link
Collaborator

Thanks for the issue, @JustinaPetr will get back to you about it soon!

You may find help in the docs and the forum, too 🤗

@ArjaanBuijk ArjaanBuijk self-assigned this Jan 19, 2021
@humcasma
Copy link
Author

The bug was introduced as part of the solution for this issue #6977
Up to v2.0.8, the validate() method in forms.py raised an ActionExecutionRejection exception if a slot extraction failed, and the form execution was immediately interrupted. As it is now, if the user manually interrupts the form, triggering an ActionExecutionRejected event, the exception is not raised, the execution of the form continues and request_next_slot() is executed once more.
It is correct not to raise an ActionExecutionRejection exception, because that "erases" all the triggered events in the validation code, including any potential LoopInterrupted event. But request_next_slot() should only be called inside the do() method if the user did not manually interrupt the form:

    async def do(
        self,
        output_channel: "OutputChannel",
        nlg: "NaturalLanguageGenerator",
        tracker: "DialogueStateTracker",
        domain: "Domain",
        events_so_far: List[Event],
    ) -> List[Event]:
        events = await self._validate_if_required(tracker, domain, output_channel, nlg)

        user_rejected_manually = any(
            isinstance(event, ActionExecutionRejected) for event in events
        )
        if not user_rejected_manually:
          events += await self.request_next_slot(
              tracker, domain, output_channel, nlg, events_so_far + events
          )

        return events

ArjaanBuijk added a commit that referenced this issue Jan 21, 2021
@ArjaanBuijk
Do the ActionExecutionRejection independent if user rejected manually
7de47da 
Fix for issue #7751 to avoid that a form will re-ask for the slot prior to switching to another form.
@ArjaanBuijk ArjaanBuijk mentioned this issue Jan 21, 2021
Merged
4 tasks
ArjaanBuijk added a commit that referenced this issue Jan 22, 2021
@ArjaanBuijk
Correct fix for issue #7751 where rasa ass for slot upon form switch
3b27bcc 
forms is first called for the original form. 
- The slot validation is called on the sentence that will trigger the form switching, but the original form is still active. The slot validation will fail and will also return ActionExecutionRejected event
- Need to avoid raising the ActionExecutionRejected event too soon, because other slots might be set by the custom action. Reverted that change with a comment to clarify.
- Need to avoid asking for the requested slot by checking if ActionExecutionRejected event was returned. The form is not yet done, so the is_done() method is still False.
@alwx alwx added the area:rasa-oss/forms Issues focused around rasa forms label Jan 28, 2021
@ma7555
Copy link

ma7555 commented Feb 9, 2021

is this fixed in 3b27bcc ?

@TyDunn
Copy link
Contributor

TyDunn commented Apr 16, 2021

@ArjaanBuijk Closing this issue because it looks like it was fixed in your PR

@TyDunn TyDunn closed this as completed Apr 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ArjaanBuijk ArjaanBuijk

Labels
area:rasa-oss 🎡 Anything related to the open source Rasa framework area:rasa-oss/forms Issues focused around rasa forms type:bug 🐛 Inconsistencies or issues which will cause an issue or problem for users or implementors.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@alwx @humcasma @ma7555 @TyDunn @ArjaanBuijk @sara-tagger