Add tokens to events

[joejuzl]

Description of Problem:
Background: #8533
Proposed solution: https://www.notion.so/rasa/Sending-tokens-with-events-to-Rasa-X-05fe8895e4aa416e8ca8e3e1f339bc91

Summary:
We don't want to re-calculate tokens in Rasa X so we want to send the message tokens along with the event through the event broker.

Overview of the Solution:

• Add the text tokens to the events by updating output_properties
• Convert them to a serialisable form

Related Issues:
Adding model identifier to events: #8914

Definition of Done:

• UserUttered with text tokens are serializable.
• All UserUttered events have text tokens attached when pushed to the event broker.
• Integration tested.

[ancalita]

@joejuzl might need a quick crash course in integration testing 😬 to make sure I understand what's expected to be completed. I also left a few questions in the Notion page + double-checking on deserialisation of tokens, if that's something i need to take care of anywhere in Rasa? Or that's something that happens at Rasa-X end?

[joejuzl]

double-checking on deserialisation of tokens, if that's something i need to take care of anywhere in Rasa? Or that's something that happens at Rasa-X end?

It will be up to Rasa X to read the tokens from the events that it consumes from the event broker.

might need a quick crash course in integration testing

We have so few integration tests right now. Currently only for one broker, and it just checks initial connection...
Thinking about it now I'm not 100% sure if adding integration tests should be in scope of this issue.
For now I think just unit and manual test (with local Rasa X instance!)

Also have a look at tests/integration_tests/core/brokers/test_pika.py , and if we do decide to do more tests we can have a call to discuss it properly.

[ancalita]

@joejuzl Quick q on running a manual test with rasa x: When I initialised the development set-up, I created separate virtual envs for rasa-oss and rasa x. Will the rasa x command run in the rasa-oss env or should I pip install rasa-x in this environment too?

[joejuzl]

@joejuzl Quick q on running a manual test with rasa x: When I initialised the development set-up, I created separate virtual envs for rasa-oss and rasa x. Will the rasa x command run in the rasa-oss env or should I pip install rasa-x in this environment too?

@ancalita you need to make sure you have both rasa and rasa-x installed in the same environment and that the rasa installation is using the source code on your machine not a pypi package. The way to do this is to run make install from the rasa directory after install rasa-x.
This guide shows one way of getting to this state of affairs: https://www.notion.so/rasa/Python-Development-Setup-6a7308797283415b9a66b4d5985437fd