Markers docs

changelog/10095.doc.md

@@ -0,0 +1 @@
+Added new docs for Markers.

docs/docs/command-line-interface.mdx

@@ -23,6 +23,7 @@ abstract: The command line interface (CLI) gives you easy-to-remember commands f
 |`rasa data convert`     |Converts training data between different formats.                                                                                         |
 |`rasa data validate`    |Checks the domain, NLU and conversation data for inconsistencies.                                                                                             |
 |`rasa export`           |Exports conversations from a tracker store to an event broker.                                                                            |
+|`rasa evaluate markers` |Extracts markers from an existing tracker store.                                                                                                |
 |`rasa x`                |Launches Rasa X in local mode.                                                                                                            |
 |`rasa -h`               |Shows all available commands.                                                                                                             |
 
@@ -419,6 +420,54 @@ This command is most commonly used to import old conversations into Rasa X to an
 them. Read more about [importing conversations into Rasa X](https://rasa.com/docs/rasa-x/installation-and-setup/deploy#1-import-existing-conversations-from-rasa-open-source).
 :::
 
+## rasa evaluate markers
+
+:::caution
+
+This feature is currently experimental and might change or be removed in the future. Share your feedback in the forum to help us make it production-ready.
+
+:::
+
+The following command applies the [markers](./markers.mdx) you defined in your marker configuration file,
+to pre-existing dialogues stored in your [tracker store](./tracker-stores.mdx), and produces `CSV` files containing
+the extracted markers and summary statistics:
+
+```bash
+rasa evaluate markers all extracted_markers.csv
+```
+
+Use the following arguments to configure the marker extraction process:
+
+```
+usage: rasa evaluate markers [-h] [-v] [-vv] [--quiet] [--config CONFIG] [--no-stats | --stats-file-prefix [STATS_FILE_PREFIX]] [--endpoints ENDPOINTS] [-d DOMAIN] output_filename {first_n,sample,all} ...
+
+positional arguments:
+  output_filename       The filename to write the extracted markers to (CSV format).
+  {first_n,sample,all}
+    first_n             Select trackers sequentially until N are taken.
+    sample              Select trackers by sampling N.
+    all                 Select all trackers.
+
+optional arguments:
+  -h, --help            show this help message and exit
+  --config CONFIG       The config file(s) containing marker definitions. This can be a single YAML file, or a directory that contains several files with marker definitions in it. The content of these files will be read and
+                        merged together. (default: markers.yml)
+  --no-stats            Do not compute summary statistics. (default: True)
+  --stats-file-prefix [STATS_FILE_PREFIX]
+                        The common file prefix of the files where we write out the compute statistics. More precisely, the file prefix must consist of a common path plus a common file prefix, to which suffixes `-overall.csv` and
+                        `-per-session.csv` will be added automatically. (default: stats)
+  --endpoints ENDPOINTS
+                        Configuration file for the tracker store as a yml file. (default: endpoints.yml)
+  -d DOMAIN, --domain DOMAIN
+                        Domain specification. This can be a single YAML file, or a directory that contains several files with domain specifications in it. The content of these files will be read and merged together. (default:
+                        domain.yml)
+
+Python Logging Options:
+  -v, --verbose         Be verbose. Sets logging level to INFO. (default: None)
+  -vv, --debug          Print lots of debugging statements. Sets logging level to DEBUG. (default: None)
+  --quiet               Be quiet! Sets logging level to WARNING. (default: None)
+```
+
 ## rasa x
 
 Rasa X is a tool for practicing Conversation-Driven Development.

docs/docs/markers.mdx

@@ -0,0 +1,291 @@
+---
+id: markers
+sidebar_label: Markers
+title: Markers
+description: Find out how to mark points of interest in dialogues using Marker conditions.
+abstract: Markers are conditions that allow you to describe and mark points of interest in dialogues.
+---
+
+
+
+:::caution
+
+This feature is currently experimental and might change or be removed in the future. Share your feedback in the forum to help us make it production-ready.
+
+:::
+
+## Overview
+
+Markers are conditions that allow you to describe and mark points of interest in dialogues. In Rasa, a dialogue is represented as a sequence of events,
+which include bot actions that were executed, intents that were detected, and slots that were set.
+Markers allow you to describe conditions over such events, and when these conditions are met, the dialogues are "marked" for further analysis or inspection.
+
+There are several applications for Markers. For example, they can be used to define your bot's **Key Performance Indicators (KPIs)**,
+such as dialogue completion or task success. Take [Carbon Bot](https://rasa.com/blog/using-conversation-tags-to-measure-carbon-bots-success-rate/)
+for example, which helps users offset their carbon emissions from flying. For Carbon Bot, you might define **dialogue completion** as "all mandatory slots have been filled",
+and **task success** as "all mandatory slots have been filled and a carbon estimate has been successfully computed".
+Marking when these important events occur allows you to quantify Carbon Bot's success rate.
+
+Markers also allow you to **diagnose your dialogues** by surfacing important events for further inspection.
+For example, you might observe that Carbon Bot tends to successfully set the `travel_departure` and `travel_destination` slots,
+but fails to set the `travel_flight_class` slot. You can define a marker to quantify how often this behavior occurs
+and surface relevant dialogues for review as part of
+[Conversation Driven Development (CDD)](./conversation-driven-development.mdx).
+
+Marker definitions are written in `YAML`. For example, here are the markers that define dialogue completion and task success for Carbon Bot:
+
+```yaml
+marker_dialogue_completion:
+  and:
+    - slot_was_set: travel_departure
+    - slot_was_set: travel_destination
+    - slot_was_set: travel_flight_class
+
+marker_task_success:
+  and:
+    - slot_was_set: travel_departure
+    - slot_was_set: travel_destination
+    - slot_was_set: travel_flight_class
+    - action: provide_carbon_estimate
+```
+
+And here is the marker for surfacing dialogues where all mandatory slots are set except `travel_flight_class`:
+
+```yaml
+marker_dialogue_mandatory_slot_failure:
+  and:
+    - slot_was_set: travel_departure
+    - slot_was_set: travel_destination
+    - not:
+      - slot_was_set: travel_flight_class
+```
+
+The next sections explain how to write marker definitions, how to apply them to your existing dialogues, and what the output format looks like.
+
+## Defining Markers
+
+You define your markers in a marker configuration file written in `YAML`.
+Each marker should have a **unique identifier**, and consists of at least one **event condition**.
+Markers can also contain **operators**, which allow you to express more nuanced behavior or
+combine event conditions.
+
+Consider the following marker definition:
+```yaml
+marker_mood_expressed:
+  or:
+    - intent: mood_unhappy
+    - intent: mood_great
+```
+The unique marker identifier is `marker_mood_expressed`. This marker definition contains one operator `or`,
+and two event conditions `intent: mood_unhappy` and `intent: mood_great`. This marker will be true at every point in the dialogue
+where a user expressed either a `mood_unhappy` or a `mood_great`.
+
+### Event Conditions
+
+The following event condition labels are supported:
+
+- `action`: the specified bot action was executed.
+- `intent`: the specified user intent was detected.
+- `slot_was_set`: the specified slot was set.
+
+The negated forms of the labels are also supported:
+
+- `not_action`: the event is not the specified bot action.
+- `not_intent`: the event is not the specified user intent.
+- `slot_was_not_set`: the specified slot has not been set.
+
+### Operators
+
+The following operators are supported:
+
+- `and`: all listed conditions applied.
+- `or`: any of the listed conditions applied.
+- `not`: the condition did not apply. This operator only accepts 1 condition.
+- `seq`: the list of conditions applied in the specified order, with any number of events occurring in-between.
+- `at_least_once`: the listed marker definitions occurred at least once. Only the first occurrence will be marked.
+- `never`: the listed marker definitions never occurred.
+
+### Marker Configuration
+
+Here is an example of a marker configuration file containing several marker definitions. The example is created for mood bot,
+with a new slot `name` to illustrate the use of the label `slot_was_set`:
+
+```yaml
+marker_name_provided:
+  slot_was_set: name
+
+marker_mood_expressed:
+  or:
+    - intent: mood_unhappy
+    - intent: mood_great
+
+marker_cheer_up_failed:
+  seq:
+    - intent: mood_unhappy
+    - action: utter_cheer_up
+    - action: utter_did_that_help
+    - intent: deny
+
+marker_bot_not_challenged:
+  never:
+    - intent: bot_challenge
+
+marker_cheer_up_attempted:
+  at_least_once:
+    - action: utter_cheer_up
+
+marker_mood_expressed_and_name_not_provided:
+  and:
+    - not:
+      - slot_was_set: name
+    - or:
+      - intent: mood_unhappy
+      - intent: mood_great
+```
+Note the following:
+
+- Each marker has a unique identifier (or name) such as `marker_name_provided`.
+
+- A marker definition can contain a single condition, as shown in `marker_name_provided`.
+
+- A marker definition can contain a single operator with a list of conditions, as shown in `marker_mood_expressed`,
+`marker_cheer_up_failed`, `marker_bot_not_challenged`, and `marker_cheer_up_attempted`.
+
+- A marker definition can contain nested operators, as shown in `marker_mood_expressed_and_name_not_provided`.
+
+- The values assigned to event conditions must be valid according to your bot's `domain.yml` file. For example, in
+`marker_mood_expressed`, the intents `mood_unhappy` and `mood_unhappy` are both intents listed in the mood bot's `domain.yml` file.
+
+:::note
+You cannot yet reuse an existing marker name in the definition of another marker.
+:::
+
+## Extracting Markers
+
+Markers are extracted from dialogues already stored in a tracker store. To learn how to store interactions with your bot in a tracker store,
+read the [Tracker Store](./tracker-stores.mdx) page.
+
+Once you've created your marker definitions in the marker configuration file, and have stored some dialogues in your tracker store,
+you can apply your markers to your trackers by running the following command:
+
+```bash
+rasa evaluate markers all --config marker.yml extracted_markers.csv
+```
+This script will process the marker definitions you provide your in the marker configuration file: `marker.yml`.
+The script will output the extracted markers in the specified output file: `extracted_markers.csv`.
+It will also produce two summary statistics files. The format of the output files are described in the next section.
+
+By default, the script will validate your marker definitions against your bot's `domain.yml` file. To specify a different
+domain file, use the optional `--domain` argument.
+
+By default, the script will process the tracker store in your bot's `endpoint.yml`. However, you can specify a different
+endpoint file using the optional `--endpoint` argument.
+
+Three different tracker loading strategies are supported: `all`, `sample_n`, and `first_n`. The option `all` will process
+all the trackers in your tracker store. The other two strategies process a subset of `N` trackers, either sequentially
+(by using `first_n`), or by sampling uniformly without replacement (using `sample_n`). The sampling strategy also allows you to set the random seed.
+For more information on the usage of each strategy, type the following command, replacing `<strategy>` with one of: `all`, `first_n`, and `sample_n`:
+```bash
+rasa evaluate markers <strategy> --help
+```
+
+:::note
+Each tracker in the tracker store can contain multiple sessions. The script will process each session separately, indexing them by `session_idx`.
+:::
+The next two sections describe the formats of the extracted markers and computed statistics.
+
+
+### Extracted Markers
+
+For each marker defined in your marker configuration file, the following information is extracted:
+1. The number of user turns preceding the event at which a marker applied. Each `UserUttered` event is treated as a user turn.
+2. The index of the event at which a marker applied.
+
+For the `at_least_once` operator, the information above will only be extracted for the first occurrence.
+
+The extracted markers will be output to the `.csv` file you specify in the script, for example, `extracted_markers.csv`.
+The extracted markers output file contains the following columns:
+
+- `sender_id`: taken from the trackers.
+- `session_idx`: an integer indexing sessions, starting with `0`.
+- `marker`: the unique identifier you specified for the marker.
+- `event_idx`: an integer indexing events, starting with `0`.
+- `num_preceding_user_turns`: an integer specifying the number of user turns which preceded the event at which the marker applied.
+
+Here is a sample of the extracted markers output file:
+
+```
+sender_id,session_idx,marker_name,event_id,num_preceding_user_turns
+1309ed25d6fa45beb74d0862d37289a5,0,marker_mood_expressed,7,1
+2cf4106e76a041dfb2d4f75bd6ae804b,0,marker_mood_expressed,7,1
+:
+```
+
+### Computed Statistics
+
+By default, the command computes summary statistics about the information gathered. To disable the statistics computation, use the optional flag `--no-stats`.
+
+The script computes the following statistics:
+
+1. **For each session and each marker**: "per-session statistics" which include the arithmetic mean, median, minimum, and maximum number of user turns preceding the event at which the marker applied.
+2. **For all sessions and for each marker**:
+    1. Overall statistics including the arithmetic mean, median, minimum, and maximum number of user turns preceding the event where the marker applied in any session.
+    2. The number of sessions and the percentage of sessions where each marker applied at least once.
+
+The results are stored in tabular format in `stats-overall.csv` and `stats-per-session.csv`. You can change prefix `stats` in the file names using the optional argument `--stats-file-prefix `.
+For example, the following script will produce the files: `marker-statistics-overall.csv` and `marker-statistics-per-session.csv`:
+```bash
+rasa evaluate markers all --stats-file-prefix "marker-statistics" extracted_markers.csv
+```
+
+
+Both files contain the following columns:
+
+- `sender_id`: taken from the trackers.
+- `session_idx`: an integer indexing sessions, starting with `0`.
+- `marker`: the unique identifier you specified for the marker.
+- `statistic`: a description of the statistic computed.
+- `value`: an integer or float value represented the computed statistic.
+
+Here is a sample `stats-per-session.csv` output:
+
+```
+sender_id,session_idx,marker,statistic,value
+1309ed25d6fa45beb74d0862d37289a5,0,marker_mood_expressed,count(number of preceding user turns),1
+2cf4106e76a041dfb2d4f75bd6ae804b,0,marker_mood_expressed,count(number of preceding user turns),1
+79e7604d2acc442d9f78e00b173786c9,0,marker_mood_expressed,count(number of preceding user turns),0
+1309ed25d6fa45beb74d0862d37289a5,0,marker_mood_expressed,max(number of preceding user turns),1
+2cf4106e76a041dfb2d4f75bd6ae804b,0,marker_mood_expressed,max(number of preceding user turns),1
+79e7604d2acc442d9f78e00b173786c9,0,marker_mood_expressed,max(number of preceding user turns),nan
+1309ed25d6fa45beb74d0862d37289a5,0,marker_mood_expressed,mean(number of preceding user turns),1.0
+2cf4106e76a041dfb2d4f75bd6ae804b,0,marker_mood_expressed,mean(number of preceding user turns),1.0
+79e7604d2acc442d9f78e00b173786c9,0,marker_mood_expressed,mean(number of preceding user turns),nan
+1309ed25d6fa45beb74d0862d37289a5,0,marker_mood_expressed,median(number of preceding user turns),1.0
+2cf4106e76a041dfb2d4f75bd6ae804b,0,marker_mood_expressed,median(number of preceding user turns),1.0
+79e7604d2acc442d9f78e00b173786c9,0,marker_mood_expressed,median(number of preceding user turns),nan
+1309ed25d6fa45beb74d0862d37289a5,0,marker_mood_expressed,min(number of preceding user turns),1
+2cf4106e76a041dfb2d4f75bd6ae804b,0,marker_mood_expressed,min(number of preceding user turns),1
+79e7604d2acc442d9f78e00b173786c9,0,marker_mood_expressed,min(number of preceding user turns),nan
+:
+```
+Note that some statistics will be `nan`, for example if a marker never occurred, then the number of preceding user turns will be `nan`.
+
+Here is a sample `stats-overall.csv` output.
+
+```
+sender_id,session_idx,marker,statistic,value
+all,nan,-,total_number_of_sessions,3
+all,nan,marker_cheer_up,number_of_sessions_where_marker_applies_at_least_once,0
+all,nan,marker_cheer_up,percentage_of_sessions_where_marker_applies_at_least_once,0.0
+all,nan,marker_mood_expressed,number_of_sessions_where_marker_applies_at_least_once,2
+all,nan,marker_mood_expressed,percentage_of_sessions_where_marker_applies_at_least_once,66.667
+all,nan,marker_mood_expressed_and_name_provided,number_of_sessions_where_marker_applies_at_least_once,0
+all,nan,marker_mood_expressed_and_name_provided,percentage_of_sessions_where_marker_applies_at_least_once,0.0
+:
+```
+Note that because each row computes a statistics over all session, the `sender_id` is equal to `all`,
+and the `session_idx` is equal to `nan`.
+
+## Configuring the CLI command
+
+Visit our [CLI page](./command-line-interface.mdx#rasa-evaluate-markers) for more information on configuring the marker extraction and statistics computation process.

docs/sidebars.js

@@ -77,6 +77,7 @@ module.exports = {
             'default-actions',
           ],
         },
+        'markers',
         {
           type: 'category',
           label: 'Channel Connectors',