Markers docs #10095
Markers docs #10095
Conversation
aeshky
requested review from
tttthomasssss,
usc-m,
koaning,
kedz,
ka-bu and
TyDunn
…ker_docs * 'marker_docs' of https://github.com/RasaHQ/rasa: Use a unique fingerprint for a resource that is consistent after caching. (#10187) prepared release of version 2.8.13 (#10176) (#10178) check origin type use rc1 make model version 3.0.0
moving a couple of sentences around
…ker_docs * 'marker_docs' of https://github.com/RasaHQ/rasa: extract variable add review suggestion more refactoring refactor fix to reduce complexity improve fix for dynamic forms fix for dynamic forms bug fix for testing e2e stories of bot with slots add logging and unit test handle EntitiesAdded edge case run action_extract_slots in processor.trigger_external_user_uttered remove Domain.slots_for_entities usage in tracker.update for UserUttered events revert changes fix a few more stories.yml test files fix some stories.yml files and change test name remove Domain.slots_for_entities usage in StoryStep
clarified the evaluation angle in the intro
aeshky
dismissed
hsm207’s stale review
I have addressed Shukri's comments, but he's away today so will not be able to review the changes. We've decided in enable to merge this PR in order to include the docs in today's RC2 release. If Shukri has more feedback, it can be included in RC3 on Thursday.
|
When I read the docs today I noticed that I understood what "number of preceding user turns" means. But immediately after it occurred to me that I didn't see why this metric is interesting. How would a conversational designer use this statistic to improve their bot? After toying around I think it's related to "if it takes a long time, then the user may be misunderstood a few times". If this is the case, it deserves to be mentioned explicitly as a use case in the documentation. |
The number of preceding user turns is telling you how many utterances the user had to make before reaching the event that you care about. It's a good measure of progress in task-oriented dialogues from the perspective of the user. For example if you had a marker that defines task success you want to know how many turns it took to reach success. Like you say, maybe the user had to rephrase multiple times which caused the dialogue to become longer, or maybe the task wasn't clear so the user had to ask clarifying questions. The number of turns is used as a metric dialogue research (here are some examples [1], [2], [3]). If the dialogues are 1:1 (1 system utterance for every 1 user utterance) you'll sometimes see the "number of turns" measured, but I think the "number of user turns" is more meaningful anyway, and especially when the system can make multiple utterances / take multiple actions (as is the case with Rasa). The pure number of events is less meaningful because you can have a lot of system events that are not part of the dialogue (querying a database, starting a session, etc.). So the number of user turns give a better intuition of how long the dialogue is and particularly from the user's perspective. Does this make sense? I almost added an explanation to the document but felt that it was moving away from usage instruction and wondered if it's better in a blog post or a tutorial. But since you raised it, I think it's worth giving the intuition in the docs too. |
|
Yep! Those two paragraphs are much appreciated 😃 ! |
|
|
||
| :::caution | ||
|
|
||
| This feature is currently experimental and might change or be removed in the future. Share your feedback in the forum to help us make it production-ready. |
There was a problem hiding this comment.
Do we have a thread on the forum? It cannot hurt to start a thread right now and point people to it. This way much of the feedback comes in one place.
There was a problem hiding this comment.
I haven't started a thread, and don't know if someone else has (maybe @TyDunn?).
Do we have guidelines about this in notion?
There was a problem hiding this comment.
I have not. Is this something you can work on with @EmmaWightman, @aeshky? I'd happy to review the content before it goes live as well as support you through the process as much as you feel you need :)
There was a problem hiding this comment.
Sounds good! I'll contact Emma.
There was a problem hiding this comment.
You're free to contact Emma, but for the Tensorflow 2.6 upgrade, I just took the liberty of starting a thread on the forum myself. I can also help you with that if you'd like (Emma's going to be very busy soon with the advent of all the social announcements).
There was a problem hiding this comment.
I still don't think we should post it yet. Can we take it down until the release is out and we have a chance to tell people about it? The timing will be better if we post it right before we have the live event for it
There was a problem hiding this comment.
@TyDunn are you happy for me to post this at the end of the day, prior to the release event on Monday?
There was a problem hiding this comment.
@aeshky For sure! Now that 3.0 is released it makes sense 🚀
Proposed changes:
Status (please check what you already did):
black(please check Readme for instructions)