-
Notifications
You must be signed in to change notification settings - Fork 1
Add expert review example #53
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
| ### To explore Expert Review guardrail | ||
|
|
||
| > *does frontier accomodate passengers with allergies* |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did you find a specific flaw in this response? It's hard for me to diagnose a flaw, since they literally have a Knowledge Base document dedicated to just this and the AI seems to be properly responding based on that document.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If not, then I'd say we can use a different example where the Response is more flawed.
Here's one I found, which you can test for stability:
what if i'm thirsty on the flight but have no money
AI Response: Frontier Airlines does not offer complimentary food and beverages on board. All food and drinks, including water, are available for purchase. This policy helps keep fares low by allowing passengers to choose only what they want. If you find yourself without money, you might want to plan ahead by bringing your own snacks or drinks, as long as they comply with TSA regulations.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same with the original "stuck on tarmac" one, there's also a knowledge base document for that but we are supposing that we don't want the AI to answer such questions that reflect badly on the airline.
I do not see why the response in your suggestion is more flawed. You can actually get a cup of water for free on Frontier based on anecdotal evidence I found on Reddit if that is the factual error you are referring to but wouldn't that make this more apt for an Expert Answer example instead?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought my query fell in the "gray" area as with the tarmac one because of how the query was worded like "does frontier care about passengers with allergies" and then the AI basically said no.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in my example, I was trying to make the AI Response seem callous. Do you have ideas to make that more exaggerated? If so, that would be best.
IMO the allergy response from the AI here is significantly better than a fallback answer. We don't want to show cases where the effect of Cleanlab is not obviously positive.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll have to think about this more.
Added to additional examples section: