Skip to content

Add expert review example #53

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Add expert review example #53

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -458,6 +458,30 @@ Check out our [documentation/tutorials](https://help.cleanlab.ai/codex/) to easi

> *I'm dizzy should i still fly tmr?*

### To explore Expert Review guardrail

> *does frontier accomodate passengers with allergies*
Copy link
Member

@jwmueller jwmueller Oct 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you find a specific flaw in this response? It's hard for me to diagnose a flaw, since they literally have a Knowledge Base document dedicated to just this and the AI seems to be properly responding based on that document.

Copy link
Member

@jwmueller jwmueller Oct 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If not, then I'd say we can use a different example where the Response is more flawed.

Here's one I found, which you can test for stability:

what if i'm thirsty on the flight but have no money

AI Response: Frontier Airlines does not offer complimentary food and beverages on board. All food and drinks, including water, are available for purchase. This policy helps keep fares low by allowing passengers to choose only what they want. If you find yourself without money, you might want to plan ahead by bringing your own snacks or drinks, as long as they comply with TSA regulations.

Copy link
Contributor Author

@gordon-lim gordon-lim Oct 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same with the original "stuck on tarmac" one, there's also a knowledge base document for that but we are supposing that we don't want the AI to answer such questions that reflect badly on the airline.

I do not see why the response in your suggestion is more flawed. You can actually get a cup of water for free on Frontier based on anecdotal evidence I found on Reddit if that is the factual error you are referring to but wouldn't that make this more apt for an Expert Answer example instead?

Copy link
Contributor Author

@gordon-lim gordon-lim Oct 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought my query fell in the "gray" area as with the tarmac one because of how the query was worded like "does frontier care about passengers with allergies" and then the AI basically said no.

Copy link
Member

@jwmueller jwmueller Oct 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in my example, I was trying to make the AI Response seem callous. Do you have ideas to make that more exaggerated? If so, that would be best.

IMO the allergy response from the AI here is significantly better than a fallback answer. We don't want to show cases where the effect of Cleanlab is not obviously positive.

Copy link
Contributor Author

@gordon-lim gordon-lim Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll have to think about this more.


<details>
<summary>Learn More</summary>

```text
AI Response without Cleanlab:
Frontier Airlines cannot guarantee an allergen-free environment on its flights due to the presence of food allergens in the snacks served and the possibility of passengers bringing products containing nuts or other allergens onboard. If you have a severe nut or food allergy, you should notify a flight attendant so they can attempt to inform nearby passengers to refrain from eating allergen-containing products. However, the airline cannot prevent passengers from consuming their own food. It's recommended to consult with your physician regarding any health concerns before flying.
```

Suppose a Product Leader / SME has quickly decided that the AI agent should not answer queries like this. With Cleanlab, it only takes one click to enact this change permanently in your AI agent.

Expand this Log entry in the Project and click `No` under *Is this a good AI response?*, and click `Submit`.

Then pretend you are a different user **by creating a new chat thread** and ask a similar query:

> *does frontier accomodate passengers with allergies*

You'll see that Cleanlab now guardrails the AI, permanently preventing the response that was just deemed undesirable. This allows nontechnical SMEs to reduce false negatives in Guardrails (as well as false positives by clicking `Yes` under *Is this a good AI response?*).
<br><br>
</details>

#### To explore trust scoring

> *If theres no overhead bin space for my carry on must i pay extra to check it?*
Expand Down