Skip to content

Conversation

@gordon-lim
Copy link
Contributor

Added to additional examples section:

### To explore Expert Review guardrail

> *does frontier accomodate passengers with allergies*

<details>
  <summary>Learn More</summary>

\```text
AI Response without Cleanlab:
Frontier Airlines cannot guarantee an allergen-free environment on its flights due to the presence of food allergens in the snacks served and the possibility of passengers bringing products containing nuts or other allergens onboard. If you have a severe nut or food allergy, you should notify a flight attendant so they can attempt to inform nearby passengers to refrain from eating allergen-containing products. However, the airline cannot prevent passengers from consuming their own food. It's recommended to consult with your physician regarding any health concerns before flying.
\```

Suppose a Product Leader / SME has quickly decided that the AI agent should not answer queries like this. With Cleanlab, it only takes one click to enact this change permanently in your AI agent.

Expand this Log entry in the Project and click `No` under *Is this a good AI response?*, and click `Submit`.

Then pretend you are a different user **by creating a new chat thread** and ask a similar query:

> *does frontier accomodate passengers with allergies*

You'll see that Cleanlab now guardrails the AI, permanently preventing the response that was just deemed undesirable. This allows nontechnical SMEs to reduce false negatives in Guardrails (as well as false positives by clicking `Yes` under *Is this a good AI response?*).
<br><br>
</details>

@gordon-lim gordon-lim requested a review from jwmueller October 30, 2025 20:47
### To explore Expert Review guardrail

> *does frontier accomodate passengers with allergies*
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you find a specific flaw in this response? It's hard for me to diagnose a flaw, since they literally have a Knowledge Base document dedicated to just this and the AI seems to be properly responding based on that document.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If not, then I'd say we can use a different example where the Response is more flawed.

Here's one I found, which you can test for stability:

what if i'm thirsty on the flight but have no money

AI Response: Frontier Airlines does not offer complimentary food and beverages on board. All food and drinks, including water, are available for purchase. This policy helps keep fares low by allowing passengers to choose only what they want. If you find yourself without money, you might want to plan ahead by bringing your own snacks or drinks, as long as they comply with TSA regulations.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same with the original "stuck on tarmac" one, there's also a knowledge base document for that but we are supposing that we don't want the AI to answer such questions that reflect badly on the airline.

I do not see why the response in your suggestion is more flawed. You can actually get a cup of water for free on Frontier based on anecdotal evidence I found on Reddit if that is the factual error you are referring to but wouldn't that make this more apt for an Expert Answer example instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought my query fell in the "gray" area as with the tarmac one because of how the query was worded like "does frontier care about passengers with allergies" and then the AI basically said no.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in my example, I was trying to make the AI Response seem callous. Do you have ideas to make that more exaggerated? If so, that would be best.

IMO the allergy response from the AI here is significantly better than a fallback answer. We don't want to show cases where the effect of Cleanlab is not obviously positive.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll have to think about this more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants