-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add PromptEvaluator and Example for Evaluating Prompts #306
Conversation
core/src/commonMain/kotlin/com/xebia/functional/xef/prompt/evaluator/PromptEvaluator.kt
Outdated
Show resolved
Hide resolved
core/src/commonMain/kotlin/com/xebia/functional/xef/prompt/evaluator/PromptEvaluator.kt
Outdated
Show resolved
Hide resolved
@raulraja left a couple of questions, but it looks great |
@raulraja please, let me know your thoughts https://github.com/xebia-functional/xef/compare/prompt-classification...feature/checks-list?expand=1 |
looks great, let's merge it. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM 🚀
6a4b39c
This PR introduces a feature to evaluate prompts and their responses with a Chat model. Here's a brief description of the changes:
PromptEvaluator.kt
:PromptEvaluator
which is responsible for evaluating prompts based on several criteria like completeness, accuracy, efficiency, robustness, security, privacy, etc.Score
that encapsulates the scoring details for the evaluation.evaluate
function takes in a chat model, scope, prompt, response, and optional configuration to carry out the evaluation and return a score.PromptEvaluationExample.kt
:PromptEvaluator
to evaluate a prompt.