eval: data structure as per multi-turn datamodel #72
Conversation
WalkthroughThe evaluation data YAML file was restructured from a flat list of evaluations to a nested format, grouping each evaluation under a "conversation_group" with a "conversation" list. Additionally, a typo in the expected response for the "list_versions" evaluation was corrected. The README documentation was updated to reflect support for multi-turn evaluation and broadened the Python version requirement. Changes
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~5 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Note ⚡️ Unit Test Generation is now available in beta!Learn more here, or try it out under "Finishing Touches" below. 📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (2)
✅ Files skipped from review due to trivial changes (1)
🚧 Files skipped from review as they are similar to previous changes (1)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
✨ Finishing Touches🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
|
/hold |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
test/evals/eval_data.yaml (1)
18-21: Tighten wording in the “list_versions” expected response
Minor phrasing + hyphenation fix.- expected_response: "There are several versions of OpenShift available. The most recent production version is 4.19, 4.20 pre release versions are available as well as several previous versions." + expected_response: "There are several versions of OpenShift available. The most recent production version is 4.19. Pre-release versions of 4.20 are also available, as well as several previous versions."
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
test/evals/eval_data.yaml(1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Red Hat Konflux / assisted-chat-saas-main-on-pull-request
🔇 Additional comments (1)
test/evals/eval_data.yaml (1)
1-28: Double-check that the evaluation loader expects a list of conversation-group mappingsThe previous format was a flat list; now each item is a mapping with the keys
conversation_groupandconversation.
Before merging, verify that whatever code consumes this YAML (e.g., the evaluation runner) iterates over a list rather than expecting a single top-level mapping keyed by group name – otherwise it will silently load nothing.If the loader still expects a mapping, wrap the whole list under a new top-level key or switch to:
basic_introduction_conv: conversation: - … basic_cluster_request_conv: conversation: - …
|
/unhold |
|
Does this need to change the version in https://github.com/rh-ecosystem-edge/assisted-chat/tree/main/test/evals#prerequisites? |
eval: data structure as per multi-turn datamodel
asamal4
force-pushed
the
eval-multiturn-datamodel
branch
from
99a8968 to
880a8a9
Compare
|
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: asamal4, carbonin The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-merge-bot
bot
merged commit 7c342f8
into
rh-ecosystem-edge:main
2 participants
modify eval data structure as per latest multi-turn data model
Summary by CodeRabbit