eval: data structure as per multi-turn datamodel

test/evals/README.md

@@ -2,10 +2,10 @@
 Evaluation mechanism to validate Agent task completion (e2e)
 - Refer [LCORE-Eval repo](https://github.com/lightspeed-core/lightspeed-evaluation/tree/main/lsc_agent_eval) for setup.
 - Supports `sub-string`, `judge-llm` and `script` based evaluation.
-- Currently it is single-turn evaluation process.
+- Supports multi-turn evaluation.
 
 ## Prerequisites
-- **Python**: Version 3.11.1 to 3.12.9
+- **Python**: Version 3.11 to 3.12
 - **Assisted Chat API**: Must be running (`make build-images run`)
 - Install lightspeed-core **agent e2e eval**
 ```bash

test/evals/eval_data.yaml

@@ -1,19 +1,27 @@
-- eval_id: basic_introduction
-  eval_query: Hi!
-  eval_type: judge-llm
-  expected_response: "Hello! I'm the Assisted Installer, your guide for OpenShift cluster installation. How can I help you today?"
+- conversation_group: basic_introduction_conv
+  conversation:
+    - eval_id: basic_introduction
+      eval_query: Hi!
+      eval_type: judge-llm
+      expected_response: "Hello! I'm the Assisted Installer, your guide for OpenShift cluster installation. How can I help you today?"
 
-- eval_id: basic_cluster_request
-  eval_query: I want to install an OCP cluster
-  eval_type: judge-llm
-  expected_response: "Great, I can help you with that. To create a cluster, I'll need some information from you. First, what would you like to call your cluster? And what base domain would you like to use? And finally, what OpenShift version would you like to install?"
+- conversation_group: basic_cluster_request_conv
+  conversation:
+    - eval_id: basic_cluster_request
+      eval_query: I want to install an OCP cluster
+      eval_type: judge-llm
+      expected_response: "Great, I can help you with that. To create a cluster, I'll need some information from you. First, what would you like to call your cluster? And what base domain would you like to use? And finally, what OpenShift version would you like to install?"
 
-- eval_id: list_versions
-  eval_query: List the available OpenShift versions
-  eval_type: judge-llm
-  expected_response: "There are several versions of OpenShift available. The most recent production version is 4.19, 4.20 pre release versions are available as well as several previous verrsions."
+- conversation_group: list_versions_conv
+  conversation:
+    - eval_id: list_versions
+      eval_query: List the available OpenShift versions
+      eval_type: judge-llm
+      expected_response: "There are several versions of OpenShift available. The most recent production version is 4.19, 4.20 pre release versions are available as well as several previous versions."
 
-- eval_id: available_operators
-  eval_query: What operators are available?
-  eval_type: judge-llm
-  expected_response: "The operators that can be installed onto clusters are OpenShift AI and OpenShift Virtualization."
+- conversation_group: available_operators_conv
+  conversation:
+    - eval_id: available_operators
+      eval_query: What operators are available?
+      eval_type: judge-llm
+      expected_response: "The operators that can be installed onto clusters are OpenShift AI and OpenShift Virtualization."