diff --git a/template.yaml b/template.yaml index 2d2d73c..02e8478 100644 --- a/template.yaml +++ b/template.yaml @@ -305,7 +305,8 @@ objects: **General Proactive Principles:** * Always anticipate the user's next logical step in the installation process and offer to assist with it. * **Prioritize Informed Information Gathering:** During initial cluster creation, focus on efficiently collecting the four required parameters, **NEVER asking for what is already known.** - * If a step requires specific information (e.g., cluster ID, host ID, VIPs, openshift version), explicitly ask for it, unless you can obtain it from the conversation context or by using a tool. + * If a step requires specific information (e.g., cluster ID, host ID, VIPs, openshift version), explicitly ask for it, unless you already know it. + * If the user specifies a cluster by name (not a UUID), map it to its cluster ID by internally listing the clusters and finding an exact match of the name (do not consider similar names, only exact matches). If the name exactly maps to multiple clusters, ask the user to clarify which one. * If the user deviates from the standard flow, adapt your suggestions to their current request while still being ready to guide them back to the installation path. * After completing a step, confirm its success (if possible via tool output) and then immediately suggest the next logical action based on the workflow. * In case of failure, clearly state the failure and provide actionable troubleshooting options. diff --git a/test/evals/eval_data.yaml b/test/evals/eval_data.yaml index 5533aa7..126f9c8 100644 --- a/test/evals/eval_data.yaml +++ b/test/evals/eval_data.yaml @@ -129,14 +129,14 @@ - conversation_group: cluster_info_conv conversation: - eval_id: cluster_info_tool_call - eval_query: Give me details about cluster abc123 + eval_query: Give me details about cluster named abc123 eval_types: [tool_eval, response_eval:sub-string] expected_tool_calls: - - - tool_name: cluster_info - arguments: - cluster_id: "abc123" - expected_keywords: ["cluster", "abc123", "error", "not found"] - description: Test error handling for non-existent cluster ID + # It should list the clusters to try to match up the name + - - tool_name: list_clusters + arguments: {} + expected_keywords: ["cluster", "abc123"] + description: Test error handling for non-existent cluster ID/Name - conversation_group: error_handling_conv description: Validate graceful handling of invalid SSH key format @@ -168,4 +168,22 @@ eval_query: Which platforms can you help me install on? eval_types: [response_eval:accuracy] expected_response: I can help you install OpenShift on-premise using the Assisted Installer, either on bare metal servers or virtual machines (e.g., vSphere, KVM, libvirt). I do not support public cloud platforms like AWS, Azure, or GCP. - description: Ensure the assistant concisely states supported on-prem baremetal/VM scope and references Assisted Installer \ No newline at end of file + description: Ensure the assistant concisely states supported on-prem baremetal/VM scope and references Assisted Installer + +- conversation_group: cluster_id_from_name + conversation: + - eval_id: create_single_node_cluser + eval_query: Create a multi-node cluster named 'eval-test-cluster-name' with OpenShift 4.18.22 and domain test.local. I do not have an SSH key to provide. + eval_types: [response_eval:sub-string] + expected_keywords: ["cluster", "eval-test-cluster-name", "cluster ID"] + - eval_id: cluster_name_tool_call + eval_query: Show me information on cluster eval-test-cluster-name + eval_types: [tool_eval, response_eval:sub-string] + expected_tool_calls: + - - tool_name: list_clusters + arguments: {} + - - tool_name: cluster_info + arguments: + cluster_id: "[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}" + expected_keywords: ["cluster", "eval-test-cluster-name", "test.local", "4.18.22"] + description: Test handling requesting a cluster by name