MGMT-22245: eval-test update for the SSH key fix

template.yaml

@@ -251,10 +251,17 @@ objects:
       1.  **Start Installation / Cluster Creation:**
           * If the user expresses an interest in installing OpenShift, suggest **creating a new cluster**.
           * Prompt for necessary details like **cluster name**, **OpenShift version**, **base domain**, and whether it's a **single-node cluster**. These things must be specified before the cluster is created.
-          * Before offering the Discovery ISO, if there is no static network configuration present in the cluster, let the user know that the cluster will use DHCP for host networking config by default but if they want to configure static network config for each host, they should do it before downloading the Discovery ISO. If the user has static networking config present, do not remind them. Always check if static networking config is already present.
           * Upon successful cluster creation, inform the user and provide the **cluster ID**.
 
-          **Static Network Configuration**
+          **Pre-ISO Configuration (Always Check and Offer)**
+          * After cluster creation and before offering the Discovery ISO, proactively check which of the following optional configuration steps have not yet been addressed with the user, and offer those that remain:
+            1. **SSH Key Configuration**: If the user has not already mentioned, provided, or declined an SSH public key earlier in the conversation, offer to add one. Explain that it enables SSH access to cluster nodes during and after installation for troubleshooting and administration. If already addressed, skip this offer.
+            2. **Static Network Configuration**: Check if static network configuration is already present using the appropriate tool. If not present and the user has not already mentioned or declined static networking earlier in the conversation, offer to configure it. Explain that by default the cluster will use DHCP for host networking, but static network configuration can be set up now before downloading the Discovery ISO if needed. If already addressed, skip this offer.
+          * These are independent optional configurations - offer whichever have not been addressed yet (could be both, one, or neither).
+          * Present any unanswered configuration options in the same response before offering the Discovery ISO.
+          * These are optional - if the user doesn't explicitly respond to these offers, you can proceed with offering the Discovery ISO in your next response.
+
+          **Static Network Configuration Details**
           * If the user wants static network configuration, you should first remind them of any existing static network configuration already present on the cluster by using the appropriate tool call. Show them the YAML only and not the mac_interface_map.
           * Then generate the nmstate configuration for the desired hosts by calling the proper tool. Don't make any assumptions about best or common practices unless told to.
           * If the user does not provide interface names, use a reasonable default based on the type of the interface (e.g. for ethernet use eth0, eth1, etc).

test/evals/eval_data.yaml

@@ -72,11 +72,11 @@
   cleanup_script: ../scripts/delete_cluster.sh
   conversation:
     - eval_id: create_eval_test_multinode
-      eval_types: [response_eval:accuracy, response_eval:sub-string, action_eval]
+      eval_types: [response_eval:intent, response_eval:sub-string, action_eval]
       eval_query: Create a multi-node cluster named 'eval-test-multinode-uniq-cluster-name' with OpenShift 4.18.22 and domain test.local and with the x86_64 CPU architecture.
       eval_verify_script: ../scripts/verify_create_eval_test_multinode.sh
       expected_keywords: ["eval-test-multinode-uniq-cluster-name", "ID", "Discovery ISO", "cluster"]
-      expected_response: I have created a cluster with name eval-test-multinode-uniq-cluster-name. Before downloading the Discovery ISO, would you like to configure static network configuration for the hosts? If not, the cluster will use Dynamic Host Configuration Protocol (DHCP) for host networking configuration by default.
+      expected_intent: A confirmation that the cluster was successfully created with the specified configuration, including offers to configure both SSH key and static network configuration before proceeding to the Discovery ISO
     - eval_id: set_ssh_key_eval_test_ssh
       eval_query: Set the SSH key for the cluster you just created to "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQCmeaBFhSJ/MLECmqUaKweRgo10ABpwdvJ7v76qLYfP0pzfzYsF3hGP/fH5OQfHi9pTbWynjaEcPHVfaTaFWHvyMtv8PEMUIDgQPWlBSYzb+3AgQ5AsChhzTJCYnRdmCdzENlV+azgtb3mVfXiyCfjxhyy3QAV4hRrMaVtJGuUQfQ== [email protected]"
       eval_types: [tool_eval, response_eval:accuracy]
@@ -125,7 +125,7 @@
     - eval_id: invalid_ssh_key_handling
       eval_query: create a new single node cluster named eval-test-sno, running on version 4.19.7, configured under the base domain example.com, and with cpu architecture x86_64 using the provided SSH key abc123def456.
       eval_types: [response_eval:intent]
-      expected_intent: A response indicating that the provided SSH key is not in a valid format and asking the user to provide a valid SSH key
+      expected_intent: A response indicating that the cluster was successfully created, but the provided SSH key is not in a valid format so it could not be set for the cluster
 
 - conversation_group: non_disclosure_conv
   description: Validate refusal to reveal internal details per system prompt