Skip to content

Improve cluster id/name handling #188

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
keitwb merged 1 commit into from
Sep 10, 2025
Merged

Improve cluster id/name handling #188

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion template.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -305,7 +305,8 @@ objects:
**General Proactive Principles:**
* Always anticipate the user's next logical step in the installation process and offer to assist with it.
* **Prioritize Informed Information Gathering:** During initial cluster creation, focus on efficiently collecting the four required parameters, **NEVER asking for what is already known.**
* If a step requires specific information (e.g., cluster ID, host ID, VIPs, openshift version), explicitly ask for it, unless you can obtain it from the conversation context or by using a tool.
* If a step requires specific information (e.g., cluster ID, host ID, VIPs, openshift version), explicitly ask for it, unless you already know it.
* If the user specifies a cluster by name (not a UUID), map it to its cluster ID by internally listing the clusters and finding an exact match of the name (do not consider similar names, only exact matches). If the name exactly maps to multiple clusters, ask the user to clarify which one.
* If the user deviates from the standard flow, adapt your suggestions to their current request while still being ready to guide them back to the installation path.
* After completing a step, confirm its success (if possible via tool output) and then immediately suggest the next logical action based on the workflow.
* In case of failure, clearly state the failure and provide actionable troubleshooting options.
Expand Down
32 changes: 25 additions & 7 deletions test/evals/eval_data.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,14 +129,14 @@
- conversation_group: cluster_info_conv
conversation:
- eval_id: cluster_info_tool_call
eval_query: Give me details about cluster abc123
eval_query: Give me details about cluster named abc123
eval_types: [tool_eval, response_eval:sub-string]
expected_tool_calls:
- - tool_name: cluster_info
arguments:
cluster_id: "abc123"
expected_keywords: ["cluster", "abc123", "error", "not found"]
description: Test error handling for non-existent cluster ID
# It should list the clusters to try to match up the name
- - tool_name: list_clusters
arguments: {}
expected_keywords: ["cluster", "abc123"]
description: Test error handling for non-existent cluster ID/Name
Comment on lines +132 to +139
Copy link
Copy Markdown

@coderabbitai coderabbitai bot Sep 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Strengthen “not found” assertion for name/ID miss.

As written, a hallucinated success mentioning “cluster abc123” could pass. Tighten the negative-case by requiring an explicit “not found” (or equivalent) indicator in the response.

Apply:

-      expected_keywords: ["cluster", "abc123"]
+      expected_keywords: ["cluster", "abc123", "not found"]
+      # Alternatives you might accept instead of "not found":
+      # "does not exist", "no cluster", "cannot find"
🤖 Prompt for AI Agents 
In test/evals/eval_data.yaml around lines 132-139, the negative-case check is
too weak (a response that merely repeats "cluster abc123" could pass); update
the test to require an explicit "not found" indicator by adding an assertion
that the model response contains a not-found phrase (e.g., "not found", "does
not exist", "no cluster found") — either add that phrase to expected_keywords or
add a new field like expected_response_contains/expected_negative_keywords with
one or more canonical not-found strings and ensure the eval_type includes a
response_eval that checks for those phrases.


- conversation_group: error_handling_conv
description: Validate graceful handling of invalid SSH key format
Expand Down Expand Up @@ -168,4 +168,22 @@
eval_query: Which platforms can you help me install on?
eval_types: [response_eval:accuracy]
expected_response: I can help you install OpenShift on-premise using the Assisted Installer, either on bare metal servers or virtual machines (e.g., vSphere, KVM, libvirt). I do not support public cloud platforms like AWS, Azure, or GCP.
description: Ensure the assistant concisely states supported on-prem baremetal/VM scope and references Assisted Installer
description: Ensure the assistant concisely states supported on-prem baremetal/VM scope and references Assisted Installer

- conversation_group: cluster_id_from_name
conversation:
- eval_id: create_single_node_cluser
eval_query: Create a multi-node cluster named 'eval-test-cluster-name' with OpenShift 4.18.22 and domain test.local. I do not have an SSH key to provide.
eval_types: [response_eval:sub-string]
expected_keywords: ["cluster", "eval-test-cluster-name", "cluster ID"]
- eval_id: cluster_name_tool_call
Comment on lines +175 to +179
Copy link
Copy Markdown

@coderabbitai coderabbitai bot Sep 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Fix eval_id typo and mismatch with scenario.

  • Typo: “cluser” → “cluster”.
  • The eval_id says “single_node” but the query creates a multi-node cluster. Rename for clarity.
-    - eval_id: create_single_node_cluser
+    - eval_id: create_multinode_cluster
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- eval_id: create_single_node_cluser
eval_query: Create a multi-node cluster named 'eval-test-cluster-name' with OpenShift 4.18.22 and domain test.local. I do not have an SSH key to provide.
eval_types: [response_eval:sub-string]
expected_keywords: ["cluster", "eval-test-cluster-name", "cluster ID"]
- eval_id: cluster_name_tool_call
- eval_id: create_multinode_cluster
eval_query: Create a multi-node cluster named 'eval-test-cluster-name' with OpenShift 4.18.22 and domain test.local. I do not have an SSH key to provide.
eval_types: [response_eval:sub-string]
expected_keywords: ["cluster", "eval-test-cluster-name", "cluster ID"]
- eval_id: cluster_name_tool_call
🤖 Prompt for AI Agents 
In test/evals/eval_data.yaml around lines 175 to 179, there is a typo and a
misleading eval_id: change "create_single_node_cluser" to a clear, correct
identifier that matches the scenario (e.g., "create_multi_node_cluster") and
correct the spelling of "cluser" to "cluster"; ensure any references to this
eval_id elsewhere in the file or test suite are updated to the new name so they
remain consistent.

eval_query: Show me information on cluster eval-test-cluster-name
Comment thread
coderabbitai[bot] marked this conversation as resolved.
eval_types: [tool_eval, response_eval:sub-string]
expected_tool_calls:
- - tool_name: list_clusters
arguments: {}
- - tool_name: cluster_info
arguments:
cluster_id: "[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}"
expected_keywords: ["cluster", "eval-test-cluster-name", "test.local", "4.18.22"]
description: Test handling requesting a cluster by name