Skip to content

MGMT-21690: chatbot installation asks again for host and cluster IDs #195

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
openshift-merge-bot merged 1 commit into from
Sep 15, 2025
Merged

MGMT-21690: chatbot installation asks again for host and cluster IDs #195

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions template.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -247,8 +247,7 @@ objects:
* Upon successful cluster creation, inform the user and provide the **cluster ID**.

**Mandatory Pre-Flight Checks for Cluster Creation**
* **Unsupported Network Configuration:** Before attempting to create a cluster, you **must first check** if the user's request includes instructions for unsupported network configurations, such as **static networking**, **static IP addresses**, or **node-level network settings** (like MAC addresses, gateways, or DNS servers).
* **If an unsupported configuration is detected, you must refuse the entire request.** Do not proceed with cluster creation. Your response must state that you don't support the requested static networking configuration and then instruct the user to use the assisted-installer web-based wizard instead.
* **Unsupported Network Configuration:** If the user's request includes instructions for unsupported network configurations, such as **static networking**, **static IP addresses**, or **node-level network settings** (like MAC addresses, gateways, or or DNS servers), you must refuse the entire request.** Do not proceed with cluster creation. Your response must state that you don't support the requested static networking configuration and then instruct the user to use the assisted-installer web-based wizard instead.
* **Important Distinction:** Do not confuse unsupported static networking with setting API and Ingress VIPs. VIPs are a different concept and are supported for multi-node clusters with user-managed networking disabled.

Comment on lines 249 to 252
Copy link
Copy Markdown

@coderabbitai coderabbitai bot Sep 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Fix grammar/markdown in “Unsupported Network Configuration” bullet.

There’s a duplicated word (“or or”) and an extra closing bold marker that will render incorrectly.

Apply:

-          * **Unsupported Network Configuration:** If the user's request includes instructions for unsupported network configurations, such as **static networking**, **static IP addresses**, or **node-level network settings** (like MAC addresses, gateways, or or DNS servers), you must refuse the entire request.** Do not proceed with cluster creation. Your response must state that you don't support the requested static networking configuration and then instruct the user to use the assisted-installer web-based wizard instead.
+          * **Unsupported Network Configuration:** If the user's request includes instructions for unsupported network configurations, such as **static networking**, **static IP addresses**, or **node-level network settings** (like MAC addresses, gateways, or DNS servers), you must refuse the entire request. Do not proceed with cluster creation. Your response must state that you don't support the requested static networking configuration and then instruct the user to use the assisted-installer web-based wizard instead.
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
**Mandatory Pre-Flight Checks for Cluster Creation**
* **Unsupported Network Configuration:** Before attempting to create a cluster, you **must first check** if the user's request includes instructions for unsupported network configurations, such as **static networking**, **static IP addresses**, or **node-level network settings** (like MAC addresses, gateways, or DNS servers).
* **If an unsupported configuration is detected, you must refuse the entire request.** Do not proceed with cluster creation. Your response must state that you don't support the requested static networking configuration and then instruct the user to use the assisted-installer web-based wizard instead.
* **Unsupported Network Configuration:** If the user's request includes instructions for unsupported network configurations, such as **static networking**, **static IP addresses**, or **node-level network settings** (like MAC addresses, gateways, or or DNS servers), you must refuse the entire request.** Do not proceed with cluster creation. Your response must state that you don't support the requested static networking configuration and then instruct the user to use the assisted-installer web-based wizard instead.
* **Important Distinction:** Do not confuse unsupported static networking with setting API and Ingress VIPs. VIPs are a different concept and are supported for multi-node clusters with user-managed networking disabled.
**Mandatory Pre-Flight Checks for Cluster Creation**
* **Unsupported Network Configuration:** If the user's request includes instructions for unsupported network configurations, such as **static networking**, **static IP addresses**, or **node-level network settings** (like MAC addresses, gateways, or DNS servers), you must refuse the entire request. Do not proceed with cluster creation. Your response must state that you don't support the requested static networking configuration and then instruct the user to use the assisted-installer web-based wizard instead.
* **Important Distinction:** Do not confuse unsupported static networking with setting API and Ingress VIPs. VIPs are a different concept and are supported for multi-node clusters with user-managed networking disabled.
🤖 Prompt for AI Agents 
In template.yaml around lines 249 to 252, the "Unsupported Network
Configuration" bullet contains a duplicated word ("or or") and an extra closing
bold marker that breaks markdown; remove the duplicate "or", fix punctuation and
bold markup so the sentence reads cleanly (e.g., list unsupported items as
"static networking, static IP addresses, or node-level network settings (like
MAC addresses, gateways, or DNS servers)" and remove the stray "**" after
"request"), preserving the instruction to refuse static networking and to direct
users to the assisted-installer web-based wizard.

2. **Infrastructure Setup / ISO Download:**
Expand All @@ -258,7 +257,11 @@ objects:

3. **Host Discovery and Configuration:**
* Once the Discovery ISO is generated, the user needs to boot hosts with it.
* After hosts are discovered and appear in the cluster's hosts list, offer to help **assign roles to the hosts** (e.g., master, worker).
* When a user indicates that hosts have been booted, first check for discovered hosts for that cluster and the cluster status.
* After hosts are discovered and appear in the hosts list, present the full list of discovered hosts to the user.
* Proactively offer the next steps based on the cluster type:
* **For a multi-node cluster:** Inform the user that roles can be automatically assigned or they can manually assign them. Offer to help with **manual role assignment** to a specific host (e.g., master, worker).
* **For a Single Node OpenShift (SNO) cluster:** Inform the user that the host is automatically assigned the `master` role and no further manual role assignment is needed. Propose the next logical step, such as initiating the installation.
* If the user wants to monitor host-specific issues, offer to retrieve **host events**.
* Different cluster types and host roles have different hardware requirements:
* For a multi-node cluster:
Expand Down
13 changes: 10 additions & 3 deletions test/evals/eval_data.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,14 @@
arguments:
cluster_id: "[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}"
expected_keywords: ["Discovery ISO"]
- eval_id: host_booted_but_not_discovered
eval_query: I booted the hosts
eval_types: [tool_eval, response_eval:accuracy]
expected_tool_calls:
- - tool_name: cluster_info
arguments:
cluster_id: "[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}"
expected_response: "hosts hasn't been discovered yet."
Comment on lines +120 to +127
Copy link
Copy Markdown

@coderabbitai coderabbitai bot Sep 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Make the “hosts not discovered” assertion less brittle (and fix grammar).

Exact-phrase accuracy on a short sentence will be fragile; also grammar should be “haven’t.”

-      eval_types: [tool_eval, response_eval:accuracy]
+      eval_types: [tool_eval, response_eval:sub-string]
       expected_tool_calls:
         - - tool_name: cluster_info
             arguments:
               cluster_id: "[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}"
-      expected_response: "hosts hasn't been discovered yet."
+      expected_keywords: ["no hosts discovered", "booted", "waiting"]

If you prefer accuracy, then at least:

-      expected_response: "hosts hasn't been discovered yet."
+      expected_response: "hosts haven't been discovered yet."
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- eval_id: host_booted_but_not_discovered
eval_query: I booted the hosts
eval_types: [tool_eval, response_eval:accuracy]
expected_tool_calls:
- - tool_name: cluster_info
arguments:
cluster_id: "[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}"
expected_response: "hosts hasn't been discovered yet."
- eval_id: host_booted_but_not_discovered
eval_query: I booted the hosts
eval_types: [tool_eval, response_eval:sub-string]
expected_tool_calls:
- - tool_name: cluster_info
arguments:
cluster_id: "[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}"
expected_keywords: ["no hosts discovered", "booted", "waiting"]
🤖 Prompt for AI Agents 
In test/evals/eval_data.yaml around lines 120-127, the expected_response is
brittle and grammatically incorrect ("hosts hasn't been discovered yet.");
change it to a less fragile assertion and correct the grammar by either
replacing the exact string with a substring/regex match (e.g. a case-insensitive
pattern like "haven't been discovered" or "haven't been discovered yet") or
update the literal to the corrected sentence "hosts haven't been discovered
yet." Ensure the test harness uses substring or regex matching rather than
exact-phrase equality so short response variations don't break the eval.


- conversation_group: list_clusters_conv
conversation:
Expand All @@ -131,13 +139,12 @@
conversation:
- eval_id: cluster_info_tool_call
eval_query: Give me details about cluster named 'abc123abc'
description: Test handling for non-existent cluster name
eval_types: [tool_eval, response_eval:accuracy]
expected_response: Retrieval failed for cluster 'abc123abc' because the resource was not found.
expected_tool_calls:
# It should list the clusters to try to match up the name
- - tool_name: list_clusters
arguments: {}
description: Test handling for non-existent cluster ID/Name
expected_response: Retrieval failed for cluster 'abc123abc' because the resource was not found.

- conversation_group: error_handling_conv
description: Validate graceful handling of invalid SSH key format
Expand Down