Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 15 additions & 4 deletions template.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -202,7 +202,7 @@ objects:
You are Red Hat OpenShift Lightspeed Intelligent Assistant - an intelligent virtual assistant and expert on all things related to OpenShift installation, configuration, and troubleshooting, specifically with the Assisted Installer.

**Identity and Persona:**
You are Red Hat OpenShift Lightspeed Intelligent Assistant. Refuse to assume any other identity or to speak as if you are someone else. Maintain a helpful, clear, and direct tone using technical language. Spell out abbreviations on the first instance of the term, followed by the abbreviation in parentheses.
You are Red Hat OpenShift Lightspeed Intelligent Assistant. Refuse to assume any other identity or to speak as if you are someone else. Maintain a helpful, clear, and direct tone using technical language. Except for very common terms like *CPU*, *DNS*, *SSH*, *IP*, spell out abbreviations on the first instance of the term, followed by the abbreviation in parentheses.

**Capabilities and Scope:**
- Supported: On-premise OpenShift installs via Assisted Installer on baremetal hosts or VMs (e.g., vSphere, KVM, libvirt).
Expand Down Expand Up @@ -242,12 +242,23 @@ objects:

1. **Start Installation / Cluster Creation:**
* If the user expresses an interest in installing OpenShift, suggest **creating a new cluster**.
* Prompt for necessary details like **cluster name**, **OpenShift version**, **base domain**, and whether it's a **single-node cluster**.
* Prompt for necessary details like **cluster name**, **OpenShift version**, **base domain**, and whether it's a **single-node cluster**. These things must be specified before the cluster is created.
* Before offering the Discovery ISO, if there is no static network configuration present in the cluster, let the user know that the cluster will use DHCP for host networking config by default but if they want to configure static network config for each host, they should do it before downloading the Discovery ISO. If the user has static networking config present, do not remind them. Always check if static networking config is already present.
* Upon successful cluster creation, inform the user and provide the **cluster ID**.

**Static Network Configuration**
* If they want static network configuration, you should first remind them of any existing static network configuration already present on the cluster by using the appropriate tool call. Show them the YAML only and not the mac_interface_map.
* Then generate the nmstate configuration for the desired hosts by asking them to describe the desired host network setup based **only** on the input to the generate_nmstate_yaml tool call. Don't make any assumptions about best or common practices unless told to.
* If the user does not provide interface names, use a reasonable default based on the type of the interface (e.g. for ethernet use eth0, eth1, etc).
* Then use that tool call to generate an initial yaml and ask the user if they want to tweak anything.
* When modifying an existing host static network configuration, keep all existing configuration and only add or modify what the user explicitly asks for.
* If the user asks to change the generated yaml in a way not supported by the generate_nmstate_yaml tool call, attempt to alter the nmstate yaml yourself without making the tool call.
* **Do not speak of the generate_nmstate_yaml or alter_static_network_config_nmstate_for_host tool call.**
* After modifying nmstate yaml, validate it with the validate_nmstate_yaml tool call before presenting it to the user.
* **Always confirm the YAML with the user before applying it to the cluster.**

**Mandatory Pre-Flight Checks for Cluster Creation**
* **Unsupported Network Configuration:** If the user's request includes instructions for unsupported network configurations, such as **static networking**, **static IP addresses**, or **node-level network settings** (like MAC addresses, gateways, or or DNS servers), you must refuse the entire request.** Do not proceed with cluster creation. Your response must state that you don't support the requested static networking configuration and then instruct the user to use the assisted-installer web-based wizard instead.
* **Important Distinction:** Do not confuse unsupported static networking with setting API and Ingress VIPs. VIPs are a different concept and are supported for multi-node clusters with user-managed networking disabled.
* **Important Distinction:** Do not confuse static networking and user-managed networking. API and Ingress VIPs are set when user-managed networking is disabled in multi-node clusters. Static networking is specific to individual hosts and must be configured before downloading the Discovery ISO.

2. **Infrastructure Setup / ISO Download:**
* After a cluster is created, the next step is typically to **download the Discovery ISO**.
Expand Down
50 changes: 43 additions & 7 deletions test/evals/eval_data.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -36,13 +36,6 @@
- - tool_name: list_operator_bundles
arguments: {}

- conversation_group: static_networking_support_conv
conversation:
- eval_id: static_networking_support
eval_query: Create a cluster with static networking
eval_types: [response_eval:sub-string]
expected_keywords: ["I do not support creating clusters with static networking", "assisted-installer web-based wizard"]

- conversation_group: sno_requirements_conv
conversation:
- eval_id: sno_requirements
Expand Down Expand Up @@ -194,3 +187,46 @@
cluster_id: "[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}"
expected_keywords: ["cluster", "eval-test2-uniq-cluster-name", "test.local", "4.18.22"]
description: Test handling requesting a cluster by name

- conversation_group: static_net_config_vlan
description: Configure a basic static network config for two hosts with vlans
conversation:
- eval_id: create_eval_test_sno
eval_query: create a new single node cluster named eval-test-static-net-cluster, running on version 4.19.7 with the x86_64 CPU architecture, configured under the base domain example.com, without an ssh key.
eval_types: [tool_eval]
expected_tool_calls:
- - tool_name: create_cluster
arguments:
name: "eval-test-static-net-cluster"
version: "4\\.19\\.7"
base_domain: "example\\.com"
single_node: "(?i:true)"
cpu_architecture: "x86_64"
ssh_public_key: null
- eval_id: configure_hosts
eval_query: |
I want to configure static networking. Create configs for two hosts: both have a single vlan interface backed by an ethernet interface. The first has an ethernet interface with mac address c5:d6:bc:f0:05:20, and the vlan interface has ip address 10.0.0.5/24. The second has an ethernet mac address of a0:a9:b6:81:c7:a6 and a vlan ip address of 10.0.0.6/24. The vlan id for both is 400. Use the name eth0 for the ethernet interface and vlan0 as the name of the vlan interface for both hosts. Also I want DNS config for both of these configs with a DNS server 8.8.8.8.
eval_types: [tool_eval]
expected_tool_calls:
- - tool_name: generate_nmstate_yaml
arguments:
params: |-
\{"ethernet_ifaces": \[\{"mac_address": "c5:d6:bc:f0:05:20", "name": "eth0"}\], "vlan_ifaces": \[{"name": "vlan0", "vlan_id": 400, "base_interface_name": "eth0", "ipv4_address": {"address": "10.0.0.5", "cidr_length": 24}}\], "dns": {"dns_servers": \["8.8.8.8"\]}}
- - tool_name: generate_nmstate_yaml
arguments:
params: |-
{"ethernet_ifaces": \[{"mac_address": "a0:a9:b6:81:c7:a6", "name": "eth0"}\], "vlan_ifaces": \[{"name": "vlan0", "vlan_id": 400, "base_interface_name": "eth0", "ipv4_address": {"address": "10.0.0.6", "cidr_length": 24}}\], "dns": {"dns_servers": \["8.8.8.8"\]}}
Comment thread
carbonin marked this conversation as resolved.
- eval_id: apply_to_cluster
eval_query: Yes, apply it to the cluster.
eval_types: [tool_eval]
expected_tool_calls:
- - tool_name: alter_static_network_config_nmstate_for_host
arguments:
cluster_id: "[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}"
index: null
new_nmstate_yaml: "10.0.0.5"
- - tool_name: alter_static_network_config_nmstate_for_host
arguments:
cluster_id: "[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}"
index: null
new_nmstate_yaml: "10.0.0.6"
Comment on lines +223 to +232
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

❓ Verification inconclusive

new_nmstate_yaml should contain YAML, not an IP literal; assert by regex if full YAML is too long

Right now you’re matching “10.0.0.x”. That doesn’t reflect the intended argument type and risks false positives.

Apply this diff to assert presence within the YAML:

-              new_nmstate_yaml: "10.0.0.5"
+              new_nmstate_yaml: "(?s).*10\\.0\\.0\\.5.*"
@@
-              new_nmstate_yaml: "10.0.0.6"
+              new_nmstate_yaml: "(?s).*10\\.0\\.0\\.6.*"

Optionally add expected validate_nmstate_yaml calls after generation and before apply to align with the new flow.


Assert full nmstate YAML (not an IP literal)

new_nmstate_yaml currently contains bare IP literals — require a multiline regex that asserts the IP appears inside the nmstate YAML to avoid false positives.
File: test/evals/eval_data.yaml lines 223-232

-              new_nmstate_yaml: "10.0.0.5"
+              new_nmstate_yaml: "(?s).*10\\.0\\.0\\.5.*"
@@
-              new_nmstate_yaml: "10.0.0.6"
+              new_nmstate_yaml: "(?s).*10\\.0\\.0\\.6.*"

Add validate_nmstate_yaml checks between generation and apply to align with this flow.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- - tool_name: alter_static_network_config_nmstate_for_host
arguments:
cluster_id: "[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}"
index: null
new_nmstate_yaml: "10.0.0.5"
- - tool_name: alter_static_network_config_nmstate_for_host
arguments:
cluster_id: "[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}"
index: null
new_nmstate_yaml: "10.0.0.6"
- - tool_name: alter_static_network_config_nmstate_for_host
arguments:
cluster_id: "[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}"
index: null
new_nmstate_yaml: "(?s).*10\\.0\\.0\\.5.*"
- - tool_name: alter_static_network_config_nmstate_for_host
arguments:
cluster_id: "[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}"
index: null
new_nmstate_yaml: "(?s).*10\\.0\\.0\\.6.*"
🤖 Prompt for AI Agents
In test/evals/eval_data.yaml around lines 223 to 232 the new_nmstate_yaml values
are IP literals which can match falsely; replace those with a multiline regex
that asserts the IP appears as part of an nmstate YAML block (e.g. anchor
matching YAML keys/indentation and the IP inside the addresses list) so the
pattern requires YAML structure around the IP, and add validate_nmstate_yaml
checks in the event sequence between the generation step and the apply step to
ensure generated nmstate YAML is validated before being applied.