MGMT-22783: add pytest e2e test suite for vmaas

.gitignore

@@ -6,3 +6,16 @@ fulfillment-cli
 
 # Temporary files
 *.retry
+
+# Python
+__pycache__/
+*.pyc
+*.pyo
+.pytest_cache/
+.venv/
+*.egg-info/
+dist/
+build/
+
+# Test reports
+reports/

Makefile

@@ -0,0 +1,13 @@
+REPORTS_DIR ?= reports
+
+.PHONY: test-vmaas lint format
+
+test-vmaas:
+	pytest tests/vmaas/ -v $(if $(TEST),-k "$(TEST)") --junitxml=$(REPORTS_DIR)/vmaas.xml
+
+lint:
+	ruff check tests/
+	ruff format --check tests/
+
+format:
+	ruff format tests/

README.md

@@ -1,97 +1,96 @@
 # OSAC Test Infrastructure
 
-Ansible-based test infrastructure for OSAC end-to-end testing.
+End-to-end test suite for OSAC. Tests the full stack: fulfillment CLI/API, operator, AAP provisioning, and KubeVirt VM lifecycle.
 
-## Overview
+## Test Framework
 
-This repository contains automated testing capabilities for OSAC hub creation and management through the OSAC fulfillment service.
+Tests are written in pytest. The existing Ansible playbooks remain in `playbooks/` and `roles/` for reference during the migration period.
 
-## Architecture
+## Directory Structure
 
-### Directory Structure
 ```
-playbooks/          # Main Ansible playbooks for executing tests
-roles/              # Reusable Ansible roles for test functionality
-├── fulfillment_cli_base/     # Base role for fulfillment CLI operations
-├── test_hub_creation/        # Hub creation/deletion testing
+tests/
+├── conftest.py          # Session fixtures: cli, grpc, k8s, k8s_vm
+├── runner.py            # Execution primitives: run, run_unchecked, poll_until, env
+├── k8s_client.py        # K8sClient — kubectl wrapper (hub + VM cluster)
+├── grpc_client.py       # GRPCClient — grpcurl wrapper
+├── fulfillment_cli.py   # FulfillmentCLI — fulfillment-cli wrapper
+└── vmaas/               # VMaaS test suite
+    ├── test_compute_instance_creation.py          # Full VM lifecycle
+    ├── test_compute_instance_delete_during_provision.py  # Delete while provisioning
+    ├── test_compute_instance_restart.py           # Restart via gRPC
+    ├── test_compute_instance_restart_negative.py  # Past timestamp ignored
+    ├── test_compute_instance_api_fields.py        # Mutability/immutability
+    └── test_compute_instance_cli_fields.py        # CLI fields + K8s verification
+playbooks/    # Legacy Ansible tests
+roles/        # Legacy Ansible roles
 ```
 
-### Core Components
-
-**Hub Creation Testing**
-- Hub creation and registration using fulfillment-cli
-- Hub verification through gRPC API calls
-- Automated cleanup of hub resources and temporary files
-
-**Communication Methods**
-- `fulfillment-cli` for hub creation operations
-- `grpcurl` for direct API communication with fulfillment service
-- gRPC authentication using Bearer tokens
-
 ## Quick Start
 
 ### Prerequisites
-- Ansible installed
-- Access to OSAC fulfillment service
-- `fulfillment-cli` binary
-- `grpcurl` for gRPC API calls
 
-### Running Tests
+- Python 3.11+
+- `fulfillment-cli` binary (matching the deployed fulfillment-service version)
+- `grpcurl` (Go binary: `go install github.com/fullstorydev/grpcurl/cmd/grpcurl@latest`)
+- `oc` / `kubectl` with cluster-admin access
+- A running OSAC deployment
+
+### Install
 
-Execute hub creation test:
 ```bash
-ansible-playbook playbooks/test_hub_creation.yml
+uv sync
 ```
 
-## Configuration
+### Run Tests
+
+```bash
+# Run VMaaS tests against your cluster
+OSAC_NAMESPACE=osac-devel make test-vmaas
+
+# Run a single test by name
+TEST=test_compute_instance_lifecycle make test-vmaas
+```
 
-### Variables (Found in group_vars/all.yml)
+### Makefile Targets
 
-- KUBECONFIG: Path to your KUBECONFIG
-- fulfillment_cli_path: Where to find fulfillment-cli
-- test_namespace: What namespace to use
-- testing_workspace: What directory to put your test results in
-- cluster_domain_suffix: The service dns entry for your cluster. (ie apps.hcp.local.lab)
-- fulfillment_app_name: Kubernetes app name for your fulfillment api service.
-- fulfillment_port: Listening port for your fulfillment-api
+```
+make test-vmaas
+make lint
+make format
+```
 
-- osac_installer_fulfillment_address: "{{ fulfillment_app_name }}-{{ test_namespace }}.{{ cluster_domain_suffix }}:{{ fulfillment_port }}"
-- fulfillment_token_script: "oc create token fulfillment-admin -n {{ test_namespace }} --duration 1h --as system:admin"
+## Configuration
 
-### Hub creation specific overrides
-- hub_service_account: "fulfillment-admin"
-- hub_token_namespace: "{{ test_namespace }}"
-- hub_token_duration: "1h"
+All configuration via environment variables. Same vars work in local dev and CI.
 
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `OSAC_NAMESPACE` | `osac-devel` | Namespace where OSAC is deployed |
+| `KUBECONFIG` | `~/.kube/config` | Kubeconfig for the hub (management cluster) |
+| `OSAC_VM_KUBECONFIG` | **(required)** | Kubeconfig for the VM cluster (where VirtualMachines run). In single-cluster setups, set this to the same value as `KUBECONFIG`. |
+| `OSAC_FULFILLMENT_ADDRESS` | auto-derived | Fulfillment API address (`host:port`) |
+| `OSAC_VM_TEMPLATE` | `osac.templates.ocp_virt_vm` | ComputeInstance template to use |
+| `OSAC_SERVICE_ACCOUNT` | `admin` | ServiceAccount for token generation |
+| `FULFILLMENT_CLI_PATH` | `fulfillment-cli` | Path to the CLI binary |
+| `TEST` | (none) | pytest `-k` filter — run only tests matching this name substring |
 
-### Fulfillment Address Configuration
+### Two-Kubeconfig Design
 
-The fulfillment address follows standard OpenShift application naming:
-```
-<app-name>-<namespace>.<cluster-domain>:<port>
-```
+Tests access two clusters:
+- **Hub** (`KUBECONFIG`) — where ComputeInstance CRs, jobs, and the fulfillment service live
+- **VM cluster** (`OSAC_VM_KUBECONFIG`) — where VirtualMachine and VirtualMachineInstance resources live
 
-Configuration variables:
-- `fulfillment_app_name` - Application name (default: "fulfillment-api")
-- `test_namespace` - Target namespace
-- `cluster_domain_suffix` - User-configurable cluster domain (e.g., "apps.hcp.local.lab")
-- `fulfillment_port` - Service port (default: "443")
+In single-cluster dev setups (VMs run on the hub): set `OSAC_VM_KUBECONFIG` to the same value as `KUBECONFIG`.
 
-The complete address is automatically constructed as:
-```
-fulfillment-api-foobar.apps.hcp.local.lab:443
-```
+In two-cluster setups: set `OSAC_VM_KUBECONFIG` to the virt cluster kubeconfig. The hub kubeconfig manages CRs, the VM kubeconfig verifies VM state.
 
-## Test Execution Flow
+## Legacy Ansible Tests
 
-1. **Setup**: Display test information and parameters
-2. **Creation**: Create hub using fulfillment-cli
-3. **Verification**: Verify hub registration via gRPC API
-4. **Cleanup**: Delete hub and remove temporary files
-5. **Logging**: Record test results to test-execution.log
+The Ansible playbooks in `playbooks/` and `roles/` are the original test implementations. They will be removed once pytest reaches full parity and is verified in CI.
 
-## Error Handling
+To run Ansible tests:
 
-- Hub deletion verification ensures proper cleanup
-- All operations include proper error logging and failure messages
-- Retry functionality available through Ansible retry files
+```bash
+ansible-playbook playbooks/test_compute_instance_creation.yml -e test_namespace=osac-devel
+```

pyproject.toml

@@ -0,0 +1,37 @@
+[project]
+name = "osac-test-infra"
+version = "0.1.0"
+description = "OSAC end-to-end test infrastructure"
+requires-python = ">=3.11"
+dependencies = [
+    "pytest>=8.0",
+    "pyyaml>=6.0",
+]
+
+[tool.setuptools.packages.find]
+include = ["tests*"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+pythonpath = ["."]
+
+[tool.basedpyright]
+reportUnusedCallResult = false
+reportAny = false
+reportExplicitAny = false
+
+[tool.ruff]
+target-version = "py311"
+line-length = 120
+
+[tool.ruff.format]
+skip-magic-trailing-comma = true
+
+[tool.ruff.lint.isort]
+split-on-trailing-comma = false
+
+[tool.ruff.lint]
+select = ["E", "F", "W", "I", "UP", "ANN", "B", "SIM", "RUF"]
+
+[tool.ruff.lint.per-file-ignores]
+"tests/*" = ["ANN201"]

tests/conftest.py

@@ -0,0 +1,64 @@
+from __future__ import annotations
+
+import os
+
+import pytest
+
+from tests.fulfillment_cli import FulfillmentCLI
+from tests.grpc_client import GRPCClient
+from tests.k8s_client import K8sClient
+from tests.runner import env, run
+
+
+@pytest.fixture(scope="session")
+def namespace() -> str:
+    return env("OSAC_NAMESPACE", "osac-devel")
+
+
+@pytest.fixture(scope="session")
+def cluster_domain() -> str:
+    return run("kubectl", "get", "ingress.config.openshift.io", "cluster", "-o", "jsonpath={.spec.domain}")
+
+
+@pytest.fixture(scope="session")
+def fulfillment_address(namespace: str, cluster_domain: str) -> str:
+    return env("OSAC_FULFILLMENT_ADDRESS", f"fulfillment-api-{namespace}.{cluster_domain}:443")
+
+
+@pytest.fixture(scope="session")
+def service_account() -> str:
+    return env("OSAC_SERVICE_ACCOUNT", "admin")
+
+
+@pytest.fixture(scope="session")
+def grpc(fulfillment_address: str, namespace: str, service_account: str) -> GRPCClient:
+    token: str = run(
+        "oc", "create", "token", service_account, "-n", namespace, "--duration", "1h", "--as", "system:admin"
+    )
+    return GRPCClient(address=fulfillment_address, token=token)
+
+
+@pytest.fixture(scope="session")
+def k8s_hub_client(namespace: str) -> K8sClient:
+    return K8sClient(namespace=namespace)
+
+
+@pytest.fixture(scope="session")
+def k8s_virt_client(namespace: str) -> K8sClient:
+    vm_kubeconfig: str = os.environ["OSAC_VM_KUBECONFIG"]
+    return K8sClient(namespace=namespace, kubeconfig=vm_kubeconfig)
+
+
+@pytest.fixture(scope="session")
+def cli(namespace: str, fulfillment_address: str, service_account: str) -> FulfillmentCLI:
+    return FulfillmentCLI(
+        binary=env("FULFILLMENT_CLI_PATH", "fulfillment-cli"),
+        address=f"https://{fulfillment_address.rsplit(':', 1)[0]}",
+        token_script=f"oc create token -n {namespace} {service_account} --as system:admin",
+        namespace=namespace,
+    )
+
+
+@pytest.fixture(scope="session")
+def vm_template() -> str:
+    return env("OSAC_VM_TEMPLATE", "osac.templates.ocp_virt_vm")

tests/fulfillment_cli.py

@@ -0,0 +1,57 @@
+from __future__ import annotations
+
+import re
+
+from tests.runner import run
+
+
+class FulfillmentCLI:
+    def __init__(self, *, binary: str, address: str, token_script: str, namespace: str) -> None:
+        self.binary: str = binary
+        self.namespace: str = namespace
+        run(binary, "login", "--address", address, "--insecure", "--token-script", token_script)
+
+    def create_hub(self, *, hub_id: str, kubeconfig: str) -> None:
+        run(self.binary, "create", "hub", "--id", hub_id, "--kubeconfig", kubeconfig, "--namespace", self.namespace)
+
+    def create_compute_instance(
+        self,
+        *,
+        template: str,
+        cores: int = 2,
+        memory_gib: int = 4,
+        boot_disk_size: int = 20,
+        image: str = "quay.io/containerdisks/fedora:latest",
+        image_source_type: str = "registry",
+        run_strategy: str = "Always",
+        user_data_secret_ref: str | None = None,
+    ) -> str:
+        args: list[str] = [
+            self.binary,
+            "create",
+            "computeinstance",
+            "--template",
+            template,
+            "--cores",
+            str(cores),
+            "--memory-gib",
+            str(memory_gib),
+            "--boot-disk-size",
+            str(boot_disk_size),
+            "--image",
+            image,
+            "--image-source-type",
+            image_source_type,
+            "--run-strategy",
+            run_strategy,
+        ]
+        if user_data_secret_ref is not None:
+            args.extend(["--user-data", user_data_secret_ref])
+
+        stdout: str = run(*args)
+        match: re.Match[str] | None = re.search(r"'([^']+)'", stdout)
+        assert match is not None, f"Failed to parse UUID from CLI output: {stdout}"
+        return match.group(1)
+
+    def delete_compute_instance(self, *, uuid: str) -> None:
+        run(self.binary, "delete", "computeinstance", uuid)

tests/grpc_client.py

@@ -0,0 +1,38 @@
+from __future__ import annotations
+
+import json
+from typing import Any
+
+from tests.runner import run
+
+PUBLIC_API: str = "osac.public.v1"
+PRIVATE_API: str = "osac.private.v1"
+
+
+class GRPCClient:
+    def __init__(self, *, address: str, token: str) -> None:
+        self.address: str = address
+        self.token: str = token
+
+    def call(self, *, service: str, data: dict[str, Any] | None = None) -> dict[str, Any]:
+        args: list[str] = ["grpcurl", "-insecure", "-H", f"Authorization: Bearer {self.token}"]
+        if data is not None:
+            args.extend(["-d", json.dumps(data)])
+        args.extend([self.address, service])
+        return json.loads(run(*args))
+
+    def list_compute_instance_ids(self) -> list[str]:
+        response: dict[str, Any] = self.call(service=f"{PUBLIC_API}.ComputeInstances/List")
+        return [item["id"] for item in response.get("items", [])]
+
+    def get_hub(self, *, hub_id: str) -> dict[str, Any]:
+        return self.call(service=f"{PRIVATE_API}.Hubs/Get", data={"id": hub_id})
+
+    def update_restart(self, *, uuid: str, template: str, timestamp: str) -> dict[str, Any]:
+        return self.call(
+            service=f"{PUBLIC_API}.ComputeInstances/Update",
+            data={
+                "object": {"id": uuid, "spec": {"template": template, "restart_requested_at": timestamp}},
+                "updateMask": {"paths": ["spec.restart_requested_at"]},
+            },
+        )