MGMT-22783: add pytest e2e test suite for vmaas by omer-vishlitzky · Pull Request #18 · osac-project/osac-test-infra

omer-vishlitzky · 2026-03-22T02:44:14Z

https://redhat.atlassian.net/browse/MGMT-22783

Port all ansible e2e tests to pytest with 1:1 parity. The hub creation test was removed — hub creation is implicitly verified by every compute instance test since they all go through the fulfillment api on the hub.

Tests support a two-cluster topology where the hub cluster runs the osac operator and the remote cluster runs the kubevirt vms. The k8s client takes a kubeconfig parameter, and tests that need to inspect vms on the remote cluster use a separate OSAC_VM_KUBECONFIG env var. See MGMT-23623 for the remote cluster setup script.

Tests:

compute instance lifecycle (create, wait for running, delete)
delete during provision
restart (trigger via grpc, verify new vmi creation timestamp)
restart negative (past timestamp ignored)
api fields (explicit cpu/memory/disk via grpc)
cli fields (explicit cpu/memory/disk via fulfillment-cli)

openshift-ci-robot · 2026-03-22T02:44:18Z

@omer-vishlitzky: This pull request references MGMT-22783 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.22.0" version, but no target version was set.

Details

In response to this:

https://redhat.atlassian.net/browse/MGMT-22783

Port all ansible e2e tests to pytest with 1:1 parity. The hub creation test was removed — hub creation is implicitly verified by every compute instance test since they all go through the fulfillment api on the hub.

Tests support a two-cluster topology where the hub cluster runs the osac operator and the remote cluster runs the kubevirt VMs. The k8s client takes a kubeconfig parameter, and tests that need to inspect VMs on the remote cluster use a separate OSAC_VM_KUBECONFIG env var. See MGMT-23623 for the remote cluster setup script.

Tests

compute instance lifecycle (create, wait for running, delete)

delete during provision

restart (trigger via grpc, verify new vmi creation timestamp)

restart negative (past timestamp ignored)

api fields (explicit cpu/memory/disk via grpc)

cli fields (explicit cpu/memory/disk via fulfillment-cli)

Infrastructure

k8s client with two-kubeconfig support (hub + remote)

grpc client wrapping grpcurl

fulfillment-cli wrapper with typed methods

poll_until helper matching ansible retry semantics

Makefile with MAKEFILE_TARGET dispatch for ci

pyproject.toml with ruff and basedpyright config

Verified

All 6 tests pass on a two-cluster deployment (hub SNO + remote virt cluster) with fulfillment-service v0.0.48 and latest operator.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2026-03-22T02:58:57Z

@omer-vishlitzky: This pull request references MGMT-22783 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.22.0" version, but no target version was set.

Details

In response to this:

https://redhat.atlassian.net/browse/MGMT-22783

Port all ansible e2e tests to pytest with 1:1 parity. The hub creation test was removed — hub creation is implicitly verified by every compute instance test since they all go through the fulfillment api on the hub.

Tests support a two-cluster topology where the hub cluster runs the osac operator and the remote cluster runs the kubevirt vms. The k8s client takes a kubeconfig parameter, and tests that need to inspect vms on the remote cluster use a separate OSAC_VM_KUBECONFIG env var. See MGMT-23623 for the remote cluster setup script.

Tests:

compute instance lifecycle (create, wait for running, delete)

delete during provision

restart (trigger via grpc, verify new vmi creation timestamp)

restart negative (past timestamp ignored)

api fields (explicit cpu/memory/disk via grpc)

cli fields (explicit cpu/memory/disk via fulfillment-cli)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

eranco74

Well-structured pytest port with solid architecture (runner/client abstraction layers, two-kubeconfig design). A few items to address:

Must fix

k8s_virt_client crashes with KeyError when OSAC_VM_KUBECONFIG is not set (conftest.py:51)
os.environ["OSAC_VM_KUBECONFIG"] raises a raw KeyError if the env var is missing, but the README says it defaults to the hub kubeconfig. Fix:

vm_kubeconfig = os.environ.get("OSAC_VM_KUBECONFIG")
return K8sClient(namespace=namespace, kubeconfig=vm_kubeconfig)

Should fix

No cleanup on test failure
If a test fails after creating a compute instance but before the cleanup section, the resource leaks. Over time this causes resource exhaustion in CI. Use request.addfinalizer() or yield fixtures for cleanup.

cli fixture duplicates fulfillment_address derivation (conftest.py)
The cli fixture re-derives the fulfillment address instead of depending on the existing fulfillment_address fixture. Use the fixture to avoid duplication and potential inconsistency.

Worth noting

Session-scoped token with --duration 1h may expire during long test runs (total possible poll time across all tests exceeds 1h)
run_unchecked mixes stdout/stderr into one string — works now but fragile for assertions that match on error messages
UUID parsing from CLI output (re.search(r"'([^']+)'", stdout)) could silently grab wrong value if output format changes; a UUID regex would be safer
Inconsistent resource type references: "computeinstance" vs "computeinstance.osac.openshift.io" — pick one convention
No lock file committed despite uv sync in README — builds not reproducible

omer-vishlitzky · 2026-03-22T21:59:25Z

k8s_virt_client crashes intentionally on a missing env var, this is a test setup, I want to fail fast. changed the readme to reflect that this env var should always be set though
cleanup is redundant, this is going to run in the ci environment, baremetal machines are simply returned to the pool
fixed the fixtures

Self-contained image with all tools needed to run the e2e test suite on a bare metal machine via podman. Depends on osac-project#18 (pyproject.toml and uv.lock must exist).

omer-vishlitzky · 2026-03-23T06:39:39Z

/hold
waiting for @adriengentil review

adriengentil · 2026-03-23T08:55:52Z

what do you think about BDD? I think it would be great to have way to express test in this way, for example like in NetworkManager repo: https://gitlab.freedesktop.org/NetworkManager/NetworkManager-ci/-/merge_requests/1971/diffs

adriengentil · 2026-03-23T08:58:45Z

can't we build a python fulfillment-service client from the proto files?

adriengentil · 2026-03-23T09:00:53Z

+        self.kubeconfig: str | None = kubeconfig
+
+    def _base(self) -> list[str]:
+        args: list[str] = ["kubectl"]


why not use python k8s client?

adriengentil · 2026-03-23T09:06:08Z

+            run_strategy,
+        ]
+        if user_data_secret_ref is not None:
+            args.extend(["--user-data-secret-ref", user_data_secret_ref])


Suggested change

args.extend(["--user-data-secret-ref", user_data_secret_ref])

args.extend(["--user-data", user_data_secret_ref])

it was renamed in osac-project/fulfillment-service#333

eranco74

Previous concerns addressed or reasonably justified:

✅ cli fixture duplication — fixed, now uses fulfillment_address fixture
✅ Resource type consistency — standardized on "computeinstance"
✅ Lock file — uv.lock committed
⚠️ k8s_virt_client KeyError — intentional fail-fast, README updated to mark as required
❌ Cleanup on failure — CI-only, machines returned to pool, acceptable

LGTM. The remaining items (token expiry, stdout/stderr mixing, UUID parsing) are minor and can be improved in follow-ups. adriengentil's architectural suggestions (native Python clients, BDD) are good long-term goals but out of scope here.

Port all ansible e2e tests to pytest with 1:1 parity. The hub creation test was removed — hub creation is implicitly verified by every compute instance test since they all go through the fulfillment api on the hub. Tests support a two-cluster topology where the hub cluster runs the osac operator and the remote cluster runs the kubevirt VMs. The k8s client takes a kubeconfig parameter, and tests that need to inspect VMs on the remote cluster use a separate OSAC_VM_KUBECONFIG env var. See MGMT-23623 for the remote cluster setup script. Tests: - compute instance lifecycle (create, wait for running, delete) - delete during provision - restart (trigger via grpc, verify new vmi creation timestamp) - restart negative (past timestamp ignored) - api fields (explicit cpu/memory/disk via grpc) - cli fields (explicit cpu/memory/disk via fulfillment-cli) Infrastructure: - k8s client with two-kubeconfig support (hub + remote) - grpc client wrapping grpcurl - fulfillment-cli wrapper with typed methods - poll_until helper matching ansible retry semantics - Makefile with MAKEFILE_TARGET dispatch for ci - pyproject.toml with ruff and basedpyright config

openshift-ci · 2026-03-24T09:07:54Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: adriengentil, eranco74, omer-vishlitzky

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [adriengentil]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

omer-vishlitzky · 2026-03-24T09:13:57Z

/unhold

openshift-ci-robot added the jira/valid-reference label Mar 22, 2026

openshift-ci Bot requested review from jhernand and trewest March 22, 2026 02:44

omer-vishlitzky force-pushed the feature/pytest-e2e-mgmt-22783 branch 6 times, most recently from b535310 to fbbdafe Compare March 22, 2026 10:07

eranco74 reviewed Mar 22, 2026

View reviewed changes

omer-vishlitzky force-pushed the feature/pytest-e2e-mgmt-22783 branch from fbbdafe to 9666318 Compare March 22, 2026 21:59

omer-vishlitzky mentioned this pull request Mar 23, 2026

MGMT-22799: add Containerfile for pytest test image #19

Merged

omer-vishlitzky force-pushed the feature/pytest-e2e-mgmt-22783 branch 2 times, most recently from fc33812 to 2f75918 Compare March 23, 2026 06:35

openshift-ci Bot added the do-not-merge/hold Block merge until the label is removed label Mar 23, 2026

adriengentil reviewed Mar 23, 2026

View reviewed changes

Comment thread tests/grpc_client.py

adriengentil Mar 23, 2026

Copy link
Copy Markdown

Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can't we build a python fulfillment-service client from the proto files?

adriengentil reviewed Mar 23, 2026

View reviewed changes

eranco74 approved these changes Mar 23, 2026

View reviewed changes

openshift-ci Bot assigned eranco74 Mar 23, 2026

openshift-ci Bot added the lgtm label Mar 23, 2026

omer-vishlitzky force-pushed the feature/pytest-e2e-mgmt-22783 branch from 2f75918 to 5525d27 Compare March 23, 2026 14:35

openshift-ci Bot removed the lgtm label Mar 23, 2026

omer-vishlitzky force-pushed the feature/pytest-e2e-mgmt-22783 branch from 5525d27 to 19c02cf Compare March 23, 2026 14:50

adriengentil approved these changes Mar 24, 2026

View reviewed changes

openshift-ci Bot assigned adriengentil Mar 24, 2026

openshift-ci Bot added the lgtm label Mar 24, 2026

openshift-ci Bot added the approved label Mar 24, 2026

openshift-ci Bot removed the do-not-merge/hold Block merge until the label is removed label Mar 24, 2026

openshift-merge-bot Bot merged commit 2af0881 into osac-project:main Mar 24, 2026
3 checks passed

	args.extend(["--user-data-secret-ref", user_data_secret_ref])
	args.extend(["--user-data", user_data_secret_ref])

Conversation

omer-vishlitzky commented Mar 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Mar 22, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tests

Infrastructure

Verified

Uh oh!

openshift-ci-robot commented Mar 22, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eranco74 left a comment

Choose a reason for hiding this comment

Must fix

Should fix

Worth noting

Uh oh!

omer-vishlitzky commented Mar 22, 2026

Uh oh!

omer-vishlitzky commented Mar 23, 2026

Uh oh!

adriengentil Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

adriengentil Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

adriengentil Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

adriengentil Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

eranco74 left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci Bot commented Mar 24, 2026

Uh oh!

omer-vishlitzky commented Mar 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

omer-vishlitzky commented Mar 22, 2026 •

edited

Loading

openshift-ci-robot commented Mar 22, 2026 •

edited by openshift-ci Bot

Loading

openshift-ci-robot commented Mar 22, 2026 •

edited by openshift-ci Bot

Loading