TESTING HELM INSTALLATION IGNORE by omer-vishlitzky · Pull Request #348 · osac-project/osac-installer

omer-vishlitzky · 2026-06-25T18:01:45Z

This is a test PR, please ignore.

@sha256

…pply The kustomize overlay changes the fulfillment-database StatefulSet image reference from a :latest tag to a @sha256: digest. This triggers a StatefulSet pod recreation, killing the database mid-connection. If the grpc-server is running database migrations at that moment, golang-migrate leaves the schema_migrations table in a dirty state and all subsequent grpc-server starts refuse to run — causing a boot failure. Fix: use `kustomize edit set replicas` to set fulfillment-controller and fulfillment-grpc-server to 0 before applying the overlay. The apply itself deploys with zero replicas, eliminating the race entirely. At step [5/9], the script now: 1. Waits for TLS certificates 2. Waits for the database StatefulSet rollout to complete 3. Scales grpc-server and controller back to 1 4. Waits for all fulfillment deployment rollouts This ensures the database is healthy before any migration-running pod starts, and that grpc-server is available before rest-gateway's readiness probe checks the gRPC upstream. Depends on: openshift/release#80431 (adds kustomize binary to the installer container image). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

openshift-ci · 2026-06-25T18:01:52Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: omer-vishlitzky

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [omer-vishlitzky]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

coderabbitai · 2026-06-25T18:01:55Z

Warning

Review limit reached

@omer-vishlitzky, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 59 minutes and 57 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository: osac-project/coderabbit/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: fe4591b3-7b8d-496f-ab71-06c36ddf43b1

📥 Commits

Reviewing files that changed from the base of the PR and between e6b3066 and 5542bcb.

📒 Files selected for processing (2)

empty-test-file-3.txt
scripts/refresh-after-snapshot.sh

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

omer-vishlitzky · 2026-06-25T18:02:01Z

/test vmaas-e2e-full-setup-helm

omer-vishlitzky · 2026-06-25T18:13:52Z

/test e2e-vmaas-full-setup-helm

openshift-ci · 2026-06-25T21:00:17Z

@omer-vishlitzky: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-vmaas-full-setup-helm	`5542bcb`	link	false	`/test e2e-vmaas-full-setup-helm`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

omer-vishlitzky · 2026-06-25T21:12:02Z

💀 CI Triage: broken_main | Category: TEST

Root cause: The e2e-vmaas-full-setup-helm job has a 0% pass rate (42 runs, 0 passes in 7 days) due to OSAC operator controller race conditions causing CR reconciliation conflicts.

Explanation: Root Cause Chain:

The e2e-vmaas-full-setup-helm job is completely broken - 42 runs over 7 days, ZERO passes. This is not a PR-specific issue; it's broken_main affecting this specific job type.
The immediate failure mechanism in build 2070208898819690496 was a 60-minute timeout SIGKILL (exit code 137). The test suite collected 47 tests, passed 6, failed 3, then was killed while running the 4th test.
The underlying root cause is operator controller race conditions. The operator logs show 35 "Operation cannot be fulfilled" resource version conflict errors across PublicIPPool, PublicIP, VirtualNetwork, Subnet, and ComputeInstance controllers. These conflicts occur when both the main controller (e.g., publicip) and the feedback controller (e.g., publicip-feedback) attempt to update the same CR's status simultaneously.
The conflict pattern breaks CR progression: When CRs cannot update their status due to conflicts, they remain stuck in Progressing phase indefinitely. Tests timeout waiting for CRs to reach Ready state (as seen in PR 311 where VirtualNetworks timed out at 300s stuck in Progressing). Multiple stuck tests exhaust the 60-minute job timeout.
Why this is broken_main, not pr_issue:
- PR 348 (this build) and PR 346 both failed identically on e2e-vmaas-full-setup-helm
- The base e2e-vmaas job (non-helm variant) is passing with 74% success rate
- Other PRs pass e2e-vmaas but fail e2e-vmaas-full-setup-helm
- This indicates the helm setup variant has a specific broken configuration that triggers the race condition
Hypothesis on the trigger: The -full-setup-helm variant likely deploys the operator with a configuration that enables both main and feedback controllers simultaneously, or deploys multiple operator replicas without proper leader election, causing the race. The standard vmaas job uses a different deployment method (kustomize) that doesn't trigger this race.

Evidence:

[ci-obs data](https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/osac-project_osac-installer/348/pull-ci-osac-project-osac-installer-main-e2e-vmaas-full-setup-helm/2070208898819690496/artifacts/e2e-vmaas-full-setup-helm/osac-project-gather/artifacts/osac-logs/ci-obs data):

Total full-setup-helm jobs in last 7 days: 42, Passed: 0, Failed: 35

[ci-obs data](https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/osac-project_osac-installer/348/pull-ci-osac-project-osac-installer-main-e2e-vmaas-full-setup-helm/2070208898819690496/artifacts/e2e-vmaas-full-setup-helm/osac-project-gather/artifacts/osac-logs/ci-obs data):

osac-installer e2e-vmaas base variant: 35/47 passed (74%)

pod-osac-operator-6775d44448-kcl69-manager.log:

2026-06-25T19:57:45Z ERROR Reconciler error {"controller": "publicippool-feedback", "error": "Operation cannot be fulfilled on publicippools.osac.openshift.io \"publicippool-lshkk\": the object has been modified"}

pod-osac-operator-6775d44448-kcl69-manager.log:

2026-06-25T19:58:21Z ERROR failed to update status {"controller": "publicip", "error": "Operation cannot be fulfilled on publicips.osac.openshift.io \"publicip-c2glj\": the object has been modified"}

[operator log summary](https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/osac-project_osac-installer/348/pull-ci-osac-project-osac-installer-main-e2e-vmaas-full-setup-helm/2070208898819690496/artifacts/e2e-vmaas-full-setup-helm/osac-project-gather/artifacts/osac-logs/operator log summary):

35 'Operation cannot be fulfilled' conflicts: 18 publicip, 6 publicip-feedback, 3 publicippool-feedback, 3 computeinstance-feedback, 2 virtualnetwork-feedback, 2 subnet-feedback

[build-log.txt PR 311](https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/osac-project_osac-installer/348/pull-ci-osac-project-osac-installer-main-e2e-vmaas-full-setup-helm/2070208898819690496/artifacts/e2e-vmaas-full-setup-helm/osac-project-gather/artifacts/osac-logs/build-log.txt PR 311):

TimeoutError: virtualnetwork-wg4g7 VirtualNetwork Ready — timeout after 300s, last value: 'Progressing'

[build-log.txt current build](https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/osac-project_osac-installer/348/pull-ci-osac-project-osac-installer-main-e2e-vmaas-full-setup-helm/2070208898819690496/artifacts/e2e-vmaas-full-setup-helm/osac-project-gather/artifacts/osac-logs/build-log.txt current build):

/bin/bash: line 59:    23 Killed                  timeout -s 9 60m ... exit code: 137

Suggestion: Immediate Actions:

Disable the e2e-vmaas-full-setup-helm job until fixed - it has a 0% pass rate and is blocking all PRs that trigger it.

Compare operator deployment between variants:

# Investigate the helm deployment
cd osac-installer
grep -r "replicas" charts/osac-operator/
grep -r "leaderElection" charts/osac-operator/

# Compare with kustomize variant
diff <(yq eval '.spec.replicas' kustomize/overlays/vmaas-ci/operator-deployment.yaml) \
     <(yq eval '.spec.replicas' charts/osac-operator/templates/deployment.yaml)

Check for duplicate controller registration: Review osac-operator code to confirm feedback controllers aren't being registered alongside main controllers in a way that causes conflicts. The pattern suggests both controller types are active simultaneously when only one should be reconciling.
Verify leader election config: Ensure the helm deployment enables proper leader election to prevent multiple replicas from reconciling simultaneously.
Add resource version retry logic: The operator should retry with exponential backoff on conflict errors rather than logging and giving up, but this is a mitigation, not a fix.

Long-term Fix:

The helm-based deployment likely has a misconfiguration causing either:

Multiple operator replicas without leader election
Both main and feedback controllers active when only one should run
Incorrect RBAC allowing external processes to modify CR status

Review osac-installer PR history around when e2e-vmaas-full-setup-helm was introduced to identify what configuration diverged from the working kustomize variant.

Prow job | Build 2070208898819690496 | 🤖 triagent | Cost: $1.1791

_{For deeper investigation, use the /osac-debug-e2e skill with this build ID.}

omer-vishlitzky and others added 2 commits June 14, 2026 12:32

empty test file 3

5542bcb

openshift-ci Bot requested review from sk-ilya and tzvatot June 25, 2026 18:01

openshift-ci Bot added the approved label Jun 25, 2026

This was referenced Jun 25, 2026

TESTING HELM INSTALLATION IGNORE #349

Closed

TESTING HELM INSTALLATION IGNORE #350

Closed

omer-vishlitzky closed this Jun 28, 2026

Navigation Menu

Uh oh!

TESTING HELM INSTALLATION IGNORE - #348

TESTING HELM INSTALLATION IGNORE#348
omer-vishlitzky wants to merge 2 commits into
osac-project:mainfrom
omer-vishlitzky:test-helm-ignore-3

omer-vishlitzky commented Jun 25, 2026

Uh oh!

openshift-ci Bot commented Jun 25, 2026

Uh oh!

coderabbitai Bot commented Jun 25, 2026

Review limit reached

Uh oh!

omer-vishlitzky commented Jun 25, 2026

Uh oh!

omer-vishlitzky commented Jun 25, 2026

Uh oh!

openshift-ci Bot commented Jun 25, 2026

Uh oh!

omer-vishlitzky commented Jun 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

omer-vishlitzky commented Jun 25, 2026

Uh oh!

openshift-ci Bot commented Jun 25, 2026

Uh oh!

coderabbitai Bot commented Jun 25, 2026

Review limit reached

Uh oh!

omer-vishlitzky commented Jun 25, 2026

Uh oh!

omer-vishlitzky commented Jun 25, 2026

Uh oh!

openshift-ci Bot commented Jun 25, 2026

Uh oh!

omer-vishlitzky commented Jun 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant