Skip to content
This repository was archived by the owner on Sep 9, 2026. It is now read-only.

TESTING HELM INSTALLATION IGNORE - #349

Closed
omer-vishlitzky wants to merge 2 commits into
osac-project:mainfrom
omer-vishlitzky:test-helm-ignore-4
Closed

TESTING HELM INSTALLATION IGNORE#349
omer-vishlitzky wants to merge 2 commits into
osac-project:mainfrom
omer-vishlitzky:test-helm-ignore-4

Conversation

@omer-vishlitzky

@omer-vishlitzky omer-vishlitzky commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

This is a test PR, please ignore.

omer-vishlitzky and others added 2 commits June 14, 2026 12:32
@omer-vishlitzky @claude
NO-ISSUE: deploy fulfillment pods at zero replicas during kustomize a…
d0ee776 
…pply

The kustomize overlay changes the fulfillment-database StatefulSet image
reference from a :latest tag to a @sha256: digest. This triggers a
StatefulSet pod recreation, killing the database mid-connection. If the
grpc-server is running database migrations at that moment, golang-migrate
leaves the schema_migrations table in a dirty state and all subsequent
grpc-server starts refuse to run — causing a boot failure.

Fix: use `kustomize edit set replicas` to set fulfillment-controller and
fulfillment-grpc-server to 0 before applying the overlay. The apply
itself deploys with zero replicas, eliminating the race entirely.

At step [5/9], the script now:
1. Waits for TLS certificates
2. Waits for the database StatefulSet rollout to complete
3. Scales grpc-server and controller back to 1
4. Waits for all fulfillment deployment rollouts

This ensures the database is healthy before any migration-running pod
starts, and that grpc-server is available before rest-gateway's
readiness probe checks the gRPC upstream.

Depends on: openshift/release#80431 (adds kustomize binary to the
installer container image).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@omer-vishlitzky
empty test file 4
5f0f30a
@openshift-ci
openshift-ci Bot requested review from eliorerz and tzvatot June 25, 2026 18:01
@openshift-ci

openshift-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: omer-vishlitzky

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai

coderabbitai Bot commented Jun 25, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@omer-vishlitzky, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 59 minutes and 54 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: osac-project/coderabbit/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: a7e65a4b-fccd-48d0-b977-0b93654e19af

📥 Commits

Reviewing files that changed from the base of the PR and between e6b3066 and 5f0f30a.

📒 Files selected for processing (2)
  • empty-test-file-4.txt
  • scripts/refresh-after-snapshot.sh
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@omer-vishlitzky

omer-vishlitzky commented Jun 25, 2026

Copy link
Copy Markdown
Contributor Author

/test vmaas-e2e-full-setup-helm

@omer-vishlitzky

omer-vishlitzky commented Jun 25, 2026

Copy link
Copy Markdown
Contributor Author

/test e2e-vmaas-full-setup-helm

@openshift-ci

openshift-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown

@omer-vishlitzky: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-vmaas-full-setup-helm 5f0f30a link false /test e2e-vmaas-full-setup-helm

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@omer-vishlitzky

omer-vishlitzky commented Jun 25, 2026

Copy link
Copy Markdown
Contributor Author

💀 CI Triage: broken_main | Category: TEST

Root cause: The e2e-vmaas-full-setup-helm job has a 0% pass rate (0/26 runs) across all PRs in the last 72 hours, while the regular e2e-vmaas job maintains a 70% pass rate - indicating the helm-specific workflow/flavor is completely broken, not this PR.

Explanation: ## Investigation Summary

Differential Analysis (MANDATORY - Step 0):

This PR (349):

  • Changes: empty-test-file-4.txt, scripts/refresh-after-snapshot.sh (testing/development changes)
  • Job results: e2e-vmaas-full-setup-helm FAILED, e2e-vmaas PASSED, images PASSED
  • The PR itself is NOT the cause - it passed the regular e2e-vmaas job

Failure Pattern Across All Runs:
Every single e2e-vmaas-full-setup-helm failure shows:

  1. First 6 tests PASS (public_ip pool capacity tests: 2%, 4%, 6%, 8%, 10%, 12%)
  2. Next 3 tests immediately FAIL (test_attach_detach_reattach, test_validation_rejections, test_compute_instance_api_fields: 14%, 17%, 19%)
  3. Test suite hangs and hits 60-minute timeout before completing remaining 38 tests
  4. Process killed with exit code 137 (SIGKILL from timeout -s 9 60m)

Timeline Analysis:

  • Week of June 19-22: Mostly BOOT and INFRA failures (flavor/infrastructure issues)
  • June 23-25: Started reaching TEST phase but timing out with same test failure pattern
  • Recent osac-installer changes include MetalLB reconfiguration, Helm chart updates, AAP token configuration changes (commits from June 19-25)

Root Cause:
The helm-specific installation/deployment path is broken. Likely causes:

  1. Stale/incompatible flavor snapshot - The vmaas-helm flavor hasn't been rebuilt to include recent Helm chart or MetalLB changes
  2. Helm-specific configuration bug - Recent changes to Helm values, MetalLB CIDR configuration, or AAP token handling broke the Helm deployment path
  3. Test incompatibility with Helm deployment - The public_ip tests that fail may have dependencies on resources/configurations that exist in kustomize deployments but not in Helm deployments

Why Tests Hang:
The 3 failing tests appear to be stuck waiting for resources that never become ready in the Helm-deployed environment, causing pytest to hang until the 60-minute timeout kills the process. This prevents the remaining 38 tests from running.

Evidence This Is NOT the PR:

  • PR 349's only substantive change is to scripts/refresh-after-snapshot.sh
  • PR 349 PASSED the e2e-vmaas job (non-helm) successfully
  • ALL other PRs show identical failure pattern in e2e-vmaas-full-setup-helm
  • The failure pattern predates this PR by several days

Why NOT Infra/Flake:

  • 100% reproducible across all runs (0/26 passing)
  • Identical test progression every time (not random)
  • Same 3 tests fail at same progression points
  • Regular e2e-vmaas jobs pass at 70% rate on same infrastructure
  • No OOM, no node resource exhaustion (node at 50% memory, pods healthy)

Evidence:

[ci-obs differential analysis](https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/osac-project_osac-installer/349/pull-ci-osac-project-osac-installer-main-e2e-vmaas-full-setup-helm/2070208909544525824/artifacts/e2e-vmaas-full-setup-helm/osac-project-gather/artifacts/osac-logs/ci-obs differential analysis):

e2e-vmaas-full-setup-helm: 0/26 passed (0%) in 72h; e2e-vmaas: 31/44 passed (70%) in 24h

[build log timestamp analysis](https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/osac-project_osac-installer/349/pull-ci-osac-project-osac-installer-main-e2e-vmaas-full-setup-helm/2070208909544525824/artifacts/e2e-vmaas-full-setup-helm/osac-project-gather/artifacts/osac-logs/build log timestamp analysis):

Test step started 2026-06-25T20:26:21Z, killed at 2026-06-25T21:26:31Z = 60m10s (timeout was 60m)

[build-log.txt test progression](https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/osac-project_osac-installer/349/pull-ci-osac-project-osac-installer-main-e2e-vmaas-full-setup-helm/2070208909544525824/artifacts/e2e-vmaas-full-setup-helm/osac-project-gather/artifacts/osac-logs/build-log.txt test progression):

6 tests PASSED (2%-12%), then 3 FAILED (14%-19%), then timeout: test_attach_detach_reattach FAILED, test_validation_rejections FAILED, test_compute_instance_api_fields FAILED

[PR 347 build log](https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/osac-project_osac-installer/349/pull-ci-osac-project-osac-installer-main-e2e-vmaas-full-setup-helm/2070208909544525824/artifacts/e2e-vmaas-full-setup-helm/osac-project-gather/artifacts/osac-logs/PR 347 build log):

Identical pattern: same 6 PASS, same 3 FAIL at 14%/17%/19%

[PR 346 build log](https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/osac-project_osac-installer/349/pull-ci-osac-project-osac-installer-main-e2e-vmaas-full-setup-helm/2070208909544525824/artifacts/e2e-vmaas-full-setup-helm/osac-project-gather/artifacts/osac-logs/PR 346 build log):

Identical pattern: same 6 PASS, same 3 FAIL at 14%/17%/19%

[This PR results](https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/osac-project_osac-installer/349/pull-ci-osac-project-osac-installer-main-e2e-vmaas-full-setup-helm/2070208909544525824/artifacts/e2e-vmaas-full-setup-helm/osac-project-gather/artifacts/osac-logs/This PR results):

e2e-vmaas job: SUCCESS; e2e-vmaas-full-setup-helm: FAILURE - proves PR code is not the issue

[osac-installer git log](https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/osac-project_osac-installer/349/pull-ci-osac-project-osac-installer-main-e2e-vmaas-full-setup-helm/2070208909544525824/artifacts/e2e-vmaas-full-setup-helm/osac-project-gather/artifacts/osac-logs/osac-installer git log):

Recent helm-related changes: 768174e (MetalLB pool), 6d43fd2 (config-as-code AAP), e6b3066 (caas helm values), 1db1e7c (AAP token secret)

node-resources.txt:

Node CPU: 2627m/11%, Memory: 27902Mi/50% - no resource exhaustion

Suggestion: Immediate Actions:

  1. Disable or Skip the e2e-vmaas-full-setup-helm Job
    Since it has a 0% pass rate and is blocking all PRs unnecessarily, mark it as non-blocking in CI config until fixed.

  2. Investigate Helm vs Kustomize Deployment Differences

    • Compare what resources exist in a working e2e-vmaas cluster vs a failing e2e-vmaas-full-setup-helm cluster
    • Check if PublicIPPool CRs or related network resources are missing/misconfigured in Helm deployments
    • Verify MetalLB configuration is applied correctly via Helm

  3. Rebuild the vmaas-helm Flavor
    The flavor snapshot may be stale and incompatible with recent Helm chart changes. Rebuild from current main with:

    • Latest osac-installer Helm charts
    • Latest MetalLB pool configuration
    • Latest AAP token configuration
    • Latest component images

  4. Add Diagnostic Logging to Failing Tests
    Modify test_attach_detach_reattach, test_validation_rejections, and test_compute_instance_api_fields to log what resource they're waiting for and its current state before hanging.

  5. Check for Missing Helm Chart Resources
    Review recent commits to charts/ and kustomize/ in osac-installer:

    • Ensure PublicIPPool CRDs and related network resources are templated in Helm
    • Verify the Helm chart includes all resources that kustomize overlays provide
    • Compare rendered Helm templates vs kustomize output

  6. Bisect Recent Changes
    If rebuilding doesn't fix it, bisect osac-installer commits from June 19-25 to find which Helm/MetalLB change broke the deployment.

Do NOT:

  • Blame this PR - it's not the cause
  • Classify as flake - it's 100% reproducible
  • Classify as infra - regular e2e-vmaas works fine on same infrastructure

Prow job | Build 2070208909544525824 | 🤖 triagent | Cost: $0.9903

For deeper investigation, use the /osac-debug-e2e skill with this build ID.

@omer-vishlitzky omer-vishlitzky mentioned this pull request Jun 25, 2026
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@eliorerz eliorerz Awaiting requested review from eliorerz
@tzvatot tzvatot Awaiting requested review from tzvatot

Assignees

No one assigned

Labels

approved

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@omer-vishlitzky