Skip to content
This repository was archived by the owner on Sep 9, 2026. It is now read-only.

TESTING HELM INSTALLATION IGNORE - #350

Closed
omer-vishlitzky wants to merge 2 commits into
osac-project:mainfrom
omer-vishlitzky:test-helm-ignore-5
Closed

TESTING HELM INSTALLATION IGNORE#350
omer-vishlitzky wants to merge 2 commits into
osac-project:mainfrom
omer-vishlitzky:test-helm-ignore-5

Conversation

@omer-vishlitzky

@omer-vishlitzky omer-vishlitzky commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

This is a test PR, please ignore.

omer-vishlitzky and others added 2 commits June 14, 2026 12:32
@omer-vishlitzky @claude
NO-ISSUE: deploy fulfillment pods at zero replicas during kustomize a…
d0ee776 
…pply

The kustomize overlay changes the fulfillment-database StatefulSet image
reference from a :latest tag to a @sha256: digest. This triggers a
StatefulSet pod recreation, killing the database mid-connection. If the
grpc-server is running database migrations at that moment, golang-migrate
leaves the schema_migrations table in a dirty state and all subsequent
grpc-server starts refuse to run — causing a boot failure.

Fix: use `kustomize edit set replicas` to set fulfillment-controller and
fulfillment-grpc-server to 0 before applying the overlay. The apply
itself deploys with zero replicas, eliminating the race entirely.

At step [5/9], the script now:
1. Waits for TLS certificates
2. Waits for the database StatefulSet rollout to complete
3. Scales grpc-server and controller back to 1
4. Waits for all fulfillment deployment rollouts

This ensures the database is healthy before any migration-running pod
starts, and that grpc-server is available before rest-gateway's
readiness probe checks the gRPC upstream.

Depends on: openshift/release#80431 (adds kustomize binary to the
installer container image).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@omer-vishlitzky
empty test file 5
fa4ff0e
@openshift-ci
openshift-ci Bot requested review from danmanor and larsks June 25, 2026 18:01
@coderabbitai

coderabbitai Bot commented Jun 25, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@omer-vishlitzky, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 59 minutes and 53 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: osac-project/coderabbit/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: a4464f77-a0c7-4524-b328-0cf828e27fe0

📥 Commits

Reviewing files that changed from the base of the PR and between e6b3066 and fa4ff0e.

📒 Files selected for processing (2)
  • empty-test-file-5.txt
  • scripts/refresh-after-snapshot.sh
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci

openshift-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: omer-vishlitzky

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@omer-vishlitzky

omer-vishlitzky commented Jun 25, 2026

Copy link
Copy Markdown
Contributor Author

/test vmaas-e2e-full-setup-helm

@omer-vishlitzky

omer-vishlitzky commented Jun 25, 2026

Copy link
Copy Markdown
Contributor Author

/test e2e-vmaas-full-setup-helm

@openshift-ci

openshift-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown

@omer-vishlitzky: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-vmaas-full-setup-helm fa4ff0e link false /test e2e-vmaas-full-setup-helm

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@omer-vishlitzky omer-vishlitzky mentioned this pull request Jun 25, 2026
Closed
@omer-vishlitzky

omer-vishlitzky commented Jun 25, 2026

Copy link
Copy Markdown
Contributor Author

💀 CI Triage: broken_main | Category: TEST

Root cause: The e2e-vmaas-full-setup-helm job has a 60-minute test step timeout that is insufficient for the test suite, which requires 45-60+ minutes due to slow VM provisioning and long polling intervals in the tests.

Explanation: ## Root Cause Analysis

The e2e-vmaas-full-setup-helm job is systematically failing due to a mismatch between test execution time and the CI timeout. This is a broken_main issue affecting all PRs for at least 2 weeks.

Failure Pattern

Exit code 137 (SIGKILL) - the test process is killed by the 60-minute timeout (timeout -s 9 60m) in the test step script. The test suite doesn't complete within the allocated time.

The build log shows the same pattern across all failures:

  • 6 tests pass (public_ip pool capacity tests - fast, no VM creation)
  • 3 tests fail (public_ip lifecycle tests + compute_instance_api_fields)
  • Test Create hub-access serviceaccount for use when creating hubs #10 (test_compute_instance_cli_explicit_fields) is running when the 60m timeout hits
  • Process killed with SIGKILL (exit code 137)

Why Tests Are Slow

Analysis of the test code reveals cumulative timeout issues:

Per-test overhead from fixtures:

  • make_compute_instances fixture (used by public_ip lifecycle tests): Creates 1-2 VMs, waits up to 15 minutes EACH for Running state (30min total)
  • default_networking fixture (session-scoped): Creates VirtualNetwork + Subnet, up to 10 minutes
  • Each test then creates additional resources with long waits

Individual test timing estimates:

  • test_attach_detach_reattach: 45+ minutes (2 VMs + 4 attach/detach cycles with 5-10min timeouts each)
  • test_validation_rejections: 20+ minutes (1 VM + validation tests)
  • test_compute_instance_api_fields: 25+ minutes (VM lifecycle: provision → run → halt → run)
  • test_compute_instance_cli_explicit_fields: 26+ minutes (provision + run + cleanup)

Total expected runtime: 45-60+ minutes for just these 4 slow tests, not counting the 43 other tests in the suite. The 60-minute timeout is being hit before the suite completes.

Evidence This Is Broken Main

Differential analysis:

  • 49 total runs in past 2 weeks: 0 successes, 45 failures (91% failure rate)
  • Failures span multiple PRs (346, 347, 348, 349, 350, 311) and periodic runs
  • Identical failure pattern across all runs: same 3 tests fail, same timeout on test Create hub-access serviceaccount for use when creating hubs #10
  • No PR-specific correlation - even unrelated test PRs fail the same way

NOT a flake:

NOT infra:

  • Cluster boots successfully (pre step passes)
  • All pods healthy (no CrashLoopBackOff, OOMKills, or resource exhaustion)
  • Node resources normal (11% CPU, 49% memory)
  • Tests execute normally until timeout

NOT PR issue:

Why This Started

The job configuration likely never had adequate timeout for the full test suite. Contributing factors:

  1. Tests were written with very long defensive timeouts (5-15 min per operation)
  2. VM provisioning in CI is slower than expected
  3. Session/class-scoped fixtures add hidden overhead
  4. No pytest timeout configured to fail fast on hung tests

This became visible when:

  • Test suite grew to 47 tests (more tests = more cumulative time)
  • More PRs started using this job variant (helm vs kustomize)
  • Infrastructure slowdowns made VM provisioning take longer

Evidence:

[ci-obs differential analysis](https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/osac-project_osac-installer/350/pull-ci-osac-project-osac-installer-main-e2e-vmaas-full-setup-helm/2070208912728002560/artifacts/e2e-vmaas-full-setup-helm/osac-project-gather/artifacts/osac-logs/ci-obs differential analysis):

49 total runs in past 2 weeks: 0 successes, 45 failures (91% failure rate)

[Build log 2070208912728002560](https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/osac-project_osac-installer/350/pull-ci-osac-project-osac-installer-main-e2e-vmaas-full-setup-helm/2070208912728002560/artifacts/e2e-vmaas-full-setup-helm/osac-project-gather/artifacts/osac-logs/Build log 2070208912728002560):

/bin/bash: line 59:    23 Killed                  timeout -s 9 60m ssh ... exit code: 137

[Build log 2070208909544525824 (PR#349)](https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/osac-project_osac-installer/350/pull-ci-osac-project-osac-installer-main-e2e-vmaas-full-setup-helm/2070208912728002560/artifacts/e2e-vmaas-full-setup-helm/osac-project-gather/artifacts/osac-logs/Build log 2070208909544525824 (PR#349)):

/bin/bash: line 59:    23 Killed                  timeout -s 9 60m ssh ... exit code: 137 (identical pattern)

[osac-test-infra tests/vmaas/public_ip/test_public_ip_pool_lifecycle.py:24-103](https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/osac-project_osac-installer/350/pull-ci-osac-project-osac-installer-main-e2e-vmaas-full-setup-helm/2070208912728002560/artifacts/e2e-vmaas-full-setup-helm/osac-project-gather/artifacts/osac-logs/osac-test-infra tests/vmaas/public_ip/test_public_ip_pool_lifecycle.py:24-103):

test_attach_detach_reattach uses make_compute_instances fixture (2 VMs, 15min each) + 4 attachment operations (5-10min timeouts each)

[osac-test-infra tests/vmaas/test_compute_instance_api_fields.py:17-111](https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/osac-project_osac-installer/350/pull-ci-osac-project-osac-installer-main-e2e-vmaas-full-setup-helm/2070208912728002560/artifacts/e2e-vmaas-full-setup-helm/osac-project-gather/artifacts/osac-logs/osac-test-infra tests/vmaas/test_compute_instance_api_fields.py:17-111):

wait_for_provision: 600s, wait_for_running: 900s, VM halt/restart cycles: 300s each = 25+ min total

[osac-test-infra tests/vmaas/test_compute_instance_cli_fields.py:20-70](https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/osac-project_osac-installer/350/pull-ci-osac-project-osac-installer-main-e2e-vmaas-full-setup-helm/2070208912728002560/artifacts/e2e-vmaas-full-setup-helm/osac-project-gather/artifacts/osac-logs/osac-test-infra tests/vmaas/test_compute_instance_cli_fields.py:20-70):

wait_for_running: 90 retries × 10s = 900s (15 min) - test running when timeout hits

/tmp/debug/2070208912728002560/osac-logs/pods.txt:

All osac pods Running/Completed (except osac-aap-bootstrap Error from earlier refresh) - no infra issues

/tmp/debug/2070208912728002560/osac-logs/node-resources.txt:

CPU(cores): 2616m (11%), MEMORY: 27525Mi (49%) - no resource exhaustion

[PR#350 changes](https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/osac-project_osac-installer/350/pull-ci-osac-project-osac-installer-main-e2e-vmaas-full-setup-helm/2070208912728002560/artifacts/e2e-vmaas-full-setup-helm/osac-project-gather/artifacts/osac-logs/PR#350 changes):

empty-test-file-5.txt, scripts/refresh-after-snapshot.sh - unrelated to test execution time

Suggestion: Immediate fixes:

  1. Increase test step timeout to 120 minutes:

    • Edit the test step registry entry to change timeout from 60m to 120m
    • File: openshift/release ci-operator/step-registry/osac-project/baremetal-test/osac-project-baremetal-test-ref.yaml
    • Change: timeout: 3600 → timeout: 7200 (or use step script's timeout param)

  2. Optimize test suite for CI:

    • Add pytest-timeout plugin to fail fast on hung tests (e.g., 10min per test)
    • Reduce polling intervals from 5-10s to 2-3s (cuts wait time in half)
    • Reduce retry counts for fast operations (CR creation: 30→10 retries)
    • Use module/session fixtures more aggressively to share VMs across tests

  3. Parallelize tests:

    • Run pytest with -n auto (xdist plugin) to run tests in parallel
    • May require test isolation fixes (namespace conflicts)

  4. Skip slow tests in PR checks:

    • Mark slow lifecycle tests with @pytest.mark.slow
    • Run only fast smoke tests in presubmit
    • Run full suite in periodic/post-submit jobs

Long-term architectural fixes:

  1. Mock heavy fixtures for validation tests:

    • test_validation_rejections doesn't need real VMs to test error codes
    • Use mocked gRPC responses or lightweight fixtures

  2. Pre-provision test VMs:

    • Create a pool of ready VMs in the test snapshot
    • Tests claim/release from pool instead of provisioning from scratch
    • Cuts 15-30 min per test

  3. Snapshot flavors optimized for fast boot:

    • Pre-create VirtualNetworks, Subnets in the snapshot
    • Reduce default_networking fixture overhead

  4. Add monitoring:

    • Log cumulative test time and slowest tests
    • Alert if suite approaches timeout threshold

Recommended priority:

  • P0 (unblock CI now): Increase timeout to 120m
  • P1 (reduce flakiness): Add pytest-timeout, reduce polling intervals
  • P2 (speed up): Parallelize with xdist
  • P3 (long-term): Pre-provision VM pool, mock validation tests

Prow job | Build 2070208912728002560 | 🤖 triagent | Cost: $1.2267

For deeper investigation, use the /osac-debug-e2e skill with this build ID.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@danmanor danmanor Awaiting requested review from danmanor
@larsks larsks Awaiting requested review from larsks

Assignees

No one assigned

Labels

approved

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@omer-vishlitzky