[proposal] Test Bucketing Support for HyperShift E2E Tests

[devguyio]

Test Bucketing Support for HyperShift E2E Tests

This PR introduces test bucketing support to improve CI efficiency and resource utilization in HyperShift's E2E test suite. Full proposal can be found here: E2E Test Bucketing Proposal

Key Changes

1. E2E Test Bucketing (test/e2e_exp/)

• Added build tag e2e_batch and bucketing support
• Tests can be split into N equal buckets using:
  • batch-total: Total number of buckets
  • batch-number: Current bucket (1-N)
• Deterministic file-based distribution across buckets
• No changes to existing test code required

2. CI Job Generation (hack/tools/jobtemplate/)

• New tool to transform single test job into N bucketed jobs
• Preserves all job configuration (env vars, dependencies, etc.)
• Adds bucket-specific environment variables:

  env:
    BUCKET_NUM: N      # Current bucket number
    BUCKET_TOTAL: M    # Total number of buckets

• Maintains exact YAML structure and style of input

Usage

Generate bucketed jobs:

make generate-bucketed-jobs TOTAL_BUCKETS=3 CONFIG_FILE=config.yaml TEST_BLOCKS=e2e-aws

Run a specific test bucket:

go test -tags=e2e_batch -e2e.batch-total=3 -e2e.batch-number=1 ./test/e2e_exp/...

Checklist

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: devguyio
Once this PR has been reviewed and has the lgtm label, please assign enxebre for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-merge-robot]

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-ci]

@devguyio: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws	ef153fd	link	true	/test e2e-aws
ci/prow/e2e-kubevirt-aws-ovn-reduced	ef153fd	link	true	/test e2e-kubevirt-aws-ovn-reduced
ci/prow/e2e-aks	ef153fd	link	true	/test e2e-aks
ci/prow/e2e-aws-upgrade-hypershift-operator	ef153fd	link	true	/test e2e-aws-upgrade-hypershift-operator
ci/prow/e2e-aws-4-20	ef153fd	link	true	/test e2e-aws-4-20
ci/prow/e2e-aks-4-20	ef153fd	link	true	/test e2e-aks-4-20
ci/prow/verify	ef153fd	link	true	/test verify
ci/prow/e2e-aws-4-21	ef153fd	link	true	/test e2e-aws-4-21
ci/prow/e2e-aks-4-21	ef153fd	link	true	/test e2e-aks-4-21
ci/prow/unit	ef153fd	link	true	/test unit
ci/prow/e2e-azure-self-managed	ef153fd	link	true	/test e2e-azure-self-managed
ci/prow/verify-workflows	ef153fd	link	true	/test verify-workflows
ci/prow/security	ef153fd	link	true	/test security

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci]

Stale PRs are closed after 21d of inactivity.

If this PR is still relevant, comment to refresh it or remove the stale label.
Mark the PR as fresh by commenting /remove-lifecycle stale.

If this PR is safe to close now please do so with /close.

/lifecycle stale

[openshift-ci]

Stale PRs rot after 14d of inactivity.

Mark the PR as fresh by commenting /remove-lifecycle rotten.
Rotten PRs close after an additional 7d of inactivity.

If this PR is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[hypershift-jira-solve-ci]

All jobs consistently fail with the same root cause. Now I have all the evidence to produce the final report.

Test Failure Analysis Complete

Job Information

• Prow Jobs: e2e-aks, e2e-aws-upgrade-hypershift-operator, e2e-kubevirt-aws-ovn-reduced, e2e-aws, verify, unit, security
• PR: #6096 — "[proposal] Test Bucketing Support for HyperShift E2E Tests"
• Branch: batched-e2e (head SHA: ef153fd313ce9d377d8fc427c32e617680d319d9)
• PR State: OPEN, mergeable: CONFLICTING
• PR Created: 2025-04-29 — the branch has not been rebased in ~14 months

Test Failure Analysis

Error

$ git merge --no-ff ef153fd313ce9d377d8fc427c32e617680d319d9
Auto-merging Makefile
CONFLICT (content): Merge conflict in Makefile
Automatic merge failed; fix conflicts and then commit the result.
# Error: exit status 1
# Final SHA: 
# Total runtime: 0s

Summary

All 7 Prow CI jobs fail identically during the git merge phase before any compilation, testing, or scanning can begin. Prow attempts to merge the PR branch (ef153fd) into the current main branch via git merge --no-ff, and the merge fails due to conflicts — primarily in Makefile, but later runs against newer main SHAs also show conflicts in go.mod, go.sum, vendor/modules.txt, API types, generated CRD manifests, and controller source files. The PR branch (batched-e2e) was created on 2025-04-29 and has never been rebased, making it ~14 months stale. GitHub confirms the PR's merge state is DIRTY / CONFLICTING. No CI step (unit tests, e2e tests, verify, security scan) has ever executed for this PR in its current state.

Root Cause

The PR branch batched-e2e (SHA ef153fd313ce9d377d8fc427c32e617680d319d9) is severely out of date with the main branch of openshift/hypershift. The PR was opened on 2025-04-29 and has not been rebased since, accumulating ~14 months of drift.

When Prow CI runs any job for this PR, the first step is to create a merge commit between the PR branch and the current main. This merge fails because of conflicting changes in files that have been modified on both branches:

Minimum conflict set (present in all runs):

• Makefile — The PR appends new targets (generate-bucketed-jobs, $(JOB_TEMPLATE)) at the end of the Makefile, but main has also added content in the same region.

Extended conflict set (visible in runs against newer main SHAs):

• go.mod, go.sum, vendor/modules.txt — dependency updates on main
• api/hypershift/v1beta1/hostedcluster_types.go — API type changes
• hypershift-operator/controllers/hostedcluster/hostedcluster_controller.go and related test files
• hypershift-operator/controllers/nodepool/ — conditions.go, config.go, config_test.go, nodepool_controller.go, token.go, token_test.go
• Multiple generated CRD manifests under zz_generated.featuregated-crd-manifests/
• docs/content/reference/api.md

Because the merge fails at the git level, the CI job exits with Total runtime: 0s and an empty Final SHA. No code is compiled, no tests are run, and no meaningful CI signal is produced.

Recommendations

1. Rebase the PR branch onto current main: The PR author (devguyio) must rebase the batched-e2e branch onto the current main to resolve all merge conflicts. Given the 14-month drift, this will likely require significant manual conflict resolution.

2. Address potential Go compatibility issues after rebase: The PR's new code in test/e2e_exp/ uses strings.Title() (deprecated since Go 1.18) and rand.Seed() (deprecated since Go 1.20). After rebasing, these should be updated to avoid linter/vet failures in the verify job:

   • Replace strings.Title(base) with cases.Title(language.Und).String(base) from golang.org/x/text
   • Remove rand.Seed() calls (the global rand is automatically seeded since Go 1.20)

3. Re-evaluate the proposal scope: Given the significant drift, review whether the new files added by this PR (test/e2e_exp/, hack/tools/jobtemplate/) are still aligned with the current codebase architecture and testing patterns.

4. Consider closing and re-opening: If the rebase effort is too large, it may be more practical to close this PR and open a fresh one with the bucketing changes applied cleanly to the current main.

Evidence

Evidence	Detail
PR merge state	GitHub reports mergeable: CONFLICTING, mergeStateStatus: DIRTY
PR age	Created 2025-04-29, head SHA unchanged since then (ef153fd)
Unit job (build 2031423207088590848)	CONFLICT (content): Merge conflict in Makefile — exit status 1, runtime 0s
Security job (build 2053902169832165376)	Conflicts in Makefile, API types, CRD manifests, controllers, docs — exit status 1, runtime 0s
Security job (build 2031423207046647808)	CONFLICT (content): Merge conflict in Makefile — exit status 1, runtime 0s
Verify-workflows job (build 2045130496471470080)	Conflicts in Makefile, go.mod, go.sum, vendor/modules.txt, nodepool token.go — exit status 1, runtime 0s
E2E Azure job (build 2031423207008899072)	CONFLICT (content): Merge conflict in Makefile — exit status 1, runtime 0s
E2E-aks, e2e-aws, e2e-kubevirt jobs	Build logs garbage-collected from GCS (jobs ran 2025-05-28), but failure state consistent with merge conflicts based on the same unchanged PR SHA
Conflicting files	Makefile, go.mod, go.sum, vendor/modules.txt, hostedcluster_types.go, hostedcluster_controller.go, nodepool_controller.go, api.md, multiple CRD manifests

---

[openshift-ci]

Rotten PRs close after 7d of inactivity.

Reopen the PR by commenting /reopen.
Mark the PR as fresh by commenting /remove-lifecycle rotten.

/close

[openshift-ci]

@openshift-ci[bot]: Closed this PR.

Details

In response to this:

Rotten PRs close after 7d of inactivity.

Reopen the PR by commenting /reopen.
Mark the PR as fresh by commenting /remove-lifecycle rotten.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.