Skip to content

Run k8sgpt as part of hypershift-dump chain - #68540

Merged
openshift-merge-bot[bot] merged 4 commits into
openshift:masterfrom
mgencur:test_k8s_gpt_with_hypershift
Sep 23, 2025
Merged

Run k8sgpt as part of hypershift-dump chain#68540
openshift-merge-bot[bot] merged 4 commits into
openshift:masterfrom
mgencur:test_k8s_gpt_with_hypershift

Conversation

@mgencur

@mgencur mgencur commented Aug 27, 2025

Copy link
Copy Markdown
Contributor

Adds k8sgpt integration to the hypershift-dump workflow by:

  • Adding k8sgpt step to the dump chain configuration
  • Creating new k8sgpt step registry files (commands, metadata, yaml config, and OWNERS)

The step produces files under hypershift-k8sgpt/artifacts/:

  • namespaces/hypershift/result.json (errors from the management cluster)
  • namespaces/clusters/result.json (errors from the management cluster)
  • namespaces/clusters-<hc_name>/result.json (errors from the management cluster)
  • hostedcluster/result.json (errors from all namespace in hosted/guest cluster)

See example with errors from the latest run. It shows an error in the HCP namespace and no errors in the hosted cluster.

The step currently doesn't run the analysis through AI (via --explain flag for k8sgpt) as it is currently tied to an OpenShift OpenAI account that reached its limits. The plan is to change this to use Gemini (see slack thread). When this account is available we can enable AI analysis. Until that time, please see openshift/hypershift#6720 which adds a custom Claude command to analyze the output of these errors (instead of doing it directly by k8sgpt by calling AI).

Note: The current usage of k8sgpt doesn't gather errors from Pod logs as it doesn't explicitly use the Log analyzer (available through --filter Log). This needs further investigation as there are usually errors that are "OK" such as some temporary reconciler error).

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 27, 2025
@mgencur

mgencur commented Aug 27, 2025

Copy link
Copy Markdown
Contributor Author

/pj-rehearse ?

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@mgencur: job(s): ? either don't exist or were not found to be affected, and cannot be rehearsed

@mgencur

mgencur commented Aug 27, 2025

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-aws-conformance-cilium

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@mgencur

mgencur commented Aug 27, 2025

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-aws-conformance-cilium-k8sgpt

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@mgencur

mgencur commented Aug 28, 2025

Copy link
Copy Markdown
Contributor Author

The k8sgpt returns this error:

openai added to the AI backend provider list
Error: exhausted API quota for AI provider openai: error, status code: 429, status: 429 Too Many Requests, message: You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.

As can be seen here

@mgencur

mgencur commented Aug 28, 2025

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-aws-conformance-cilium-k8sgpt

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@mgencur

mgencur commented Aug 28, 2025

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-aws-conformance-cilium-k8sgpt

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@mgencur

mgencur commented Aug 28, 2025

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-aws-conformance-cilium-k8sgpt

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@mgencur, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

could not determine changed registry steps: could not load step registry: workflow/hypershift-aws-conformance-cilium: parameter "TEST_SKIPS" is overridden in [workflow/hypershift-aws-conformance-cilium] but not declared in any step
Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@mgencur, pj-rehearse: unable to determine affected jobs ERROR:

could not determine changed registry steps: could not load step registry: workflow/hypershift-aws-conformance-cilium: parameter "TEST_SKIPS" is overridden in [workflow/hypershift-aws-conformance-cilium] but not declared in any step

If the problem persists, please contact Test Platform.

@mgencur

mgencur commented Aug 28, 2025

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-aws-conformance-cilium-k8sgpt

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@mgencur

mgencur commented Aug 29, 2025

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-aws-conformance-cilium-k8sgpt

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@mgencur

mgencur commented Aug 29, 2025

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-aws-conformance-cilium-k8sgpt

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@mgencur

mgencur commented Aug 29, 2025

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-aws-conformance-cilium-k8sgpt

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@mgencur

mgencur commented Aug 29, 2025

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-aws-conformance-cilium-k8sgpt

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@mgencur
mgencur force-pushed the test_k8s_gpt_with_hypershift branch from 95969ab to b569d2f Compare August 29, 2025 11:25
@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 29, 2025
@mgencur

mgencur commented Aug 29, 2025

Copy link
Copy Markdown
Contributor Author

/hold

For rehearsal jobs to pass.

@openshift-ci openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 29, 2025
@mgencur

mgencur commented Sep 1, 2025

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-aws-ovn-conformance

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci

openshift-ci Bot commented Sep 1, 2025

Copy link
Copy Markdown
Contributor

@mgencur: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/rehearse/periodic-ci-openshift-hypershift-release-4.19-periodics-e2e-kubevirt-aws-ovn c14fd26 link unknown /pj-rehearse periodic-ci-openshift-hypershift-release-4.19-periodics-e2e-kubevirt-aws-ovn

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@mgencur

mgencur commented Sep 1, 2025

Copy link
Copy Markdown
Contributor Author

/unhold

@openshift-ci openshift-ci Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 1, 2025
This commit adds k8sgpt integration to the hypershift-dump workflow by:
- Adding k8sgpt step to the dump chain configuration
- Creating new k8sgpt step registry files (commands, metadata, yaml config, and OWNERS)
@mgencur
mgencur force-pushed the test_k8s_gpt_with_hypershift branch from ba76675 to b9bfd7c Compare September 16, 2025 09:58
@mgencur

mgencur commented Sep 16, 2025

Copy link
Copy Markdown
Contributor Author

/pj-rehearse periodic-ci-openshift-hypershift-release-4.21-periodics-e2e-aws-ovn-conformance

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

[REHEARSALNOTIFIER]
@mgencur: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name Repo Type Reason
pull-ci-openshift-kubevirt-csi-driver-main-e2e-kubevirt-aws-csi openshift/kubevirt-csi-driver presubmit Registry content changed
pull-ci-openshift-kubevirt-csi-driver-release-4.22-e2e-kubevirt-aws-csi openshift/kubevirt-csi-driver presubmit Registry content changed
pull-ci-openshift-kubevirt-csi-driver-release-4.21-e2e-kubevirt-aws-csi openshift/kubevirt-csi-driver presubmit Registry content changed
pull-ci-openshift-kubevirt-csi-driver-release-4.20-e2e-kubevirt-aws-csi openshift/kubevirt-csi-driver presubmit Registry content changed
pull-ci-openshift-kubevirt-csi-driver-release-4.19-e2e-kubevirt-aws-csi openshift/kubevirt-csi-driver presubmit Registry content changed
pull-ci-openshift-kubevirt-csi-driver-release-4.18-e2e-kubevirt-aws-csi openshift/kubevirt-csi-driver presubmit Registry content changed
pull-ci-openshift-kubevirt-csi-driver-release-4.17-e2e-kubevirt-aws-csi openshift/kubevirt-csi-driver presubmit Registry content changed
pull-ci-openshift-kubevirt-csi-driver-release-4.16-e2e-kubevirt-aws-csi openshift/kubevirt-csi-driver presubmit Registry content changed
pull-ci-openshift-kubevirt-csi-driver-release-4.15-e2e-kubevirt-aws-csi openshift/kubevirt-csi-driver presubmit Registry content changed
pull-ci-openshift-kubevirt-csi-driver-release-4.14-e2e-kubevirt-aws-csi openshift/kubevirt-csi-driver presubmit Registry content changed
pull-ci-openshift-cluster-api-provider-kubevirt-main-e2e-hypershift-kubevirt openshift/cluster-api-provider-kubevirt presubmit Registry content changed
pull-ci-openshift-cluster-api-provider-kubevirt-release-4.22-e2e-hypershift-kubevirt openshift/cluster-api-provider-kubevirt presubmit Registry content changed
pull-ci-openshift-cluster-api-provider-kubevirt-release-4.21-e2e-hypershift-kubevirt openshift/cluster-api-provider-kubevirt presubmit Registry content changed
pull-ci-openshift-cluster-api-provider-kubevirt-release-4.20-e2e-hypershift-kubevirt openshift/cluster-api-provider-kubevirt presubmit Registry content changed
pull-ci-openshift-cluster-api-provider-kubevirt-release-4.19-e2e-hypershift-kubevirt openshift/cluster-api-provider-kubevirt presubmit Registry content changed
pull-ci-openshift-cluster-api-provider-kubevirt-release-4.18-e2e-hypershift-kubevirt openshift/cluster-api-provider-kubevirt presubmit Registry content changed
pull-ci-openshift-cluster-api-provider-kubevirt-release-4.17-e2e-hypershift-kubevirt openshift/cluster-api-provider-kubevirt presubmit Registry content changed
pull-ci-openshift-cluster-api-provider-kubevirt-release-4.16-e2e-hypershift-kubevirt openshift/cluster-api-provider-kubevirt presubmit Registry content changed
pull-ci-openshift-cluster-api-provider-kubevirt-release-4.15-e2e-hypershift-kubevirt openshift/cluster-api-provider-kubevirt presubmit Registry content changed
pull-ci-openshift-cluster-api-provider-kubevirt-release-4.14-e2e-hypershift-kubevirt openshift/cluster-api-provider-kubevirt presubmit Registry content changed
pull-ci-openshift-origin-main-e2e-hypershift-conformance openshift/origin presubmit Registry content changed
pull-ci-openshift-origin-release-4.22-e2e-hypershift-conformance openshift/origin presubmit Registry content changed
pull-ci-openshift-origin-release-4.21-e2e-hypershift-conformance openshift/origin presubmit Registry content changed
pull-ci-openshift-origin-release-4.20-e2e-hypershift-conformance openshift/origin presubmit Registry content changed
pull-ci-openshift-origin-release-4.19-e2e-hypershift-conformance openshift/origin presubmit Registry content changed

A total of 621 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

@LiangquanLi930

Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Sep 23, 2025
@openshift-ci

openshift-ci Bot commented Sep 23, 2025

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: LiangquanLi930, mgencur

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 23, 2025
@mgencur

mgencur commented Sep 23, 2025

Copy link
Copy Markdown
Contributor Author

/pj-rehearse ack

@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@mgencur: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

@openshift-ci-robot openshift-ci-robot added the rehearsals-ack Signifies that rehearsal jobs have been acknowledged label Sep 23, 2025
@openshift-merge-bot
openshift-merge-bot Bot merged commit d7a328c into openshift:master Sep 23, 2025
11 checks passed
mkdir -p "${ARTIFACT_DIR}/namespaces" "${ARTIFACT_DIR}/hostedcluster"

CLUSTER_NAME="$(echo -n $PROW_JOB_ID|sha256sum|cut -c-20)"
HOSTED_CLUSTER_NS=$(oc get hostedcluster -A -ojsonpath='{.items[0].metadata.namespace}')

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kubevirt presubmit job is failing here

xiaojiey pushed a commit to xiaojiey/release that referenced this pull request Sep 25, 2025
* Run k8sgpt as part of hypershift-dump chain

This commit adds k8sgpt integration to the hypershift-dump workflow by:
- Adding k8sgpt step to the dump chain configuration
- Creating new k8sgpt step registry files (commands, metadata, yaml config, and OWNERS)

* Make sure the script doesn't fail if there's no guest cluster

* Collect hypershift-related namespaces from mgmt cluster

* Download k8sgpt binary directly to get latest version
abhay-nutanix pushed a commit to abhay-nutanix/release that referenced this pull request Oct 16, 2025
* Run k8sgpt as part of hypershift-dump chain

This commit adds k8sgpt integration to the hypershift-dump workflow by:
- Adding k8sgpt step to the dump chain configuration
- Creating new k8sgpt step registry files (commands, metadata, yaml config, and OWNERS)

* Make sure the script doesn't fail if there's no guest cluster

* Collect hypershift-related namespaces from mgmt cluster

* Download k8sgpt binary directly to get latest version
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. rehearsals-ack Signifies that rehearsal jobs have been acknowledged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants