Skip to content

fix: clear FailureReason/FailureMessage when VM recovers from terminal state - #382

Merged
k8s-ci-robot merged 1 commit into
kubernetes-sigs:mainfrom
qinqon:fix/clear-failure-reason-on-recovery
Apr 9, 2026
Merged

k8s-ci-robot merged 1 commit into
kubernetes-sigs:mainfrom
qinqon:fix/clear-failure-reason-on-recovery

Conversation

@qinqon

@qinqon qinqon commented Apr 8, 2026 •

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

When a KubeVirt VM is stopped (virtctl stop changes runStrategy to Halted), the VMI reaches a finalized state and IsTerminal() correctly sets FailureReason=UpdateError on the KubevirtMachine. However, when the VM is started back (virtctl start changes runStrategy back to Always), IsTerminal() returns false but the FailureReason/FailureMessage fields are never cleared.

This causes the CAPI Machine status.phase to stay stuck at Failed even though all Machine conditions (Ready, InfrastructureReady, NodeHealthy) correctly show True. Downstream consumers that rely on the Machine phase — such as HyperShift's endpointslice reconciler for default-ingress-passthrough-service — then permanently mark endpoints as not ready, breaking ingress into hosted clusters.

The fix adds an else clause to nil out FailureReason and FailureMessage when IsTerminal() returns false, allowing VMs to properly recover from a previously terminal state.

Which issue this PR fixes: OCPBUGS-77929

Special notes for your reviewer:

The bug manifests as follows:

  1. virtctl stop VM → runStrategy: Halted → VMI finalizes → IsTerminal()=true → FailureReason set
  2. virtctl start VM → runStrategy: Always → new VMI created → IsTerminal()=false → but FailureReason NOT cleared (no else clause)
  3. KubevirtMachine.Status.FailureReason stays set → CAPI Machine phase: Failed → stuck permanently

Evidence from the customer environment:

  • KubevirtMachine: ready=true, failureReason=UpdateError (contradictory)
  • Machine: conditions all True, phase=Failed (contradictory)
  • EndpointSlice: ready=false (downstream breakage)

Release notes:

Fix KubevirtMachine FailureReason/FailureMessage not being cleared when a VM recovers from a terminal state, which caused the CAPI Machine phase to stay stuck at "Failed" permanently.

@linux-foundation-easycla

linux-foundation-easycla Bot commented Apr 8, 2026 •

Copy link
Copy Markdown

CLA Signed

The committers listed above are authorized under a signed CLA.

  • ✅ login: qinqon / name: Enrique Llorente (ef6c89b)

@k8s-ci-robot k8s-ci-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 8, 2026
@qinqon
qinqon force-pushed the fix/clear-failure-reason-on-recovery branch 3 times, most recently from ce5a12e to c3806aa Compare April 8, 2026 08:49
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 8, 2026
…l state

When a KubeVirt VM is stopped (runStrategy changed to Halted), the VMI
reaches a finalized state and IsTerminal() correctly sets FailureReason
on the KubevirtMachine. However, when the VM is started back
(runStrategy changed to Always), IsTerminal() returns false but the
FailureReason/FailureMessage fields are never cleared.

This causes the CAPI Machine phase to stay stuck at "Failed" even though
all Machine conditions (Ready, InfrastructureReady, NodeHealthy) are
True. Downstream consumers that rely on the Machine phase (such as
HyperShift's endpointslice reconciler) then permanently mark endpoints
as not ready, breaking ingress into hosted clusters.

The fix adds an else clause to nil out FailureReason and FailureMessage
when IsTerminal() returns false, allowing VMs to properly recover.

Signed-off-by: Enrique Llorente <ellorent@redhat.com>
@qinqon
qinqon force-pushed the fix/clear-failure-reason-on-recovery branch from c3806aa to ef6c89b Compare April 8, 2026 08:50
@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Apr 8, 2026

@nunnatsa nunnatsa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve

@k8s-ci-robot

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: nunnatsa, qinqon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 9, 2026
@nunnatsa

nunnatsa commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

/hold

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 9, 2026
@nunnatsa

nunnatsa commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

/ok-to-test

@k8s-ci-robot k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Apr 9, 2026
@nunnatsa

nunnatsa commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

ci passed

/unhold

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 9, 2026
@nunnatsa

nunnatsa commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 9, 2026
@k8s-ci-robot
k8s-ci-robot merged commit b7582ae into kubernetes-sigs:main Apr 9, 2026
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants