Fork Sync: Update from parent repository - #73
Merged
Merged
Conversation
Co-authored-by: hhpatel14 <hitpatel@redhat.com>
kaovilai
added a commit
that referenced
this pull request
Aug 5, 2026
Adds a second temporary image override (KubeVirtDatamoverControllerImageKey -> quay.io/tkaovila/kdm-controller:issue73-phase3) alongside the existing plugin override, to exercise migtools/kubevirt-datamover-controller#124 (DataDownload controller for VM restore, issue #73 phase 3) pre-merge -- directly relevant to restore-from-CBT coverage. Both overrides are temporary and should be removed once their respective PRs merge. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
kaovilai
added a commit
that referenced
this pull request
Aug 5, 2026
Closes the #99 "Restore from KDM CBT backup" gap, now unblocked by the overridden kubevirt-datamover-controller image (migtools/kubevirt-datamover- controller#124, DataDownload controller, issue #73 phase 3). Per docs/design/kubevirt-datamover.md and the plugin's own README, restore just needs a normal Velero Restore -- the VirtualMachine RestoreItemAction plugin creates the DataDownload CR automatically from backup-recorded annotations, and separately discards the restored VMB/VMBT so restore doesn't re-trigger a backup. No manual CR driving needed, mirroring how backup already works. Verifies success via the existing generic lib.CreateRestoreFromBackup/IsRestoreDone/IsRestoreCompletedSuccessfully helpers plus a new lib.GetDataDownloadForRestore (mirrors GetDataUploadForBackup) confirming the DataDownload itself reached status.phase=Completed -- proving the datamover path actually engaged, not just that the restore completed some other way. Extracted runKubevirtDMBackup (shared create+wait+verify-backup helper) out of runSequenceBackup to avoid duplicating that logic a second time for the restore test. Also scaffolds two ginkgo.PIt (real, compiling, not run) for gaps explicitly still open per kubevirt-datamover-controller#73's phased plan: multi-disk restore (phase 4, not yet hardened -- new cirros-test-multipvc-cbt.yaml template ready for when it lands) and restore from an incremental (not full) backup (phase 5, not yet e2e-validated upstream even though the chain-rebase logic may already support it internally). Not yet run against a real cluster. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
kaovilai
added a commit
that referenced
this pull request
Aug 5, 2026
The manager ClusterRole shipped with OADP only granted access to datauploads/datauploads.status -- there was no datadownloads, datadownloads/status, or events permission at all. Without this, the controller would hit RBAC-denied errors reconciling any DataDownload, regardless of image correctness, once a real velero restore actually tried to drive it (which migtools/kubevirt-datamover-plugin#41 now makes possible). Synced config/kubevirt-datamover-controller_rbac/role.yaml and the matching block in bundle/manifests/oadp-operator.clusterserviceversion.yaml (serviceAccountName: oadp-kubevirt-datamover-controller-manager) to byte-match config/rbac/role.yaml from migtools/kubevirt-datamover-controller PR #124 (issue #73 phase 3, commit 825d176), which added these rules on the source side but were never pulled into OADP's bundled copy -- normally done via `make update-kubevirt-datamover-manifests KUBEVIRT_DATAMOVER_PATH=...`, done here by hand since no local checkout of that repo is available in this environment. This is a real, pre-existing gap (not introduced by this branch's other changes) that this branch's e2e work would otherwise have hit blind, so fixing it here rather than filing it separately. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
kaovilai
added a commit
that referenced
this pull request
Aug 5, 2026
Adds a second temporary image override (KubeVirtDatamoverControllerImageKey -> quay.io/tkaovila/kdm-controller:issue73-phase3) alongside the existing plugin override, to exercise migtools/kubevirt-datamover-controller#124 (DataDownload controller for VM restore, issue #73 phase 3) pre-merge -- directly relevant to restore-from-CBT coverage. Both overrides are temporary and should be removed once their respective PRs merge. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
kaovilai
added a commit
that referenced
this pull request
Aug 5, 2026
Closes the #99 "Restore from KDM CBT backup" gap, now unblocked by the overridden kubevirt-datamover-controller image (migtools/kubevirt-datamover- controller#124, DataDownload controller, issue #73 phase 3). Per docs/design/kubevirt-datamover.md and the plugin's own README, restore just needs a normal Velero Restore -- the VirtualMachine RestoreItemAction plugin creates the DataDownload CR automatically from backup-recorded annotations, and separately discards the restored VMB/VMBT so restore doesn't re-trigger a backup. No manual CR driving needed, mirroring how backup already works. Verifies success via the existing generic lib.CreateRestoreFromBackup/IsRestoreDone/IsRestoreCompletedSuccessfully helpers plus a new lib.GetDataDownloadForRestore (mirrors GetDataUploadForBackup) confirming the DataDownload itself reached status.phase=Completed -- proving the datamover path actually engaged, not just that the restore completed some other way. Extracted runKubevirtDMBackup (shared create+wait+verify-backup helper) out of runSequenceBackup to avoid duplicating that logic a second time for the restore test. Also scaffolds two ginkgo.PIt (real, compiling, not run) for gaps explicitly still open per kubevirt-datamover-controller#73's phased plan: multi-disk restore (phase 4, not yet hardened -- new cirros-test-multipvc-cbt.yaml template ready for when it lands) and restore from an incremental (not full) backup (phase 5, not yet e2e-validated upstream even though the chain-rebase logic may already support it internally). Not yet run against a real cluster. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
kaovilai
added a commit
that referenced
this pull request
Aug 5, 2026
The manager ClusterRole shipped with OADP only granted access to datauploads/datauploads.status -- there was no datadownloads, datadownloads/status, or events permission at all. Without this, the controller would hit RBAC-denied errors reconciling any DataDownload, regardless of image correctness, once a real velero restore actually tried to drive it (which migtools/kubevirt-datamover-plugin#41 now makes possible). Synced config/kubevirt-datamover-controller_rbac/role.yaml and the matching block in bundle/manifests/oadp-operator.clusterserviceversion.yaml (serviceAccountName: oadp-kubevirt-datamover-controller-manager) to byte-match config/rbac/role.yaml from migtools/kubevirt-datamover-controller PR #124 (issue #73 phase 3, commit 825d176), which added these rules on the source side but were never pulled into OADP's bundled copy -- normally done via `make update-kubevirt-datamover-manifests KUBEVIRT_DATAMOVER_PATH=...`, done here by hand since no local checkout of that repo is available in this environment. This is a real, pre-existing gap (not introduced by this branch's other changes) that this branch's e2e work would otherwise have hit blind, so fixing it here rather than filing it separately. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
kaovilai
added a commit
that referenced
this pull request
Aug 5, 2026
Adds a second temporary image override (KubeVirtDatamoverControllerImageKey -> quay.io/tkaovila/kdm-controller:issue73-phase3) alongside the existing plugin override, to exercise migtools/kubevirt-datamover-controller#124 (DataDownload controller for VM restore, issue #73 phase 3) pre-merge -- directly relevant to restore-from-CBT coverage. Both overrides are temporary and should be removed once their respective PRs merge. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
kaovilai
added a commit
that referenced
this pull request
Aug 5, 2026
Closes the #99 "Restore from KDM CBT backup" gap, now unblocked by the overridden kubevirt-datamover-controller image (migtools/kubevirt-datamover- controller#124, DataDownload controller, issue #73 phase 3). Per docs/design/kubevirt-datamover.md and the plugin's own README, restore just needs a normal Velero Restore -- the VirtualMachine RestoreItemAction plugin creates the DataDownload CR automatically from backup-recorded annotations, and separately discards the restored VMB/VMBT so restore doesn't re-trigger a backup. No manual CR driving needed, mirroring how backup already works. Verifies success via the existing generic lib.CreateRestoreFromBackup/IsRestoreDone/IsRestoreCompletedSuccessfully helpers plus a new lib.GetDataDownloadForRestore (mirrors GetDataUploadForBackup) confirming the DataDownload itself reached status.phase=Completed -- proving the datamover path actually engaged, not just that the restore completed some other way. Extracted runKubevirtDMBackup (shared create+wait+verify-backup helper) out of runSequenceBackup to avoid duplicating that logic a second time for the restore test. Also scaffolds two ginkgo.PIt (real, compiling, not run) for gaps explicitly still open per kubevirt-datamover-controller#73's phased plan: multi-disk restore (phase 4, not yet hardened -- new cirros-test-multipvc-cbt.yaml template ready for when it lands) and restore from an incremental (not full) backup (phase 5, not yet e2e-validated upstream even though the chain-rebase logic may already support it internally). Not yet run against a real cluster. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
kaovilai
added a commit
that referenced
this pull request
Aug 5, 2026
The manager ClusterRole shipped with OADP only granted access to datauploads/datauploads.status -- there was no datadownloads, datadownloads/status, or events permission at all. Without this, the controller would hit RBAC-denied errors reconciling any DataDownload, regardless of image correctness, once a real velero restore actually tried to drive it (which migtools/kubevirt-datamover-plugin#41 now makes possible). Synced config/kubevirt-datamover-controller_rbac/role.yaml and the matching block in bundle/manifests/oadp-operator.clusterserviceversion.yaml (serviceAccountName: oadp-kubevirt-datamover-controller-manager) to byte-match config/rbac/role.yaml from migtools/kubevirt-datamover-controller PR #124 (issue #73 phase 3, commit 825d176), which added these rules on the source side but were never pulled into OADP's bundled copy -- normally done via `make update-kubevirt-datamover-manifests KUBEVIRT_DATAMOVER_PATH=...`, done here by hand since no local checkout of that repo is available in this environment. This is a real, pre-existing gap (not introduced by this branch's other changes) that this branch's e2e work would otherwise have hit blind, so fixing it here rather than filing it separately. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
kaovilai
added a commit
that referenced
this pull request
Aug 13, 2026
Both fixes are now merged upstream: migtools/kubevirt-datamover-plugin#41 and #44 (VM-eager-start race, restore RIAs), and migtools/kubevirt-datamover-controller#124 (issue #73 phase 3 DataDownload controller). The suite now exercises the default images per the TODO this override block left for itself. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
kaovilai
added a commit
that referenced
this pull request
Aug 25, 2026
landed migtools/kubevirt-datamover-controller#124 fixed openshift#169 (stale-sibling DataDownload correlation now keys off velero.io/restore-name, not just VM identity) and openshift#186 (issue #73 Phase 4) hardened per-disk DataDownload isolation for multi-disk restores -- both scaffolded PIt placeholders can now run as real specs. A third PIt in the same file (CNV-85377, an external KubeVirt/CNV bug) is unrelated and left pending. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
kaovilai
added a commit
that referenced
this pull request
Aug 26, 2026
landed migtools/kubevirt-datamover-controller#124 fixed openshift#169 (stale-sibling DataDownload correlation now keys off velero.io/restore-name, not just VM identity) and openshift#186 (issue #73 Phase 4) hardened per-disk DataDownload isolation for multi-disk restores -- both scaffolded PIt placeholders can now run as real specs. A third PIt in the same file (CNV-85377, an external KubeVirt/CNV bug) is unrelated and left pending. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
kaovilai
added a commit
that referenced
this pull request
Aug 27, 2026
landed migtools/kubevirt-datamover-controller#124 fixed openshift#169 (stale-sibling DataDownload correlation now keys off velero.io/restore-name, not just VM identity) and openshift#186 (issue #73 Phase 4) hardened per-disk DataDownload isolation for multi-disk restores -- both scaffolded PIt placeholders can now run as real specs. A third PIt in the same file (CNV-85377, an external KubeVirt/CNV bug) is unrelated and left pending. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
kaovilai
added a commit
that referenced
this pull request
Aug 27, 2026
…idation Upstream blockers (kubevirt-datamover-controller#169, #73 phase 4) have landed, so this un-pends 'restore run-state flip is not blocked by a stale sibling DataDownload from a different restore attempt' and 'restore a multi-PVC VM from a kubevirt-datamover CBT backup'. Live validation against real AWS/GCP/Azure clusters surfaced and fixed several bugs along the way: - the decoy DataDownload used in the run-state-flip test correlated by restore-uid instead of restore-name (the actual key the real fix uses), and used Status().Update() against a CRD version with no status subresource, which unconditionally 404s - GetDataUploadForBackup could return before kubevirt_dataupload_controller had stamped the expected-backup-type annotation, racing the caller - virt-controller's VirtualMachineBackup status can permanently stop advancing after successfully attaching the backup target PVC (its attach branch returns without writing a status condition or requeuing, see kubevirt/kubevirt pkg/storage/cbt/backup.go startBackup()), tracked as CNV-85377/CNV-89684 and reported upstream with a fix at kubevirt/kubevirt#18949. Until that merges, runKubevirtDMBackup polls manually and nudges the VMI (a harmless annotation patch forcing a fresh watch event) whenever kdm-controller's logs show the frozen pattern, giving the stuck reconcile a real chance to recover instead of waiting out or retrying the whole timeout. Confirmed working across dozens of live hits on both GCP and Azure. If the nudge doesn't unstick it before the timeout, the spec marks pending (no ginkgo-level retry) rather than failing on a known, tracked upstream bug. - EnsureCommunityHcoCatalog/GetVirtOperator derived the OLM channel by guessing from the HCO index tag's numeric shape, which breaks for a moving tag like 'nightly' (channel has no relationship to the tag string). Channel is now discovered from the live PackageManifest instead, so hco_index_tag=nightly works through the existing community-HCO path -- letting future runs pick up kubevirt/kubevirt fixes (like #18949) as soon as they land in a nightly build, without pinning to a numbered release. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
kaovilai
added a commit
that referenced
this pull request
Aug 27, 2026
…idation Upstream blockers (kubevirt-datamover-controller#169, #73 phase 4) have landed, so this un-pends 'restore run-state flip is not blocked by a stale sibling DataDownload from a different restore attempt' and 'restore a multi-PVC VM from a kubevirt-datamover CBT backup'. Live validation against real AWS/GCP/Azure clusters surfaced and fixed several bugs along the way: - the decoy DataDownload used in the run-state-flip test correlated by restore-uid instead of restore-name (the actual key the real fix uses), and used Status().Update() against a CRD version with no status subresource, which unconditionally 404s - GetDataUploadForBackup could return before kubevirt_dataupload_controller had stamped the expected-backup-type annotation, racing the caller - virt-controller's VirtualMachineBackup status can permanently stop advancing after successfully attaching the backup target PVC (its attach branch returns without writing a status condition or requeuing, see kubevirt/kubevirt pkg/storage/cbt/backup.go startBackup()), tracked as CNV-85377/CNV-89684 and reported upstream with a fix at kubevirt/kubevirt#18949. Until that merges, runKubevirtDMBackup polls manually and nudges the VMI (a harmless annotation patch forcing a fresh watch event) whenever kdm-controller's logs show the frozen pattern, giving the stuck reconcile a real chance to recover instead of waiting out or retrying the whole timeout. Confirmed working across dozens of live hits on both GCP and Azure. If the nudge doesn't unstick it before the timeout, the spec marks pending (no ginkgo-level retry) rather than failing on a known, tracked upstream bug. - EnsureCommunityHcoCatalog/GetVirtOperator derived the OLM channel by guessing from the HCO index tag's numeric shape, which breaks for a moving tag like 'nightly' (channel has no relationship to the tag string). Channel is now discovered from the live PackageManifest instead (filtered by its catalog= label, since more than one CatalogSource can publish a manifest under the same package name -- getCsvFromPackageManifest's own separate, unfiltered lookup had the exact same bug, fixed the same way), so hco_index_tag=nightly works through the existing community-HCO path. - HCO_INDEX_TAG's Makefile default is now "nightly" instead of a pinned "1.18.0", so every virt/kdm e2e run (local and CI) picks up kubevirt/kubevirt fixes like #18949 automatically as soon as they land in a nightly build, with no version bump needed on our side. Override to a pinned release for a reproducible/stable run instead. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
kaovilai
added a commit
that referenced
this pull request
Aug 27, 2026
…idation Upstream blockers (kubevirt-datamover-controller#169, #73 phase 4) have landed, so this un-pends 'restore run-state flip is not blocked by a stale sibling DataDownload from a different restore attempt' and 'restore a multi-PVC VM from a kubevirt-datamover CBT backup'. Live validation against real AWS/GCP/Azure clusters surfaced and fixed several bugs along the way: - the decoy DataDownload used in the run-state-flip test correlated by restore-uid instead of restore-name (the actual key the real fix uses), and used Status().Update() against a CRD version with no status subresource, which unconditionally 404s - GetDataUploadForBackup could return before kubevirt_dataupload_controller had stamped the expected-backup-type annotation, racing the caller - virt-controller's VirtualMachineBackup status can permanently stop advancing after successfully attaching the backup target PVC (its attach branch returns without writing a status condition or requeuing, see kubevirt/kubevirt pkg/storage/cbt/backup.go startBackup()), tracked as CNV-85377/CNV-89684 and reported upstream with a fix at kubevirt/kubevirt#18949. Until that merges, runKubevirtDMBackup polls manually and nudges the VMI (a harmless annotation patch forcing a fresh watch event) whenever kdm-controller's logs show the frozen pattern, giving the stuck reconcile a real chance to recover instead of waiting out or retrying the whole timeout. Confirmed working across dozens of live hits on both GCP and Azure. If the nudge doesn't unstick it before the timeout, the spec marks pending (no ginkgo-level retry) rather than failing on a known, tracked upstream bug. - a second, distinct upstream bug found testing against kubevirt nightly: reconcileStart() (same file) can mark an already-successfully-completed VirtualMachineBackup Failed with reason SourceLost -- vmi.Status.ChangedBlockTracking.BackupStatus being nil is treated unconditionally as "status lost mid-flight", but virt-handler also clears that same field as part of normal post-completion cleanup. Introduced by kubevirt's June 2026 "observation-driven dispatch" restructure; distinct from the attach-freeze bug above. This only ever surfaces as a Kubernetes Event (never in any pod's log), so runKubevirtDMBackup's poll now also checks the VM namespace's own events, not just kdm-controller's log, for known-flake patterns. - EnsureCommunityHcoCatalog/GetVirtOperator derived the OLM channel by guessing from the HCO index tag's numeric shape, which breaks for a moving tag like 'nightly' (channel has no relationship to the tag string). Channel is now discovered from the live PackageManifest instead (filtered by its catalog= label, since more than one CatalogSource can publish a manifest under the same package name -- getCsvFromPackageManifest's own separate, unfiltered lookup had the exact same bug, fixed the same way), so hco_index_tag=nightly works through the existing community-HCO path. - HCO_INDEX_TAG's Makefile default is now "nightly" instead of a pinned "1.18.0", so every virt/kdm e2e run (local and CI) picks up kubevirt/kubevirt fixes like #18949 automatically as soon as they land in a nightly build, with no version bump needed on our side. Override to a pinned release for a reproducible/stable run instead. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
kaovilai
added a commit
that referenced
this pull request
Aug 27, 2026
…idation Upstream blockers (kubevirt-datamover-controller#169, #73 phase 4) have landed, so this un-pends 'restore run-state flip is not blocked by a stale sibling DataDownload from a different restore attempt' and 'restore a multi-PVC VM from a kubevirt-datamover CBT backup'. Live validation against real AWS/GCP/Azure clusters surfaced and fixed several bugs along the way: - the decoy DataDownload used in the run-state-flip test correlated by restore-uid instead of restore-name (the actual key the real fix uses), and used Status().Update() against a CRD version with no status subresource, which unconditionally 404s - GetDataUploadForBackup could return before kubevirt_dataupload_controller had stamped the expected-backup-type annotation, racing the caller - virt-controller's VirtualMachineBackup status can permanently stop advancing after successfully attaching the backup target PVC (its attach branch returns without writing a status condition or requeuing, see kubevirt/kubevirt pkg/storage/cbt/backup.go startBackup()), tracked as CNV-85377/CNV-89684 and reported upstream with a fix at kubevirt/kubevirt#18949. Until that merges, runKubevirtDMBackup polls manually and nudges the VMI (a harmless annotation patch forcing a fresh watch event) whenever kdm-controller's logs show the frozen pattern, giving the stuck reconcile a real chance to recover instead of waiting out or retrying the whole timeout. Confirmed working across dozens of live hits on both GCP and Azure. If the nudge doesn't unstick it before the timeout, the spec marks pending (no ginkgo-level retry) rather than failing on a known, tracked upstream bug. - a second, distinct upstream bug found testing against kubevirt nightly: reconcileStart() (same file) can mark an already-successfully-completed VirtualMachineBackup Failed with reason SourceLost -- vmi.Status.ChangedBlockTracking.BackupStatus being nil is treated unconditionally as "status lost mid-flight", but virt-handler also clears that same field as part of normal post-completion cleanup. Introduced by kubevirt's June 2026 "observation-driven dispatch" restructure; distinct from the attach-freeze bug above. This only ever surfaces as a Kubernetes Event (never in any pod's log), so runKubevirtDMBackup's poll now also checks the VM namespace's own events, not just kdm-controller's log, for known-flake patterns. - EnsureCommunityHcoCatalog/GetVirtOperator derived the OLM channel by guessing from the HCO index tag's numeric shape, which breaks for a moving tag like 'nightly' (channel has no relationship to the tag string). Channel is now discovered from the live PackageManifest instead (filtered by its catalog= label, since more than one CatalogSource can publish a manifest under the same package name -- getCsvFromPackageManifest's own separate, unfiltered lookup had the exact same bug, fixed the same way), so hco_index_tag=nightly works through the existing community-HCO path. - HCO_INDEX_TAG's Makefile default is now "nightly" instead of a pinned "1.18.0", so every virt/kdm e2e run (local and CI) picks up kubevirt/kubevirt fixes like #18949 automatically as soon as they land in a nightly build, with no version bump needed on our side. Override to a pinned release for a reproducible/stable run instead. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
kaovilai
added a commit
that referenced
this pull request
Aug 27, 2026
…idation Upstream blockers (kubevirt-datamover-controller#169, #73 phase 4) have landed, so this un-pends 'restore run-state flip is not blocked by a stale sibling DataDownload from a different restore attempt' and 'restore a multi-PVC VM from a kubevirt-datamover CBT backup'. Live validation against real AWS/GCP/Azure clusters surfaced and fixed several bugs along the way: - the decoy DataDownload used in the run-state-flip test correlated by restore-uid instead of restore-name (the actual key the real fix uses), and used Status().Update() against a CRD version with no status subresource, which unconditionally 404s - GetDataUploadForBackup could return before kubevirt_dataupload_controller had stamped the expected-backup-type annotation, racing the caller - virt-controller's VirtualMachineBackup status can permanently stop advancing after successfully attaching the backup target PVC (its attach branch returns without writing a status condition or requeuing, see kubevirt/kubevirt pkg/storage/cbt/backup.go startBackup()), tracked as CNV-85377/CNV-89684 and reported upstream with a fix at kubevirt/kubevirt#18949. Until that merges, runKubevirtDMBackup polls manually and nudges the VMI (a harmless annotation patch forcing a fresh watch event) whenever kdm-controller's logs show the frozen pattern, giving the stuck reconcile a real chance to recover instead of waiting out or retrying the whole timeout. Confirmed working across dozens of live hits on both GCP and Azure. If the nudge doesn't unstick it before the timeout, the spec marks pending (no ginkgo-level retry) rather than failing on a known, tracked upstream bug. - a second, distinct upstream bug found testing against kubevirt nightly: reconcileStart() (same file) can mark an already-successfully-completed VirtualMachineBackup Failed with reason SourceLost -- vmi.Status.ChangedBlockTracking.BackupStatus being nil is treated unconditionally as "status lost mid-flight", but virt-handler also clears that same field as part of normal post-completion cleanup. Introduced by kubevirt's June 2026 "observation-driven dispatch" restructure; distinct from the attach-freeze bug above. This only ever surfaces as a Kubernetes Event (never in any pod's log), so runKubevirtDMBackup's poll now also checks the VM namespace's own events, not just kdm-controller's log, for known-flake patterns. - EnsureCommunityHcoCatalog/GetVirtOperator derived the OLM channel by guessing from the HCO index tag's numeric shape, which breaks for a moving tag like 'nightly' (channel has no relationship to the tag string). Channel is now discovered from the live PackageManifest instead (filtered by its catalog= label, since more than one CatalogSource can publish a manifest under the same package name -- getCsvFromPackageManifest's own separate, unfiltered lookup had the exact same bug, fixed the same way), so hco_index_tag=nightly works through the existing community-HCO path. - HCO_INDEX_TAG's Makefile default is now "nightly" instead of a pinned "1.18.0", so every virt/kdm e2e run (local and CI) picks up kubevirt/kubevirt fixes like #18949 automatically as soon as they land in a nightly build, with no version bump needed on our side. Override to a pinned release for a reproducible/stable run instead. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
kaovilai
added a commit
that referenced
this pull request
Aug 27, 2026
…idation Upstream blockers (kubevirt-datamover-controller#169, #73 phase 4) have landed, so this un-pends 'restore run-state flip is not blocked by a stale sibling DataDownload from a different restore attempt' and 'restore a multi-PVC VM from a kubevirt-datamover CBT backup'. Live validation against real AWS/GCP/Azure clusters surfaced and fixed several bugs along the way: - the decoy DataDownload used in the run-state-flip test correlated by restore-uid instead of restore-name (the actual key the real fix uses), and used Status().Update() against a CRD version with no status subresource, which unconditionally 404s - GetDataUploadForBackup could return before kubevirt_dataupload_controller had stamped the expected-backup-type annotation, racing the caller - virt-controller's VirtualMachineBackup status can permanently stop advancing after successfully attaching the backup target PVC (its attach branch returns without writing a status condition or requeuing, see kubevirt/kubevirt pkg/storage/cbt/backup.go startBackup()), tracked as CNV-85377/CNV-89684 and reported upstream with a fix at kubevirt/kubevirt#18949. Until that merges, runKubevirtDMBackup polls manually and nudges the VMI (a harmless annotation patch forcing a fresh watch event) whenever kdm-controller's logs show the frozen pattern, giving the stuck reconcile a real chance to recover instead of waiting out or retrying the whole timeout. Confirmed working across dozens of live hits on both GCP and Azure. If the nudge doesn't unstick it before the timeout, the spec marks pending (no ginkgo-level retry) rather than failing on a known, tracked upstream bug. - a second, distinct upstream bug found testing against kubevirt nightly: reconcileStart() (same file) can mark an already-successfully-completed VirtualMachineBackup Failed with reason SourceLost -- vmi.Status.ChangedBlockTracking.BackupStatus being nil is treated unconditionally as "status lost mid-flight", but virt-handler also clears that same field as part of normal post-completion cleanup. Introduced by kubevirt's June 2026 "observation-driven dispatch" restructure; distinct from the attach-freeze bug above. This only ever surfaces as a Kubernetes Event (never in any pod's log), so runKubevirtDMBackup's poll now also checks the VM namespace's own events, not just kdm-controller's log, for known-flake patterns. - EnsureCommunityHcoCatalog/GetVirtOperator derived the OLM channel by guessing from the HCO index tag's numeric shape, which breaks for a moving tag like 'nightly' (channel has no relationship to the tag string). Channel is now discovered from the live PackageManifest instead (filtered by its catalog= label, since more than one CatalogSource can publish a manifest under the same package name -- getCsvFromPackageManifest's own separate, unfiltered lookup had the exact same bug, fixed the same way), so hco_index_tag=nightly works through the existing community-HCO path. - HCO_INDEX_TAG's Makefile default is now "nightly" instead of a pinned "1.18.0", so every virt/kdm e2e run (local and CI) picks up kubevirt/kubevirt fixes like #18949 automatically as soon as they land in a nightly build, with no version bump needed on our side. Override to a pinned release for a reproducible/stable run instead. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
kaovilai
added a commit
that referenced
this pull request
Aug 27, 2026
…idation Upstream blockers (kubevirt-datamover-controller#169, #73 phase 4) have landed, so this un-pends 'restore run-state flip is not blocked by a stale sibling DataDownload from a different restore attempt' and 'restore a multi-PVC VM from a kubevirt-datamover CBT backup'. Live validation against real AWS/GCP/Azure clusters surfaced and fixed several bugs along the way: - the decoy DataDownload used in the run-state-flip test correlated by restore-uid instead of restore-name (the actual key the real fix uses), and used Status().Update() against a CRD version with no status subresource, which unconditionally 404s - GetDataUploadForBackup could return before kubevirt_dataupload_controller had stamped the expected-backup-type annotation, racing the caller - virt-controller's VirtualMachineBackup status can permanently stop advancing after successfully attaching the backup target PVC (its attach branch returns without writing a status condition or requeuing, see kubevirt/kubevirt pkg/storage/cbt/backup.go startBackup()), tracked as CNV-85377/CNV-89684 and reported upstream with a fix at kubevirt/kubevirt#18949. Until that merges, runKubevirtDMBackup polls manually and nudges the VMI (a harmless annotation patch forcing a fresh watch event) whenever kdm-controller's logs show the frozen pattern, giving the stuck reconcile a real chance to recover instead of waiting out or retrying the whole timeout. Confirmed working across dozens of live hits on both GCP and Azure. If the nudge doesn't unstick it before the timeout, the spec marks pending (no ginkgo-level retry) rather than failing on a known, tracked upstream bug. - a second, distinct upstream bug found testing against kubevirt nightly: reconcileStart() (same file) can mark an already-successfully-completed VirtualMachineBackup Failed with reason SourceLost -- vmi.Status.ChangedBlockTracking.BackupStatus being nil is treated unconditionally as "status lost mid-flight", but virt-handler also clears that same field as part of normal post-completion cleanup. Introduced by kubevirt's June 2026 "observation-driven dispatch" restructure; distinct from the attach-freeze bug above. This only ever surfaces as a Kubernetes Event (never in any pod's log), so runKubevirtDMBackup's poll now also checks the VM namespace's own events, not just kdm-controller's log, for known-flake patterns. - EnsureCommunityHcoCatalog/GetVirtOperator derived the OLM channel by guessing from the HCO index tag's numeric shape, which breaks for a moving tag like 'nightly' (channel has no relationship to the tag string). Channel is now discovered from the live PackageManifest instead (filtered by its catalog= label, since more than one CatalogSource can publish a manifest under the same package name -- getCsvFromPackageManifest's own separate, unfiltered lookup had the exact same bug, fixed the same way), so hco_index_tag=nightly works through the existing community-HCO path. - HCO_INDEX_TAG's Makefile default is now "nightly" instead of a pinned "1.18.0", so every virt/kdm e2e run (local and CI) picks up kubevirt/kubevirt fixes like #18949 automatically as soon as they land in a nightly build, with no version bump needed on our side. Override to a pinned release for a reproducible/stable run instead. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
kaovilai
added a commit
that referenced
this pull request
Aug 27, 2026
…idation Upstream blockers (kubevirt-datamover-controller#169, #73 phase 4) have landed, so this un-pends 'restore run-state flip is not blocked by a stale sibling DataDownload from a different restore attempt' and 'restore a multi-PVC VM from a kubevirt-datamover CBT backup'. Live validation against real AWS/GCP/Azure clusters surfaced and fixed several bugs along the way: - the decoy DataDownload used in the run-state-flip test correlated by restore-uid instead of restore-name (the actual key the real fix uses), and used Status().Update() against a CRD version with no status subresource, which unconditionally 404s - GetDataUploadForBackup could return before kubevirt_dataupload_controller had stamped the expected-backup-type annotation, racing the caller - virt-controller's VirtualMachineBackup status can permanently stop advancing after successfully attaching the backup target PVC (its attach branch returns without writing a status condition or requeuing, see kubevirt/kubevirt pkg/storage/cbt/backup.go startBackup()), tracked as CNV-85377/CNV-89684 and reported upstream with a fix at kubevirt/kubevirt#18949. Until that merges, runKubevirtDMBackup polls manually and nudges the VMI (a harmless annotation patch forcing a fresh watch event) whenever kdm-controller's logs show the frozen pattern, giving the stuck reconcile a real chance to recover instead of waiting out or retrying the whole timeout. Confirmed working across dozens of live hits on both GCP and Azure. If the nudge doesn't unstick it before the timeout, the spec marks pending (no ginkgo-level retry) rather than failing on a known, tracked upstream bug. - a second, distinct upstream bug found testing against kubevirt nightly: reconcileStart() (same file) can mark an already-successfully-completed VirtualMachineBackup Failed with reason SourceLost -- vmi.Status.ChangedBlockTracking.BackupStatus being nil is treated unconditionally as "status lost mid-flight", but virt-handler also clears that same field as part of normal post-completion cleanup. Introduced by kubevirt's June 2026 "observation-driven dispatch" restructure; distinct from the attach-freeze bug above. This only ever surfaces as a Kubernetes Event (never in any pod's log), so runKubevirtDMBackup's poll now also checks the VM namespace's own events, not just kdm-controller's log, for known-flake patterns. - EnsureCommunityHcoCatalog/GetVirtOperator derived the OLM channel by guessing from the HCO index tag's numeric shape, which breaks for a moving tag like 'nightly' (channel has no relationship to the tag string). Channel is now discovered from the live PackageManifest instead (filtered by its catalog= label, since more than one CatalogSource can publish a manifest under the same package name -- getCsvFromPackageManifest's own separate, unfiltered lookup had the exact same bug, fixed the same way), so hco_index_tag=nightly works through the existing community-HCO path. - HCO_INDEX_TAG's Makefile default is now "nightly" instead of a pinned "1.18.0", so every virt/kdm e2e run (local and CI) picks up kubevirt/kubevirt fixes like #18949 automatically as soon as they land in a nightly build, with no version bump needed on our side. Override to a pinned release for a reproducible/stable run instead. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
kaovilai
added a commit
that referenced
this pull request
Aug 27, 2026
…idation Upstream blockers (kubevirt-datamover-controller#169, #73 phase 4) have landed, so this un-pends 'restore run-state flip is not blocked by a stale sibling DataDownload from a different restore attempt' and 'restore a multi-PVC VM from a kubevirt-datamover CBT backup'. Live validation against real AWS/GCP/Azure clusters surfaced and fixed several bugs along the way: - the decoy DataDownload used in the run-state-flip test correlated by restore-uid instead of restore-name (the actual key the real fix uses), and used Status().Update() against a CRD version with no status subresource, which unconditionally 404s - GetDataUploadForBackup could return before kubevirt_dataupload_controller had stamped the expected-backup-type annotation, racing the caller - virt-controller's VirtualMachineBackup status can permanently stop advancing after successfully attaching the backup target PVC (its attach branch returns without writing a status condition or requeuing, see kubevirt/kubevirt pkg/storage/cbt/backup.go startBackup()), tracked as CNV-85377/CNV-89684 and reported upstream with a fix at kubevirt/kubevirt#18949. Until that merges, runKubevirtDMBackup polls manually and nudges the VMI (a harmless annotation patch forcing a fresh watch event) whenever kdm-controller's logs show the frozen pattern, giving the stuck reconcile a real chance to recover instead of waiting out or retrying the whole timeout. Confirmed working across dozens of live hits on both GCP and Azure. If the nudge doesn't unstick it before the timeout, the spec marks pending (no ginkgo-level retry) rather than failing on a known, tracked upstream bug. - a second, distinct upstream bug found testing against kubevirt nightly: reconcileStart() (same file) can mark an already-successfully-completed VirtualMachineBackup Failed with reason SourceLost -- vmi.Status.ChangedBlockTracking.BackupStatus being nil is treated unconditionally as "status lost mid-flight", but virt-handler also clears that same field as part of normal post-completion cleanup. Introduced by kubevirt's June 2026 "observation-driven dispatch" restructure; distinct from the attach-freeze bug above. This only ever surfaces as a Kubernetes Event (never in any pod's log), so runKubevirtDMBackup's poll now also checks the VM namespace's own events, not just kdm-controller's log, for known-flake patterns. - EnsureCommunityHcoCatalog/GetVirtOperator derived the OLM channel by guessing from the HCO index tag's numeric shape, which breaks for a moving tag like 'nightly' (channel has no relationship to the tag string). Channel is now discovered from the live PackageManifest instead (filtered by its catalog= label, since more than one CatalogSource can publish a manifest under the same package name -- getCsvFromPackageManifest's own separate, unfiltered lookup had the exact same bug, fixed the same way), so hco_index_tag=nightly works through the existing community-HCO path. - HCO_INDEX_TAG's Makefile default is now "nightly" instead of a pinned "1.18.0", so every virt/kdm e2e run (local and CI) picks up kubevirt/kubevirt fixes like #18949 automatically as soon as they land in a nightly build, with no version bump needed on our side. Override to a pinned release for a reproducible/stable run instead. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
kaovilai
added a commit
that referenced
this pull request
Aug 27, 2026
…idation Upstream blockers (kubevirt-datamover-controller#169, #73 phase 4) have landed, so this un-pends 'restore run-state flip is not blocked by a stale sibling DataDownload from a different restore attempt' and 'restore a multi-PVC VM from a kubevirt-datamover CBT backup'. Live validation against real AWS/GCP/Azure clusters surfaced and fixed several bugs along the way: - the decoy DataDownload used in the run-state-flip test correlated by restore-uid instead of restore-name (the actual key the real fix uses), and used Status().Update() against a CRD version with no status subresource, which unconditionally 404s - GetDataUploadForBackup could return before kubevirt_dataupload_controller had stamped the expected-backup-type annotation, racing the caller - virt-controller's VirtualMachineBackup status can permanently stop advancing after successfully attaching the backup target PVC (its attach branch returns without writing a status condition or requeuing, see kubevirt/kubevirt pkg/storage/cbt/backup.go startBackup()), tracked as CNV-85377/CNV-89684 and reported upstream with a fix at kubevirt/kubevirt#18949. Until that merges, runKubevirtDMBackup polls manually and nudges the VMI (a harmless annotation patch forcing a fresh watch event) whenever kdm-controller's logs show the frozen pattern, giving the stuck reconcile a real chance to recover instead of waiting out or retrying the whole timeout. Confirmed working across dozens of live hits on both GCP and Azure. If the nudge doesn't unstick it before the timeout, the spec marks pending (no ginkgo-level retry) rather than failing on a known, tracked upstream bug. - a second, distinct upstream bug found testing against kubevirt nightly: reconcileStart() (same file) can mark an already-successfully-completed VirtualMachineBackup Failed with reason SourceLost -- vmi.Status.ChangedBlockTracking.BackupStatus being nil is treated unconditionally as "status lost mid-flight", but virt-handler also clears that same field as part of normal post-completion cleanup. Introduced by kubevirt's June 2026 "observation-driven dispatch" restructure; distinct from the attach-freeze bug above. This only ever surfaces as a Kubernetes Event (never in any pod's log), so runKubevirtDMBackup's poll now also checks the VM namespace's own events, not just kdm-controller's log, for known-flake patterns. - EnsureCommunityHcoCatalog/GetVirtOperator derived the OLM channel by guessing from the HCO index tag's numeric shape, which breaks for a moving tag like 'nightly' (channel has no relationship to the tag string). Channel is now discovered from the live PackageManifest instead (filtered by its catalog= label, since more than one CatalogSource can publish a manifest under the same package name -- getCsvFromPackageManifest's own separate, unfiltered lookup had the exact same bug, fixed the same way), so hco_index_tag=nightly works through the existing community-HCO path. - HCO_INDEX_TAG's Makefile default is now "nightly" instead of a pinned "1.18.0", so every virt/kdm e2e run (local and CI) picks up kubevirt/kubevirt fixes like #18949 automatically as soon as they land in a nightly build, with no version bump needed on our side. Override to a pinned release for a reproducible/stable run instead. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
kaovilai
added a commit
that referenced
this pull request
Aug 28, 2026
…idation Upstream blockers (kubevirt-datamover-controller#169, #73 phase 4) have landed, so this un-pends 'restore run-state flip is not blocked by a stale sibling DataDownload from a different restore attempt' and 'restore a multi-PVC VM from a kubevirt-datamover CBT backup'. Live validation against real AWS/GCP/Azure clusters surfaced and fixed several bugs along the way: - the decoy DataDownload used in the run-state-flip test correlated by restore-uid instead of restore-name (the actual key the real fix uses), and used Status().Update() against a CRD version with no status subresource, which unconditionally 404s - GetDataUploadForBackup could return before kubevirt_dataupload_controller had stamped the expected-backup-type annotation, racing the caller - virt-controller's VirtualMachineBackup status can permanently stop advancing after successfully attaching the backup target PVC (its attach branch returns without writing a status condition or requeuing, see kubevirt/kubevirt pkg/storage/cbt/backup.go startBackup()), tracked as CNV-85377/CNV-89684 and reported upstream with a fix at kubevirt/kubevirt#18949. Until that merges, runKubevirtDMBackup polls manually and nudges the VMI (a harmless annotation patch forcing a fresh watch event) whenever kdm-controller's logs show the frozen pattern, giving the stuck reconcile a real chance to recover instead of waiting out or retrying the whole timeout. Confirmed working across dozens of live hits on both GCP and Azure. If the nudge doesn't unstick it before the timeout, the spec marks pending (no ginkgo-level retry) rather than failing on a known, tracked upstream bug. - a second, distinct upstream bug found testing against kubevirt nightly: reconcileStart() (same file) can mark an already-successfully-completed VirtualMachineBackup Failed with reason SourceLost -- vmi.Status.ChangedBlockTracking.BackupStatus being nil is treated unconditionally as "status lost mid-flight", but virt-handler also clears that same field as part of normal post-completion cleanup. Introduced by kubevirt's June 2026 "observation-driven dispatch" restructure; distinct from the attach-freeze bug above. This only ever surfaces as a Kubernetes Event (never in any pod's log), so runKubevirtDMBackup's poll now also checks the VM namespace's own events, not just kdm-controller's log, for known-flake patterns. - EnsureCommunityHcoCatalog/GetVirtOperator derived the OLM channel by guessing from the HCO index tag's numeric shape, which breaks for a moving tag like 'nightly' (channel has no relationship to the tag string). Channel is now discovered from the live PackageManifest instead (filtered by its catalog= label, since more than one CatalogSource can publish a manifest under the same package name -- getCsvFromPackageManifest's own separate, unfiltered lookup had the exact same bug, fixed the same way), so hco_index_tag=nightly works through the existing community-HCO path. - HCO_INDEX_TAG's Makefile default is now "nightly" instead of a pinned "1.18.0", so every virt/kdm e2e run (local and CI) picks up kubevirt/kubevirt fixes like #18949 automatically as soon as they land in a nightly build, with no version bump needed on our side. Override to a pinned release for a reproducible/stable run instead. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
kaovilai
added a commit
that referenced
this pull request
Aug 28, 2026
…idation Upstream blockers (kubevirt-datamover-controller#169, #73 phase 4) have landed, so this un-pends 'restore run-state flip is not blocked by a stale sibling DataDownload from a different restore attempt' and 'restore a multi-PVC VM from a kubevirt-datamover CBT backup'. Live validation against real AWS/GCP/Azure clusters surfaced and fixed several bugs along the way: - the decoy DataDownload used in the run-state-flip test correlated by restore-uid instead of restore-name (the actual key the real fix uses), and used Status().Update() against a CRD version with no status subresource, which unconditionally 404s - GetDataUploadForBackup could return before kubevirt_dataupload_controller had stamped the expected-backup-type annotation, racing the caller - virt-controller's VirtualMachineBackup status can permanently stop advancing after successfully attaching the backup target PVC (its attach branch returns without writing a status condition or requeuing, see kubevirt/kubevirt pkg/storage/cbt/backup.go startBackup()), tracked as CNV-85377/CNV-89684 and reported upstream with a fix at kubevirt/kubevirt#18949. Until that merges, runKubevirtDMBackup polls manually and nudges the VMI (a harmless annotation patch forcing a fresh watch event) whenever kdm-controller's logs show the frozen pattern, giving the stuck reconcile a real chance to recover instead of waiting out or retrying the whole timeout. Confirmed working across dozens of live hits on both GCP and Azure. If the nudge doesn't unstick it before the timeout, the spec marks pending (no ginkgo-level retry) rather than failing on a known, tracked upstream bug. - a second, distinct upstream bug found testing against kubevirt nightly: reconcileStart() (same file) can mark an already-successfully-completed VirtualMachineBackup Failed with reason SourceLost -- vmi.Status.ChangedBlockTracking.BackupStatus being nil is treated unconditionally as "status lost mid-flight", but virt-handler also clears that same field as part of normal post-completion cleanup. Introduced by kubevirt's June 2026 "observation-driven dispatch" restructure; distinct from the attach-freeze bug above. This only ever surfaces as a Kubernetes Event (never in any pod's log), so runKubevirtDMBackup's poll now also checks the VM namespace's own events, not just kdm-controller's log, for known-flake patterns. - EnsureCommunityHcoCatalog/GetVirtOperator derived the OLM channel by guessing from the HCO index tag's numeric shape, which breaks for a moving tag like 'nightly' (channel has no relationship to the tag string). Channel is now discovered from the live PackageManifest instead (filtered by its catalog= label, since more than one CatalogSource can publish a manifest under the same package name -- getCsvFromPackageManifest's own separate, unfiltered lookup had the exact same bug, fixed the same way), so hco_index_tag=nightly works through the existing community-HCO path. - HCO_INDEX_TAG's Makefile default is now "nightly" instead of a pinned "1.18.0", so every virt/kdm e2e run (local and CI) picks up kubevirt/kubevirt fixes like #18949 automatically as soon as they land in a nightly build, with no version bump needed on our side. Override to a pinned release for a reproducible/stable run instead. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
kaovilai
added a commit
that referenced
this pull request
Aug 28, 2026
…idation Upstream blockers (kubevirt-datamover-controller#169, #73 phase 4) have landed, so this un-pends 'restore run-state flip is not blocked by a stale sibling DataDownload from a different restore attempt' and 'restore a multi-PVC VM from a kubevirt-datamover CBT backup'. Live validation against real AWS/GCP/Azure clusters surfaced and fixed several bugs along the way: - the decoy DataDownload used in the run-state-flip test correlated by restore-uid instead of restore-name (the actual key the real fix uses), and used Status().Update() against a CRD version with no status subresource, which unconditionally 404s - GetDataUploadForBackup could return before kubevirt_dataupload_controller had stamped the expected-backup-type annotation, racing the caller - virt-controller's VirtualMachineBackup status can permanently stop advancing after successfully attaching the backup target PVC (its attach branch returns without writing a status condition or requeuing, see kubevirt/kubevirt pkg/storage/cbt/backup.go startBackup()), tracked as CNV-85377/CNV-89684 and reported upstream with a fix at kubevirt/kubevirt#18949. Until that merges, runKubevirtDMBackup polls manually and nudges the VMI (a harmless annotation patch forcing a fresh watch event) whenever kdm-controller's logs show the frozen pattern, giving the stuck reconcile a real chance to recover instead of waiting out or retrying the whole timeout. Confirmed working across dozens of live hits on both GCP and Azure. If the nudge doesn't unstick it before the timeout, the spec marks pending (no ginkgo-level retry) rather than failing on a known, tracked upstream bug. - a second, distinct upstream bug found testing against kubevirt nightly: reconcileStart() (same file) can mark an already-successfully-completed VirtualMachineBackup Failed with reason SourceLost -- vmi.Status.ChangedBlockTracking.BackupStatus being nil is treated unconditionally as "status lost mid-flight", but virt-handler also clears that same field as part of normal post-completion cleanup. Introduced by kubevirt's June 2026 "observation-driven dispatch" restructure; distinct from the attach-freeze bug above. This only ever surfaces as a Kubernetes Event (never in any pod's log), so runKubevirtDMBackup's poll now also checks the VM namespace's own events, not just kdm-controller's log, for known-flake patterns. - EnsureCommunityHcoCatalog/GetVirtOperator derived the OLM channel by guessing from the HCO index tag's numeric shape, which breaks for a moving tag like 'nightly' (channel has no relationship to the tag string). Channel is now discovered from the live PackageManifest instead (filtered by its catalog= label, since more than one CatalogSource can publish a manifest under the same package name -- getCsvFromPackageManifest's own separate, unfiltered lookup had the exact same bug, fixed the same way), so hco_index_tag=nightly works through the existing community-HCO path. - HCO_INDEX_TAG's Makefile default is now "nightly" instead of a pinned "1.18.0", so every virt/kdm e2e run (local and CI) picks up kubevirt/kubevirt fixes like #18949 automatically as soon as they land in a nightly build, with no version bump needed on our side. Override to a pinned release for a reproducible/stable run instead. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
kaovilai
added a commit
that referenced
this pull request
Aug 28, 2026
…idation Upstream blockers (kubevirt-datamover-controller#169, #73 phase 4) have landed, so this un-pends 'restore run-state flip is not blocked by a stale sibling DataDownload from a different restore attempt' and 'restore a multi-PVC VM from a kubevirt-datamover CBT backup'. Live validation against real AWS/GCP/Azure clusters surfaced and fixed several bugs along the way: - the decoy DataDownload used in the run-state-flip test correlated by restore-uid instead of restore-name (the actual key the real fix uses), and used Status().Update() against a CRD version with no status subresource, which unconditionally 404s - GetDataUploadForBackup could return before kubevirt_dataupload_controller had stamped the expected-backup-type annotation, racing the caller - virt-controller's VirtualMachineBackup status can permanently stop advancing after successfully attaching the backup target PVC (its attach branch returns without writing a status condition or requeuing, see kubevirt/kubevirt pkg/storage/cbt/backup.go startBackup()), tracked as CNV-85377/CNV-89684 and reported upstream with a fix at kubevirt/kubevirt#18949. Until that merges, runKubevirtDMBackup polls manually and nudges the VMI (a harmless annotation patch forcing a fresh watch event) whenever kdm-controller's logs show the frozen pattern, giving the stuck reconcile a real chance to recover instead of waiting out or retrying the whole timeout. Confirmed working across dozens of live hits on both GCP and Azure. If the nudge doesn't unstick it before the timeout, the spec marks pending (no ginkgo-level retry) rather than failing on a known, tracked upstream bug. - a second, distinct upstream bug found testing against kubevirt nightly: reconcileStart() (same file) can mark an already-successfully-completed VirtualMachineBackup Failed with reason SourceLost -- vmi.Status.ChangedBlockTracking.BackupStatus being nil is treated unconditionally as "status lost mid-flight", but virt-handler also clears that same field as part of normal post-completion cleanup. Introduced by kubevirt's June 2026 "observation-driven dispatch" restructure; distinct from the attach-freeze bug above. This only ever surfaces as a Kubernetes Event (never in any pod's log), so runKubevirtDMBackup's poll now also checks the VM namespace's own events, not just kdm-controller's log, for known-flake patterns. - EnsureCommunityHcoCatalog/GetVirtOperator derived the OLM channel by guessing from the HCO index tag's numeric shape, which breaks for a moving tag like 'nightly' (channel has no relationship to the tag string). Channel is now discovered from the live PackageManifest instead (filtered by its catalog= label, since more than one CatalogSource can publish a manifest under the same package name -- getCsvFromPackageManifest's own separate, unfiltered lookup had the exact same bug, fixed the same way), so hco_index_tag=nightly works through the existing community-HCO path. - HCO_INDEX_TAG's Makefile default is now "nightly" instead of a pinned "1.18.0", so every virt/kdm e2e run (local and CI) picks up kubevirt/kubevirt fixes like #18949 automatically as soon as they land in a nightly build, with no version bump needed on our side. Override to a pinned release for a reproducible/stable run instead. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
kaovilai
added a commit
that referenced
this pull request
Aug 28, 2026
…idation Upstream blockers (kubevirt-datamover-controller#169, #73 phase 4) have landed, so this un-pends 'restore run-state flip is not blocked by a stale sibling DataDownload from a different restore attempt' and 'restore a multi-PVC VM from a kubevirt-datamover CBT backup'. Live validation against real AWS/GCP/Azure clusters surfaced and fixed several bugs along the way: - the decoy DataDownload used in the run-state-flip test correlated by restore-uid instead of restore-name (the actual key the real fix uses), and used Status().Update() against a CRD version with no status subresource, which unconditionally 404s - GetDataUploadForBackup could return before kubevirt_dataupload_controller had stamped the expected-backup-type annotation, racing the caller - virt-controller's VirtualMachineBackup status can permanently stop advancing after successfully attaching the backup target PVC (its attach branch returns without writing a status condition or requeuing, see kubevirt/kubevirt pkg/storage/cbt/backup.go startBackup()), tracked as CNV-85377/CNV-89684 and reported upstream with a fix at kubevirt/kubevirt#18949. Until that merges, runKubevirtDMBackup polls manually and nudges the VMI (a harmless annotation patch forcing a fresh watch event) whenever kdm-controller's logs show the frozen pattern, giving the stuck reconcile a real chance to recover instead of waiting out or retrying the whole timeout. Confirmed working across dozens of live hits on both GCP and Azure. If the nudge doesn't unstick it before the timeout, the spec marks pending (no ginkgo-level retry) rather than failing on a known, tracked upstream bug. - a second, distinct upstream bug found testing against kubevirt nightly: reconcileStart() (same file) can mark an already-successfully-completed VirtualMachineBackup Failed with reason SourceLost -- vmi.Status.ChangedBlockTracking.BackupStatus being nil is treated unconditionally as "status lost mid-flight", but virt-handler also clears that same field as part of normal post-completion cleanup. Introduced by kubevirt's June 2026 "observation-driven dispatch" restructure; distinct from the attach-freeze bug above. This only ever surfaces as a Kubernetes Event (never in any pod's log), so runKubevirtDMBackup's poll now also checks the VM namespace's own events, not just kdm-controller's log, for known-flake patterns. - EnsureCommunityHcoCatalog/GetVirtOperator derived the OLM channel by guessing from the HCO index tag's numeric shape, which breaks for a moving tag like 'nightly' (channel has no relationship to the tag string). Channel is now discovered from the live PackageManifest instead (filtered by its catalog= label, since more than one CatalogSource can publish a manifest under the same package name -- getCsvFromPackageManifest's own separate, unfiltered lookup had the exact same bug, fixed the same way), so hco_index_tag=nightly works through the existing community-HCO path. - HCO_INDEX_TAG's Makefile default is now "nightly" instead of a pinned "1.18.0", so every virt/kdm e2e run (local and CI) picks up kubevirt/kubevirt fixes like #18949 automatically as soon as they land in a nightly build, with no version bump needed on our side. Override to a pinned release for a reproducible/stable run instead. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
kaovilai
added a commit
that referenced
this pull request
Aug 29, 2026
…idation Upstream blockers (kubevirt-datamover-controller#169, #73 phase 4) have landed, so this un-pends 'restore run-state flip is not blocked by a stale sibling DataDownload from a different restore attempt' and 'restore a multi-PVC VM from a kubevirt-datamover CBT backup'. Live validation against real AWS/GCP/Azure clusters surfaced and fixed several bugs along the way: - the decoy DataDownload used in the run-state-flip test correlated by restore-uid instead of restore-name (the actual key the real fix uses), and used Status().Update() against a CRD version with no status subresource, which unconditionally 404s - GetDataUploadForBackup could return before kubevirt_dataupload_controller had stamped the expected-backup-type annotation, racing the caller - virt-controller's VirtualMachineBackup status can permanently stop advancing after successfully attaching the backup target PVC (its attach branch returns without writing a status condition or requeuing, see kubevirt/kubevirt pkg/storage/cbt/backup.go startBackup()), tracked as CNV-85377/CNV-89684 and reported upstream with a fix at kubevirt/kubevirt#18949. Until that merges, runKubevirtDMBackup polls manually and nudges the VMI (a harmless annotation patch forcing a fresh watch event) whenever kdm-controller's logs show the frozen pattern, giving the stuck reconcile a real chance to recover instead of waiting out or retrying the whole timeout. Confirmed working across dozens of live hits on both GCP and Azure. If the nudge doesn't unstick it before the timeout, the spec marks pending (no ginkgo-level retry) rather than failing on a known, tracked upstream bug. - a second, distinct upstream bug found testing against kubevirt nightly: reconcileStart() (same file) can mark an already-successfully-completed VirtualMachineBackup Failed with reason SourceLost -- vmi.Status.ChangedBlockTracking.BackupStatus being nil is treated unconditionally as "status lost mid-flight", but virt-handler also clears that same field as part of normal post-completion cleanup. Introduced by kubevirt's June 2026 "observation-driven dispatch" restructure; distinct from the attach-freeze bug above. This only ever surfaces as a Kubernetes Event (never in any pod's log), so runKubevirtDMBackup's poll now also checks the VM namespace's own events, not just kdm-controller's log, for known-flake patterns. - EnsureCommunityHcoCatalog/GetVirtOperator derived the OLM channel by guessing from the HCO index tag's numeric shape, which breaks for a moving tag like 'nightly' (channel has no relationship to the tag string). Channel is now discovered from the live PackageManifest instead (filtered by its catalog= label, since more than one CatalogSource can publish a manifest under the same package name -- getCsvFromPackageManifest's own separate, unfiltered lookup had the exact same bug, fixed the same way), so hco_index_tag=nightly works through the existing community-HCO path. - HCO_INDEX_TAG's Makefile default is now "nightly" instead of a pinned "1.18.0", so every virt/kdm e2e run (local and CI) picks up kubevirt/kubevirt fixes like #18949 automatically as soon as they land in a nightly build, with no version bump needed on our side. Override to a pinned release for a reproducible/stable run instead. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
kaovilai
added a commit
that referenced
this pull request
Aug 29, 2026
…idation Upstream blockers (kubevirt-datamover-controller#169, #73 phase 4) have landed, so this un-pends 'restore run-state flip is not blocked by a stale sibling DataDownload from a different restore attempt' and 'restore a multi-PVC VM from a kubevirt-datamover CBT backup'. Live validation against real AWS/GCP/Azure clusters surfaced and fixed several bugs along the way: - the decoy DataDownload used in the run-state-flip test correlated by restore-uid instead of restore-name (the actual key the real fix uses), and used Status().Update() against a CRD version with no status subresource, which unconditionally 404s - GetDataUploadForBackup could return before kubevirt_dataupload_controller had stamped the expected-backup-type annotation, racing the caller - virt-controller's VirtualMachineBackup status can permanently stop advancing after successfully attaching the backup target PVC (its attach branch returns without writing a status condition or requeuing, see kubevirt/kubevirt pkg/storage/cbt/backup.go startBackup()), tracked as CNV-85377/CNV-89684 and reported upstream with a fix at kubevirt/kubevirt#18949. Until that merges, runKubevirtDMBackup polls manually and nudges the VMI (a harmless annotation patch forcing a fresh watch event) whenever kdm-controller's logs show the frozen pattern, giving the stuck reconcile a real chance to recover instead of waiting out or retrying the whole timeout. Confirmed working across dozens of live hits on both GCP and Azure. If the nudge doesn't unstick it before the timeout, the spec marks pending (no ginkgo-level retry) rather than failing on a known, tracked upstream bug. - a second, distinct upstream bug found testing against kubevirt nightly: reconcileStart() (same file) can mark an already-successfully-completed VirtualMachineBackup Failed with reason SourceLost -- vmi.Status.ChangedBlockTracking.BackupStatus being nil is treated unconditionally as "status lost mid-flight", but virt-handler also clears that same field as part of normal post-completion cleanup. Introduced by kubevirt's June 2026 "observation-driven dispatch" restructure; distinct from the attach-freeze bug above. This only ever surfaces as a Kubernetes Event (never in any pod's log), so runKubevirtDMBackup's poll now also checks the VM namespace's own events, not just kdm-controller's log, for known-flake patterns. - EnsureCommunityHcoCatalog/GetVirtOperator derived the OLM channel by guessing from the HCO index tag's numeric shape, which breaks for a moving tag like 'nightly' (channel has no relationship to the tag string). Channel is now discovered from the live PackageManifest instead (filtered by its catalog= label, since more than one CatalogSource can publish a manifest under the same package name -- getCsvFromPackageManifest's own separate, unfiltered lookup had the exact same bug, fixed the same way), so hco_index_tag=nightly works through the existing community-HCO path. - HCO_INDEX_TAG's Makefile default is now "nightly" instead of a pinned "1.18.0", so every virt/kdm e2e run (local and CI) picks up kubevirt/kubevirt fixes like #18949 automatically as soon as they land in a nightly build, with no version bump needed on our side. Override to a pinned release for a reproducible/stable run instead. Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.