PVC Image Upload Volume Node Affinity Conflict

[CodeBeholderKASM]

I have KubeVirt and CDI running on OKE, but when attempting to use virtctl to do an image-upload of a PVC the upload pod reports this:

  - lastProbeTime: null
    lastTransitionTime: "2025-02-10T18:49:37Z"
    message: '0/3 nodes are available: 3 node(s) had volume node affinity conflict.
      preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.'
    reason: Unschedulable
    status: "False"
    type: PodScheduled

I followed the advice of this ticket (#3287) and changed my storageclass to be waitForFirstConsumer with the annotation mentioned in the comment, but my PV are still assigned to different nodes.

storageclass:

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
    cdi.kubevirt.io/storage.bind.immediate.requested: "true"
  name: kasm-storage
provisioner: blockvolume.csi.oraclecloud.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

The PVs:

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                     STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
csi-d0dbc61d-b8c5-48a4-9c6d-c74f072d9c0d   50Gi       RWO            Delete           Bound    cdi/kasm-ubuntu           kasm-storage   <unset>                          153m
csi-e7bf2880-5d50-4227-aca0-e91bfb22b42b   50Gi       RWO            Delete           Bound    cdi/kasm-ubuntu-scratch   kasm-storage   <unset>                          153m

Thanks for the help!

k8s version:

$ kubectl version
Client Version: v1.31.1
Kustomize Version: v5.4.2
Server Version: v1.31.1

CDI Version:

$ kubectl get deployments -n cdi -o jsonpath="{.items[0].metadata.labels.operator\.cdi\.kubevirt\.io/createVersion}"
v1.61.1

KubeVirt Version:

$ kubectl get deployments -n kubevirt -o jsonpath="{.items[0].metadata.labels.app\.kubernetes\.io/version}"
v1.4.0

VirtCTL Version:

$ virtctl version
Client Version: version.Info{GitVersion:"v1.4.0", GitCommit:"9b9b3d4e7b681af96ca1b6b6a5cea75e2f05ce5b", GitTreeState:"clean", BuildDate:"2024-11-13T08:23:36Z", GoVersion:"go1.22.8 X:nocoverageredesign", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{GitVersion:"v1.4.0", GitCommit:"9b9b3d4e7b681af96ca1b6b6a5cea75e2f05ce5b", GitTreeState:"clean", BuildDate:"2024-11-13T09:51:17Z", GoVersion:"go1.22.8 X:nocoverageredesign", Compiler:"gc", Platform:"linux/amd64"}

[akalenyu]

Is the -scratch PVC also on the same storage class?

BTW, the point of WaitForFirstConsumer is that the scheduling is delayed until a workload tries to use the volume.
By using cdi.kubevirt.io/storage.bind.immediate.requested, you're bypassing this functionality and just have the volumes end up on a random node, instead of the node where the VM is going to be scheduled.

[CodeBeholderKASM]

Is the -scratch PVC also on the same storage class?

BTW, the point of WaitForFirstConsumer is that the scheduling is delayed until a workload tries to use the volume. By using cdi.kubevirt.io/storage.bind.immediate.requested, you're bypassing this functionality and just have the volumes end up on a random node, instead of the node where the VM is going to be scheduled.

Yes, I've updated the post to include the PV details.

I initially had the storageclass set to Immediate because as far as I'm aware using virtctl image-upload requires this setting. I only changed it to use the annotation because the other ticket that had a similar issue marked it as resolved after changing the settings.

[akalenyu]

Looks like they end up using the wrong storage class, kasm-storage instead of custom-storage?

I initially had the storageclass set to Immediate because as far as I'm aware using virtctl image-upload requires this setting

Nope, that is not a requirement. You can force bind if there's no workload going to consume the volume using --force-bind.
However, according to your storage provider topology constraints, WaitForFirstConsumer may still make more sense:
https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode

[CodeBeholderKASM]

Looks like they end up using the wrong storage class, kasm-storage instead of custom-storage?

I initially had the storageclass set to Immediate because as far as I'm aware using virtctl image-upload requires this setting

Nope, that is not a requirement. You can force bind if there's no workload going to consume the volume using --force-bind. However, according to your storage provider topology constraints, WaitForFirstConsumer may still make more sense: https://kubernetes.io/docs/concepts/storage/storage-classes/#volume-binding-mode

That was actually an artifact from an older run, they're all kasm-storage now... So my full workflow is, use image-upload to create a 'golden-image' to the cluster, that I then use when creating multiple copies of a VM from the same image, using DataVolume cloning.

This all works on GKE no problems, but the last time I tested it was KubeVirt v1.2.2.

I changed the storageclass back to Immediate without the annotation, and get the same issue.

[akalenyu]

That was actually an artifact from an older run, they're all kasm-storage now... So my full workflow is, use image-upload to create a 'golden-image' to the cluster, that I then use when creating multiple copies of a VM from the same image, using DataVolume cloning.

Right, in that case, you create the golden image using virtctl image-upload with the --force-bind flag.
Does this not succeed? do the PVs somehow end up on different nodes, even though the storageclass is set to WaitForFirstConsumer?

[CodeBeholderKASM]

That was actually an artifact from an older run, they're all kasm-storage now... So my full workflow is, use image-upload to create a 'golden-image' to the cluster, that I then use when creating multiple copies of a VM from the same image, using DataVolume cloning.

Right, in that case, you create the golden image using virtctl image-upload with the --force-bind flag. Does this not succeed? do the PVs somehow end up on different nodes, even though the storageclass is set to WaitForFirstConsumer?

Yes, using --force-bind with virtctl worked, I had been following this guidance (https://github.com/kubevirt/containerized-data-importer/blob/main/doc/waitforfirstconsumer-storage-handling.md#forcing-immediate-binding) and hadn't seen anything about using that flag instead, so thanks!